ClickHouse Jemalloc Heap Profiling: Memory Analysis Guide

ClickHouse links against jemalloc, which exposes a built-in heap profiler. When a query spikes memory unexpectedly or you suspect a leak, capturing a heap profile gives you a per-allocation breakdown that traditional metrics cannot. This guide shows how to enable profiling, expose pprof-compatible endpoints, and analyze snapshots with jeprof.

Enable Profiling at Runtime

Heap profiling must be available in the running binary, which requires prof:true in the MALLOC_CONF environment variable at server startup. A typical value:

MALLOC_CONF="prof:true,prof_active:false,lg_prof_sample:18,prof_prefix:/tmp/jeprof"

Official ClickHouse packages and Docker images set this for you. With prof_active:false, the profiler is loaded but quiescent; the SQL statements below toggle it on and off without a restart:

SYSTEM JEMALLOC ENABLE PROFILE;

Run the workload you want to investigate, then flush the in-memory buffers to disk:

SYSTEM JEMALLOC FLUSH PROFILE;

Flushed snapshots are written under the configured prof_prefix (for example /tmp/jeprof.* with the MALLOC_CONF above). When you are done, turn profiling off so it stops adding overhead:

SYSTEM JEMALLOC DISABLE PROFILE;

Expose Pprof-Style HTTP Endpoints

For remote analysis, register custom HTTP handlers and executable dictionaries in a config file such as /etc/clickhouse-server/config.d/jemalloc_dict.xml. You need three endpoints:

The configuration also defines two executable dictionaries:

• jemalloc_ls lists snapshot files under the configured prof_prefix (for example /tmp/jeprof.*) with filename, size, and timestamp.
• jemalloc_cp copies a selected snapshot into the user_files directory so the /pprof/heap handler can serve it.

Capture and Download a Profile

List available snapshots:

SELECT * FROM dictionary('jemalloc_ls');

Copy a chosen snapshot into the serving location:

SELECT dictGet('jemalloc_cp', 'status', 3);

Render an SVG flame graph from your workstation:

jeprof --svg https://user:password@cluster.example.com:8443/pprof/heap > mem.svg

The output highlights which call stacks hold the most live bytes at the moment the snapshot was flushed.

Differential Profiles

Comparing two snapshots is often more useful than a single one. Capture a baseline before the suspect workload, then a second snapshot after, and diff them:

jeprof --svg \
  --base /path/to/baseline.heap \
  https://user:password@cluster.example.com:8443/pprof/heap \
  > mem_diff.svg

The differential view shows only allocations that grew between the two points, which is the most direct way to isolate a leaking code path.

Symbol Resolution

jeprof posts batches of raw addresses to /pprof/symbol. The handler runs introspection functions (demangle, addressToSymbol, addressToLine) against the live process to translate them into function names. This avoids shipping the ClickHouse binary or debug symbols to the analyst's machine.

Common Pitfalls

• Forgetting to disable profiling. The runtime overhead is small but not zero, and snapshots accumulate under /tmp.
• Running jeprof against a different ClickHouse version than the running server. Symbol addresses will not match.
• Exposing the pprof endpoints without authentication. Treat them as privileged, since they can leak code layout and memory contents.
• Only capturing a single snapshot. Without a baseline, large allocations from steady-state caches dominate the view.

Frequently Asked Questions

Q: Do I need to restart ClickHouse to enable jemalloc profiling? A: Only if MALLOC_CONF does not already include prof:true. Once the server is started with profiling support compiled in and prof:true set, SYSTEM JEMALLOC ENABLE PROFILE toggles collection at runtime. The HTTP handler config also needs a restart, but only the first time.

Q: Where do the heap snapshots go? A: Flushed snapshots land under the prefix configured by prof_prefix in MALLOC_CONF (for example /tmp/jeprof.* when prof_prefix:/tmp/jeprof is set). Without an explicit prof_prefix, jemalloc writes to the current working directory with default prefix jeprof.

Q: Can I use this on ClickHouse Cloud or managed services? A: Only if the provider exposes the jemalloc system statements and lets you mount custom config files. Many managed offerings do not.

Q: Is heap profiling safe for production? A: Yes, the sampling overhead is low. The bigger risk is disk consumption from accumulated profile files, so disable profiling when you are done.

Q: Why is jeprof reporting unresolved addresses? A: The /pprof/symbol handler is missing or the running ClickHouse binary does not match. Verify the handler returns demangled names for a known address.