ClickHouse system.asynchronous_metrics: Reference and Usage

system.asynchronous_metrics holds the values that are too expensive to compute inline on every query. A background thread refreshes them on an interval (one second by default, controlled by asynchronous_metrics_update_period_s), and the most recent snapshot lives in the table. Compared with system.metrics (instantaneous gauges) and system.events (cumulative counters), asynchronous_metrics is where you find OS-level numbers, replica lag, cache occupancy, and other aggregates. The matching history table system.asynchronous_metric_log writes those snapshots to disk so you can graph them.

What's in the table

The schema is two columns: metric (String) and value (Float64). A live ClickHouse server exposes several hundred entries. Names follow a loose convention: camel-case for ClickHouse-specific aggregates (ReplicasMaxAbsoluteDelay) and dotted lowercase for jemalloc/OS metrics (jemalloc.resident, OSCPUVirtualTimeMicroseconds).

SELECT metric, value
FROM system.asynchronous_metrics
ORDER BY metric
FORMAT Vertical;

Memory metrics

These come from process-level introspection and the jemalloc allocator.

MemoryResident is the number to alert on for capacity. MemoryResident minus the sum of caches (mark, uncompressed, primary-key) and active query allocations should be small. A large gap usually means jemalloc has not returned freed memory to the OS; SYSTEM JEMALLOC PURGE triggers a release.

Cache metrics

If MarkCacheBytes is consistently close to its limit and you see disk reads on hot queries, increase mark_cache_size. The uncompressed cache only helps when the same granules are read repeatedly, which is rare outside dashboard-style workloads.

Replication metrics

For replicated tables only:

ReplicasMaxAbsoluteDelay is the single most important number for replicated clusters. Burst spikes are normal during heavy inserts. Sustained values above 30 seconds indicate the replica cannot keep up.

Inventory metrics

Filesystem and OS metrics

These mirror what you would get from vmstat and df. Having them inside ClickHouse means alert rules can join them with internal state, for example raising an alarm only when both FilesystemMainPathAvailableBytes < 10 GiB and active merges are large.

Joining with the history table

SELECT
    toStartOfMinute(event_time) AS minute,
    avg(value) AS resident_mb
FROM system.asynchronous_metric_log
WHERE metric = 'MemoryResident'
  AND event_time > now() - INTERVAL 6 HOUR
GROUP BY minute
ORDER BY minute;

system.asynchronous_metric_log is enabled by default since 21.8. To disable it, set <asynchronous_metric_log remove="1"/> in config.d/.

Common Pitfalls

• Treating MemoryResident as the only memory number. The breakdown across mark cache, uncompressed cache, query memory, and merge memory is what tells you why RSS is high.
• Alerting on ReplicasMaxAbsoluteDelay without joining against system.replicas.is_readonly. A read-only replica reports high delay because it isn't replicating at all, not because it is slow.
• Forgetting that asynchronous metrics lag real state by up to one second. For sub-second precision use system.metrics instead.
• Setting alerts on raw jemalloc counters without understanding that jemalloc.retained is intentional fragmentation slack.

Frequently Asked Questions

Q: How often is system.asynchronous_metrics updated? A: Once per second by default. The setting asynchronous_metrics_update_period_s in config.xml controls it. Older versions defaulted to 60 seconds.

Q: What's the difference between system.metrics and system.asynchronous_metrics? A: system.metrics is instantaneous current values (running queries, open connections). system.asynchronous_metrics holds aggregates calculated periodically in the background because they are too expensive to compute on every read.

Q: Where do I see history? A: system.asynchronous_metric_log stores one row per metric per update interval. Enabled by default since 21.8.

Q: My MemoryResident keeps growing but no query is running. Is this a leak? A: Usually not. Run SYSTEM JEMALLOC PURGE and check again. jemalloc holds onto freed memory to avoid the cost of reacquiring it from the kernel.

Q: Why does ReplicasMaxAbsoluteDelay stay at zero on a read-only replica? A: The metric is computed against the replica's own queue. A read-only replica accepts no replication tasks, so its queue is empty. Cross-check system.replicas.is_readonly and is_session_expired.