ClickHouse Concurrency and Threading Misconfiguration

Q: What is the difference between `max_threads` and `max_thread_pool_size`?

max_threads is a per-query, user-level setting that caps parallelism for a single query (default 0, auto-detects CPU cores). max_thread_pool_size is a server-level hard limit on all threads across every concurrent query and background operation (default 10,000). Many queries each using max_threads threads draw from the same global pool. If max_threads * active_query_count exceeds max_thread_pool_size , task submissions fail with CANNOT_SCHEDULE_TASK .

ClickHouse relies on aggressive parallelism to deliver analytical query performance. When threading and concurrency settings are misconfigured, the result is one of two opposite failure modes: queries that use only a single CPU core and run far slower than the hardware allows, or a system overwhelmed by too many simultaneous threads that exhausts the global thread pool and emits DB::Exception: Cannot schedule a task (CANNOT_SCHEDULE_TASK).

Understanding which layer is misconfigured -- per-query parallelism, total query concurrency, or background pool sizing -- is the prerequisite for any fix.

What This Error Means

ClickHouse maintains several overlapping thread budgets:

Per-query parallelism: max_threads controls how many processing lanes a single query uses. Its default is 0, which at runtime resolves to the number of CPU cores visible to ClickHouse.
Global thread pool: max_thread_pool_size (default 10,000) is the hard ceiling on all threads across every concurrent activity -- queries, background merges, replication, and internal tasks. Exceeding this ceiling causes CANNOT_SCHEDULE_TASK (error code 439).
Query concurrency: max_concurrent_queries (default 1,000 since ClickHouse 23.9; previously 100) limits how many queries can execute simultaneously. Queries beyond this limit are rejected immediately.
Background merge pool: background_pool_size (default 16, server-level only) controls threads dedicated to MergeTree background merges and mutations. This setting is separate from the query thread pool.

A useful rule of thumb: max_threads * max_concurrent_queries should stay well below max_thread_pool_size. At default settings -- max_threads resolving to, say, 16 CPU cores, with max_concurrent_queries at 1,000 -- the theoretical thread demand is 16,000, which exceeds the 10,000-thread global pool. In practice, not every query uses max_threads threads, but this product is a useful sizing guide.

Common Causes

Query underparallelism from data size thresholds. ClickHouse intentionally reduces thread count for small datasets. A query will use only one thread if the scanned data is below merge_tree_min_rows_for_concurrent_read (default 163,840 rows per lane) or merge_tree_min_bytes_for_concurrent_read (default 240 MiB per lane). This is the most frequent reason for single-threaded queries and is expected behavior, not a bug.
Explicit max_threads restriction in a user profile. A value such as <max_threads>1</max_threads> in users.xml or a settings profile silently caps all queries for that user to one thread, regardless of data size or server CPU count.
Global thread pool exhaustion. Under high concurrent query load, the product max_threads * active_queries can exhaust max_thread_pool_size, causing subsequent task submissions to fail with CANNOT_SCHEDULE_TASK. Known bug in ClickHouse 23.7: queries against distributed tables with many parts and LowCardinality columns with DISTINCT could spawn 10,000+ threads even when max_threads=5 with 25 concurrent queries.
background_pool_size set at the wrong scope. This setting appears as Obsolete in system.settings (user-level settings) because it was moved to server-level configuration in ClickHouse 23.3. Setting it in a user profile has no effect. It must appear in config.xml or a config.d/ drop-in. The confusion leads operators to believe merges are unconstrained when in fact the server is using the default of 16.
max_concurrent_queries set too low for the workload. Installations running pre-23.9 ClickHouse with the historical default of 100 will reject queries under moderate load. Raising this without also checking the max_threads * max_concurrent_queries product can push thread demand past max_thread_pool_size.
OS-level thread limits too low. ClickHouse emits a startup warning if the OS maximum thread count is below 30,000: Maximum number of threads is lower than 30000. The relevant limits are ulimit -u (nproc) and /proc/sys/kernel/threads-max. Each thread also consumes a file descriptor, so ulimit -n matters too.
Thread pool bottleneck before ClickHouse 24.10. Prior to version 24.10, thread creation occurred inside a critical section (kernel mmap semaphore contention). This caused high load average with low CPU utilization -- thousands of threads created but only a fraction running. Fixes in PRs #68694 and #68674 reduced lock wait from 24.5 billion to 28.2 million microseconds. Upgrading to 24.10+ resolves this class of issue.
concurrent_threads_soft_limit_num not configured. This server setting (default 0, unlimited) provides a soft cap on total query-processing threads across all concurrent queries. Without it, nothing constrains total thread usage at the query level short of the hard global pool limit.

How to Fix

1. Diagnose actual thread usage per query

-- Active queries with thread details
SELECT
    query_id,
    user,
    elapsed,
    read_rows,
    formatReadableSize(memory_usage) AS memory,
    peak_threads_usage,
    length(thread_ids) AS thread_count,
    substring(query, 1, 100) AS query_snippet
FROM system.processes
ORDER BY elapsed DESC;

For completed queries, use system.query_log:

SELECT
    query_id,
    type,
    event_time,
    query_duration_ms,
    peak_threads_usage,
    length(thread_ids) AS threads_participated,
    read_rows,
    normalizeQuery(query) AS normalized_query
FROM system.query_log
WHERE type = 'QueryFinish'
  AND event_time >= now() - INTERVAL 1 HOUR
ORDER BY peak_threads_usage DESC
LIMIT 20;

2. Check effective session settings and verify `max_threads`

-- Effective max_threads for the current session
SELECT getSetting('max_threads');

-- All thread-related session settings
SELECT name, value, default, changed, description
FROM system.settings
WHERE name IN ('max_threads', 'max_insert_threads', 'max_final_threads')
ORDER BY name;

If max_threads appears as 1 and changed = 1, a user profile is overriding the default. Locate and correct it in users.xml or via ALTER SETTINGS PROFILE.

3. Check server-level thread and concurrency settings

SELECT name, value, default, changed, description
FROM system.server_settings
WHERE name IN (
    'max_thread_pool_size',
    'max_thread_pool_free_size',
    'thread_pool_queue_size',
    'max_concurrent_queries',
    'max_concurrent_select_queries',
    'max_concurrent_insert_queries',
    'background_pool_size',
    'background_merges_mutations_concurrency_ratio',
    'concurrent_threads_soft_limit_num',
    'concurrent_threads_soft_limit_ratio_to_cores'
)
ORDER BY name;

The changed column is 1 for settings that have been explicitly configured. Settings where value != default but changed = 0 indicate the compiled default has changed between versions.

4. Use `EXPLAIN PIPELINE` to inspect planned parallelism

EXPLAIN PIPELINE shows the query plan before execution and reveals how many parallel lanes ClickHouse intends to use:

EXPLAIN PIPELINE
SELECT count() FROM my_table WHERE date >= today() - 7;

Look for the × N notation (for example, MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread) × 8). A result of × 1 means planned single-threaded execution. If this contradicts expectations, check whether the table has enough data to exceed the concurrent-read thresholds.

To force maximum parallelism for a single test query (lowering the per-lane thresholds to essentially zero):

SELECT count()
FROM my_table
SETTINGS
    max_threads = 16,
    merge_tree_min_rows_for_concurrent_read = 1,
    merge_tree_min_bytes_for_concurrent_read = 1;

5. Monitor real-time thread pool utilization

SELECT metric, value, description
FROM system.metrics
WHERE metric IN (
    'GlobalThread',
    'GlobalThreadActive',
    'GlobalThreadScheduled',
    'BackgroundMergesAndMutationsPoolTask',
    'BackgroundMergesAndMutationsPoolSize',
    'QueryThread'
)
ORDER BY metric;

GlobalThread approaching max_thread_pool_size (10,000) indicates impending exhaustion. BackgroundMergesAndMutationsPoolTask at the pool size ceiling means the merge pool is saturated.

6. Fix `background_pool_size` misconfiguration

If background_pool_size is being set in a user profile and showing as Obsolete in system.settings, move it to the server configuration:

<!-- /etc/clickhouse-server/config.d/background-pools.xml -->
<clickhouse>
    <background_pool_size>16</background_pool_size>
    <background_merges_mutations_concurrency_ratio>2</background_merges_mutations_concurrency_ratio>
</clickhouse>

Remove the setting from any user profiles where it has no effect. Verify via system.server_settings rather than system.settings.

7. Configure `concurrent_threads_soft_limit_num`

To prevent thread pool exhaustion under mixed workloads, set a soft limit that reduces parallelism for new queries when the total thread count is high:

<!-- Limit total query-processing threads to 4× CPU cores -->
<clickhouse>
    <concurrent_threads_soft_limit_ratio_to_cores>4</concurrent_threads_soft_limit_ratio_to_cores>
</clickhouse>

This is a soft limit: queries will still receive at least one thread even when the limit is exceeded. It does not reject queries; it reduces per-query parallelism under load.

8. Address OS thread limits

# Check current per-process thread limit
ulimit -u

# Check ClickHouse thread count
ps -eLf | grep clickhouse-server | wc -l

# Check OS-level limits
cat /proc/sys/kernel/threads-max

Add to /etc/security/limits.conf:

clickhouse soft nproc 131072
clickhouse hard nproc 131072
clickhouse soft nofile 262144
clickhouse hard nofile 262144

Root-Cause Analysis

Find queries that ran single-threaded

SELECT
    normalizeQuery(query) AS query_pattern,
    count() AS occurrences,
    avg(query_duration_ms) AS avg_duration_ms,
    avg(read_rows) AS avg_read_rows
FROM system.query_log
WHERE type = 'QueryFinish'
  AND event_time >= now() - INTERVAL 24 HOUR
  AND peak_threads_usage = 1
  AND query_kind = 'Select'
GROUP BY query_pattern
ORDER BY occurrences DESC
LIMIT 20;

If avg_read_rows is consistently below 163,840, the single-threaded execution is caused by the small-dataset threshold and is expected. If avg_read_rows is large, a profile-level max_threads restriction is likely.

Detect thread pool saturation over time

-- Requires metric_log enabled in config.xml
SELECT
    toStartOfMinute(event_time) AS minute,
    max(CurrentMetrics_GlobalThread) AS max_threads,
    max(CurrentMetrics_GlobalThreadActive) AS max_active_threads,
    max(CurrentMetrics_GlobalThreadScheduled) AS max_scheduled_jobs
FROM system.metric_log
WHERE event_time >= now() - INTERVAL 1 HOUR
GROUP BY minute
ORDER BY minute;

Cluster-wide thread usage

SELECT
    hostName() AS host,
    count() AS active_queries,
    sum(peak_threads_usage) AS total_threads
FROM clusterAllReplicas('default', system.processes)
GROUP BY host
ORDER BY active_queries DESC;

Check OS thread counts from async metrics

SELECT metric, value
FROM system.asynchronous_metrics
WHERE metric IN ('OSThreadsTotal', 'OSThreadsRunnable', 'OSProcessesRunning', 'OSProcessesBlocked')
ORDER BY metric;

Preventive Measures

Keep the product of max_threads and max_concurrent_queries below approximately 8,000--10,000 to stay within the default max_thread_pool_size of 10,000.
Set background_pool_size in config.xml or config.d/, never in user profiles; verify it is active via system.server_settings.
Enable metric_log to retain historical thread pool usage; it is the only way to diagnose saturation events after they occur.
Configure concurrent_threads_soft_limit_ratio_to_cores to 3--4 on mixed-workload nodes, so query parallelism gracefully degrades under load before hitting the hard pool limit.
Set OS nproc and nofile limits well above 30,000 for the clickhouse user and verify ClickHouse does not emit the startup warning about thread count.
Upgrade to ClickHouse 24.10 or later to eliminate the thread-creation bottleneck that caused high load average with low CPU utilization.
Use EXPLAIN PIPELINE before deploying new queries to verify expected parallelism at the query design stage.

Resolve Concurrency and Threading Issues Automatically with Pulse

Pulse continuously monitors thread pool utilization, per-query parallelism, and background pool saturation across your ClickHouse clusters. It surfaces single-threaded query patterns, identifies profile-level max_threads restrictions, and alerts when GlobalThread approaches max_thread_pool_size -- before you hit CANNOT_SCHEDULE_TASK in production. Pulse also tracks peak_threads_usage trends over time, giving you the historical context needed to size thread pools correctly for your workload.

Frequently Asked Questions

Q: Why does my query use only 1 thread even though max_threads is set to 16?
A: The most common reason is that the scanned data is below the concurrent-read threshold. ClickHouse requires at least merge_tree_min_rows_for_concurrent_read rows (default 163,840) per parallel lane. If your table has 100,000 rows total, the query runs single-threaded regardless of max_threads. Use EXPLAIN PIPELINE to confirm -- a × 1 annotation means planned single-threaded execution. You can also check peak_threads_usage in system.query_log for completed queries.

Q: What is the difference between max_threads and max_thread_pool_size?
A: max_threads is a per-query, user-level setting that caps parallelism for a single query (default 0, auto-detects CPU cores). max_thread_pool_size is a server-level hard limit on all threads across every concurrent query and background operation (default 10,000). Many queries each using max_threads threads draw from the same global pool. If max_threads * active_query_count exceeds max_thread_pool_size, task submissions fail with CANNOT_SCHEDULE_TASK.

Q: background_pool_size shows as Obsolete in system.settings -- does it still work?
A: Yes, but it must be set at the server level, not in user profiles. The Obsolete flag in system.settings (user-level) indicates that the setting was moved to server-level configuration in ClickHouse 23.3. It is fully active when set in config.xml or config.d/ and verified via system.server_settings. Setting it in a user profile has no effect.

Q: max_concurrent_queries was 100 in my config -- is that the default?
A: The default was 100 for many years and was raised to 1,000 in ClickHouse 23.9/23.11 (PR #53285). If your installation predates 23.9 or explicitly set max_concurrent_queries to 100, queries beyond the 100th concurrent execution are rejected. Increasing this requires checking that max_threads * new_limit remains well below max_thread_pool_size.

Q: Is concurrent_threads_soft_limit_num a hard cap that rejects queries?
A: No. It is explicitly a soft limit. When the total query-processing thread count exceeds this value, ClickHouse reduces the thread allocation for new queries, but every query will still receive at least one thread. It does not reject queries. Use it to gracefully degrade parallelism under load rather than as a strict concurrency gate.

Q: I see high load average but low CPU utilization -- are these related?
A: Yes, and this was a known issue before ClickHouse 24.10. Thread creation happened inside a critical section, so creating threads for one query could block the entire thread pool. Thousands of threads would be allocated but only a fraction would be actively executing, producing high load average with low CPU. The fix landed in PRs #68694 and #68674 in 24.10. Upgrading resolves this class of issue.