ClickHouse DB::Exception: Async load wait failed

The "DB::Exception: Async load wait failed" error in ClickHouse occurs when the server or a query tries to wait for an asynchronous loading operation to complete, but the wait itself fails — typically due to a timeout. The ASYNC_LOAD_WAIT_FAILED error code indicates that the expected table, database, or dictionary did not finish loading within the allowed time.

Impact

Queries that depend on the table or dictionary being loaded will fail until the object becomes available. In severe cases, if many tables are affected, the server may appear to be running but remain partially unusable. Background processes that depend on loaded tables, such as materialized views or scheduled merges, will also stall.

Common Causes

The server is under heavy load during startup and table loading takes longer than the configured wait timeout
Disk I/O bottlenecks slow down metadata reading and part enumeration for large tables
ZooKeeper or ClickHouse Keeper connectivity issues delay metadata synchronization for replicated tables
A dependent object (such as a dictionary source table) has not finished loading, causing a cascading wait failure
An extremely large number of tables or parts that overwhelms the async loading thread pool

Troubleshooting and Resolution Steps

Examine the server logs for details about which object timed out:

grep -i 'ASYNC_LOAD_WAIT_FAILED\|async.*wait\|loading.*timeout' /var/log/clickhouse-server/clickhouse-server.log | tail -30

Check the current state of table loading after startup:

SELECT database, name, metadata_modification_time
FROM system.tables
WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA')
ORDER BY metadata_modification_time DESC
LIMIT 20;

If ZooKeeper connectivity is the bottleneck, verify the connection:
```
SELECT * FROM system.zookeeper WHERE path = '/';
```

Increase the number of async loading threads if the default pool is too small for your table count:

<clickhouse>
    <async_load_databases>true</async_load_databases>
    <tables_loader_foreground_pool_size>4</tables_loader_foreground_pool_size>
    <tables_loader_background_pool_size>8</tables_loader_background_pool_size>
</clickhouse>

Reduce startup load by cleaning up unused tables, detaching tables you do not need immediately, or reducing the number of parts through manual optimization:
```
OPTIMIZE TABLE my_database.my_table FINAL;
```
As a temporary workaround, disable async loading to force synchronous startup, which avoids wait timeouts at the cost of a longer startup time:
```
<clickhouse>
    <async_load_databases>false</async_load_databases>
</clickhouse>
```

Best Practices

Keep the number of parts per table manageable through proper partitioning and regular OPTIMIZE operations.
Provision adequate disk I/O for the ClickHouse data directory, especially on nodes with many tables.
Monitor ZooKeeper or ClickHouse Keeper latency, as replica metadata sync is a common startup bottleneck.
Consider staggering node restarts in a cluster rather than restarting all nodes simultaneously.
Drop or detach tables and databases that are no longer needed to speed up server startup.

Frequently Asked Questions

Q: How long does ClickHouse wait for async loading before failing?
A: The wait duration depends on the context. Queries that access a table being loaded will wait according to lock_acquire_timeout and other session-level timeouts. During startup, internal timeouts govern how long the server waits before marking a load as failed.

Q: Can I query tables that are still loading?
A: A query will block and wait for the table to finish loading. If the load takes too long, the query fails with ASYNC_LOAD_WAIT_FAILED. You cannot query a partially loaded table.

Q: Does this error mean my data is corrupted?
A: Not necessarily. The error indicates a timeout or wait failure, not data corruption. However, if a specific table consistently fails to load, investigate its metadata and parts for corruption.