ClickHouse DB::Exception: HTTP connection limit reached

The "DB::Exception: HTTP connection limit reached" error in ClickHouse indicates that the server has reached its maximum allowed number of concurrent HTTP connections. The error code is HTTP_CONNECTION_LIMIT_REACHED. When this limit is exceeded, any new HTTP connection attempt is rejected until an existing connection is freed.

Impact

All HTTP-based clients, including dashboard tools, REST API consumers, JDBC/ODBC drivers using the HTTP interface, and monitoring queries, will be unable to connect. This can take down entire visualization layers and API services while leaving native protocol connections unaffected. The server itself remains operational but inaccessible over HTTP.

Common Causes

1. The server's HTTP connection limit is set too low relative to the number of concurrent HTTP clients.
2. Applications holding HTTP connections open for extended periods, such as long-running queries or streaming inserts.
3. Connection leaks in HTTP clients that do not properly close or release connections after use.
4. A sudden surge of requests from dashboards or automated jobs that exceed the normal connection count.
5. Reverse proxies or load balancers multiplying connections by opening multiple backend connections per frontend request.

Troubleshooting and Resolution Steps

1. Check the current HTTP connection count:

   SELECT * FROM system.metrics WHERE metric = 'HTTPConnection';
   

2. Review and increase the HTTP connection limit if necessary. This is controlled by the max_connections setting or specific HTTP settings depending on your ClickHouse version:

   <max_connections>4096</max_connections>
   

3. Identify which processes are consuming connections:

   SELECT query, elapsed, read_rows, client_name
   FROM system.processes
   ORDER BY elapsed DESC
   LIMIT 20;
   

4. Look for long-running queries that are holding HTTP connections open and consider adding timeouts:

   SET max_execution_time = 300;
   

5. Configure keep-alive settings to reclaim idle HTTP connections more aggressively:

   <keep_alive_timeout>10</keep_alive_timeout>
   

6. If a reverse proxy sits in front of ClickHouse, review its connection pooling configuration to prevent connection multiplication.

7. Ensure the OS file descriptor limit supports the desired maximum connections:

   ulimit -n
   

Best Practices

• Monitor HTTPConnection metrics and set up alerting at a threshold below the maximum to get early warning.
• Configure HTTP keep-alive timeouts to balance connection reuse with resource reclamation.
• Use connection pooling in HTTP client libraries and set reasonable pool sizes.
• Set query-level timeouts (max_execution_time) to prevent any single query from holding a connection indefinitely.
• Separate heavy batch workloads from interactive/dashboard workloads, potentially using different ClickHouse users with different connection limits.

Frequently Asked Questions

Q: Does this limit also affect native TCP connections?
A: No. HTTP and native TCP connections are tracked separately. Hitting the HTTP limit does not prevent native protocol clients from connecting, and vice versa.

Q: How do I find the right HTTP connection limit for my setup?
A: Start by monitoring peak HTTP connection usage under normal load. Set the limit to at least 2x your observed peak to handle traffic spikes. Adjust based on available system resources and file descriptor limits.

Q: Can I set different connection limits for different users?
A: ClickHouse allows you to set max_concurrent_queries_for_user in user profiles, which indirectly limits how many active connections a single user can consume. This helps prevent one user from exhausting the shared pool.

Q: Will HTTP/2 help with this issue?
A: ClickHouse's HTTP interface operates over HTTP/1.1. If you are using a reverse proxy that supports HTTP/2 on the client side and multiplexes to HTTP/1.1 on the backend, it can reduce the number of physical connections to ClickHouse.