ClickHouse Connection Refused / Cannot Connect: Diagnosis Guide

Failure to connect to a ClickHouse server has a small number of root causes, but they hide behind nearly identical error messages. Connection refused, Connection reset by peer, while reading from socket, and silent hangs all point to the same first question: is the server actually accepting traffic on the address and port the client is trying to use? This guide walks through the diagnosis path that resolves the majority of cases without guesswork.

Confirm the server is running and ready

Most "cannot connect" reports turn out to be a server that is still starting, has crashed, or never started cleanly after a config change.

systemctl status clickhouse-server

If the process is running, look for the readiness marker in the log:

grep "Ready for connections" /var/log/clickhouse-server/clickhouse-server.log | tail

Startup can take a long time on instances with many parts, many databases, or slow storage (NFS, EBS cold volumes). During this window the process exists but the listener is not yet bound. clickhouse-client will hit Connection refused until the readiness line appears.

If the process is not running, the error log is the next stop:

tail -200 /var/log/clickhouse-server/clickhouse-server.err.log

Common startup blockers include invalid XML in config.d/, missing TLS certificate files, and a <path> directory that the clickhouse user cannot write to.

Check which ports and interfaces are bound

The next question is whether ClickHouse is listening where you think it is. List sockets owned by the server:

sudo lsof -i -P -n | grep LISTEN

You should see entries for the configured ports, typically 9000 (native TCP), 8123 (HTTP), 9009 (interserver), and the secure variants if TLS is enabled. Cross-check against the runtime config:

sudo clickhouse-extract-from-config --config=/etc/clickhouse-server/config.xml --key=listen_host
sudo clickhouse-extract-from-config --config=/etc/clickhouse-server/config.xml --key=tcp_port
sudo clickhouse-extract-from-config --config=/etc/clickhouse-server/config.xml --key=tcp_port_secure
sudo clickhouse-extract-from-config --config=/etc/clickhouse-server/config.xml --key=http_port

clickhouse-extract-from-config reads the assembled configuration after includes and overrides are merged, which is what the server actually uses.

A quick loopback check confirms the HTTP and TCP listeners respond:

curl http://127.0.0.1:8123/
curl http://127.0.0.1:9000/

The HTTP port should reply with Ok. The TCP port will return a protocol mismatch banner, which is fine: it proves a listener is bound.

The IPv4 / IPv6 trap

This causes a disproportionate share of "but my config says listen on everything" tickets. If config.xml has only:

<listen_host>0.0.0.0</listen_host>

ClickHouse binds to IPv4 only. On most modern Linux distros, localhost resolves to ::1 (IPv6) first, so clickhouse-client against localhost fails with Connection refused while clickhouse-client --host 127.0.0.1 works. Bind both families:

<listen_host>0.0.0.0</listen_host>
<listen_host>::</listen_host>

Or restrict to specific addresses if you have a hardening requirement.

TLS and secure ports

If the client is using tcp_port_secure (9440) or https_port (8443), a missing or invalid certificate on the server causes the handshake to fail before any ClickHouse-level response is returned, which surfaces to the client as a generic connection error. Check the server log for SSL errors and verify the cert chain matches the hostname the client is connecting to. Disabling certificate verification on the client is a workaround, not a fix, and should not stay in production.

Connection limits

ClickHouse caps total open connections at 4096 by default (max_connections). Once exceeded, new connections are refused. Count current sockets owned by the server:

sudo lsof -i -a -p $(pidof clickhouse-server) | wc -l

For a more useful breakdown of socket states on the native port:

netstat -tn 2>/dev/null | tail -n +3 \
  | awk '{ printf("%s\t%s\t%s\t%s\t%s\t%s\n", $1, $2, $3, $4, $5, $6) }' \
  | clickhouse-local -S "Proto String, RecvQ Int64, SendQ Int64, LocalAddress String, ForeignAddress String, State LowCardinality(String)" \
    --query="SELECT State, count() FROM table WHERE LocalAddress LIKE '%:9000' GROUP BY State"

A large number of CLOSE_WAIT or TIME_WAIT sockets usually means a client is creating and abandoning connections instead of reusing them. Fix the client, not the limit.

Also confirm the OS-level file descriptor ceiling has not been hit:

cat /proc/$(pidof -s clickhouse-server)/limits | grep "open files"

Firewall, SELinux, and cloud security groups

If the listener is correct and local connections work but remote connections fail, the issue is between the client and the host. Walk the path:

1. telnet <host> 9000 or nc -vz <host> 9000 from the client.
2. Check iptables -L -n or nft list ruleset on the server.
3. Check cloud security groups (AWS SG, GCP firewall rules, Azure NSG).
4. If SELinux is enforcing, check audit.log for denials on the ClickHouse binary or its data path.

Common Pitfalls

• Binding only to 0.0.0.0 and then connecting via localhost (resolves to IPv6).
• Modifying /etc/clickhouse-server/config.xml directly, then having a package upgrade revert it. Use config.d/ overrides instead.
• Restarting the server with a broken config: startup fails, but the previous process keeps running until it is actually stopped. The "running" status can be misleading.
• Treating Connection reset by peer as a network issue when it is actually the server closing the socket because of an auth failure or protocol mismatch (check the server log).
• Pointing clickhouse-client at the HTTP port (8123) or vice versa. Wrong port produces immediate errors.

Frequently Asked Questions

Q: What does "Connection reset by peer, while reading from socket" mean? A: The remote side closed the TCP connection mid-stream. With ClickHouse this is usually the server shutting down, hitting a connection limit, rejecting an auth attempt, or terminating an idle client past keep_alive_timeout. Check the server error log around the timestamp of the failed client request.

Q: My server is running but clickhouse-client returns "Connection refused". What now? A: In order: check listen_host covers both IPv4 and IPv6 if you connect via localhost, verify the port with lsof, then check max_connections and OS file descriptor limits. If local works and remote does not, the issue is firewall or routing.

Q: How do I see the assembled config the server is actually using? A: Look in /var/lib/clickhouse/preprocessed_configs/. The files there reflect the merged result of config.xml plus everything in config.d/. This is the source of truth for what clickhouse-server loaded.

Q: Can a full disk cause connection failures? A: Yes. Once the data path fills, the server can fail to write to its journal or temporary files and may stop accepting new sessions. Check df -h on the <path> directory before chasing network causes.

Q: Does ClickHouse log every rejected connection? A: Connection attempts that are rejected at the TCP layer (firewall, no listener) leave no trace in ClickHouse logs. Attempts that reach the server and fail authentication or exceed limits are logged at the Information or Warning level depending on the cause.