ClickHouse DB::Exception: Interserver scheme doesn't match

The "DB::Exception: Interserver scheme doesn't match" error in ClickHouse arises when nodes in a cluster attempt to communicate using different protocols for inter-server operations. The error code is INTERSERVER_SCHEME_DOESNT_MATCH. For example, one node may be configured to use HTTP for interserver communication while another expects HTTPS. This mismatch prevents replication and distributed query execution from functioning properly.

Impact

Replication between nodes will fail, meaning data inserted on one replica will not propagate to others. Distributed queries that require fetching data from other shards or replicas will also fail. In a replicated environment, this effectively breaks the cluster's high availability and data redundancy until the configuration is unified.

Common Causes

Inconsistent interserver_http_port and interserver_https_port settings across cluster nodes.
Partially completed migration from HTTP to HTTPS for inter-server communication, where some nodes have been updated and others have not.
One node configured with <interserver_http_host> using http:// while another uses https://.
Mismatched configuration files deployed across the cluster, often due to configuration management errors.
A new node added to the cluster with default settings that differ from the existing nodes' customized configuration.

Troubleshooting and Resolution Steps

Check the interserver configuration on each node. The settings should be consistent across all nodes in the cluster:
```
grep -i "interserver" /etc/clickhouse-server/config.xml
```

Verify the interserver_http_port or interserver_https_port settings. All nodes must use the same protocol:

<!-- For HTTP (all nodes must match) -->
<interserver_http_port>9009</interserver_http_port>

<!-- OR for HTTPS (all nodes must match) -->
<interserver_https_port>9010</interserver_https_port>

Check the interserver_http_host setting on each node. This should be the hostname or IP that other nodes use to reach this node:
```
<interserver_http_host>node1.example.com</interserver_http_host>
```
If migrating from HTTP to HTTPS, plan a coordinated update. All nodes need to be switched at the same time, or you should use a rolling approach where you temporarily enable both ports:
```

<interserver_http_port>9009</interserver_http_port>
<interserver_https_port>9010</interserver_https_port>
```
After making configuration changes, restart the ClickHouse service on the affected nodes:
```
sudo systemctl restart clickhouse-server
```

Verify that inter-server communication is working by checking the replication status:

SELECT database, table, is_leader, is_readonly, future_parts, queue_size
FROM system.replicas
WHERE is_readonly = 1 OR queue_size > 0;

Best Practices

Use a configuration management tool (Ansible, Chef, Puppet, etc.) to ensure consistent ClickHouse configuration across all cluster nodes.
Plan protocol migrations carefully and execute them as coordinated cluster-wide changes.
Prefer HTTPS for inter-server communication in production environments to encrypt replication traffic.
After any cluster configuration change, verify replication health using the system.replicas table.

Frequently Asked Questions

Q: Can I run some nodes with HTTP and others with HTTPS for interserver communication?
A: No. All nodes in a cluster must use the same protocol for interserver communication. Mixed configurations will result in this error.

Q: How do I safely migrate from HTTP to HTTPS for interserver traffic?
A: The safest approach is to temporarily enable both interserver_http_port and interserver_https_port on all nodes, then switch clients to HTTPS, and finally disable the HTTP port once all nodes are communicating over HTTPS.

Q: Does this error affect only replicated tables?
A: It primarily affects replicated tables and distributed queries that require data exchange between nodes. Non-replicated, local tables on each node are not affected.

Q: What port does ClickHouse use for interserver communication by default?
A: The default interserver HTTP port is 9009. There is no default HTTPS interserver port; it must be explicitly configured.