ClickHouse DB::Exception: Replica is not in quorum

The DB::Exception: Replica is not in quorum error (code REPLICA_IS_NOT_IN_QUORUM) arises when ClickHouse determines that the current replica is not part of the write quorum for a given operation. This typically surfaces during reads with select_sequential_consistency enabled, where the server verifies that the replica has received all quorum-confirmed writes before returning results.

Impact

Queries with select_sequential_consistency = 1 will fail on this replica, meaning applications expecting linearizable reads cannot be served from this node. Writes to the table are not directly affected, but the replica is considered stale for consistency-sensitive workloads until it catches up with the quorum.

Common Causes

1. Replica is lagging behind -- it has not yet fetched all parts that were confirmed by the quorum, so it cannot guarantee sequential consistency.
2. Replica was recently restarted and has not finished processing its replication queue.
3. Network delays between this replica and ZooKeeper prevent it from updating its quorum participation status.
4. The quorum record in ZooKeeper has not been updated to include this replica after a recent write.
5. Misconfigured select_sequential_consistency applied globally when only certain queries need it.

Troubleshooting and Resolution Steps

1. Check the replica's replication status

   SELECT database, table, is_leader, absolute_delay, queue_size,
          last_queue_update
   FROM system.replicas
   WHERE table = 'my_table';
   

   A large absolute_delay or queue_size indicates the replica is behind.

2. Force a sync to catch up

   SYSTEM SYNC REPLICA db.my_table;
   

   Wait for this to complete before retrying the query.

3. Verify quorum status in ZooKeeper

   SELECT *
   FROM system.zookeeper
   WHERE path = '/clickhouse/tables/01/my_table/quorum/status';
   

4. Retry on a different replica If you need immediate results, route the query to a replica that is part of the quorum. Use a load balancer or explicitly target a healthy node.

5. Disable sequential consistency for this query If your application can tolerate eventual consistency:

   SET select_sequential_consistency = 0;
   SELECT * FROM my_table WHERE ...;
   

6. Check ZooKeeper session health

   SELECT database, table, zookeeper_name, is_session_expired
   FROM system.replicas
   WHERE table = 'my_table';
   

   An expired session means the replica has been disconnected from ZooKeeper.

Best Practices

• Only enable select_sequential_consistency for queries that truly require linearizable reads; applying it globally adds overhead and increases the chance of hitting this error.
• Monitor replication lag and alert when absolute_delay exceeds a few seconds.
• Ensure replicas have adequate resources (network, disk I/O) to keep up with the write rate.
• Use load-balancer health checks that verify replication status so that lagging replicas are temporarily removed from the read pool.
• After planned maintenance or restarts, wait for SYSTEM SYNC REPLICA to complete before routing read traffic to the node.

Frequently Asked Questions

Q: Does this error affect writes to the table?
A: No. This error is specific to reads that require sequential consistency. Writes proceed independently based on the insert_quorum setting.

Q: What is select_sequential_consistency?
A: It is a ClickHouse setting that, when enabled, ensures a SELECT query only returns data that has been confirmed by the write quorum. This guarantees that you always read the most recently committed data.

Q: Will the replica eventually rejoin the quorum automatically?
A: Yes. Once the replica catches up with the replication queue and confirms receipt of all quorum parts, it will be considered part of the quorum again.

Q: Is there a timeout for SYSTEM SYNC REPLICA?
A: Yes, you can specify a timeout: SYSTEM SYNC REPLICA db.my_table STRICT TIMEOUT 60. If the sync does not complete within the given seconds, the command returns an error.