ClickHouse DB::Exception: Too few live replicas for quorum

The DB::Exception: Too few live replicas error (code TOO_FEW_LIVE_REPLICAS) occurs when an INSERT statement requires quorum confirmation but the number of currently active replicas is below the quorum threshold. ClickHouse refuses the write rather than risk data being committed to fewer nodes than the quorum demands.

Impact

All INSERT operations targeting the affected table will fail as long as the quorum cannot be met. This effectively makes the table read-only until enough replicas come back online. SELECT queries continue to work against the data already present. Applications that depend on writes will experience errors or timeouts during this window.

Common Causes

1. Replica nodes are down -- one or more replicas have crashed, are being restarted, or are undergoing maintenance, dropping the live count below the quorum.
2. Network partition -- replicas are running but cannot communicate with ZooKeeper, so they appear as inactive.
3. ZooKeeper or Keeper unavailability -- if the coordination service itself is degraded, replicas cannot register as alive.
4. Quorum set too high -- the insert_quorum setting is configured to require more replicas than the cluster can realistically sustain during normal operations.
5. Slow replica startup -- after a rolling restart, nodes may take time to re-register in ZooKeeper, temporarily reducing the active count.

Troubleshooting and Resolution Steps

1. Check how many replicas are active

   SELECT database, table, total_replicas, active_replicas, is_session_expired
   FROM system.replicas
   WHERE table = 'my_table';
   

2. Review the quorum settings

   SELECT name, value
   FROM system.settings
   WHERE name IN ('insert_quorum', 'insert_quorum_parallel');
   

   If insert_quorum equals the total replica count, even a single node failure will trigger this error.

3. Bring offline replicas back up Check the status of ClickHouse on the down nodes:

   sudo systemctl status clickhouse-server
   

   Review logs for startup errors and fix any issues preventing the service from running.

4. Verify ZooKeeper connectivity from each replica

   SELECT * FROM system.zookeeper WHERE path = '/';
   

   If this query fails, the node has lost its ZooKeeper session.

5. Temporarily lower the quorum requirement If you need to resume writes urgently and accept reduced durability:

   SET insert_quorum = 1;
   INSERT INTO my_table VALUES (...);
   

   Or disable quorum writes entirely at the session level:

   SET insert_quorum = 0;
   

6. Wait for replicas to recover After a rolling restart, give all nodes time to re-establish their ZooKeeper sessions and appear as active. Monitor progress with:

   SELECT hostName(), uptime()
   FROM clusterAllReplicas('my_cluster', system.one);
   

Best Practices

• Set insert_quorum to a majority (e.g., 2 out of 3 replicas) rather than requiring all replicas, so that one node can be down for maintenance without blocking writes.
• Use insert_quorum_parallel (available in recent versions) to allow parallel quorum writes, which improves throughput.
• Implement health checks and alerting on system.replicas.active_replicas to catch replica failures early.
• Plan rolling restarts to keep at least quorum-many replicas active at all times.
• Consider using insert_quorum_timeout to control how long ClickHouse waits before giving up on a quorum write.

Frequently Asked Questions

Q: What is the default value of insert_quorum?
A: The default is 0, which means quorum writes are disabled. You must explicitly set it to enable this feature.

Q: Can I set insert_quorum per query rather than globally?
A: Yes. You can set it at the session, user, or query level using SETTINGS insert_quorum = N appended to the INSERT statement.

Q: Does lowering the quorum risk data loss?
A: Lowering the quorum means writes are confirmed by fewer replicas. If the replica holding the only copy fails before replication catches up, that data could be lost. It is a trade-off between availability and durability.

Q: How does insert_quorum interact with insert_quorum_parallel?
A: When insert_quorum_parallel is enabled (default in ClickHouse 22.3+), multiple quorum inserts can proceed concurrently without waiting for previous quorum confirmations.