ClickHouse DB::Exception: Zero-copy replication error

The DB::Exception: Zero-copy replication error (code ZERO_COPY_REPLICATION_ERROR) is triggered when ClickHouse encounters a failure in its zero-copy replication mechanism. Zero-copy replication allows replicas to share data parts stored on a common backend (such as S3 or HDFS) without physically copying the data between nodes. When the coordination or metadata management for this shared storage breaks down, this error appears.

Impact

Replication for the affected table stalls because the replica cannot correctly reference shared data parts. New inserts may fail if the replication queue is blocked. Depending on the failure mode, some parts may become inaccessible to one or more replicas, causing query inconsistencies across the cluster.

Common Causes

1. S3 or HDFS connectivity issues -- the shared storage backend is unreachable or returning errors.
2. ZooKeeper metadata inconsistency -- the zero-copy lock or reference count in ZooKeeper is corrupted or out of sync with the actual objects in storage.
3. Concurrent operations conflicting -- multiple replicas attempting to merge or mutate the same shared part simultaneously.
4. Permissions errors on the shared storage bucket or path preventing read/write operations.
5. Object storage eventual consistency -- in rare cases, a newly written object is not yet visible to all replicas.
6. ClickHouse version bug -- zero-copy replication is a relatively newer feature and has seen fixes across releases.

Troubleshooting and Resolution Steps

1. Check shared storage connectivity Verify the ClickHouse node can reach S3/HDFS:

   aws s3 ls s3://your-clickhouse-bucket/data/ --region your-region
   

   For HDFS:

   hdfs dfs -ls /clickhouse/data/
   

2. Review ClickHouse logs for details The error log will contain specifics about which zero-copy operation failed and the underlying storage error. Look for S3 error codes or HDFS exceptions.

3. Inspect ZooKeeper zero-copy metadata

   SELECT *
   FROM system.zookeeper
   WHERE path LIKE '%/zero_copy%' AND path LIKE '%my_table%';
   

   Look for stale locks or incorrect reference counts.

4. Verify storage credentials and permissions Ensure the IAM role, access keys, or HDFS keytab used by ClickHouse has read/write access to the relevant paths. Check for recently rotated credentials.

5. Restart replication

   SYSTEM RESTART REPLICA db.my_table;
   

6. Disable zero-copy replication temporarily If you need to unblock operations while investigating:

   ALTER TABLE db.my_table MODIFY SETTING allow_remote_fs_zero_copy_replication = 0;
   

   This falls back to traditional replication where each replica stores its own copy.

7. Upgrade ClickHouse If you are on an older version, check the changelog for zero-copy replication fixes. This feature has matured significantly across recent releases.

Best Practices

• Test zero-copy replication thoroughly in a staging environment before enabling it in production.
• Monitor S3/HDFS latency and error rates alongside ClickHouse replication metrics.
• Keep ClickHouse updated to benefit from the latest zero-copy bug fixes and improvements.
• Use versioned S3 buckets to protect against accidental object deletion.
• Set up alerting on zero-copy ZooKeeper paths for anomalies like stale locks.
• Consider the maturity of zero-copy replication when deciding whether to enable it -- as of recent versions it is more stable but still evolving.

Frequently Asked Questions

Q: What is zero-copy replication?
A: Zero-copy replication is a ClickHouse feature where replicas share data parts on a common object store (S3, GCS, HDFS) instead of each replica maintaining its own copy. Only metadata is replicated, not the actual data, which saves storage and bandwidth.

Q: Does disabling zero-copy replication cause data loss?
A: No. Disabling it means ClickHouse will start copying parts to each replica's local or dedicated storage. Existing shared parts remain accessible.

Q: Is zero-copy replication production-ready?
A: It has improved significantly in recent ClickHouse releases, but it is still considered an advanced feature. Evaluate it carefully for your workload and keep your ClickHouse version current.

Q: Can I mix zero-copy and regular replication in the same cluster?
A: Zero-copy replication is a per-table setting. You can have some tables using it and others using traditional replication within the same cluster.