ClickHouse DB::Exception: Cannot decompress data

The "DB::Exception: Cannot decompress data" error in ClickHouse means that a decompression operation failed while reading data. The CANNOT_DECOMPRESS error fires when ClickHouse attempts to decompress a block and the decompression library returns a failure. This usually indicates that the compressed data is corrupted, was compressed with a different codec than expected, or that the block metadata is inconsistent with the actual data.

Impact

Any query that needs to read the affected data blocks will fail with this error. If the corruption is limited to specific parts, only queries scanning those parts are impacted. Background operations like merges will also fail if they involve corrupted parts, leading to growing part counts. In a replicated setup, this can stall replication queues for the affected partitions.

Common Causes

1. Data corruption on disk due to hardware failures, bad sectors, or filesystem bugs
2. The data was compressed with a different codec than what the metadata indicates, often due to a failed or interrupted codec change
3. Incomplete writes from a crash or power loss during an insert or merge operation
4. Network corruption during replication that was not caught by higher-level checksums
5. Manual modification or truncation of data files on disk
6. Version mismatch where a newer compression format is not fully compatible with an older decompressor

Troubleshooting and Resolution Steps

1. Examine the error details in the ClickHouse logs to identify the specific part and column:

   grep -i "cannot decompress" /var/log/clickhouse-server/clickhouse-server.log
   

2. Check the part integrity:

   CHECK TABLE your_database.your_table;
   

   This will report parts with integrity issues.

3. For replicated tables, detach the corrupted part so it can be re-fetched from a healthy replica:

   ALTER TABLE your_table DETACH PART 'corrupted_part_name';
   

   The replication mechanism will automatically download a new copy.

4. Check disk health for underlying hardware issues:

   smartctl -a /dev/sda
   dmesg | grep -i "i/o error"
   

5. For non-replicated tables, restore from a backup. If no backup exists, you may need to drop the affected part:

   ALTER TABLE your_table DROP PART 'corrupted_part_name';
   

   Accept that data in that part will be lost.

6. Verify codec consistency by checking what codec is assigned to the columns:

   SELECT name, compression_codec
   FROM system.columns
   WHERE database = 'your_database' AND table = 'your_table';
   

7. If the issue appeared after a version change, consider whether a rollback or upgrade might resolve a codec compatibility problem.

Best Practices

• Use ReplicatedMergeTree engines to enable automatic recovery of corrupted parts from replicas.
• Run CHECK TABLE periodically to detect corruption before it impacts production queries.
• Implement proper UPS and storage redundancy to prevent corruption from power loss.
• Keep backups current so that non-replicated tables can be restored when corruption occurs.
• Monitor system logs and system.part_log for early signs of I/O or decompression errors.

Frequently Asked Questions

Q: Is CANNOT_DECOMPRESS the same as CHECKSUM_DOESNT_MATCH?
A: They are related but distinct. CHECKSUM_DOESNT_MATCH means the checksum verification failed before decompression. CANNOT_DECOMPRESS means the decompression library itself returned an error during the decompression process. Both typically indicate data corruption, but they fail at different stages.

Q: Can I recover data from a part that fails to decompress?
A: Generally, no. If the compressed data is corrupted beyond what the decompression library can handle, the data in that block is unrecoverable. Your best options are replicas or backups.

Q: Why did this happen even though my disks seem healthy?
A: Disk health checks may not catch all issues, especially with SSDs that have firmware bugs or controllers that silently corrupt data. RAM errors can also cause corruption during the original compression that only manifests later during decompression. Consider running memory tests if disk diagnostics come back clean.