ClickHouse DB::Exception: LZ4 decoder failed

The "DB::Exception: LZ4 decoder failed" error in ClickHouse indicates that the LZ4 decompression library could not successfully decode a compressed data block. The LZ4_DECODER_FAILED error strongly suggests data corruption, since LZ4 decompression rarely fails for any other reason. The compressed data on disk does not match what the LZ4 decoder expects, and it cannot produce valid output.

Impact

Queries that access the corrupted parts will fail. Since LZ4 is ClickHouse's default codec, this can affect many tables. Merges involving corrupted parts will also fail, and if enough parts are affected, query performance may degrade due to part count accumulation. In replicated environments, this can block replication progress.

Common Causes

1. On-disk data corruption from storage hardware failures or bad sectors
2. Incomplete writes due to a crash, power failure, or forced process termination during a merge or insert
3. Memory corruption during the original compression that produced invalid LZ4 output
4. Network-level corruption during replication data transfer
5. Filesystem corruption from unexpected shutdowns or kernel bugs
6. Manual tampering with or accidental modification of data files in the ClickHouse data directory

Troubleshooting and Resolution Steps

1. Identify the corrupted part from the error output:

   grep -i "LZ4.*decoder\|LZ4_DECODER" /var/log/clickhouse-server/clickhouse-server.log
   

2. Check all parts in the affected table:

   CHECK TABLE your_database.your_table;
   

   This will list which parts pass or fail integrity checks.

3. For ReplicatedMergeTree tables, detach the bad part and let replication heal it:

   ALTER TABLE your_table DETACH PART 'corrupted_part';
   

   The replication mechanism will automatically download a fresh copy from a healthy replica.

4. For non-replicated tables, restore from backup or drop the corrupted part:

   -- Last resort if no backup is available
   ALTER TABLE your_table DROP PART 'corrupted_part';
   

5. Inspect storage health:

   smartctl -a /dev/sda
   dmesg | grep -i "error\|i/o\|fault"
   

   Replace failed or degraded disks promptly.

6. Check for filesystem errors:

   # Run on unmounted filesystem
   fsck /dev/sda1
   

7. Test system memory to rule out RAM-related corruption:

   memtest86+
   

Best Practices

• Use ReplicatedMergeTree for all production tables to enable automatic corruption recovery.
• Deploy storage with redundancy (RAID, distributed storage) to protect against single-disk failures.
• Use ECC RAM to prevent bit-flip corruption during compression and decompression.
• Schedule periodic CHECK TABLE operations and alert on failures.
• Maintain current backups for all critical non-replicated tables.

Frequently Asked Questions

Q: Is LZ4_DECODER_FAILED always caused by disk corruption?
A: Disk corruption is the most common cause, but not the only one. The data could have been corrupted in RAM during the original write, during network transfer in replication, or by a filesystem bug. The key point is that the compressed bytes on disk are not valid LZ4 data.

Q: Can I recover the data from a corrupted LZ4 part?
A: If the LZ4 decoder cannot process the data, that specific block is unrecoverable from the corrupted copy. Your recovery options are replicas and backups. There is no partial recovery mechanism for LZ4-compressed blocks.

Q: How does this differ from a checksum mismatch?
A: A checksum mismatch is detected before decompression begins -- the checksum of the compressed block does not match. LZ4_DECODER_FAILED means the checksum may have passed (or was not checked at that stage) but the decompressor itself failed. Both indicate corruption but at different validation stages.