ClickHouse DB::Exception: Backup is damaged

The "DB::Exception: Backup is damaged" error means ClickHouse detected that the backup data is corrupted, incomplete, or otherwise inconsistent. The BACKUP_DAMAGED error code is raised during a RESTORE operation when internal checksums do not match, required files are missing, or the backup metadata is malformed.

Impact

A damaged backup cannot be restored. This is one of the most critical backup-related errors because it may mean your recovery point is lost. If this is your only backup, you could face data loss. Discovering a damaged backup during an actual disaster recovery scenario is especially costly because it delays restoration and may force you to fall back to an older or less complete backup.

Common Causes

1. Interrupted backup process -- a backup that was killed mid-write (server crash, OOM kill, manual cancellation) may leave files in an incomplete state.
2. Storage corruption -- bit rot, disk failures, or unreliable network storage can silently corrupt backup files after they were written.
3. Incomplete file transfer -- copying or syncing backup files between storage systems (e.g., rsync, S3 multipart upload) that did not complete fully.
4. Manual modification of backup files -- someone edited, moved, or deleted individual files within the backup directory.
5. Filesystem full during backup -- running out of disk space partway through writing the backup produces truncated files.

Troubleshooting and Resolution Steps

1. Check the error details. ClickHouse usually reports which specific file or checksum failed:

   DB::Exception: Backup is damaged: Checksum mismatch for file 'data/my_db/my_table/all_1_1_0/data.bin'
   

2. Verify backup file integrity. If checksums were stored alongside the backup, validate them:

   cd /var/lib/clickhouse/backups/my_backup/
   find . -name "*.bin" | while read f; do md5sum "$f"; done
   

3. Check for incomplete files. Look for zero-byte or unexpectedly small files:

   find /var/lib/clickhouse/backups/my_backup/ -size 0
   

4. Re-download or re-sync the backup if it was transferred from remote storage. Ensure the transfer tool verifies checksums:

   aws s3 sync s3://my-bucket/backups/my_backup/ /var/lib/clickhouse/backups/my_backup/ --exact-timestamps
   

5. Try restoring individual tables rather than the entire backup. Some tables may be intact even if others are damaged:

   RESTORE TABLE my_db.healthy_table FROM Disk('backups', 'my_backup');
   

6. Fall back to an older backup. If the damaged backup cannot be salvaged, use the most recent undamaged backup:

   RESTORE DATABASE my_db FROM Disk('backups', 'previous_backup');
   

7. If data exists on replicas, you may not need the backup at all. ClickHouse replicated tables will re-sync from healthy replicas automatically.

Best Practices

• Verify backups immediately after creation by running a test restore on a staging server.
• Store backup checksums separately from the backup files themselves so you can detect corruption independently.
• Use storage with built-in integrity guarantees (e.g., S3 with checksums enabled, ZFS with scrubbing).
• Maintain multiple backup generations so a single damaged backup does not leave you without options.
• Monitor disk space on backup destinations to prevent mid-write failures from full disks.
• Set up alerts on backup job exit codes so you catch failures before you need the backup.

Frequently Asked Questions

Q: Can I repair a damaged backup?
A: In most cases, no. ClickHouse does not provide a built-in repair tool for backups. If specific files are corrupted, you may be able to replace them from another copy of the same backup, but this requires identical file-level granularity.

Q: How can I tell if a backup is damaged before I need to restore it?
A: The most reliable method is to perform a test restore to a temporary database or a staging server. You can also verify checksums if they were stored during backup creation.

Q: Does ClickHouse validate backups at creation time?
A: ClickHouse writes checksums as part of the backup process, but it does not re-read and verify the entire backup after writing. A post-backup validation step is your responsibility.

Q: Will replication protect me from needing backups?
A: Replication protects against node failure but not against logical errors like accidental DROP TABLE or bad data mutations. Backups remain essential even with a fully replicated cluster.