ClickHouse DB::Exception: Cannot create hard link

The "DB::Exception: Cannot create hard link" error in ClickHouse occurs when the server fails to create a hard link between files. Identified by the CANNOT_LINK error code, this problem arises because ClickHouse uses hard links extensively for efficient data management -- particularly during mutations, freezing tables for backups, and certain merge operations. Hard links allow ClickHouse to reference the same physical data from multiple part directories without duplicating it on disk.

Impact

When hard link creation fails, the effects include:

• ALTER TABLE FREEZE (backup snapshots) will not complete
• Mutations that create new parts while preserving unchanged columns may fail
• Certain internal data reorganization operations will be blocked
• Increased disk usage if ClickHouse falls back to copying data instead of linking (where supported)

Common Causes

1. Attempting to create a hard link across different filesystems or mount points (hard links cannot span devices)
2. The filesystem does not support hard links (some FUSE or overlay filesystems)
3. The maximum number of hard links per inode has been reached (ext4 limit: 65,000)
4. Permission denied on the source or destination directory
5. The filesystem is mounted read-only
6. SELinux or AppArmor blocking the link operation
7. Storage policies that place data on different physical volumes, making cross-device links impossible

Troubleshooting and Resolution Steps

1. Identify the source and target paths from the error log:

   grep "Cannot create hard link\|Cannot link" /var/log/clickhouse-server/clickhouse-server.err.log | tail -5
   

2. Check whether both paths are on the same filesystem:

   df /path/to/source
   df /path/to/destination
   

   If the device differs, hard links are impossible. You need to reorganize your storage so that the operation stays within a single filesystem.

3. Verify filesystem support for hard links:

   touch /var/lib/clickhouse/test_link_source
   ln /var/lib/clickhouse/test_link_source /var/lib/clickhouse/test_link_dest
   rm /var/lib/clickhouse/test_link_source /var/lib/clickhouse/test_link_dest
   

   If the ln command fails, the filesystem doesn't support hard links.

4. Check the hard link count on the source file:

   stat /path/to/source/file
   

   If the link count is near the filesystem's maximum, the inode is exhausted.

5. Verify permissions:

   ls -la /path/to/source/
   ls -la /path/to/destination/
   

   Ensure the ClickHouse user has write access to both directories:

   sudo chown -R clickhouse:clickhouse /var/lib/clickhouse/
   

6. Review ClickHouse storage policies: If you use a multi-volume storage policy, confirm that operations requiring hard links (like FREEZE) operate within a single volume:

   <storage_configuration>
     <disks>
       <default>
         <path>/var/lib/clickhouse/</path>
       </default>
       <shadow>
         <path>/var/lib/clickhouse/shadow/</path>  <!-- Must be on the same filesystem -->
       </shadow>
     </disks>
   </storage_configuration>
   

7. For cross-device issues, restructure storage so that the ClickHouse data directory and shadow (backup) directory reside on the same filesystem.

Best Practices

• Keep the ClickHouse data directory and shadow directory on the same filesystem to ensure hard links work for FREEZE operations
• Use filesystems that fully support hard links (ext4, xfs) rather than overlay or network filesystems
• When using multi-disk storage policies, be aware of which operations require hard links and plan volume layouts accordingly
• Monitor hard link counts on heavily mutated tables to detect approaching filesystem limits
• Ensure consistent permissions across all ClickHouse data directories

Frequently Asked Questions

Q: Why does ClickHouse use hard links instead of copying files?
A: Hard links are instantaneous and consume no additional disk space because they point to the same underlying data. This makes operations like FREEZE (for backups) extremely fast and space-efficient compared to full copies.

Q: Can I use symbolic links instead?
A: ClickHouse internally uses hard links, not symbolic links, for data management operations. You cannot substitute one for the other. Symbolic links in the data directory are supported for other purposes (like pointing to different storage), but they won't help with this specific error.

Q: Does this affect ClickHouse Cloud?
A: ClickHouse Cloud manages storage transparently, and you typically won't encounter this error. It primarily affects self-hosted deployments where you control the filesystem layout.

Q: I hit the hard link limit on ext4. What are my options?
A: You can switch to xfs, which supports a much higher hard link limit (theoretically billions). Alternatively, reduce the number of mutations on affected tables, as each mutation creates new parts linked to old column files.