ClickHouse DB::Exception: Cannot unlink file

The "DB::Exception: Cannot unlink file" error in ClickHouse is raised when the server attempts to delete a file and the operating system refuses the request. The CANNOT_UNLINK error code points to a filesystem-level problem preventing file removal. ClickHouse deletes files regularly as part of normal operations -- dropping parts after merges, cleaning up temporary files, and removing old data. When this fails, cleanup stalls and disk usage can grow unchecked.

Impact

An inability to delete files has cascading effects:

• Disk space is not reclaimed after merges, leading to gradual disk exhaustion
• Temporary files accumulate, consuming additional space
• DROP TABLE or DROP PARTITION operations may fail
• Old parts that should have been removed persist, potentially confusing metadata
• Mutations that produce new parts cannot clean up the old ones

Common Causes

1. The ClickHouse process does not have write permission on the parent directory
2. The file or directory has the immutable attribute set (chattr +i)
3. SELinux or AppArmor policy blocking unlink operations
4. The filesystem is mounted read-only
5. Another process has locked the file (some backup tools lock files during copying)
6. A filesystem bug or corruption that prevents directory entry removal
7. Insufficient permissions due to running ClickHouse in a restricted container environment

Troubleshooting and Resolution Steps

1. Identify the file from the error log:

   grep "Cannot unlink" /var/log/clickhouse-server/clickhouse-server.err.log | tail -5
   

2. Check the file and directory permissions:

   ls -la /path/to/affected/file
   ls -la /path/to/affected/
   

   The ClickHouse user needs write permission on the parent directory to unlink a file:

   sudo chown -R clickhouse:clickhouse /var/lib/clickhouse/data/your_db/your_table/
   

3. Check for immutable attributes:

   lsattr /path/to/affected/file
   

   If you see an i flag, remove it:

   sudo chattr -i /path/to/affected/file
   

4. Check for security policy denials:

   sudo ausearch -m avc -ts recent  # SELinux
   sudo journalctl | grep apparmor | tail -10  # AppArmor
   

   Update policies to allow ClickHouse to unlink files in its data directory.

5. Verify filesystem mount mode:

   mount | grep $(df /var/lib/clickhouse --output=source | tail -1)
   

   Remount read-write if necessary.

6. Check if another process holds the file:

   fuser /path/to/affected/file
   lsof /path/to/affected/file
   

   Wait for the other process to release the file or stop it.

7. Manually delete the file if ClickHouse is unable to, after confirming it is safe:

   sudo rm /path/to/affected/file
   

Best Practices

• Ensure consistent ownership of the entire ClickHouse data directory tree by the clickhouse user
• Avoid setting immutable attributes on any files within the data directory
• Configure backup tools to use copy-on-read rather than file locking
• Review SELinux and AppArmor policies during deployment, not after errors occur
• Monitor disk space to catch the symptom (growing usage) even if the cause (unlink failures) goes unnoticed initially
• In container deployments, verify that the ClickHouse process has appropriate filesystem permissions on mounted volumes

Frequently Asked Questions

Q: When does ClickHouse need to delete files?
A: ClickHouse deletes files after completing merges (removing old parts), when dropping tables or partitions, during mutation cleanup, and when removing temporary files used for query processing.

Q: Will ClickHouse retry the unlink operation?
A: ClickHouse does not automatically retry failed unlink operations. The old files will remain on disk until the issue is fixed and the cleanup is triggered again (e.g., by another merge or a server restart).

Q: Can accumulated unlinked files cause the server to crash?
A: Not directly, but if enough unlinked files accumulate, the disk may fill up, which will cause write failures and potentially make the server unusable.

Q: Is it safe to manually delete files that ClickHouse failed to unlink?
A: If the file belongs to an inactive (non-active) part that has already been superseded by a merge, it is generally safe to remove. Check system.parts to confirm the part is not marked as active before deleting.