ClickHouse DB::Exception: Cannot remove directory

The "DB::Exception: Cannot remove directory" error in ClickHouse means the server tried to delete a directory and the operating system rejected the request. The CANNOT_RMDIR error code appears during operations like cleaning up merged parts, dropping tables, or removing empty partition directories. Since ClickHouse manages a tree of directories for its data storage, an inability to remove directories disrupts normal housekeeping.

Impact

Directory removal failures lead to:

Stale part directories accumulating on disk after merges
Failed DROP TABLE or DROP DATABASE commands
Growing disk usage from directories that should have been cleaned up
Potential confusion in metadata if empty directories persist
Blocked cleanup operations that keep the data directory cluttered

Common Causes

The directory is not empty -- files remain inside that prevent rmdir from succeeding
Permission issues on the directory or its parent
The filesystem is mounted read-only
Another process has a working directory or open file descriptor inside the directory
SELinux or AppArmor denying the operation
Filesystem corruption preventing directory removal
The directory is a mount point for another filesystem

Troubleshooting and Resolution Steps

Find the affected directory in the error log:

grep "Cannot remove directory\|Cannot rmdir" /var/log/clickhouse-server/clickhouse-server.err.log | tail -5

Check if the directory is empty:
```
ls -la /path/to/affected/directory/
```
If files remain, they may need to be removed first. Check whether they belong to an old part that should have been cleaned up:
```
find /path/to/affected/directory/ -type f
```

Verify permissions:

stat /path/to/affected/directory
stat $(dirname /path/to/affected/directory)

Fix ownership if needed:

sudo chown -R clickhouse:clickhouse /path/to/affected/directory

Check for processes using the directory:

fuser -m /path/to/affected/directory
lsof +D /path/to/affected/directory

Verify the directory is not a mount point:
```
mountpoint /path/to/affected/directory
```
If it is a mount point, it cannot be removed with rmdir. Unmount it first if appropriate.

Check filesystem state:

mount | grep $(df /var/lib/clickhouse --output=source | tail -1)
dmesg | grep -i error | tail -10

Manually clean up if the directory contains leftover files from a completed merge:
```
sudo rm -rf /path/to/affected/directory
```
Only do this after confirming the part is not active in system.parts.

Best Practices

Let ClickHouse manage its data directory entirely through SQL commands
Avoid running processes that create files inside ClickHouse data directories (e.g., monitoring agents writing temp files)
Monitor for growing numbers of inactive parts or stale directories
Ensure consistent ownership across the entire ClickHouse data directory
When using multiple storage volumes, ensure mount points do not overlap with ClickHouse part directories
Periodically check for orphaned directories that may have been left by failed operations

Frequently Asked Questions

Q: Why can't ClickHouse remove a directory that looks empty?
A: Hidden files (names starting with .) or filesystem metadata (like .nfs lock files on NFS) can prevent rmdir from working even when ls shows the directory as empty. Use ls -la to check for hidden entries.

Q: Is it safe to manually remove directories in the ClickHouse data path?
A: Only remove directories for parts that are confirmed inactive (not listed as active in system.parts). Removing active part directories will cause data loss.

Q: Will this error fix itself after a restart?
A: A restart may help if the issue was a transient lock or process holding the directory open. However, if the root cause is permissions or a non-empty directory, the error will persist until you address it directly.

Q: How do orphaned directories happen?
A: They can result from interrupted merges, server crashes during cleanup, or bugs in older ClickHouse versions. ClickHouse tries to clean them up during startup, but if the cleanup itself fails, the directories remain.