ClickHouse DB::Exception: Cannot create directory

The "DB::Exception: Cannot create directory" error in ClickHouse is raised when the server tries to create a directory and the operating system rejects the request. The CANNOT_CREATE_DIRECTORY error code appears during operations that require new directories, such as creating tables, writing new data parts, or setting up storage paths. Since ClickHouse organizes data in a directory-per-part structure, directory creation is a fundamental operation that must succeed for the server to function.

Impact

When directory creation fails, the consequences are immediate:

• New table creation will fail entirely
• Insert operations cannot create new data parts
• Merges that produce output in new part directories will stall
• Database creation may fail if the database directory cannot be made
• The server may refuse to start if essential directories are missing and cannot be created

Common Causes

1. The parent directory does not exist or was accidentally removed
2. The ClickHouse user does not have write permissions on the parent directory
3. Disk is full or inodes are exhausted
4. Filesystem mounted read-only
5. SELinux or AppArmor blocking directory creation
6. Storage volume not mounted at the expected path
7. A regular file exists where a directory is expected (name collision)

Troubleshooting and Resolution Steps

1. Check the error log for the target path:

   grep "Cannot create directory" /var/log/clickhouse-server/clickhouse-server.err.log | tail -5
   

2. Verify the parent directory exists and check permissions:

   ls -la /var/lib/clickhouse/data/your_db/
   

   If the parent is missing, create it:

   sudo mkdir -p /var/lib/clickhouse/data/your_db/
   sudo chown clickhouse:clickhouse /var/lib/clickhouse/data/your_db/
   

3. Check disk space and inodes:

   df -h /var/lib/clickhouse
   df -i /var/lib/clickhouse
   

4. Verify filesystem mount mode:

   mount | grep $(df /var/lib/clickhouse --output=source | tail -1)
   

5. Check for name collisions:

   file /var/lib/clickhouse/data/your_db/expected_directory_name
   

   If a regular file exists where a directory should be, remove it (after confirming it is safe).

6. Test directory creation as the ClickHouse user:

   sudo -u clickhouse mkdir /var/lib/clickhouse/test_dir && rmdir /var/lib/clickhouse/test_dir
   

7. Review security policies if the above checks pass:

   sudo ausearch -m avc -ts recent
   sudo journalctl | grep apparmor | tail -10
   

8. Ensure storage volumes are mounted before starting ClickHouse. Configure systemd dependencies:

   [Unit]
   After=var-lib-clickhouse.mount
   Requires=var-lib-clickhouse.mount
   

Best Practices

• Configure ClickHouse to start only after all required storage volumes are mounted
• Maintain proper ownership of the entire data directory tree
• Monitor inode usage alongside disk space
• Verify directory structure integrity after OS upgrades or storage migrations
• Avoid placing regular files in paths where ClickHouse expects to create directories
• Use automation to ensure consistent permissions and directory structure across cluster nodes

Frequently Asked Questions

Q: Does ClickHouse create its directory structure automatically?
A: Yes, ClickHouse creates database and table directories as needed during DDL operations. However, the root data path (e.g., /var/lib/clickhouse) must exist and be writable. Custom storage paths in storage policies must also exist beforehand.

Q: I see this error during server startup. What happened?
A: The data directory may have been deleted or the storage volume may not be mounted. Check that the path configured in <path> in config.xml exists and is writable by the clickhouse user.

Q: Can this affect ClickHouse Keeper (ZooKeeper replacement)?
A: Yes. ClickHouse Keeper stores its data in directories as well. If it cannot create directories for snapshots or logs, it will fail to operate, which in turn affects replicated tables.

Q: How do I fix this in a Docker container?
A: Ensure the volume mounted to /var/lib/clickhouse inside the container is writable. Check that the volume's filesystem permissions match the UID/GID of the clickhouse user inside the container (typically UID 101).