ClickHouse DB::Exception: Cannot statvfs

The "DB::Exception: Cannot statvfs" error in ClickHouse occurs when the server attempts to query filesystem statistics (such as free space and total capacity) and the OS call fails. The CANNOT_STATVFS error code indicates that ClickHouse was unable to determine the state of the filesystem where data is stored. This information is critical for storage policy decisions, free space checks, and preventing writes to full disks.

Impact

Without filesystem statistics, ClickHouse faces operational challenges:

• The server cannot determine available disk space, potentially leading to writes on a full disk
• Storage policies that route data based on free space will not function correctly
• Health checks and monitoring queries against system.disks may return errors
• ClickHouse may refuse to start or operate if it cannot verify storage availability

Common Causes

1. The filesystem or mount point is no longer accessible (unmounted, disconnected network volume)
2. The path configured in ClickHouse storage policies does not exist
3. Permission denied -- the ClickHouse process cannot stat the filesystem
4. A corrupted or hung filesystem that does not respond to system calls
5. The storage path is a symbolic link pointing to a nonexistent target
6. Container or chroot environment where the filesystem is not properly mounted

Troubleshooting and Resolution Steps

1. Verify the path exists and is accessible:

   ls -la /var/lib/clickhouse
   stat /var/lib/clickhouse
   

   If the path doesn't exist, the volume may need to be remounted.

2. Test statvfs manually:

   stat -f /var/lib/clickhouse
   

   If this command fails, the problem is at the OS level.

3. Check mount status:

   mount | grep /var/lib/clickhouse
   df /var/lib/clickhouse
   

   If the mount is missing, remount the volume:

   sudo mount -a
   

4. Inspect ClickHouse storage configuration: Check the storage policies in your config to confirm all paths are valid:

   grep -A 10 "<path>" /etc/clickhouse-server/config.xml
   

   Look for <path> entries that reference non-existent directories.

5. Check for permission issues:

   sudo -u clickhouse stat -f /var/lib/clickhouse
   

   If this fails but running as root succeeds, it's a permission issue.

6. For network storage, verify connectivity:

   showmount -e nfs-server  # For NFS
   ping storage-host
   

7. Restart ClickHouse after resolving the mount or path issue. The server will re-query filesystem stats on startup.

Best Practices

• Use automount or systemd mount units to ensure storage volumes are reliably mounted before ClickHouse starts
• Configure ClickHouse to depend on mount targets in systemd to prevent startup before storage is ready
• Validate all storage policy paths during configuration changes before restarting ClickHouse
• Monitor mount point availability as part of your infrastructure health checks
• Avoid symlinks in storage paths unless the target is guaranteed to be present
• Test storage configuration changes in a staging environment before applying to production

Frequently Asked Questions

Q: Why does ClickHouse need statvfs?
A: ClickHouse uses statvfs to determine how much free space is available on each configured disk. This drives storage policy decisions (such as moving data between hot and cold tiers), enforces min_free_disk_space limits, and provides the data shown in system.disks.

Q: Can a temporary network glitch cause this error?
A: Yes, if ClickHouse data resides on network-attached storage (NFS, CIFS, or a cloud disk), a brief connectivity loss can make the filesystem temporarily unavailable, triggering this error. The issue typically resolves once the network connection is restored.

Q: Will this error cause data loss?
A: Not directly. The error prevents ClickHouse from checking free space, but it does not corrupt or delete data. However, if the underlying problem is that a volume has become unmounted, data on that volume will be inaccessible until it is remounted.

Q: I use multiple disks in a storage policy. Does this error affect all of them?
A: The error is specific to the path that failed. Other disks in the storage policy may continue to function normally, though ClickHouse might not be able to route data away from the failing disk as intended.