ClickHouse DB::Exception: Cannot fork process

The "DB::Exception: Cannot fork process" error in ClickHouse means the fork() system call has failed. ClickHouse occasionally needs to spawn child processes -- for example, when running executable UDFs or dictionary sources that invoke external programs. The CANNOT_FORK error indicates the operating system refused the fork request, usually due to resource constraints.

Impact

Queries that depend on executable table functions, executable UDFs, or external dictionary sources will fail when this error occurs. The impact is typically scoped to those specific operations, but if the underlying resource issue is severe enough, it may also affect other processes on the system and indicate broader stability concerns.

Common Causes

1. The system has reached its maximum number of processes (nproc limit) for the ClickHouse user
2. Available memory is insufficient to duplicate the process address space during fork()
3. The system-wide PID limit (/proc/sys/kernel/pid_max) has been reached
4. Cgroup limits on the number of processes or memory are too restrictive
5. The kernel's vm.overcommit_memory setting is preventing the memory reservation needed for fork
6. A resource leak in external scripts or child processes has accumulated over time

Troubleshooting and Resolution Steps

1. Check current process limits:

   cat /proc/$(pidof clickhouse-server)/limits | grep "processes"
   

   Compare the current number of processes against the limit.

2. Count running processes for the ClickHouse user:

   ps -u clickhouse --no-headers | wc -l
   

   If this number is close to the nproc limit, you have found the bottleneck.

3. Check the system-wide PID limit:

   cat /proc/sys/kernel/pid_max
   ps aux | wc -l
   

4. Inspect memory availability:

   free -h
   cat /proc/meminfo | grep -E "MemAvailable|CommitLimit|Committed_AS"
   

   If Committed_AS exceeds CommitLimit and overcommit is restricted, fork will fail.

5. Check cgroup limits (containerized environments):

   cat /sys/fs/cgroup/pids/pids.max 2>/dev/null
   cat /sys/fs/cgroup/pids/pids.current 2>/dev/null
   

6. Increase the process limit if it is too low:

   # In /etc/security/limits.conf or systemd override
   # clickhouse soft nproc 65535
   # clickhouse hard nproc 65535
   sudo systemctl edit clickhouse-server
   # Add: LimitNPROC=65535
   sudo systemctl restart clickhouse-server
   

7. Adjust overcommit settings if memory reservation is the issue:

   cat /proc/sys/vm/overcommit_memory
   # Setting to 1 allows overcommit, which may help fork succeed
   sudo sysctl vm.overcommit_memory=1
   

Best Practices

• Set generous nproc limits for the ClickHouse user, especially when using executable UDFs or dictionary sources.
• Monitor process counts and memory usage to catch resource exhaustion before it causes fork failures.
• Keep external scripts invoked by ClickHouse lightweight and ensure they exit promptly to avoid process accumulation.
• In containerized environments, set the PIDs limit high enough to accommodate ClickHouse and any child processes it spawns.
• Prefer native ClickHouse functions and built-in dictionary sources over executable ones when possible, to avoid the need for forking entirely.

Frequently Asked Questions

Q: Why does ClickHouse need to fork at all?
A: ClickHouse forks to execute external programs for features like executable table functions, executable user-defined functions, and dictionary sources that call external commands. These features require spawning child processes.

Q: Will increasing RAM fix this error?
A: It can, if the root cause is insufficient memory for the fork operation. However, if the issue is a process count limit rather than memory, adding RAM alone will not help. Check both limits.

Q: Can this error happen on systems with plenty of resources?
A: Yes, if the per-user or per-cgroup process limits are set too low. Even a powerful server can hit CANNOT_FORK if the nproc limit is restrictive.

Q: Is there a way to avoid forking for UDFs?
A: You can use ClickHouse's built-in SQL-based UDFs (lambda functions) or native aggregate function combinators, which do not require forking. Executable UDFs are the ones that trigger fork operations.