ClickHouse DB::Exception: Cannot dlopen shared library

The "DB::Exception: Cannot dlopen shared library" error in ClickHouse occurs when the server attempts to load a dynamic shared library (.so file) and the dlopen() call fails. This typically happens when using user-defined functions (UDFs) backed by shared libraries, or when ClickHouse tries to load an optional library for certain functionality. The CANNOT_DLOPEN error means the library either does not exist, has incompatible dependencies, or cannot be accessed.

Impact

When this error occurs, the specific functionality that depends on the shared library will be unavailable. If the library backs a UDF, any query referencing that function will fail. In some cases, if the library is required during server startup for a configured feature, ClickHouse may log the error and disable that feature or fail to start entirely.

Common Causes

1. The shared library file does not exist at the specified path
2. The library was compiled for a different architecture or glibc version than the running system
3. The library has unresolved dependencies on other shared libraries that are not installed
4. File permissions prevent the ClickHouse process from reading the library
5. The library path is not included in the expected search directories
6. A ClickHouse upgrade changed the expected library interface, making an older library incompatible

Troubleshooting and Resolution Steps

1. Check the full error message in the ClickHouse logs:

   grep -i "CANNOT_DLOPEN\|Cannot dlopen" /var/log/clickhouse-server/clickhouse-server.err.log | tail -10
   

   The message usually includes the library path and the underlying dlerror() output.

2. Verify the library file exists:

   ls -la /path/to/library.so
   

   If the file is missing, you will need to install or rebuild it.

3. Check for missing dependencies:

   ldd /path/to/library.so
   

   Any line showing "not found" indicates a missing dependency that must be installed.

4. Verify architecture compatibility:

   file /path/to/library.so
   uname -m
   

   Ensure the library matches the system architecture (e.g., x86_64 vs. aarch64).

5. Check file permissions:

   stat /path/to/library.so
   sudo -u clickhouse test -r /path/to/library.so && echo "Readable" || echo "Not readable"
   

   The clickhouse user must have read and execute permissions on the library file.

6. Verify the UDF configuration if applicable:

   cat /etc/clickhouse-server/config.d/*_function*
   ls /var/lib/clickhouse/user_defined/
   

   Ensure the function XML or SQL definition points to the correct library path.

7. Rebuild or reinstall the library if it is incompatible:

   # Example for a custom UDF library
   g++ -shared -fPIC -o my_function.so my_function.cpp
   cp my_function.so /var/lib/clickhouse/user_defined/
   chown clickhouse:clickhouse /var/lib/clickhouse/user_defined/my_function.so
   

Best Practices

• Always use absolute paths when referencing shared libraries in ClickHouse configuration.
• Compile UDF libraries on the same OS and architecture where ClickHouse runs, or use static linking for dependencies.
• After ClickHouse upgrades, verify that custom shared libraries remain compatible with the new version.
• Store UDF libraries in a dedicated directory with appropriate permissions, such as /var/lib/clickhouse/user_defined/.
• Include shared library deployment in your infrastructure automation to prevent drift between servers.

Frequently Asked Questions

Q: Where should I place shared libraries for ClickHouse UDFs?
A: The recommended location is the user_defined directory under the ClickHouse data path, typically /var/lib/clickhouse/user_defined/. You can also configure the path using the user_defined_executable_functions_config setting.

Q: Can I use shared libraries compiled on a different Linux distribution?
A: It depends on binary compatibility. Libraries compiled on older glibc versions generally work on newer systems, but not the reverse. The safest approach is to compile on the same distribution and version where ClickHouse runs.

Q: Does ClickHouse support loading shared libraries from network paths?
A: No, the library must be available on the local filesystem. If you need to distribute libraries across a cluster, use your deployment tooling to place the files on each node.

Q: I updated my UDF library but ClickHouse still uses the old version. Why?
A: Shared libraries are loaded into memory and may be cached. You typically need to either reload the function definition or restart ClickHouse for the updated library to take effect.