ClickHouse DB::Exception: Cannot connect to RabbitMQ

The "DB::Exception: Cannot connect to RabbitMQ" error occurs when ClickHouse is unable to establish an AMQP connection to a RabbitMQ broker. The CANNOT_CONNECT_RABBITMQ error code is raised during table creation or when an existing RabbitMQ engine table tries to reconnect after a connection loss.

Impact

This error prevents ClickHouse from consuming messages from or producing messages to RabbitMQ queues. Any RabbitMQ engine table affected by this error will stop ingesting data, which can lead to message backlog in RabbitMQ and stale data in ClickHouse. Queries that attempt to read from a disconnected RabbitMQ table will return an error.

Common Causes

1. RabbitMQ broker is not running or not reachable from the ClickHouse host.
2. Incorrect hostname, port, or AMQP protocol version in the ClickHouse table definition.
3. Wrong credentials or the RabbitMQ user does not have access to the specified virtual host.
4. Firewall rules blocking the AMQP port (default 5672 for plain, 5671 for TLS).
5. TLS configuration mismatch — RabbitMQ requires TLS but ClickHouse is connecting without it, or certificate verification fails.
6. The RabbitMQ virtual host specified in the ClickHouse configuration does not exist.
7. RabbitMQ connection limits reached on the broker side.

Troubleshooting and Resolution Steps

1. Check the ClickHouse server log for the AMQP library error:

   grep -i "CANNOT_CONNECT_RABBITMQ\|RabbitMQ\|AMQP" /var/log/clickhouse-server/clickhouse-server.log | tail -20
   

2. Verify RabbitMQ is running and listening:

   rabbitmqctl status
   nc -zv rabbitmq-host 5672
   

3. Test the connection with a standalone AMQP client:

   # Using Python's pika
   python3 -c "
   import pika
   conn = pika.BlockingConnection(pika.ConnectionParameters('rabbitmq-host', 5672, '/', pika.PlainCredentials('user', 'pass')))
   print('Connected')
   conn.close()
   "
   

4. Verify the virtual host exists and the user has access:

   rabbitmqctl list_vhosts
   rabbitmqctl list_permissions -p /
   

5. Review the ClickHouse table definition for correctness:

   CREATE TABLE rabbitmq_table
   (
       message String
   )
   ENGINE = RabbitMQ
   SETTINGS
       rabbitmq_host_port = 'rabbitmq-host:5672',
       rabbitmq_exchange_name = 'my_exchange',
       rabbitmq_format = 'JSONEachRow',
       rabbitmq_username = 'user',
       rabbitmq_password = 'pass',
       rabbitmq_vhost = '/';
   

6. If TLS is required, configure the appropriate settings:

   SETTINGS
       rabbitmq_host_port = 'rabbitmq-host:5671',
       rabbitmq_secure = 1;
   

7. Check RabbitMQ connection limits in the management UI or via CLI:

   rabbitmqctl list_connections | wc -l
   

Best Practices

• Monitor RabbitMQ broker health and connection counts alongside ClickHouse to detect connectivity issues early.
• Use dedicated RabbitMQ credentials for ClickHouse with permissions scoped to the required virtual host and exchanges.
• Configure automatic reconnection behavior in ClickHouse by setting appropriate retry parameters.
• Enable TLS for RabbitMQ connections in production environments.
• Set RabbitMQ connection and channel limits high enough to accommodate all ClickHouse tables and their consumer threads.
• Use the RabbitMQ management plugin to monitor queue depths and consumer activity.

Frequently Asked Questions

Q: Does ClickHouse automatically reconnect to RabbitMQ after a transient failure?
A: Yes, the RabbitMQ engine in ClickHouse includes reconnection logic. After a connection loss, it will periodically attempt to reconnect. However, if the underlying cause (wrong credentials, missing vhost) is not fixed, reconnection attempts will continue to fail.

Q: Can ClickHouse connect to a RabbitMQ cluster?
A: ClickHouse connects to a single RabbitMQ endpoint. To achieve high availability, place a load balancer in front of the RabbitMQ cluster or use RabbitMQ's built-in client-side failover by listing multiple hosts if supported by your setup.

Q: What AMQP versions does ClickHouse support?
A: ClickHouse uses AMQP 0-9-1, which is the protocol version supported by RabbitMQ. It does not support AMQP 1.0.

Q: How do I check if messages are being consumed from RabbitMQ by ClickHouse?
A: Use the RabbitMQ management UI or rabbitmqctl list_consumers to verify that ClickHouse consumers are registered. You can also query system.rabbitmq_consumers in newer ClickHouse versions for consumer status.