ClickHouse DB::Exception: Query with same ID is already running

The "DB::Exception: Query with same ID is already running" error in ClickHouse occurs when you submit a query with a query_id that matches one already being executed. ClickHouse requires each running query to have a unique identifier. If you explicitly set a query_id and that ID is already in use, the server rejects the new query. The error code is QUERY_WITH_SAME_ID_IS_ALREADY_RUNNING.

Impact

The new query is rejected, but the existing query with that ID continues running unaffected. This error commonly surfaces in retry logic, scheduled jobs, or application code that assigns fixed query IDs. It prevents duplicate execution of the same logical query, which can actually be a useful safety mechanism.

Common Causes

1. Application retry logic reusing the same query_id -- a failed or timed-out request is retried with the same ID, but the original is still running on the server.
2. Scheduled jobs overlapping -- a cron job or scheduler fires before the previous execution completes, using the same query_id.
3. Fixed query_id in application code -- hardcoding a query_id rather than generating unique ones.
4. Load balancer retries -- a reverse proxy or load balancer retries a request to a different ClickHouse node while the original is still executing.
5. Distributed queries -- in some cluster configurations, subqueries on different shards may conflict with queries using the same ID.

Troubleshooting and Resolution Steps

1. Check which query is already running with that ID:

   SELECT query_id, query, elapsed, user FROM system.processes
   WHERE query_id = 'your_query_id';
   

2. Wait for the existing query to complete. If it is a legitimate query, let it finish:

   -- Monitor progress
   SELECT query_id, elapsed, read_rows, total_rows_approx
   FROM system.processes
   WHERE query_id = 'your_query_id';
   

3. Kill the existing query if it is stale or unwanted:

   KILL QUERY WHERE query_id = 'your_query_id';
   

4. Use the replace_running_query setting. This tells ClickHouse to kill the old query and run the new one instead:

   SET replace_running_query = 1;
   SELECT ... SETTINGS query_id = 'your_query_id';
   

   Or per-query:

   SELECT ... SETTINGS query_id = 'my_id', replace_running_query = 1;
   

5. Generate unique query IDs. Instead of fixed IDs, use UUIDs or timestamps:

   import uuid
   query_id = f"etl_daily_{uuid.uuid4()}"
   client.execute(query, query_id=query_id)
   

6. Add overlap protection to scheduled jobs. Use a lock file or check if the previous execution is still running before starting a new one:

   SELECT count() FROM system.processes
   WHERE query_id LIKE 'my_etl_job_%' AND is_cancelled = 0;
   

Best Practices

• Use unique query IDs for each query execution. Append a UUID or timestamp to any logical identifier.
• If you intentionally want to prevent duplicate execution, use fixed query IDs as a concurrency control mechanism.
• For scheduled jobs that should not overlap, check system.processes before launching a new execution.
• Use replace_running_query = 1 only when you explicitly want the new query to supersede the old one (e.g., dashboard refreshes).
• Set appropriate timeouts (max_execution_time) on long-running queries to prevent them from blocking future executions indefinitely.

Frequently Asked Questions

Q: What is the default query_id if I do not set one?
A: ClickHouse automatically generates a unique UUID for each query if you do not specify one. This means you will never see this error unless you explicitly set query IDs.

Q: Does replace_running_query immediately kill the old query?
A: It sends a cancellation signal to the old query. The old query may take a moment to stop, controlled by the replace_running_query_max_wait_ms setting (default 5000ms). If the old query does not stop in time, the new query may also fail.

Q: Can different users have the same query_id?
A: Yes. Query IDs are scoped per-user by default. Two different users can run queries with the same query_id simultaneously without conflict. This behavior is controlled by the server configuration.

Q: How do I use query_id for deduplication in a retry scenario?
A: If you want idempotent retries, you can use a fixed query_id with replace_running_query = 1. This ensures only one copy of the query runs at a time. For INSERT deduplication, consider using insert_deduplication_token instead, which is designed specifically for that purpose.