ClickHouse DB::Exception: Google Cloud Platform error

The "DB::Exception: Google Cloud Platform error" is raised when ClickHouse fails to communicate with a Google Cloud service, most commonly Google Cloud Storage (GCS). The GCP_ERROR code wraps the HTTP error returned by the GCS JSON or XML API, and the message typically includes the status code and a description of what went wrong.

Impact

When this error occurs, ClickHouse cannot read from or write to GCS-backed tables, disks, or backup destinations. Queries on tables using the GCS table function or engine will fail, and background merges on storage policies that include GCS disks will stall. Any active backup or restore job targeting a GCS bucket will also be interrupted.

Common Causes

1. Missing or invalid service account credentials — the HMAC key, JSON key file, or metadata-based credentials are not available to the ClickHouse process.
2. The target GCS bucket does not exist or is in a different project than expected.
3. Insufficient IAM permissions on the service account — it lacks storage.objects.get, storage.objects.create, or storage.objects.delete.
4. Network connectivity problems between the ClickHouse node and storage.googleapis.com.
5. The bucket has uniform bucket-level access enabled but the IAM binding is missing.
6. Request throttling when ClickHouse exceeds GCS request rate limits for a given bucket.
7. Incorrect endpoint URL when using a custom or private GCS endpoint.

Troubleshooting and Resolution Steps

1. Inspect the full error message in the ClickHouse server log for the HTTP status code and Google error description:

   grep -i "GCP_ERROR\|GoogleException\|GCS" /var/log/clickhouse-server/clickhouse-server.log | tail -30
   

2. For 403 Forbidden errors, verify the service account's IAM roles. The service account needs at minimum the Storage Object Admin role on the bucket:

   gcloud storage buckets get-iam-policy gs://your-bucket
   

3. Confirm the bucket exists and is in the expected project:

   gcloud storage buckets describe gs://your-bucket
   

4. If using HMAC keys for GCS (S3-compatible access), verify they have not been deactivated:

   gcloud storage hmac list --service-account=your-sa@project.iam.gserviceaccount.com
   

5. Check that ClickHouse can reach the GCS endpoint:

   curl -I https://storage.googleapis.com/your-bucket/
   

6. If running on GCE or GKE, verify the VM or pod has the correct service account attached and that the storage-full or storage-rw scope is assigned:

   gcloud compute instances describe your-instance --format="get(serviceAccounts)"
   

7. For throttling issues (HTTP 429), reduce ClickHouse read/write concurrency and consider distributing objects across multiple buckets or prefixes.

Best Practices

• Use Workload Identity on GKE or the VM service account on GCE rather than static HMAC keys or JSON key files.
• Grant the minimum required IAM permissions — Storage Object Admin on the specific bucket rather than project-wide roles.
• Place ClickHouse nodes in the same GCP region as the GCS bucket to minimize latency and avoid egress charges.
• Monitor GCS metrics in the Cloud Console (request counts, error rates, latency) alongside ClickHouse server metrics.
• Enable GCS audit logging to trace which requests are being denied and why.
• Set appropriate retry settings in the ClickHouse configuration to handle transient GCS failures.

Frequently Asked Questions

Q: Can I use ClickHouse's S3 table function to access GCS?
A: Yes, GCS provides an S3-compatible XML API. You can use the S3 table function or disk configuration with the https://storage.googleapis.com/your-bucket endpoint and HMAC credentials. Failures may surface as either S3_ERROR or GCP_ERROR depending on the error path.

Q: Why do I get GCP_ERROR with "Anonymous caller does not have storage.objects.get access"?
A: This means ClickHouse is not sending valid credentials. Confirm the service account key file path is correct in the configuration or that the instance metadata service is reachable for automatic credential discovery.

Q: Does ClickHouse support GCS dual-region or multi-region buckets?
A: Yes, ClickHouse interacts with GCS through standard APIs and is agnostic to the bucket location type. However, be aware of the higher latency that can come with multi-region configurations.

Q: How do I switch from HMAC keys to a service account JSON key?
A: Update your ClickHouse disk or table configuration to reference the JSON key file path instead of the access key and secret. Restart ClickHouse or reload the configuration for the change to take effect.