ClickHouse DB::Exception: Google Cloud error

The "DB::Exception: Google Cloud error" in ClickHouse surfaces when a Google Cloud Storage (GCS) operation fails. The GOOGLE_CLOUD_ERROR error code acts as a wrapper around the underlying GCS API error, which could stem from authentication failures, missing buckets, permission issues, or network problems. The full error message typically includes details from the Google Cloud API response.

Impact

When this error occurs, any ClickHouse operation that depends on GCS will fail. This includes queries against tables backed by GCS (such as the s3 table function pointed at a gs:// URL or a MergeTree table with GCS-based storage), backup and restore operations targeting GCS, and data imports or exports using GCS paths. Depending on the root cause, the impact may be intermittent (network issues) or persistent (misconfigured credentials).

Common Causes

1. Invalid or expired service account credentials or HMAC keys
2. The specified GCS bucket does not exist or the name is misspelled
3. Insufficient IAM permissions on the service account (e.g., missing storage.objects.get or storage.objects.create)
4. Network connectivity issues between the ClickHouse server and Google Cloud endpoints
5. Bucket region restrictions or VPC Service Controls blocking access
6. Exceeded GCS API rate limits or quotas
7. Incorrect endpoint URL configuration in ClickHouse settings
8. Using a deprecated authentication method or API version

Troubleshooting and Resolution Steps

1. Read the full error message carefully. The GOOGLE_CLOUD_ERROR usually includes the HTTP status code and a message from the GCS API that indicates the specific failure reason.

2. Verify your credentials are valid and correctly configured in ClickHouse:

   <storage_configuration>
       <disks>
           <gcs>
               <type>s3</type>
               <endpoint>https://storage.googleapis.com/my-bucket/data/</endpoint>
               <access_key_id>YOUR_HMAC_KEY</access_key_id>
               <secret_access_key>YOUR_HMAC_SECRET</secret_access_key>
           </gcs>
       </disks>
   </storage_configuration>
   

3. Test connectivity to GCS from the ClickHouse server:

   curl -I https://storage.googleapis.com/my-bucket/
   

4. Check IAM permissions for the service account. At minimum, you typically need:

   • storage.objects.list
   • storage.objects.get
   • storage.objects.create
   • storage.objects.delete (if ClickHouse needs to manage parts)

5. Verify the bucket exists and the name is correct:

   gsutil ls gs://my-bucket/
   

6. If using VPC Service Controls, ensure the ClickHouse server's network is within the allowed perimeter or that the appropriate access levels are configured.

7. Check for GCS quota or rate limit issues in the Google Cloud Console under the Cloud Storage API quotas page.

8. Review ClickHouse server logs for additional context about the failed request, including HTTP response codes and retry attempts.

Best Practices

• Use service account keys with the minimum required permissions following the principle of least privilege.
• Rotate HMAC keys and service account credentials on a regular schedule.
• Monitor GCS API quotas and set up alerts for approaching limits.
• Configure retry settings in ClickHouse for transient GCS errors to improve resilience.
• Test GCS connectivity and permissions as part of your deployment verification process.
• Keep ClickHouse updated to benefit from improvements in GCS integration and error handling.

Frequently Asked Questions

Q: Can I use a service account JSON key file instead of HMAC keys?
A: ClickHouse primarily supports HMAC keys for GCS access through its S3-compatible interface. Some configurations may support service account JSON keys depending on the version and setup. Check your ClickHouse version's documentation for the latest authentication options.

Q: Why do I get this error intermittently?
A: Intermittent GOOGLE_CLOUD_ERROR failures are usually caused by network instability, transient GCS outages, or API rate limiting. Configuring retries and monitoring GCS service status can help mitigate these.

Q: Does ClickHouse support the native GCS API or only the S3-compatible interface?
A: ClickHouse accesses GCS through its S3-compatible API endpoint (storage.googleapis.com). The GOOGLE_CLOUD_ERROR code is specific to errors received from Google Cloud endpoints.

Q: How do I check if my GCS bucket is accessible from the ClickHouse server?
A: Run a simple curl or gsutil command from the ClickHouse server to verify network access and authentication. If those succeed but ClickHouse still fails, the issue is likely in the ClickHouse configuration.