ClickHouse DB::Exception: AWS error

The "DB::Exception: AWS error" surfaces when the AWS SDK inside ClickHouse encounters a failure that does not map to a more specific error code like S3_ERROR. The AWS_ERROR code acts as a catch-all for SDK-level problems such as credential resolution failures, endpoint discovery issues, STS token errors, and unexpected API responses from any AWS service that ClickHouse interacts with.

Impact

This error blocks any ClickHouse operation that depends on AWS APIs. Depending on which component triggered the error, it can prevent reads and writes to S3-backed storage, stop background merges on tiered storage policies, or interrupt backup and restore operations. If the underlying cause is a credentials issue, all AWS-dependent functionality on the affected node will be impacted simultaneously.

Common Causes

1. The AWS SDK cannot locate valid credentials — no instance profile, no environment variables, and no static keys configured.
2. STS AssumeRole calls fail due to incorrect role ARN, expired session, or missing trust policy.
3. The configured AWS endpoint URL is unreachable or malformed.
4. Region mismatch between the ClickHouse configuration and the actual AWS resource location.
5. The AWS SDK encounters an unexpected HTTP error from a non-S3 AWS service (e.g., KMS, STS).
6. Proxy or network configuration prevents the SDK from reaching AWS APIs.
7. Clock skew on the ClickHouse host causes signature validation to fail.

Troubleshooting and Resolution Steps

1. Check the full error message in the ClickHouse log — the AWS SDK typically includes a detailed description and an AWS request ID:

   grep -i "AWS_ERROR\|Aws::Client" /var/log/clickhouse-server/clickhouse-server.log | tail -30
   

2. Verify that credentials are available to the ClickHouse process. If using an EC2 instance profile, confirm the metadata endpoint is reachable:

   curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
   

3. If using static credentials, make sure they are correctly configured in the ClickHouse config and have not been rotated or deactivated:

   <s3>
     <access_key_id>AKIAIOSFODNN7EXAMPLE</access_key_id>
     <secret_access_key>wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY</secret_access_key>
   </s3>
   

4. For STS / AssumeRole errors, verify the role ARN and the trust policy on the target role. Test manually:

   aws sts assume-role --role-arn arn:aws:iam::123456789012:role/ClickHouseRole --role-session-name test
   

5. Check for clock skew, which causes AWS Signature Version 4 validation to fail:

   date -u
   # Compare with an NTP server
   ntpdate -q pool.ntp.org
   

6. If using a custom endpoint or VPC endpoint, confirm the URL is correct and reachable:

   curl -I https://vpce-0abcdef1234567890-abcdefgh.s3.us-east-1.vpce.amazonaws.com
   

7. Confirm the region setting matches the AWS resource:

   <s3>
     <region>us-east-1</region>
   </s3>
   

Best Practices

• Prefer IAM roles over static credentials to eliminate credential rotation issues.
• Enable NTP synchronization on all ClickHouse nodes to prevent clock skew.
• Set the AWS region explicitly in the ClickHouse configuration rather than relying on auto-detection.
• Monitor AWS CloudTrail for denied API calls originating from ClickHouse nodes.
• Use VPC endpoints for AWS services to improve reliability and reduce data transfer costs.
• Keep the ClickHouse version up to date to benefit from AWS SDK patches and improvements.

Frequently Asked Questions

Q: What is the difference between AWS_ERROR and S3_ERROR in ClickHouse?
A: S3_ERROR is specifically for S3 API failures, while AWS_ERROR covers broader AWS SDK failures including credential resolution, STS, KMS, and endpoint issues that are not S3-specific.

Q: Can clock skew really cause an AWS_ERROR?
A: Yes. AWS requires that the timestamp in a signed request be within five minutes of the server time. If the ClickHouse host clock drifts beyond that window, every AWS API call will fail with a signature error.

Q: How do I enable debug logging for the AWS SDK in ClickHouse?
A: You can increase the ClickHouse log level to trace in the server configuration. The AWS SDK messages will appear in the standard ClickHouse log file, providing details about each API call and credential resolution step.

Q: Does ClickHouse support AWS IMDS v2 (Instance Metadata Service v2)?
A: Yes, recent versions of ClickHouse and the embedded AWS SDK support IMDSv2 with the required PUT-based token retrieval. Make sure your ClickHouse version is recent enough if you have disabled IMDSv1.