ClickHouse DB::Exception: S3 error

The "DB::Exception: S3 error" occurs when ClickHouse fails to interact with an Amazon S3 bucket or S3-compatible object storage. Because S3 is one of the most popular backends for ClickHouse external storage, MergeTree disk policies, and backup targets, this S3_ERROR is among the most frequently encountered cloud-storage errors. The accompanying message usually contains the HTTP status code and the AWS error string, which narrows down the root cause considerably.

Impact

Any query or background operation that touches S3-backed data will fail while this error persists. That includes SELECT queries on S3-backed tables, INSERT operations writing to S3 disks, merges on tiered storage policies, and backup/restore jobs. In a production cluster the ripple effect can stall merges and increase the number of active parts, potentially leading to secondary errors like "Too many parts."

Common Causes

Incorrect or missing IAM permissions — the ClickHouse process does not have s3:GetObject, s3:PutObject, s3:DeleteObject, or s3:ListBucket on the target bucket and prefix.
The S3 bucket does not exist, was deleted, or the region is wrong.
AWS request throttling (HTTP 503 SlowDown) when ClickHouse issues a high volume of requests against a single prefix.
Network connectivity issues between the ClickHouse node and the S3 endpoint, including VPC endpoint misconfigurations or proxy problems.
Expired or rotated AWS credentials, especially when using short-lived STS tokens.
Server-side encryption configuration mismatch — for example, the bucket requires SSE-KMS but ClickHouse is not sending the correct KMS key ID.
Bucket policy or S3 Object Lock settings that deny the requested operation.
Using a path-style URL when the bucket requires virtual-hosted-style addressing, or vice versa.

Troubleshooting and Resolution Steps

Read the full error message carefully — it typically contains the HTTP status code (403, 404, 503, etc.) and the AWS error code (AccessDenied, NoSuchBucket, SlowDown). This tells you exactly which category of problem you are dealing with.

For 403 AccessDenied, verify IAM permissions:

-- Check the credentials ClickHouse is using
SELECT * FROM system.storage_policies;

Then confirm the IAM role or user has the required actions on the bucket:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject",
    "s3:ListBucket"
  ],
  "Resource": [
    "arn:aws:s3:::your-bucket",
    "arn:aws:s3:::your-bucket/*"
  ]
}

For 404 NoSuchBucket or NoSuchKey, confirm the bucket name, region, and object prefix are correct in your ClickHouse configuration or table definition:

<storage_configuration>
  <disks>
    <s3_disk>
      <type>s3</type>
      <endpoint>https://your-bucket.s3.us-east-1.amazonaws.com/data/</endpoint>
    </s3_disk>
  </disks>
</storage_configuration>

For 503 SlowDown (throttling), reduce concurrency or spread data across multiple S3 prefixes. You can also adjust ClickHouse settings:
```
SET s3_max_connections = 50;        -- lower from the default
SET s3_max_single_read_retries = 6; -- increase retries
```
Check network connectivity from the ClickHouse host:
```
curl -I https://your-bucket.s3.us-east-1.amazonaws.com/
```
If you are using a VPC endpoint, make sure the endpoint policy allows the necessary S3 actions.
If credentials have expired, rotate them and restart ClickHouse, or switch to instance profile / IAM role-based authentication to avoid manual credential management.

Review the ClickHouse server log for the full stack trace:

grep -i "S3_ERROR\|S3Exception" /var/log/clickhouse-server/clickhouse-server.log | tail -50

Best Practices

Use IAM roles (instance profiles or IRSA on Kubernetes) rather than static access keys so credentials rotate automatically.
Enable S3 server access logging or AWS CloudTrail data events on the bucket to get an independent view of failed requests.
Spread objects across multiple prefixes to reduce the risk of S3 request-rate throttling on a single partition.
Set s3_max_single_read_retries and s3_retry_attempts to sensible values so transient errors are retried transparently.
Monitor the S3ReadMicroseconds, S3WriteMicroseconds, and S3ReadRequestsErrors metrics in system.events to catch problems early.
Pin the S3 endpoint to the correct region to avoid cross-region latency and unexpected redirect errors.

Frequently Asked Questions

Q: Why do I get S3_ERROR with "Access Denied" even though my IAM policy looks correct?
A: Double-check the bucket policy in addition to the IAM policy — an explicit Deny in the bucket policy overrides any IAM Allow. Also verify that the Resource ARN includes both the bucket itself (arn:aws:s3:::bucket) and the objects inside it (arn:aws:s3:::bucket/*).

Q: Can S3 throttling cause data corruption in ClickHouse?
A: No, throttling does not cause corruption. ClickHouse treats throttled requests as retriable errors, and the operation will either succeed after retries or fail cleanly with the S3_ERROR exception.

Q: How do I use S3-compatible storage like MinIO or Wasabi with ClickHouse?
A: Set the endpoint to your S3-compatible URL and, if the provider does not support virtual-hosted-style requests, enable path-style access by appending a trailing slash or adjusting the use_environment_credentials and endpoint settings accordingly. The same S3_ERROR codes apply.

Q: Does ClickHouse support S3 server-side encryption?
A: Yes. You can configure SSE-S3 or SSE-KMS by adding server_side_encryption_customer_key_base64 or the appropriate headers in the disk configuration. Make sure the IAM policy also grants kms:GenerateDataKey and kms:Decrypt if you use SSE-KMS.

Q: How can I tell if the error is transient or permanent?
A: Transient errors typically have HTTP 500, 502, 503, or 504 status codes. Permanent errors like 403 or 404 will not resolve on their own — you need to fix credentials, permissions, or the bucket configuration.