ClickHouse DB::Exception: Too many HTTP redirects

The "DB::Exception: Too many HTTP redirects" error in ClickHouse occurs when an HTTP request issued by the server follows more redirects than the allowed limit. The TOO_MANY_REDIRECTS error code is triggered by table functions like url(), s3(), and dictionary sources that fetch data over HTTP. When the target URL keeps redirecting (via 301, 302, or 307 responses) beyond ClickHouse's threshold, the server aborts the request to prevent infinite redirect loops.

Impact

This error prevents ClickHouse from fetching data from the affected HTTP endpoint:

Queries using the url() table function will fail
S3-compatible storage access may be blocked if the endpoint redirects excessively
HTTP-sourced dictionaries cannot load or refresh
External table engines that rely on HTTP will be unable to read data
Data import pipelines that fetch from URLs will stall

Common Causes

A redirect loop between two or more URLs that keep pointing to each other
Authentication or authorization middleware redirecting to a login page repeatedly
The target URL has moved and requires more redirect hops than ClickHouse allows
Misconfigured load balancer or CDN creating circular redirects
HTTP-to-HTTPS redirect followed by another redirect back (protocol mismatch loop)
URL shortener services with chain redirects exceeding the limit
S3 bucket endpoint redirecting to a region-specific endpoint multiple times

Troubleshooting and Resolution Steps

Test the URL manually to see the redirect chain:
```
curl -vL --max-redirs 20 "https://your-url.com/data" 2>&1 | grep -i "location:"
```
This reveals each redirect hop and helps identify loops.
Find the final destination URL:
```
curl -sIL "https://your-url.com/data" | grep -i "^location:"
```
Use the final resolved URL directly in your ClickHouse query to bypass redirects entirely.

Update the URL in your query or configuration:

-- Instead of the redirecting URL, use the final destination
SELECT * FROM url('https://final-destination.com/actual-data.csv', 'CSV', 'col1 String, col2 UInt32');

For S3 endpoints, use the correct regional endpoint:

-- Use the region-specific endpoint instead of the global one
SELECT * FROM s3('https://your-bucket.s3.us-east-1.amazonaws.com/data.parquet');

Using the region-specific URL avoids S3's region redirect.

Check for authentication redirects: If the URL requires authentication, ensure credentials are provided in the request so the server doesn't redirect to a login page:
```
SELECT * FROM url('https://api.example.com/data', 'JSONEachRow', 'col1 String',
  headers('Authorization' = 'Bearer your-token'));
```

Increase the redirect limit if the chain is legitimate: ClickHouse has a max_http_get_redirects setting:

SET max_http_get_redirects = 10;
SELECT * FROM url('https://your-url.com/data', 'CSV', 'col1 String');

Fix the redirect loop at the source if you control the web server or load balancer. Common fixes include correcting rewrite rules, ensuring consistent HTTP/HTTPS configuration, and removing circular redirect rules.

Best Practices

Always use direct URLs rather than shortened or redirecting URLs in ClickHouse configurations
For S3 access, use region-specific endpoints to avoid redirect hops
Set max_http_get_redirects to a reasonable value (5-10) rather than leaving it at the default if your data sources require some redirects
Test URLs with curl -L before using them in ClickHouse queries
When configuring HTTP dictionary sources, verify the endpoint is stable and does not redirect
Document the actual data URLs used in production queries to make troubleshooting easier

Frequently Asked Questions

Q: What is the default redirect limit in ClickHouse?
A: By default, max_http_get_redirects is set to 0, meaning ClickHouse does not follow redirects at all. You must explicitly set it to a positive value if your URL requires redirects.

Q: Can I follow redirects for S3 table functions?
A: Yes, the max_http_get_redirects setting applies to S3 and S3-compatible storage access as well. Set it in your query or session settings.

Q: How do I detect a redirect loop?
A: Use curl -vL --max-redirs 20 and watch the Location: headers. If you see the same URLs repeating, there is a loop. The web server or load balancer configuration needs to be fixed.

Q: Will this error resolve on its own?
A: Only if the redirect issue on the remote server is temporary (e.g., a misconfiguration that gets corrected). If the redirect loop is persistent, you must either fix the server-side configuration or use a direct URL.