ClickHouse DB::Exception: Azure Blob Storage error

The "DB::Exception: Azure Blob Storage error" is raised when ClickHouse cannot complete an operation against Azure Blob Storage. The AZURE_BLOB_STORAGE_ERROR code wraps the underlying Azure SDK failure, and the error message typically includes the HTTP status code and Azure-specific error string. This affects any table, disk, or backup target that relies on Azure Blob Storage as its backend.

Impact

While this error is active, ClickHouse cannot read from or write to Azure-backed storage. Queries against tables using the AzureBlobStorage engine or MergeTree tables on Azure disks will fail. Background merges on tiered storage policies that include Azure will stall, and backup or restore operations targeting Azure containers will not succeed.

Common Causes

1. Invalid or expired Azure credentials — wrong storage account key, expired SAS token, or misconfigured managed identity.
2. The target container does not exist or has been deleted.
3. Incorrect storage account name or endpoint URL in the ClickHouse configuration.
4. Network connectivity issues, including blocked outbound traffic to *.blob.core.windows.net or private endpoint misconfigurations.
5. Azure RBAC or access policy restrictions that deny the required operation (read, write, list, delete).
6. Throttling by Azure Storage when request rates exceed the per-account or per-container limits.
7. TLS/SSL handshake failures when custom certificates or proxy servers intercept HTTPS traffic.

Troubleshooting and Resolution Steps

1. Examine the full error message in the ClickHouse server log. The Azure SDK error usually includes an HTTP status code and a descriptive message such as AuthenticationFailed, ContainerNotFound, or BlobNotFound:

   grep -i "AZURE_BLOB_STORAGE_ERROR\|AzureException" /var/log/clickhouse-server/clickhouse-server.log | tail -30
   

2. Verify your credentials. If you are using a storage account key, confirm it has not been rotated:

   <storage_configuration>
     <disks>
       <azure_disk>
         <type>azure_blob_storage</type>
         <storage_account_url>https://youraccount.blob.core.windows.net</storage_account_url>
         <container_name>clickhouse-data</container_name>
         <account_name>youraccount</account_name>
         <account_key>YOUR_KEY_HERE</account_key>
       </azure_disk>
     </disks>
   </storage_configuration>
   

   If using a SAS token, make sure it has not expired and includes the r, w, d, and l permissions on the container.

3. Confirm the container exists and is accessible from the ClickHouse host:

   curl -I "https://youraccount.blob.core.windows.net/clickhouse-data?restype=container" \
     -H "x-ms-version: 2021-08-06"
   

4. Check Azure RBAC assignments. If you are using managed identity or Azure AD, ensure the identity has the Storage Blob Data Contributor role on the storage account or container.

5. For networking issues, verify that the storage account firewall allows traffic from the ClickHouse node's IP or virtual network. If using a private endpoint, confirm DNS resolution points to the private IP:

   nslookup youraccount.blob.core.windows.net
   

6. If the error indicates throttling (HTTP 503 or 429), reduce ClickHouse concurrency settings and consider spreading data across multiple containers or storage accounts.

7. Test connectivity with the Azure CLI:

   az storage blob list --account-name youraccount --container-name clickhouse-data --auth-mode key --account-key YOUR_KEY
   

Best Practices

• Use managed identity authentication on Azure VMs or AKS to avoid manual credential rotation.
• Monitor Azure Storage metrics (Availability, SuccessE2ELatency, Transactions) in the Azure portal to catch issues early.
• Configure geo-redundant storage (GRS) or zone-redundant storage (ZRS) for production data to improve durability.
• Keep ClickHouse updated so you benefit from the latest Azure SDK improvements and bug fixes.
• Place ClickHouse nodes in the same Azure region as the storage account to minimize latency and egress costs.
• Set appropriate retry and timeout values in the ClickHouse Azure disk configuration to handle transient failures gracefully.

Frequently Asked Questions

Q: How do I rotate Azure storage account keys without downtime?
A: Azure storage accounts have two keys. Update your ClickHouse configuration to use the secondary key, reload the config, then regenerate the primary key. This ensures uninterrupted access.

Q: Can I use Azure AD (Entra ID) authentication instead of storage account keys?
A: Yes. ClickHouse supports managed identity and workload identity authentication for Azure Blob Storage. Configure the disk without an account_key and ensure the appropriate RBAC role is assigned to the identity.

Q: Why does ClickHouse fail with AZURE_BLOB_STORAGE_ERROR after enabling a storage account firewall?
A: The firewall blocks requests from IPs not in the allow list. Add the ClickHouse node IPs or the virtual network to the storage account's network rules, or use a private endpoint.

Q: Does ClickHouse retry failed Azure Blob Storage requests automatically?
A: Yes, the Azure SDK used by ClickHouse includes built-in retry logic for transient errors. However, permanent errors like authentication failures or missing containers will not be retried.