Elasticsearch ClusterBlockException: Cluster is read-only - Common Causes & Fixes

Brief Explanation

The "ClusterBlockException: Cluster is read-only" error in Elasticsearch occurs when the cluster has been automatically set to read-only mode, typically due to low disk space or other critical issues that could compromise data integrity.

Impact

This error has a significant impact on cluster operations:

• Write operations (indexing, updates, deletions) are blocked
• Search and read operations continue to function
• New indices cannot be created
• Cluster metadata changes are prevented

Common Causes

1. Low disk space (below the configured watermark)
2. Corrupted cluster state
3. Network issues preventing proper cluster communication
4. Misconfigured cluster settings

Troubleshooting and Resolution Steps

1. Check disk space:

   GET /_cat/allocation?v
   

2. If disk space is low, free up space or add more storage.

3. Once disk space is resolved, clear the read-only index block:

   PUT /_all/_settings
   {
     "index.blocks.read_only_allow_delete": null
   }
   

4. If disk space isn't the issue, check cluster health:

   GET /_cluster/health
   

5. Review logs for any error messages or warnings.

6. Ensure all nodes can communicate properly within the cluster.

7. Verify cluster settings, particularly those related to disk watermarks:

   GET /_cluster/settings
   

8. If the issue persists, consider restarting nodes one by one.

Best Practices

• Regularly monitor disk space and set up alerts for low disk space conditions.
• Implement proper data retention and index lifecycle management policies.
• Use rolling restarts when updating cluster settings to avoid downtime.
• Maintain regular backups of your Elasticsearch data.

Frequently Asked Questions

Q: How can I prevent the cluster from going into read-only mode?
A: Monitor disk space closely, set up alerts for low disk space, implement proper data retention policies, and ensure adequate storage capacity for your cluster's needs.

Q: Can I still query data when the cluster is in read-only mode?
A: Yes, read operations like searches are still allowed. Only write operations are blocked.

Q: How do I check the current disk usage in my Elasticsearch cluster?
A: Use the GET /_cat/allocation?v API call to view disk usage across your cluster nodes.

Q: What are the default disk watermark settings in Elasticsearch?
A: By default, the low watermark is 85%, high watermark is 90%, and flood stage is 95% of disk capacity. These can be adjusted in the cluster settings.

Q: After resolving disk space issues, why might my cluster still be in read-only mode?
A: You may need to manually clear the read-only block using the PUT /_all/_settings API call as shown in the resolution steps above.