Brief Explanation
The "ClusterBlockException: Cluster is read-only" error in Elasticsearch occurs when the cluster has been automatically set to read-only mode, typically due to low disk space or other critical issues that could compromise data integrity.
Impact
This error has a significant impact on cluster operations:
- Write operations (indexing, updates, deletions) are blocked
- Search and read operations continue to function
- New indices cannot be created
- Cluster metadata changes are prevented
Common Causes
- Low disk space (below the configured watermark)
- Corrupted cluster state
- Network issues preventing proper cluster communication
- Misconfigured cluster settings
Troubleshooting and Resolution Steps
Check disk space:
GET /_cat/allocation?v
If disk space is low, free up space or add more storage.
Once disk space is resolved, clear the read-only index block:
PUT /_all/_settings { "index.blocks.read_only_allow_delete": null }
If disk space isn't the issue, check cluster health:
GET /_cluster/health
Review logs for any error messages or warnings.
Ensure all nodes can communicate properly within the cluster.
Verify cluster settings, particularly those related to disk watermarks:
GET /_cluster/settings
If the issue persists, consider restarting nodes one by one.
Best Practices
- Regularly monitor disk space and set up alerts for low disk space conditions.
- Implement proper data retention and index lifecycle management policies.
- Use rolling restarts when updating cluster settings to avoid downtime.
- Maintain regular backups of your Elasticsearch data.
Frequently Asked Questions
Q: How can I prevent the cluster from going into read-only mode?
A: Monitor disk space closely, set up alerts for low disk space, implement proper data retention policies, and ensure adequate storage capacity for your cluster's needs.
Q: Can I still query data when the cluster is in read-only mode?
A: Yes, read operations like searches are still allowed. Only write operations are blocked.
Q: How do I check the current disk usage in my Elasticsearch cluster?
A: Use the GET /_cat/allocation?v
API call to view disk usage across your cluster nodes.
Q: What are the default disk watermark settings in Elasticsearch?
A: By default, the low watermark is 85%, high watermark is 90%, and flood stage is 95% of disk capacity. These can be adjusted in the cluster settings.
Q: After resolving disk space issues, why might my cluster still be in read-only mode?
A: You may need to manually clear the read-only block using the PUT /_all/_settings
API call as shown in the resolution steps above.