The cluster.routing.allocation.disk.watermark.flood_stage
setting in Elasticsearch is a critical threshold that controls when the cluster enforces a read-only index block on indices allocated to nodes that have reached critical disk usage levels. This setting is part of Elasticsearch's disk-based shard allocation mechanism, which helps prevent nodes from running out of disk space.
- Default Value: 95%
- Possible Values: Percentage (e.g., 95%) or Byte value (e.g., 5gb)
- Recommendations: Set this value higher than
cluster.routing.allocation.disk.watermark.high
but lower than 100% to allow for some buffer.
This setting represents the absolute last resort before a node completely runs out of disk space. When a node reaches this threshold, Elasticsearch will enforce a read-only index block on every index that has at least one shard allocated on the affected node, preventing writes to these indices.
Example
To change the flood stage watermark to 97%:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.disk.watermark.flood_stage": "97%"
}
}
You might want to increase this value if you have very large indices and need more time to react to disk space issues. However, setting it too high risks node failure due to completely filled disks.
Common Issues and Misuses
- Setting the value too close to 100% can lead to node failures if disk space fills up completely before action can be taken.
- Setting it too low might cause unnecessary read-only blocks on indices, disrupting write operations prematurely.
Do's and Don'ts
- Do monitor disk usage closely and set up alerts well before reaching this threshold.
- Do ensure this value is higher than the
high
watermark setting. - Don't set this value to 100%, as it effectively disables the protection mechanism.
- Don't ignore read-only index blocks triggered by this setting; address the underlying disk space issue promptly.
Frequently Asked Questions
Q: What happens when the flood stage watermark is reached?
A: Elasticsearch enforces a read-only index block on all indices with at least one shard on the affected node, preventing further writes to these indices.
Q: Can I still read from indices when the flood stage is triggered?
A: Yes, read operations are still allowed. Only write operations are blocked to prevent further disk usage.
Q: How do I remove the read-only block once disk space is freed?
A: Once disk usage drops below the threshold, you can remove the block using the _all
index:
PUT _all/_settings
{
"index.blocks.read_only_allow_delete": null
}
Q: Is there a way to exclude certain indices from the flood stage protection?
A: No, the flood stage protection applies to all indices with shards on the affected node to ensure cluster stability.
Q: How often does Elasticsearch check disk usage against this threshold?
A: Elasticsearch continuously monitors disk usage and reacts promptly when thresholds are crossed, typically within seconds.