The cluster.routing.allocation.disk.watermark.high.max_headroom setting in Elasticsearch controls the maximum amount of disk space that should be reserved when the high disk watermark is reached. This setting helps prevent nodes from completely running out of disk space by ensuring a minimum amount of free space is always available.
Description
- Default value: 10GB
- Possible values: Any positive byte size value (e.g., 15GB, 10240MB)
- Recommendation: Adjust based on your cluster's specific needs and disk sizes
This setting works in conjunction with the cluster.routing.allocation.disk.watermark.high setting. When a node reaches the high watermark, Elasticsearch will attempt to relocate shards from that node, but will always leave at least the amount specified by max_headroom free on the disk.
Example
To change this setting using the cluster settings API:
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.high.max_headroom": "15GB"
}
}
You might want to increase this value if you have large nodes with a lot of disk space and want to ensure more free space is always available. Conversely, you might decrease it on smaller nodes where disk space is at a premium.
Common Issues
- Setting this value too high can lead to unnecessary shard relocations, especially on smaller nodes.
- Setting it too low might not leave enough free space for critical operations or sudden spikes in data ingestion.
Do's and Don'ts
- Do monitor your disk usage patterns and adjust this setting accordingly.
- Do consider the size of your largest indices when setting this value.
- Don't set this value higher than what your smallest node can reasonably accommodate.
- Don't ignore this setting; it's crucial for preventing out-of-disk-space scenarios.
Frequently Asked Questions
Q: How does this setting interact with the flood_stage watermark?
A: While this setting ensures a minimum free space when the high watermark is reached, the flood_stage watermark is a last line of defense that makes the index read-only to prevent any further writes when disk space is critically low.
Q: Can this setting be different for different nodes?
A: No, this is a cluster-wide setting. However, you can use other node-specific settings to influence shard allocation based on available disk space.
Q: What happens if the available disk space falls below this setting?
A: Elasticsearch will attempt to relocate shards from the affected node to other nodes with more available disk space, if possible.
Q: How often does Elasticsearch check disk usage against this setting?
A: Disk usage is checked periodically as part of the cluster state update process, typically every 30 seconds.
Q: Should I use a percentage or an absolute value for this setting?
A: This particular setting uses an absolute value, which can be more predictable across nodes with varying disk sizes. However, other related watermark settings can use percentages.
