The cluster.routing.allocation.disk.watermark.high setting in Elasticsearch is a threshold that controls shard allocation based on disk usage. It prevents new shards from being allocated to nodes that have reached this high watermark of disk usage.
- Default value: 85%
- Possible values: Percentage (e.g., 85%) or byte size (e.g., 10GB)
- Recommendations: Adjust based on your cluster's specific needs and disk capacities
This setting is part of Elasticsearch's disk-based shard allocation mechanism. When a node's disk usage exceeds this high watermark, Elasticsearch will not allocate new shards to that node. Additionally, it will attempt to relocate shards from this node to other nodes with more available disk space.
Example
To change the high watermark to 80%:
cluster.routing.allocation.disk.watermark.high: 80%
You might want to lower this value if you need to be more conservative with disk usage or raise it if you have large disks and want to utilize more space before triggering allocation changes.
Common Issues and Misuses
- Setting the value too low can lead to unnecessary shard relocations and cluster rebalancing
- Setting it too high might risk running out of disk space
- Forgetting to account for this setting when planning cluster capacity
Do's and Don'ts
Do's:
- Regularly monitor disk usage across your cluster
- Adjust the setting based on your specific hardware and workload
- Consider using absolute values (e.g., 20GB) for more predictable behavior on large disks
Don'ts:
- Don't set it too close to 100%, as this leaves no buffer for sudden increases in data
- Don't ignore this setting when troubleshooting allocation issues
- Don't set it without considering the
lowandflood_stagewatermarks as well
Frequently Asked Questions
Q: How does this setting differ from the low watermark?
A: While the low watermark (cluster.routing.allocation.disk.watermark.low) prevents new shards from being allocated to a node, the high watermark also triggers relocation of existing shards from the affected node.
Q: Can this setting be changed dynamically?
A: Yes, this setting can be updated dynamically using the cluster update settings API without requiring a cluster restart.
Q: What happens if all nodes exceed the high watermark?
A: If all nodes exceed the high watermark, Elasticsearch will not be able to allocate or relocate shards, potentially leading to unassigned shards and reduced cluster health.
Q: How does this setting interact with the flood stage watermark?
A: The flood stage watermark (cluster.routing.allocation.disk.watermark.flood_stage) is a higher threshold that, when reached, makes the index read-only. It's a last resort to prevent data loss due to completely full disks.
Q: Should I use percentage or absolute values for this setting?
A: It depends on your setup. Percentages work well for homogeneous clusters, while absolute values can be more predictable for clusters with varying disk sizes or very large disks.
