The cluster.routing.allocation.disk.watermark.low setting in Elasticsearch is a threshold that controls when the cluster starts to allocate shards away from a node based on its disk usage. This setting is part of Elasticsearch's disk-based shard allocation mechanism, which helps prevent nodes from running out of disk space.
- Default value: 85%
- Possible values: Percentage (e.g., 75%) or byte values (e.g., 100gb)
- Recommendations: Set according to your cluster's needs and disk capacities
This setting defines the low watermark for disk usage. When a node's disk usage exceeds this threshold, Elasticsearch will not allocate new shards to that node but will allow the allocation of shards as a result of rebalancing. This helps distribute data more evenly across the cluster and prevents nodes from becoming overloaded.
Example
To change the low watermark to 80%:
cluster.routing.allocation.disk.watermark.low: 80%
Reason for change: You might want to be more conservative with disk usage, especially if you have limited storage or experience rapid data growth.
Effects: Elasticsearch will start to allocate shards away from nodes earlier, potentially leading to more frequent rebalancing but reducing the risk of nodes running out of disk space.
Common Issues and Misuses
- Setting the value too high can lead to nodes running out of disk space unexpectedly
- Setting it too low can cause unnecessary shard movements and impact cluster performance
- Forgetting to adjust this setting when adding larger disks to the cluster
Do's and Don'ts
Do's:
- Regularly monitor disk usage across your cluster
- Adjust the setting based on your specific hardware and data growth patterns
- Consider using absolute values (e.g., 100gb) for more predictable behavior
Don'ts:
- Don't set this value too close to 100%
- Don't ignore this setting when planning cluster capacity
- Don't set it without also considering the
highandflood_stagewatermarks
Frequently Asked Questions
Q: How does this setting interact with cluster.routing.allocation.disk.watermark.high?
A: While the low watermark prevents new shard allocations, the high watermark actively attempts to relocate shards away from the node. They work together to manage disk usage effectively.
Q: Can I change this setting dynamically?
A: Yes, this setting can be updated dynamically using the cluster settings API without requiring a cluster restart.
Q: What happens if all nodes exceed the low watermark?
A: If all nodes exceed the low watermark, Elasticsearch will still allocate new shards, but it will choose the nodes with the most available disk space.
Q: How often does Elasticsearch check disk usage against this watermark?
A: Elasticsearch checks disk usage periodically, typically every 30 seconds, but this can vary based on cluster load and configuration.
Q: Should I use percentage or absolute values for this setting?
A: It depends on your setup. Percentages are more flexible across different node sizes, while absolute values provide more predictable behavior, especially in clusters with varying disk sizes.
