The cluster.routing.allocation.disk.watermark.low.max_headroom setting in Elasticsearch controls the maximum amount of disk space to reserve when using the low disk watermark for shard allocation decisions. This setting helps prevent the cluster from allocating shards to nodes that are approaching their storage capacity limits.
Description
- Default value: 150GB
- Possible values: Any valid byte size value (e.g., 100GB, 200GB)
- Recommendation: Adjust based on your cluster's storage capacity and expected growth
This setting works in conjunction with the cluster.routing.allocation.disk.watermark.low setting. When the available disk space on a node falls below the low watermark percentage or the max_headroom value (whichever is larger), Elasticsearch will stop allocating new shards to that node.
Example
To change the low watermark max headroom to 200GB:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.disk.watermark.low.max_headroom": "200GB"
}
}
Reasons for changing this value might include:
- Adapting to larger disk sizes in your cluster
- Providing more buffer space for data growth
- Balancing between efficient disk usage and preventing out-of-space issues
Effects of the change:
- Nodes with less than 200GB free space will not receive new shard allocations
- May lead to more balanced shard distribution across nodes with larger disks
Common Issues or Misuses
- Setting the value too low, which can lead to nodes running out of disk space
- Setting the value too high, which can result in inefficient use of available storage
- Not considering this setting in conjunction with other disk-based allocation settings
Do's and Don'ts
Do's:
- Regularly monitor disk usage and adjust this setting as your cluster grows
- Consider your largest expected index size when setting this value
- Use in combination with other disk watermark settings for comprehensive storage management
Don'ts:
- Don't set this value higher than necessary, as it may lead to underutilization of storage
- Don't ignore this setting when planning for cluster capacity
- Don't rely solely on this setting for managing disk space; implement proper monitoring and alerting
Frequently Asked Questions
Q: How does this setting interact with the percentage-based low watermark setting?
A: Elasticsearch uses whichever value (percentage or max_headroom) results in more free space for the low watermark threshold. This ensures that both small and large disks are protected appropriately.
Q: Can this setting be changed dynamically?
A: Yes, this setting can be updated dynamically using the cluster settings API without requiring a cluster restart.
Q: How often does Elasticsearch check disk usage against this setting?
A: Elasticsearch periodically checks disk usage as part of its cluster state updates, typically every 30 seconds.
Q: What happens if all nodes in the cluster exceed this watermark?
A: If all nodes exceed the low watermark, Elasticsearch will still allocate shards but will prefer nodes with more free space. It's crucial to address storage issues promptly in this scenario.
Q: Should this setting be the same across all nodes in a cluster?
A: While it's typically set cluster-wide, you can have different values for different node types if necessary. However, a consistent value across all nodes is generally recommended for simplicity and predictability.
