The cluster.routing.allocation.node_concurrent_outgoing_recoveries setting in Elasticsearch controls the number of shards that can be concurrently recovered from a single node to other nodes in the cluster.
Description
- Default value: 2
- Possible values: Any positive integer
- Recommendation: The default value is suitable for most scenarios, but it can be increased for nodes with high I/O capacity
This setting limits the number of outgoing recoveries that can happen simultaneously from a single node. It helps prevent a node from being overwhelmed by too many outgoing recovery operations, which could impact its performance and stability.
Example
To change the value using the cluster settings API:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.node_concurrent_outgoing_recoveries": 4
}
}
Increasing this value might be beneficial in scenarios where you have nodes with high I/O capacity and you want to speed up the recovery process during cluster rebalancing or when adding new nodes. However, setting it too high could overload the source node and negatively impact its performance.
Common Issues
- Setting the value too high can overwhelm the source node, causing performance issues
- Setting the value too low can significantly slow down the recovery process, especially in large clusters
Do's and Don'ts
- Do monitor your cluster's performance when adjusting this setting
- Do consider the I/O capacity of your nodes when setting this value
- Don't set this value excessively high without proper testing
- Don't ignore this setting when planning for cluster scaling or data rebalancing
Frequently Asked Questions
Q: How does this setting differ from cluster.routing.allocation.node_concurrent_incoming_recoveries?
A: While node_concurrent_outgoing_recoveries limits the number of shards that can be sent from a node, node_concurrent_incoming_recoveries limits the number of shards that can be received by a node concurrently.
Q: Can changing this setting impact cluster stability?
A: Yes, setting it too high can overload nodes and potentially cause stability issues. It's important to change this setting gradually and monitor the cluster's performance.
Q: Does this setting affect snapshot restore operations?
A: No, this setting specifically affects shard recovery between nodes in the cluster. Snapshot restore operations are controlled by different settings.
Q: Should I always increase this value to speed up recovery?
A: Not necessarily. While increasing the value can speed up recovery in some cases, it's important to balance this with the node's capacity and overall cluster performance.
Q: How can I monitor the impact of changing this setting?
A: You can use Elasticsearch's Cat Recovery API and monitor node performance metrics to observe the impact of changes to this setting.
