Elasticsearch gateway.recover_after_master_nodes Setting

The gateway.recover_after_master_nodes setting in Elasticsearch controls the number of master-eligible nodes that must be present before the cluster recovery process can begin after a full cluster restart.

Description

• Default value: 0
• Possible values: Integer between 0 and the total number of master-eligible nodes
• Recommendation: Set this to at least (master_eligible_nodes / 2) + 1 to ensure quorum

This setting is part of the gateway recovery process, which determines how Elasticsearch recovers its state after a full cluster restart. When set to a value greater than 0, it ensures that a minimum number of master-eligible nodes are present before the cluster starts recovering its state.

Example

To change the gateway.recover_after_master_nodes setting using the cluster settings API:

PUT /_cluster/settings
{
  "persistent": {
    "gateway.recover_after_master_nodes": 2
  }
}

In this example, we set the value to 2, meaning the cluster will wait for at least 2 master-eligible nodes to be present before starting the recovery process. This can be useful in a 3-node cluster to ensure that a majority of master-eligible nodes are available, reducing the risk of split-brain scenarios.

Common Issues and Misuses

• Setting the value too high can prevent the cluster from recovering if the required number of master-eligible nodes cannot be reached.
• Setting the value too low (or leaving it at 0) may allow the cluster to recover with an insufficient number of master nodes, potentially leading to data inconsistencies.

Do's and Don'ts

• Do set this value in conjunction with gateway.recover_after_nodes and gateway.expected_nodes for a comprehensive recovery strategy.
• Do consider your cluster topology and adjust this setting accordingly when adding or removing master-eligible nodes.
• Don't set this value higher than the total number of master-eligible nodes in your cluster.
• Don't ignore this setting in production environments, as it plays a crucial role in maintaining cluster stability during restarts.

Frequently Asked Questions

Q: How does gateway.recover_after_master_nodes differ from discovery.zen.minimum_master_nodes?
A: While both settings relate to master nodes, they serve different purposes. gateway.recover_after_master_nodes is used during cluster recovery after a full restart, whereas discovery.zen.minimum_master_nodes (deprecated in newer versions) was used to prevent split-brain scenarios during normal operation.

Q: Can changing this setting impact an already running cluster?
A: Changing this setting on a running cluster will not have an immediate effect. It only comes into play during a full cluster restart.

Q: What happens if the number of available master-eligible nodes is less than gateway.recover_after_master_nodes?
A: The cluster will wait indefinitely for the specified number of master-eligible nodes to become available before starting the recovery process.

Q: Is this setting still relevant in newer versions of Elasticsearch?
A: Yes, this setting is still relevant and important in recent versions of Elasticsearch for controlling the cluster recovery process.

Q: How should I choose the value for gateway.recover_after_master_nodes in a 5-node cluster with 3 master-eligible nodes?
A: In this scenario, a good value would be 2 (calculated as (3 / 2) + 1, rounded down). This ensures that a majority of master-eligible nodes are present before recovery begins.