Elasticsearch gateway.expected_nodes Setting

What is gateway.expected_nodes?

The gateway.expected_nodes setting in Elasticsearch controls the number of data nodes that should be present in the cluster before starting the recovery process after a full cluster restart.

Description

• Default value: 0
• Possible values: Any non-negative integer
• Recommendation: Set this to the number of data nodes in your cluster

The settings gateway.recover_after_nodes, gateway.recover_after_master_nodes, gateway.expected_nodes, and gateway.expected_master_nodes were deprecated in Elasticsearch 7.7.0 and removed in Elasticsearch 8.0.0. These settings were used to delay cluster recovery until a specified number of nodes had joined the cluster. However, it was determined that waiting for additional master-eligible nodes was unnecessary, as recovery could safely proceed once a majority of master-eligible nodes had joined.

Example

To set the gateway.expected_nodes to 3 using the cluster settings API:

PUT /_cluster/settings
{
  "persistent": {
    "gateway.expected_nodes": 3
  }
}

You might want to change this setting if you have a fixed number of data nodes in your cluster and want to ensure that all nodes are present before starting recovery. This can help prevent unnecessary shard allocations and relocations if not all nodes start simultaneously.

Common Issues

• Setting this value too high can delay cluster recovery if some nodes are permanently offline.
• Setting it too low might start recovery prematurely, leading to unnecessary shard movements when more nodes join later.

Do's and Don'ts

• Do set this to match the number of data nodes in your cluster.
• Don't set this higher than your actual number of data nodes.
• Do consider using this in conjunction with gateway.recover_after_nodes for more granular control.
• Don't confuse this with discovery.zen.minimum_master_nodes (for versions before 7.0) or discovery.seed_hosts (for version 7.0+).

Frequently Asked Questions

Q: How does gateway.expected_nodes differ from discovery.zen.minimum_master_nodes?
A: gateway.expected_nodes is about data nodes and cluster recovery, while discovery.zen.minimum_master_nodes (deprecated in 7.0+) was about master-eligible nodes and split-brain prevention.

Q: What happens if the number of nodes never reaches gateway.expected_nodes?
A: The cluster will wait indefinitely unless gateway.recover_after_time is set, which forces recovery after a specified time regardless of node count.

Q: Can gateway.expected_nodes be changed dynamically?
A: Yes, it can be changed using the cluster settings API, but it only takes effect on the next full cluster restart.

Q: Should gateway.expected_nodes be set in every node's elasticsearch.yml?
A: It's generally better to set it cluster-wide using the cluster settings API to ensure consistency across all nodes.

Q: How does this setting interact with gateway.recover_after_nodes?
A: gateway.recover_after_nodes sets the minimum number of nodes to start recovery, while gateway.expected_nodes sets the ideal number. Recovery can start after reaching recover_after_nodes but before reaching expected_nodes.