How to Remove a Node from an Elasticsearch Cluster

Removing a node from an Elasticsearch cluster may be necessary in various scenarios, such as:

• Decommissioning old or underperforming hardware
• Scaling down the cluster due to reduced workload
• Replacing a faulty node
• Performing maintenance on a specific node
• Rebalancing the cluster for optimal performance

Steps to remove a node from the cluster

1. Prepare the cluster:

   • Ensure the cluster is in a healthy state (green)
   • Verify that removing the node won't impact the cluster's availability

2. Disable shard allocation:

   PUT _cluster/settings
   {
     "persistent": {
       "cluster.routing.allocation.enable": "none"
     }
   }
   

3. Stop indexing and perform a synced flush (optional but recommended):

   POST _flush/synced
   

4. Shut down the node:

   • Stop the Elasticsearch service on the node you want to remove

5. Remove the node from the cluster configuration:

   • Update the elasticsearch.yml file on all remaining nodes to remove references to the decommissioned node

6. Re-enable shard allocation:

   PUT _cluster/settings
   {
     "persistent": {
       "cluster.routing.allocation.enable": null
     }
   }
   

7. Monitor cluster health:

   • Watch the cluster health and shard reallocation process

   GET _cluster/health
   GET _cat/shards
   

8. Update your client applications (if necessary):

   • Remove the decommissioned node from client configurations

Best Practices and Additional Information

• Always maintain enough nodes to store at least one replica of each shard
• Perform node removals during off-peak hours to minimize impact on performance
• Ensure proper backups are in place before making any cluster changes
• If removing multiple nodes, do so one at a time to maintain cluster stability
• Consider using the Cluster Update Settings API to exclude the node from shard allocation before shutting it down:

  PUT _cluster/settings
  {
    "transient" : {
      "cluster.routing.allocation.exclude._name" : "node_to_remove"
    }
  }
  

Frequently Asked Questions

Q: How long does it take to remove a node from the cluster?
A: The time varies depending on cluster size, data volume, and hardware. It can take minutes to hours. Monitor the cluster health and shard allocation to track progress.

Q: Can I remove a master-eligible node?
A: Yes, but ensure you have enough remaining master-eligible nodes to form a quorum. It's recommended to have at least three master-eligible nodes in a production cluster.

Q: What happens to the data on the removed node?
A: Data from the removed node will be redistributed to the remaining nodes in the cluster. Ensure you have enough storage capacity on other nodes before removal.

Q: Will removing a node affect my cluster's performance?
A: Temporarily, yes. The cluster will need to reallocate shards and may experience increased load. Plan the removal during off-peak hours to minimize impact.

Q: How can I ensure I'm not removing a critical node?
A: Check the cluster health, shard allocation, and node roles before removal. Ensure you have enough nodes to maintain the required number of replicas and master-eligible nodes.