Elasticsearch flood_stage Watermark: 95% Default and Read-Only Index Block

Q: What is the fastest way to diagnose and recover from a flood-stage `read_only_allow_delete` block in production?

First confirm the block is in place with GET /*/_settings | grep read_only_allow_delete , then free disk via _delete_by_query or rollover (the block allows deletes by design). For continuous coverage, Pulse acts as an AI DBA for Elasticsearch and OpenSearch that detects the 95% flood-stage trip, identifies which indices have shards on the affected node, and walks through the recovery - delete, rollover, or expand - while monitoring for auto-release back below the high watermark.

Q: What is the default value of cluster.routing.allocation.disk.watermark.flood_stage?

The default is 95%. When any data node's disk usage crosses 95%, Elasticsearch applies an index.blocks.read_only_allow_delete: true block to every index that has a shard on the affected node. Writes are blocked until disk space is freed.

Q: What is index.blocks.read_only_allow_delete?

It is the index-level block Elasticsearch applies when flood stage is reached. It blocks writes but allows reads and delete-by-query, so you can free space to recover. Once disk drops below the high watermark, Elasticsearch (7.4+) removes the block automatically.

Q: How do I remove the read-only-allow-delete block?

First free disk space. On Elasticsearch 7.4 and later the block lifts automatically. If it persists, run PUT /*/_settings {"index.blocks.read_only_allow_delete": null} to clear the block on every index.

Q: Does flood stage block reads or only writes?

Only writes. The block name is literal: read_only_allow_delete allows reads and _delete_by_query , but rejects indexing and update calls. This is intentional so operators can delete data to free disk.

Q: How is flood stage different from the high watermark?

The high watermark (default 90%) triggers shard relocation away from the over-utilised node but does not block writes. Flood stage (95%) goes further and forces the affected indices into a read-only state. Flood stage is the last line of defence before full disk.

Q: Can I disable the flood-stage watermark?

There is no clean off switch. You can set it close to 100% to delay it, but doing so risks node failure if the disk fills entirely. Better to fix the underlying capacity issue.

The cluster.routing.allocation.disk.watermark.flood_stage setting is the third and most aggressive of Elasticsearch's disk watermarks. It defaults to 95% of disk usage. Once any data node crosses it, Elasticsearch applies an index.blocks.read_only_allow_delete: true block to every index that has at least one shard on the affected node. Writes are blocked; only reads and delete-by-query operations are permitted. The flood-stage block is the cluster's emergency stop - by the time it fires, the low and high watermarks have already failed to recover the situation.

Definition

cluster.routing.allocation.disk.watermark.flood_stage is a dynamic cluster-level setting expressed as a percentage of disk used or an absolute amount of free space remaining. When a data node crosses the threshold, the master sets index.blocks.read_only_allow_delete: true on every index with a shard on the node. Unlike the low and high watermarks, flood stage does not just affect future allocation - it actively blocks user writes to indices that already exist.

Default and Allowed Values

Property	Value
Default	`95%`
Type	Percentage or absolute free space
Scope	Cluster, dynamic
Effect	Sets `index.blocks.read_only_allow_delete: true` on affected indices
Companion	`cluster.routing.allocation.disk.watermark.flood_stage.frozen` (default 95% or 20GB free, whichever is lower) for frozen tiers
Required ordering	`low <= high <= flood_stage`

When the disk drops back below the high watermark, Elasticsearch automatically removes the read-only block on affected indices. This auto-release was introduced in Elasticsearch 7.4. On older versions, the block must be removed manually.

How to Change It

Through the cluster settings API:

# Raise flood stage to 97%, leaving more disk runway
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}

Or as absolute free space (often more predictable on large disks):

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.flood_stage": "50gb"
  }
}

Reset to default with null. The value must remain greater than or equal to disk.watermark.high, or the API call is rejected.

Recovering from a Flood-Stage Block

When flood stage triggers, you will see errors like:

TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

Recovery procedure:

Free disk space. Delete or roll over old indices, expand storage, or move data to a warmer tier with snapshots.
Wait for auto-release (7.4+). Once disk usage drops below the high watermark, Elasticsearch removes the read-only block automatically. The release happens at the next disk-info update.
Manually remove the block if needed. On older versions, or if the block persists for any reason:

# Clear the read-only block on every index
PUT /*/_settings
{
  "index.blocks.read_only_allow_delete": null
}

Setting the value to null removes the block. Setting it to false also works but null is the canonical reset.

Verify writes have resumed. Index a test document and confirm cluster health is green or yellow.

Operational Impact

Flood stage is a controlled outage. The block protects the node from a full disk (which can corrupt translog and Lucene segments) but at the cost of write availability for every affected index. Even one node at 95% can take down writes across many indices, because most indices in a multi-node cluster have at least one shard on every node.

Two side effects that surprise people:

The block applies to indices, not nodes. Once an index is marked read-only, every shard of it - including shards on healthy nodes - rejects writes. Releasing the block requires explicitly clearing it on the index.
Delete-by-query is permitted. The name read_only_allow_delete is literal: you can still issue _delete_by_query to free space without first removing the block, which is the intended escape hatch.

Common Mistakes

Setting flood stage to 99% or higher. This effectively disables the safety net. Full disk leads to translog corruption and unstable nodes.
Manually clearing the block while disk is still full. The block re-applies in seconds. Free disk first.
Forgetting read_only_allow_delete after recovery. Some operators set index.blocks.read_only: true manually for a different reason and conflate it with flood stage. The two are independent.
Mixing percentages and absolute values across watermarks. Pick one form for all three.

Resolve the 95% Flood-Stage Write Block Automatically with Pulse

Pulse is an AI DBA for Elasticsearch and OpenSearch. When a data node crosses cluster.routing.allocation.disk.watermark.flood_stage (default 95%) and TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block starts rejecting writes in your environment, Pulse:

Continuously tracks per-node disk usage, the index.blocks.read_only_allow_delete: true block on every affected index, and the auto-release threshold tied to the high watermark (Elasticsearch 7.4+)
Correlates the flood-stage trip with shard placement (which indices have shards on the affected node), ILM state, snapshot age, and recent translog/Lucene merge activity competing for disk
Identifies which path applies - missing ILM delete phase, runaway translog, a node that needed expansion before the 85% low watermark warned, or a manually misset index.blocks.read_only confused with flood stage
Recommends the precise fix - run _delete_by_query (still permitted under the block) to free space, trigger rollover, take a snapshot and delete the source index, expand storage, or clear stale blocks with PUT /*/_settings {"index.blocks.read_only_allow_delete": null} once disk is below 90%
Applies low-risk fixes automatically with your approval (deleting indices past retention, clearing the read-only block after auto-release condition is met) or generates a one-click cluster settings PR

Pulse turns the controlled-outage recovery procedure above into an agentic SRE workflow that minimizes write downtime. Start a free trial.

Frequently Asked Questions

Q: What is the fastest way to diagnose and recover from a flood-stage read_only_allow_delete block in production?
A: First confirm the block is in place with GET /*/_settings | grep read_only_allow_delete, then free disk via _delete_by_query or rollover (the block allows deletes by design). For continuous coverage, Pulse acts as an AI DBA for Elasticsearch and OpenSearch that detects the 95% flood-stage trip, identifies which indices have shards on the affected node, and walks through the recovery - delete, rollover, or expand - while monitoring for auto-release back below the high watermark.

Q: What is the default value of cluster.routing.allocation.disk.watermark.flood_stage?
A: The default is 95%. When any data node's disk usage crosses 95%, Elasticsearch applies an index.blocks.read_only_allow_delete: true block to every index that has a shard on the affected node. Writes are blocked until disk space is freed.

Q: What is index.blocks.read_only_allow_delete?
A: It is the index-level block Elasticsearch applies when flood stage is reached. It blocks writes but allows reads and delete-by-query, so you can free space to recover. Once disk drops below the high watermark, Elasticsearch (7.4+) removes the block automatically.

Q: How do I remove the read-only-allow-delete block?
A: First free disk space. On Elasticsearch 7.4 and later the block lifts automatically. If it persists, run PUT /*/_settings {"index.blocks.read_only_allow_delete": null} to clear the block on every index.

Q: Does flood stage block reads or only writes?
A: Only writes. The block name is literal: read_only_allow_delete allows reads and _delete_by_query, but rejects indexing and update calls. This is intentional so operators can delete data to free disk.

Q: How is flood stage different from the high watermark?
A: The high watermark (default 90%) triggers shard relocation away from the over-utilised node but does not block writes. Flood stage (95%) goes further and forces the affected indices into a read-only state. Flood stage is the last line of defence before full disk.

Q: Can I disable the flood-stage watermark?
A: There is no clean off switch. You can set it close to 100% to delay it, but doing so risks node failure if the disk fills entirely. Better to fix the underlying capacity issue.