Elasticsearch Read-Only Block: Flood Stage Disk Watermark

Symptom

Writes fail with 403 FORBIDDEN "cluster read-only (api)"; disk over 95%.

Root cause

Flood-stage watermark trips and ES flips indices read-only.

Fix, in order

1. Free disk space first — that is the actual fixdf -h /var/lib/elasticsearch && du -sh /var/lib/elasticsearch/*
2. After cleanup, unset the read-only blockPUT _all/_settings { "index.blocks.read_only_allow_delete": null }
3. Raise watermarks only with a planPUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": "90%" } }

Prevent it coming back

Alert at 80% disk; ILM policies to expire old indices.

Run it on clean infra

Spinning up a fresh box to reproduce or escape this error?

Servers with headroom

Related

Cloud & DevOps tool comparisons · Hardware build guides

NinjaOps publishes free engineering guides. Some outbound links are affiliate links: they cost you nothing and support the site.

Last reviewed 2026-10-02 · NinjaOps SEO grid · DevOps blog