Elasticsearch Index Rebuild Fails During Jenkins Pipeline With Index_not_found

In this article

Table of Contents

Elasticsearch index rebuild fails during Jenkins pipeline with `index_not_found_exception`

Problem

The CI/CD pipeline (Jenkins or GitHub Actions) executes a scripted step that deletes existing indices and recreates them for a fresh staging deployment. During the delete‑and‑recreate phase the pipeline aborts with errors such as:


[2026-10-27T14:32:11,123][ERROR][o.e.c.r.a.AllocationService] [node-1] failed to delete index [orders-2023] (index_not_found_exception)
org.elasticsearch.indices.IndexNotFoundException: no such index [orders-2023]
    at org.elasticsearch.indices.IndicesService.indexExists(IndicesService.java:210)
    ...


[2026-10-27T14:32:13,487][ERROR][o.e.c.r.a.AllocationService] [node-2] failed to create index [orders-2023] (cluster_block_exception)
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [READ_ONLY_ALLOW_DELETE, index read-only / allow delete (api)]; 
    at org.elasticsearch.cluster.block.ClusterBlockException.ensureNotBlocked(ClusterBlockException.java:54)
    ...

Result: the staging environment is left without the expected indices, causing downstream integration tests to fail.

Root Cause

Two intertwined conditions cause the failure:

Race between delete and create operations. The pipeline issues a DELETE /my-index request and immediately follows with a PUT /my-index. Elasticsearch processes the delete asynchronously; the index metadata may still be marked as closed or deleted when the create request arrives, leading to index_not_found_exception for the delete (if the index was already removed by a previous run) or cluster_block_exception if the node is temporarily in a read‑only state while the delete task finishes.
Cluster read‑only block triggered by disk watermarks. When the node’s disk usage exceeds the cluster.routing.allocation.disk.watermark.flood_stage threshold, Elasticsearch automatically sets a global cluster.blocks.read_only_allow_delete flag. The flag permits delete operations but blocks any index creation, producing the cluster_block_exception observed.

Both symptoms manifest when the pipeline runs on a shared staging cluster that is already near its disk capacity and when the script does not verify the index state before proceeding.

Debug

Follow these steps to reproduce the failure locally and gather evidence.

1. Inspect pipeline logs


[INFO] Deleting index orders-2023
curl -XDELETE 'http://es-staging:9200/orders-2023?pretty'
[INFO] Creating index orders-2023
curl -XPUT 'http://es-staging:9200/orders-2023?pretty' -H 'Content-Type: application/json' -d @mapping.json

2. Check Elasticsearch cluster health and block status


curl -s http://es-staging:9200/_cluster/health?pretty
curl -s http://es-staging:9200/_cluster/settings?include_defaults=true&pretty

Typical output when the flood‑stage watermark is hit:


{
  "persistent" : {
    "cluster" : {
      "blocks" : {
        "read_only_allow_delete" : "true"
      }
    }
  },
  "transient" : { }
}

3. Verify index existence before deletion


curl -s http://es-staging:9200/_cat/indices/orders-2023?v

If the index is already absent, the delete call will return 404 and the pipeline treats it as an error.

4. Observe asynchronous task completion


curl -s http://es-staging:9200/_tasks?detailed=true&actions=*delete* | jq '.nodes[].tasks'

Confirm that the delete task has finished before issuing the create request.

5. Capture a packet trace (optional)


sudo tcpdump -i eth0 -nn -s 0 -w /tmp/es-rebuild.pcap host es-staging and port 9200

Inspect the capture for HTTP 404 or 403 responses that align with the pipeline timestamps.

Solution

Make the rebuild step idempotent, wait for delete completion, and guard against the read‑only block.

1. Update the Jenkins pipeline script


stage('Rebuild Elasticsearch Indices') {
    steps {
        script {
            def esHost = 'http://es-staging:9200'
            def indexName = 'orders-2023'
            def mappingFile = 'mapping.json'

            // 1️⃣ Ensure the cluster is writable
            sh """
                curl -s -XPUT \${esHost}/_cluster/settings -H 'Content-Type: application/json' -d '
                {
                  "persistent": {
                    "cluster.blocks.read_only_allow_delete": null
                  }
                }' || true
            """

            // 2️⃣ Delete if it exists, ignore 404
            sh """
                curl -s -XDELETE \${esHost}/\${indexName}?pretty -w '\\n%{http_code}' | grep -E '200|404'
            """

            // 3️⃣ Wait for the delete task to finish (max 30s)
            timeout(time: 30, unit: 'SECONDS') {
                waitUntil {
                    def resp = sh(script: "curl -s \${esHost}/_cat/indices/\${indexName}?v", returnStdout: true).trim()
                    return resp == ''
                }
            }

            // 4️⃣ Re‑create the index
            sh """
                curl -s -XPUT \${esHost}/\${indexName}?pretty \\
                    -H 'Content-Type: application/json' \\
                    -d @\${mappingFile}
            """
        }
    }
}

2. Add a retry wrapper for transient read‑only blocks


def createIndexWithRetry(esHost, indexName, mappingFile, attempts = 5) {
    for (i = 1; i <= attempts; i++) {
        def rc = sh(script: """
            curl -s -o /dev/null -w '%{http_code}' -XPUT \${esHost}/\${indexName}?pretty \\
                -H 'Content-Type: application/json' \\
                -d @\${mappingFile}
        """, returnStdout: true).trim()
        if (rc == '200') {
            return true
        }
        if (rc == '403') {
            // read‑only block still active – wait and retry
            sleep time: 5, unit: 'SECONDS'
        } else {
            error "Unexpected response ${rc} while creating index"
        }
    }
    error "Failed to create index after ${attempts} attempts"
}

3. Before‑and‑After configuration comparison

Setting	Before (failing run)	After (fixed run)
Cluster block	`cluster.blocks.read_only_allow_delete: true`	`cluster.blocks.read_only_allow_delete: null`
Delete request	Fails with `404` because index already absent	Uses `ignore_unavailable=true` (implicit via script) – treated as success
Creation timing	Immediate after delete → `cluster_block_exception`	Waits for index disappearance → `200 OK`

Verify

Run the pipeline again and confirm the following checks pass.

Cluster health is green after the stage.
Index appears in _cat/indices with the expected mapping.
No index_not_found_exception or cluster_block_exception lines in the Jenkins console output.


curl -s http://es-staging:9200/_cat/indices/orders-2023?v
# Expected line:
orders-2023 1 1 0 0 1.2gb 1.2gb open

Optionally, execute a simple search to ensure the index is functional:


curl -s -XGET 'http://es-staging:9200/orders-2023/_search?q=*:*&size=0' | jq '.hits.total'

Prevent

Guard against disk‑watermark blocks. Set up a monitoring alert on cluster.blocks.read_only_allow_delete and on cluster.routing.allocation.disk.watermark.flood_stage. Automate a cleanup job that frees space before a deployment.
Make index‑rebuild scripts idempotent. Use ?ignore_unavailable=true on delete, check existence before create, and always wait for the delete task to finish.
Leverage ILM or index aliases. Instead of dropping and recreating the same name, create a new index (e.g., orders-2023-20231027) and atomically switch an alias. This eliminates the delete‑create race entirely.
Configure pipeline timeouts. Use Jenkins timeout and waitUntil blocks to avoid indefinite hangs if the cluster is under stress.
Record the cluster block state. Add a diagnostic step that logs GET /_cluster/settings before any index operation; this makes future failures easier to triage.

FAQ

Why does the delete step sometimes return 404 even though the index should exist?
Because a previous pipeline run may have left the index removed, or the delete operation from an earlier stage is still being processed. Treat 404 as a non‑error when the intent is “ensure the index does not exist”.
What triggers cluster.blocks.read_only_allow_delete and how can I clear it?
Elasticsearch sets the flag when disk usage exceeds the flood‑stage watermark. Clear it with a PUT /_cluster/settings request that sets the flag to null, but first free disk space or increase the watermark thresholds.
Can I rely on the wait_for_active_shards parameter instead of a manual wait loop?
wait_for_active_shards only applies to the create request. It does not guarantee the previous delete has fully completed, so a separate check (e.g., _cat/indices or task API) is still required.
Is using an alias with rollover a better approach than delete‑and‑recreate?
Yes. Rollover creates a brand‑new index and swaps the alias atomically, eliminating the race condition and avoiding read‑only blocks because no delete occurs.
How do I debug a situation where the pipeline succeeds but the index is still missing?
Inspect the Elasticsearch task list (GET /_tasks) for any lingering delete tasks, check the cluster logs for disk‑watermark events, and verify that the pipeline did not skip the create step due to a premature return or error condition.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

Elasticsearch index rebuild fails during Jenkins pipeline with index_not_found_exception