LlamaIndex Index Rebuild Failure After Concurrent Kafka Event Updates

Q: Inspection commands

# Verify that the index directory is locked lsof +D /var/data/llama_index | grep .lock # Show the corrupted metadata file cat /var/data/llama_index/metadata.json | head -n 30 Solution Introduce a distributed lock around refresh Use a lightweight coordination service (e.g., redis with SETNX or Zookeeper) to guarantee that only one consumer performs a refresh at a time.

Q: Verify Functional validation

# After deploying the lock, run the concurrent test again python consumer_a.py & python consumer_b.py & wait # Expected log output 2026-10-27 14:15:02,112 INFO index_builder - Refresh succeeded (lock acquired) 2026-10-27 14:15:02,115 INFO index_builder - Refresh succeeded (lock acquired) Metadata integrity check # Verify that metadata.json is valid JSON python -c "import json, sys; json.load(open('/var/data/llama_index/metadata.json'))" # No exception → file is well‑formed Performance sanity check Measure the average latency of a refresh before and after the change:

In this article

Table of Contents

Problem

LlamaIndex is used as a searchable vector store for documents that are continuously updated via a Kafka‑driven event pipeline. After a recent deployment, the index rebuild step started failing intermittently with errors such as MetadataCorruptionError and IndexRebuildFailed. The failures appear only when multiple Kafka events are processed in parallel, causing the rebuild to abort and leaving the index in an unusable state.

Root Cause

The index rebuild process in LlamaIndex assumes exclusive access to the underlying storage directory while it rewrites the metadata.json and segment files. In the event‑driven architecture, each consumer instance processes Kafka messages concurrently and invokes index.refresh() after persisting a new document. When two or more refreshes run at the same time, the following sequence occurs:

Consumer A writes a new document chunk and calls index.refresh().
Consumer B, almost simultaneously, writes a different document chunk and also calls index.refresh().
Both refresh calls acquire a read lock on the index but attempt to acquire a write lock on the metadata file.
The first writer succeeds; the second writer overwrites the partially written metadata.json with stale data, corrupting the JSON structure.
LlamaIndex detects the malformed metadata on the next rebuild attempt and raises MetadataCorruptionError, aborting the rebuild.

In short, the race condition stems from missing synchronization around the index’s metadata updates, leading to corrupted state when concurrent refreshes occur.

Debug

Log patterns


2026-10-27 14:02:13,842 ERROR index_builder - MetadataCorruptionError: Unexpected token at line 23 column 5
2026-10-27 14:02:13,845 INFO  index_builder - Rebuild aborted after 3 attempts
2026-10-27 14:02:13,847 WARN  kafka_consumer - Event processing delayed due to index lock contention

Reproducing the race

Run two parallel consumers that each ingest a batch of documents and trigger a refresh:


# consumer_a.py
for doc in docs_a:
    index.upsert(doc)
index.refresh()   # triggers rebuild

# consumer_b.py (started ~100 ms later)
for doc in docs_b:
    index.upsert(doc)
index.refresh()

When both scripts execute concurrently, the second refresh() frequently fails with the error shown above.

Inspection commands


# Verify that the index directory is locked
lsof +D /var/data/llama_index | grep .lock

# Show the corrupted metadata file
cat /var/data/llama_index/metadata.json | head -n 30

Solution

Introduce a distributed lock around refresh

Use a lightweight coordination service (e.g., redis with SETNX or Zookeeper) to guarantee that only one consumer performs a refresh at a time.

Before


def process_event(event):
    doc = parse(event)
    index.upsert(doc)
    # Immediate refresh – no synchronization
    index.refresh()

After


import redis
import time
import uuid

REDIS_LOCK_KEY = "llama_index_refresh_lock"
LOCK_TTL = 30  # seconds
client = redis.StrictRedis(host='redis', port=6379, db=0)

def acquire_lock():
    lock_id = str(uuid.uuid4())
    while not client.set(REDIS_LOCK_KEY, lock_id, nx=True, ex=LOCK_TTL):
        time.sleep(0.1)  # back‑off
    return lock_id

def release_lock(lock_id):
    script = """
    if redis.call("GET", KEYS[1]) == ARGV[1] then
        return redis.call("DEL", KEYS[1])
    else
        return 0
    end
    """
    client.eval(script, 1, REDIS_LOCK_KEY, lock_id)

def process_event(event):
    doc = parse(event)
    index.upsert(doc)

    lock_id = acquire_lock()
    try:
        index.refresh()
    finally:
        release_lock(lock_id)

The lock guarantees exclusive access to the metadata rewrite. The TTL prevents deadlocks if a consumer crashes while holding the lock.

Batch refresh instead of per‑event

Collect incoming events in a short time window (e.g., 5 seconds) and perform a single refresh for the batch. This reduces lock contention and improves throughput.


from collections import defaultdict
from threading import Timer

batch = defaultdict(list)
timer = None
BATCH_WINDOW = 5  # seconds

def schedule_refresh():
    global timer
    if timer:
        timer.cancel()
    timer = Timer(BATCH_WINDOW, execute_batch_refresh)
    timer.start()

def process_event(event):
    doc = parse(event)
    batch['docs'].append(doc)
    schedule_refresh()

def execute_batch_refresh():
    lock_id = acquire_lock()
    try:
        for doc in batch['docs']:
            index.upsert(doc)
        index.refresh()
    finally:
        release_lock(lock_id)
        batch['docs'].clear()

Verify

Functional validation


# After deploying the lock, run the concurrent test again
python consumer_a.py &
python consumer_b.py &
wait

# Expected log output
2026-10-27 14:15:02,112 INFO  index_builder - Refresh succeeded (lock acquired)
2026-10-27 14:15:02,115 INFO  index_builder - Refresh succeeded (lock acquired)

Metadata integrity check


# Verify that metadata.json is valid JSON
python -c "import json, sys; json.load(open('/var/data/llama_index/metadata.json'))"
# No exception → file is well‑formed

Performance sanity check

Measure the average latency of a refresh before and after the change:

Metric	Before (no lock)	After (distributed lock)
Mean refresh latency	≈ 120 ms (high variance)	≈ 150 ms (stable)
Failure rate	≈ 12 %	0 %

Prevent

Always serialize metadata writes. Use a distributed lock or a single writer service for any operation that rewrites metadata.json or segment files.
Batch updates. Accumulate document changes and refresh in intervals rather than per event.
Health‑check the index directory. Deploy a watchdog that periodically parses metadata.json and raises an alert if JSON is invalid.
Monitor lock contention. Export a Prometheus metric llama_index_refresh_lock_wait_seconds to detect spikes.
Graceful shutdown. Ensure the lock release script runs in a finally block or via a Kubernetes preStop hook.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the failure only appear under load? The race condition is triggered when two refreshes overlap. Under low traffic, refreshes are serialized naturally, hiding the bug.
Can I use a file‑system lock instead of Redis? Yes, fcntl advisory locks work on a shared volume, but they do not survive container restarts. A distributed lock service provides stronger guarantees across pods.
Is it safe to increase the lock TTL? The TTL should be slightly longer than the longest expected refresh (including I/O). Setting it too high can delay recovery if a worker crashes while holding the lock.
Do I need to rebuild the entire index after fixing the lock? No. The fix only prevents future corruption. Existing valid segments remain usable; you can optionally run a one‑time index.optimize() to compact the index.
How does this interact with LlamaIndex’s async API? The same lock logic applies; wrap the async await index.refresh() call inside an async lock primitive (e.g., aioredis.lock) to preserve exclusivity.