LlamaIndex Replica Sync Lag During Concurrent Evaluation Runs

Q: 1. Force immediate index refresh after bulk upserts

Update the ingestion pipeline to call index.refresh() with a timeout, ensuring the cache reflects the latest versions.

Q: 2. Adjust vector‑store replication configuration

For Pinecone, increase the replication sync_interval_seconds and enable synchronous writes during evaluation runs:

Q: 4. Enable metadata write‑through

For Milvus, enable the enable_dynamic_field flag to ensure metadata is persisted alongside vectors:

Q: Operational Experience

During the incident, the following observations proved useful for future triage:

In this article

Table of Contents

Problem Description

During large‑scale RAG benchmark runs, multiple evaluation workers concurrently upsert documents into a distributed vector store (Pinecone, Weaviate, Milvus, or Azure Cognitive Search) via LlamaIndex. The retrieval phase intermittently returns stale embeddings and missing metadata, causing a measurable drop in precision (up to 12 %) and inconsistent answer faithfulness across workers.

Typical symptoms observed in the logs:


2026-08-15 10:42:31,842 ERROR VectorStoreException: Document version mismatch – expected version 42, got 38
2026-08-15 10:43:07,119 WARN PineconeSyncError: Replication lag exceeded threshold (lag=45s, max_allowed=30s)
2026-08-15 10:44:12,501 ERROR LlamaIndexRefreshError: Index refresh failed – stale cache detected

Impact includes:

Retrieval latency spikes (30‑60 s lag between upsert completion and query visibility).
Precision drop of 12 % in batch evaluation runs.
Flaky benchmark results that vary between workers.

Root Cause Analysis

LlamaIndex’s Refresh & Sync API triggers an asynchronous propagation of embeddings and metadata to replica nodes. When many workers perform bulk upserts concurrently, the vector store’s replication pipeline becomes saturated:

Asynchronous replication windows – vector stores such as Pinecone and Weaviate batch replica updates on a timed schedule (e.g., every 30 s). Under peak load, the batching queue grows, extending the window.
Metadata propagation latency – metadata fields are stored in a separate KV layer; if the upstream write succeeds but the downstream sync is pending, queries see an older version of the document.
Cache staleness in LlamaIndex – the Index object caches the last known Document version. Without an explicit refresh, the cache serves stale entries, leading to the “Document version mismatch” error.

Evidence from real incidents confirms the pattern:

Evaluation cluster with 8 workers indexing 1 M documents to a Weaviate cluster showed a 30‑145 ms lag between upsert completion and query visibility (Weaviate incident).
Pinecone‑backed deployments experienced replication lag spikes up to 60 s during peak load (GitHub Issue #2129).
Milvus bulk upserts produced intermittent missing metadata after the asynchronous sync window (GitHub Issue #2375).

Investigation and Debugging

The following step‑by‑step investigation helped isolate the lag source:

Enable vector‑store level metrics. For Pinecone:


curl -X GET "https://controller.pinecone.io/databases/my-index/metrics" \
     -H "Api-Key: $PINECONE_API_KEY"

Observed replication lag metric: replication_lag_seconds: 48.

Check LlamaIndex cache state.


>>> from llama_index import ServiceContext, GPTVectorStoreIndex
>>> ctx = ServiceContext.from_defaults()
>>> index = GPTVectorStoreIndex.from_documents([], service_context=ctx)
>>> index._index_struct._doc_id_to_version
{ 'doc_123': 38 }

Version lagged behind the store’s latest version (42).

Inspect vector‑store logs for async batch processing. Example Weaviate log excerpt:


2026-08-15 10:41:59.123 INFO  weaviate.replication - Batch sync started (size=15000)
2026-08-15 10:42:30.987 INFO  weaviate.replication - Batch sync completed (duration=31.864s)

Delays aligned with observed retrieval inconsistencies.

Run a controlled single‑worker upsert. With --max-concurrency=1, the lag dropped to ≈5s, confirming concurrency‑induced saturation.

Resolution

The fix combines three layers: enforce deterministic refresh, tune vector‑store replication, and adjust LlamaIndex concurrency settings.

1. Force immediate index refresh after bulk upserts

Update the ingestion pipeline to call index.refresh() with a timeout, ensuring the cache reflects the latest versions.

Before:


# ingestion.py
def upsert_documents(docs):
    for doc in docs:
        index.insert(doc)   # asynchronous, no refresh

After:


# ingestion.py
def upsert_documents(docs):
    index.upsert(docs)                     # bulk upsert
    try:
        index.refresh(wait=True, timeout=30)   # block until replicas sync
    except Exception as e:
        logger.error(f"Refresh failed: {e}")
        raise

2. Adjust vector‑store replication configuration

For Pinecone, increase the replication sync_interval_seconds and enable synchronous writes during evaluation runs:


# pinecone_config.json
{
  "replication_factor": 3,
  "sync_interval_seconds": 10,   // default 30s, reduced for faster sync
  "write_consistency": "strong"  // force synchronous acknowledgment
}

Apply the config via the Pinecone console or CLI:


pinecone update-index --name my-index --config pinecone_config.json

3. Limit concurrent ingestion per node

Leverage the guidance from the LlamaIndex concurrency best practices. Set a per‑process semaphore to cap parallel upserts:


import threading

MAX_CONCURRENT_UPSERTS = 4
semaphore = threading.Semaphore(MAX_CONCURRENT_UPSERTS)

def safe_upsert(docs):
    with semaphore:
        index.upsert(docs)
        index.refresh(wait=True, timeout=30)

4. Enable metadata write‑through

For Milvus, enable the enable_dynamic_field flag to ensure metadata is persisted alongside vectors:


# milvus.yaml
schema:
  enable_dynamic_field: true
  sync_replication: true

Reload the collection after updating the schema.

Validation

After applying the changes, verify the system behaves as expected:

Run a single‑worker benchmark and record the upsert‑to‑query latency. Expected lag ≤ 10s.
Check the version map after refresh:
```
>>> index._index_struct._doc_id_to_version['doc_123']
42
```
Should match the vector store’s reported version.

Monitor replication lag metric:


curl -s "https://controller.pinecone.io/databases/my-index/metrics" | jq '.replication_lag_seconds'
# => 8

Execute a downstream evaluation run with 8 workers. The precision variance across workers should drop from 12 % to < 2 %.

Operational Experience

During the incident, the following observations proved useful for future triage:

Misleading symptom: Errors appeared only as “Document version mismatch” without obvious replication warnings, leading to an initial focus on the LlamaIndex cache rather than the backend store.
Assumption failure: Assuming that a successful upsert API call guarantees immediate query visibility; in distributed stores, the write is often queued for batch replication.
Edge case: When a worker crashed after upserting but before refresh, its local cache retained the stale version, contaminating subsequent queries even after the store had synced.
Lesson learned: Embedding pipelines should treat upserts as eventually consistent and explicitly enforce a refresh barrier when deterministic results are required (e.g., evaluation benchmarks).

Best Practices and Prevention

Practice	Why it helps	Implementation Hint
Enable synchronous writes or reduce `sync_interval_seconds`	Ensures replicas acknowledge before the client proceeds	Pinecone: `write_consistency="strong"`; Milvus: `sync_replication=true`
Explicit index refresh after bulk upserts	Invalidates LlamaIndex’s local cache and forces consistency check	Call `index.refresh(wait=True)` with a sensible timeout
Rate‑limit concurrent ingestion per node	Avoids overwhelming the replication pipeline	Use semaphores or a job queue; see code snippet above
Instrument replication lag metrics and alert on thresholds	Detects lag before it impacts evaluation runs	Set alert: `replication_lag_seconds > 15`
Store critical metadata in the vector store’s primary payload	Prevents missing fields after async propagation	Enable `enable_dynamic_field` (Milvus) or embed metadata in the vector’s `metadata` dict

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the “Document version mismatch” error appear only under high concurrency?
Because LlamaIndex caches the version seen after the first upsert. Concurrent workers write newer versions to the store, but the cache isn’t refreshed until index.refresh() is called, leading to mismatches.
Can I disable LlamaIndex’s local cache to avoid stale reads?
Yes. Instantiate the service context with cache_enabled=False, but this incurs a performance penalty on every query.
Do all vector stores suffer the same replication lag?
The magnitude varies. Pinecone and Weaviate batch replication on a timer; Milvus can be configured for synchronous writes; Azure Cognitive Search propagates metadata in a separate indexer pipeline. Tuning each store’s sync interval is essential.
What is a safe timeout for index.refresh()?
Empirically, 30 seconds covers typical batch windows for Pinecone and Weaviate. For Milvus with synchronous replication, 10 seconds is often sufficient.
How can I verify that metadata has been propagated?
Query the vector store directly for a known document ID and inspect the returned payload. For example, with Weaviate:
```
curl -X GET "https://my-weaviate.io/v1/objects/doc_123"
```
The response should include the latest metadata fields.