RAG pipeline returns empty results during rolling update in PyTorch

Problem Description

During a Kubernetes rolling update of a TorchServe‑hosted Retrieval‑Augmented Generation (RAG) service, the RAGRetriever began returning an empty list of documents for every query. The symptom manifested as log entries such as:

INFO 2026-09-21 12:34:56,789 RAGRetriever: No documents retrieved for query
ERROR 2026-09-21 12:34:57,102 faiss index not found at path /model/index.faiss
WARNING 2026-09-21 12:34:58,210 torchserve model version 3 loaded but index file missing, falling back to empty retrieval

Clients observed a sudden drop in answer relevance, while the generation head of the RAG pipeline continued to produce text (often hallucinating) because the retrieval step yielded no context.

Root Cause Analysis

The empty retrieval results stem from a combination of three intertwined factors that become visible during a rolling update:

1. FAISS index not available at model start‑up – TorchServe loads the model and its associated RAGRetriever before the persistent volume that holds index.faiss is mounted. This mirrors the incident where a Kubernetes rolling update restarted pods before the FAISS index volume was attached, causing a 15‑minute outage.
2. Retriever cache cleared on model hot‑reload – According to the Hugging Face RAG tutorial, the RAGRetriever caches the loaded index in memory. When TorchServe performs a hot‑swap of model weights (as described in the TorchServe model versioning guide), the in‑memory cache is invalidated. If the index cannot be reloaded, the retriever falls back to an empty list.
3. Mismatch between tokenizer and index embeddings – In a separate production incident on AWS ECS, a new model version shipped with a different tokenizer version, producing out‑of‑range query embeddings that FAISS could not match. While this was not the primary cause in the current scenario, the same symptom (empty results) appears, and the logs do not differentiate the underlying mismatch.

Official PyTorch Distributed Data Parallel (DDP) documentation emphasizes that model state loading must be coordinated with external resources during rolling updates (DDP guidelines). Failure to synchronize index availability violates that recommendation.

Investigation and Debugging Steps

1. Verify index presence on pod start‑up

kubectl exec -it rag-service-5c9d7b9f6f-abcde -- ls -l /model/
total 8240
-rw-r--r-- 1 root root  8423424 Jan 12 08:15 index.faiss
-rw-r--r-- 1 root root   123456 Jan 12 08:15 config.json

If the file is missing, the retriever will log “faiss index not found”.

2. Inspect TorchServe model loading logs

2026-09-21 12:34:55,876 INFO  [ModelLoader] Loading model version 3 from /model/
2026-09-21 12:34:56,001 INFO  [RAGRetriever] Initializing FAISS index from /model/index.faiss
2026-09-21 12:34:56,102 ERROR [RAGRetriever] faiss index not found at path /model/index.faiss
2026-09-21 12:34:56,103 INFO  [RAGRetriever] Falling back to empty retrieval

3. Check volume mount lifecycle

kubectl describe pod rag-service-5c9d7b9f6f-abcde
...
Volumes:
  rag-index:
    Type:       PersistentVolumeClaim (claim "rag-index-pvc")
    Capacity:   10Gi
    Access Modes: RWO
    Mounted By: rag-service
...
Events:
  Type    Reason                 Age   From               Message
  ----    ------                 ----  ----               -------
  Normal  Pulling                2m    kubelet, node1     Pulling image "pytorch/torchserve:latest"
  Normal  Pulled                 2m    kubelet, node1     Successfully pulled image
  Normal  Created                2m    kubelet, node1     Created container rag-service
  Normal  Started                2m    kubelet, node1     Started container rag-service
  Warning VolumeMountFailed      2m    kubelet, node1     Volume rag-index not mounted: mount failed: permission denied

The warning indicates the mount failure that leads to the missing index.

4. Reproduce the empty‑retrieval condition locally

python -c "
from transformers import RagTokenizer, RagRetriever, RagSequenceForGeneration
tokenizer = RagTokenizer.from_pretrained('facebook/rag-token-nq')
retriever = RagRetriever.from_pretrained('facebook/rag-token-nq', index_name='custom', passages_path='/tmp/missing')
print(retriever.retrieve('What is the capital of France?'))
"
# Expected output: list of passages
# Actual output: []

5. Confirm tokenizer‑index compatibility

python - <<'PY'
from transformers import RagTokenizer, RagRetriever
tok_v1 = RagTokenizer.from_pretrained('facebook/rag-token-nq')
retriever_v1 = RagRetriever.from_pretrained('facebook/rag-token-nq', index_name='custom')
emb_v1 = tok_v1.encode('test query', return_tensors='pt')
print('Embedding dim:', emb_v1.shape[-1])  # e.g., 768
PY

If the new model version reports a different embedding dimension (e.g., 1024), FAISS built with 768‑dim vectors will reject the query, yielding empty results.

Resolution

Step 1 – Ensure index volume is ready before model loading

Modify the deployment manifest to add an initContainer that blocks until the index file is present.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-service
spec:
  replicas: 3
  template:
    spec:
      initContainers:
      - name: wait-for-index
        image: busybox
        command: ['sh', '-c', 'while [ ! -f /model/index.faiss ]; do echo "Waiting for FAISS index..."; sleep 2; done']
        volumeMounts:
        - name: rag-index
          mountPath: /model
      containers:
      - name: rag-service
        image: pytorch/torchserve:latest
        env:
        - name: TORCHSERVE_MODEL_DIR
          value: /model
        volumeMounts:
        - name: rag-index
          mountPath: /model
      volumes:
      - name: rag-index
        persistentVolumeClaim:
          claimName: rag-index-pvc

This guarantees that TorchServe’s model loader sees a valid index.faiss before invoking RAGRetriever.

Step 2 – Persist the retriever cache across hot‑reloads

Configure TorchServe to reuse the same model_store directory for the retriever index and disable automatic cache clearing:

# torchserve.properties
model_store=/model_store
default_workers_per_model=2
load_models_on_startup=true
# Custom environment variable read by the RAG wrapper
RAG_RETRIEVER_CACHE=/model_store/rag_cache

In the custom model handler, add:

if os.path.isdir(os.getenv('RAG_RETRIEVER_CACHE')):
    retriever = RagRetriever.from_pretrained(
        model_name_or_path,
        index_name='custom',
        passages_path=os.getenv('RAG_RETRIEVER_CACHE')
    )
else:
    # Fallback to loading from /model/index.faiss
    retriever = RagRetriever.from_pretrained(...)

Step 3 – Align tokenizer and index embedding dimensions

When a new model version is built, regenerate the FAISS index with the same embedding size:

python - <<'PY'
from transformers import RagTokenizer, RagRetriever
import faiss, numpy as np

tokenizer = RagTokenizer.from_pretrained('facebook/rag-token-nq')
retriever = RagRetriever.from_pretrained('facebook/rag-token-nq')
emb_dim = retriever.question_encoder.config.hidden_size  # e.g., 768

# Assume `documents` is a list of raw texts
doc_embeddings = []
for doc in documents:
    inputs = tokenizer(doc, return_tensors='pt')
    with torch.no_grad():
        emb = retriever.question_encoder(**inputs).pooler_output
    doc_embeddings.append(emb.squeeze().cpu().numpy())

index = faiss.IndexFlatIP(emb_dim)
index.add(np.stack(doc_embeddings))
faiss.write_index(index, '/model/index.faiss')
PY

Deploy the regenerated index alongside the new model version.

Verification

Functional test after rollout

curl -X POST http://rag-service.example.com/predictions/rag -d '{"inputs":"Who invented the telephone?"}'

Expected response snippet (excerpt):

{
  "generated_text": "Alexander Graham Bell invented the telephone..."
}

Check the logs for a successful retrieval entry:

INFO 2026-09-21 13:02:10,453 RAGRetriever: Retrieved 5 documents for query

Metrics validation

• Prometheus metric rag_retriever_documents_total should increase with each query.
• Latency histogram rag_inference_seconds should show a modest increase (≈10 ms) for the retrieval step, confirming the index is being consulted.

Operational Best Practices & Prevention

• Mount readiness probes: Use a readinessProbe that checks for /model/index.faiss existence before marking the pod ready.
• Version‑locked index artifacts: Store the FAISS index in a version‑specific sub‑directory (e.g., /model/v3/index.faiss) and reference it via an environment variable. This prevents accidental reuse of an outdated index.
• Graceful rolling update strategy: Set maxUnavailable: 0 and maxSurge: 1 to ensure at least one healthy replica serves traffic while a new pod initializes its index.
• Health check endpoint: Expose /healthz that runs a lightweight retrieval against a known query and returns HTTP 200 only if len(results) > 0.
• Monitoring alerts: Trigger an alert when the log pattern “No documents retrieved for query” exceeds a threshold (e.g., >5 per minute) or when rag_retriever_documents_total plateaus.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the RAG pipeline work locally but return empty results after a hot‑reload?
   Because the hot‑reload clears the in‑memory FAISS index. If the index file is not yet mounted or fails to load (e.g., due to OOM), the retriever falls back to an empty list.
2. Can I reuse the same FAISS index across different model versions?
   Only if the embedding dimension and tokenizer vocabulary are identical. A mismatch leads to out‑of‑range vectors and zero matches.
3. What should I do if I see “CUDA out of memory while loading retriever index”?
   Load the index on CPU first (set device='cpu' in the retriever constructor), then move the model to GPU after the index is built, or increase the pod’s GPU memory limit.
4. Is there a way to force TorchServe to wait for the index before loading the model?
   Yes, use an initContainer that blocks until the index file appears, as shown in the resolution section.
5. How can I verify which FAISS index file a running pod is using?
   Execute cat /model/index_path.txt (if you store the path in a file) or inspect the environment variable RAG_RETRIEVER_CACHE inside the container.