RAG Pipeline Empty Retrieval After Docker Container Restart In Kubernetes

Q: Problem – RAG pipeline returns empty retrieval after Docker container restart in Kubernetes

Symptom: After a pod restart (e.g., rolling update, node‑drain, or container crash) the Retrieval‑Augmented Generation (RAG) service answers every query with No documents retrieved or an empty list, even though the knowledge base was populated before the restart.

Q: Root Cause – Why the vector store disappears after a restart

The RAG pipeline relies on a persisted vector store (FAISS, Qdrant, Chroma, etc.) that holds the embeddings of the knowledge base. In Kubernetes the persistence strategy is defined by the volume configuration of the pod. The most common mis‑configurations that lead to an empty retrieval are:

Q: 4. Add a readiness probe that checks the index file exists

readinessProbe: exec: command: - sh - -c - test -f /data/index/faiss.index initialDelaySeconds: 5 periodSeconds: 10 5. If using an external vector DB (e.g., Qdrant), keep it as a separate StatefulSet and reference it via a stable service name. Verification – Confirming the fix works Deploy the updated manifest and wait for the pod to become ready. Inspect the mounted directory: kubectl exec -it rag-pod-0 -- ls -l /data/index Expected output includes faiss.index and any metadata files.

In this article

Table of Contents

Problem – RAG pipeline returns empty retrieval after Docker container restart in Kubernetes

Symptom: After a pod restart (e.g., rolling update, node‑drain, or container crash) the Retrieval‑Augmented Generation (RAG) service answers every query with No documents retrieved or an empty list, even though the knowledge base was populated before the restart.

Typical log excerpt:


2024-09-15 10:12:03,421 INFO  RetrievalQA: Query received: "What is the refund policy?"
2024-09-15 10:12:03,425 WARN  VectorStore: Query returned 0 results (distance threshold too high)
2024-09-15 10:12:03,426 INFO  RetrievalQA: No documents retrieved

Additional error messages observed in the same environment:

“FAISS index not found at /data/index/faiss.index” (Haystack)
“FileNotFoundError: [Errno 2] No such file or directory: ‘/persisted_index.pkl’” (LlamaIndex)
“Connection refused to Qdrant at http://localhost:6333” (vector DB service down)

Root Cause – Why the vector store disappears after a restart

The RAG pipeline relies on a persisted vector store (FAISS, Qdrant, Chroma, etc.) that holds the embeddings of the knowledge base. In Kubernetes the persistence strategy is defined by the volume configuration of the pod. The most common mis‑configurations that lead to an empty retrieval are:

Mis‑configuration	Effect after restart
Using `emptyDir` or a hostPath that is not backed by a PVC	Data lives only in the node’s RAM/ephemeral storage; when the pod is rescheduled the directory is recreated empty.
Mounting a PVC with `ReadWriteOnce` to a Deployment and then performing a rolling update	The new pod cannot bind the same volume; Kubernetes falls back to an empty temporary directory, wiping the index.
Container entrypoint deletes the index directory on start‑up (e.g., `rm -rf /data/index/*`)	Even if a PVC is present, the persisted files are removed before the vector store is loaded.
Vector store path not mounted (incorrect `mountPath` or missing `volumeMounts`)	LangChain/LlamaIndex falls back to creating a new in‑memory index, which is empty.

LangChain’s documentation on VectorStore Retrieval notes that the index is cached in memory and only persisted to disk when persist() is called. If the underlying directory disappears, the next start‑up will rebuild an empty index.

Similarly, LlamaIndex’s Persisting Indexes guide stresses that persist_dir must point to a volume that survives pod recreation.

Debug – Investigation steps

Confirm the volume type used by the pod.


kubectl get pod rag-pod-abcde -o jsonpath='{.spec.volumes}'

Typical output showing the problem:


[
  {
    "name": "vector-index",
    "emptyDir": {}
  }
]

Inspect the mount inside the container.


kubectl exec -it rag-pod-abcde -- sh -c 'ls -l /data/index'

Expected when the index is persisted:


total 8
-rw-r--r-- 1 root root 1234567 faiss.index
-rw-r--r-- 1 root root   56789 index_meta.json

Observed on a failing pod (empty directory):


total 0

Check container start‑up scripts. Look for commands that recreate or clean the directory.


# entrypoint.sh
mkdir -p /data/index
rm -rf /data/index/*   # <-- problematic line
python app.py

Verify that the vector store service (if external) is reachable.


curl -s http://qdrant-service:6333/collections

If the service is down, the retrieval layer will fallback to an empty local index and log “Connection refused to Qdrant …”.

Review recent rollout events.


kubectl rollout status deployment/rag-deployment
kubectl describe pod rag-pod-abcde | grep -i event

Look for events such as “FailedMount” or “Successfully attached volume”.

Solution – Restoring persistent retrieval

1. Switch to a proper PersistentVolumeClaim

Define a PVC that matches the storage class of your cluster and mount it as /data/index:


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rag-index-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard

Update the pod spec (preferably a StatefulSet to guarantee stable volume binding):


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rag-statefulset
spec:
  serviceName: rag-service
  replicas: 1
  selector:
    matchLabels:
      app: rag
  template:
    metadata:
      labels:
        app: rag
    spec:
      containers:
        - name: rag-container
          image: myorg/rag:latest
          volumeMounts:
            - name: vector-index
              mountPath: /data/index
      volumes:
        - name: vector-index
          persistentVolumeClaim:
            claimName: rag-index-pvc

2. Remove destructive clean‑up commands

Before:


# entrypoint.sh
mkdir -p /data/index
rm -rf /data/index/*   # deletes persisted embeddings
python app.py

After:


# entrypoint.sh
mkdir -p /data/index   # ensure directory exists, do NOT purge contents
python app.py

3. Ensure the vector store is persisted on shutdown

For LangChain (FAISS example):


from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
vector_store = FAISS.from_documents(docs, embeddings)

# Persist after initial indexing
vector_store.save_local("/data/index/faiss")

On container start‑up, load the persisted index:


vector_store = FAISS.load_local("/data/index/faiss", embeddings)

4. Add a readiness probe that checks the index file exists


readinessProbe:
  exec:
    command:
      - sh
      - -c
      - test -f /data/index/faiss.index
  initialDelaySeconds: 5
  periodSeconds: 10

5. If using an external vector DB (e.g., Qdrant), keep it as a separate StatefulSet and reference it via a stable service name.

Verification – Confirming the fix works

Deploy the updated manifest and wait for the pod to become ready.
Inspect the mounted directory:


kubectl exec -it rag-pod-0 -- ls -l /data/index

Expected output includes faiss.index and any metadata files.

Run a test query against the RetrievalQA endpoint.


curl -s -X POST http://rag-service:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question":"What is the refund policy?"}'

Expected JSON response contains at least one source_documents entry.

Check the logs for a successful retrieval message.


2024-09-15 10:45:12,011 INFO  RetrievalQA: Retrieved 3 documents (average distance 0.42)

Prevention – Operational guardrails

Use StatefulSets for any pod that owns a vector index. This guarantees the same PVC is re‑attached after rescheduling.
Never mount a vector store on emptyDir or a hostPath without a PVC. Those volumes are lost on node failure.
Make index persistence explicit. Call save()/persist() after every batch update and verify the file exists in a post‑indexing health check.
Add alerts on index file disappearance. A Prometheus rule that watches for fs.file_exists{path="/data/index/faiss.index"} == 0 can catch the problem before it impacts users.
Version‑pin the storage class and ensure sufficient IOPS. Low‑performance storage can cause timeouts that lead the application to fall back to an empty in‑memory index.

FAQ – Common follow‑up questions

Why does the retrieval work locally but fail after a Kubernetes rollout? Locally you are likely using a bind mount that points to a persistent directory. In Kubernetes the pod may be using an emptyDir or a PVC with the wrong access mode, causing the index to disappear when the pod is recreated.
Can I keep using a Deployment instead of a StatefulSet? Yes, if you attach a PVC with ReadWriteMany (e.g., NFS) that can be mounted by multiple pods. However, most cloud providers only offer ReadWriteOnce, making StatefulSet the safer choice.
How do I know whether my vector store is being loaded from disk or rebuilt in memory? Enable debug logging for the vector store library. LangChain logs “Loading FAISS index from /path” when load_local is called; absence of this line indicates an in‑memory rebuild.
What should I do if the external vector DB (Qdrant, Milvus) is unreachable after a restart? Verify the service’s ClusterIP or headless service name has not changed, and ensure the pod’s DNS resolves correctly. Also check that the DB pod itself is healthy and its PVC is bound.
Is there a way to automatically rebuild the index if the persisted files are missing? Implement a start‑up fallback that checks for the index file; if absent, trigger a re‑indexing job and block the Retrieval endpoint until the job completes.

Related Topic Hub: Distributed Systems Troubleshooting Hub