RAG vector store connection timeout during rolling update of Meta LLaMA

Problem Description

During a rolling update of a production Meta LLaMA RAG service, the retrieval‑augmented generation pipeline intermittently fails to retrieve embeddings. The failure manifests as:

• grpc deadline exceeded: context deadline exceeded while attempting to query vector store
• VectorStoreError: connection refused – unable to establish TCP connection to Faiss endpoint
• TimeoutError: failed to retrieve embeddings within 30s – connection pool exhausted
• CircuitBreakerOpenException – vector store circuit breaker opened due to consecutive timeouts during rolling update

Operational impact includes increased latency, failed user queries, and temporary service degradation lasting 2–3 minutes per rollout.

Root Cause Analysis

The issue originates from the interaction between Kubernetes rolling update semantics and the vector store connection handling defined in the Meta LLaMA deployment configuration:

1. Pod termination without graceful socket release – When old model pods are evicted, the preStop hook is missing, so open gRPC connections to the vector store (FAISS or Milvus) are abruptly closed.
2. Connection‑pool saturation – New pods start immediately and begin populating their own connection pools. The vector store sees a sudden surge of concurrent connection attempts while the old sockets are still in CLOSE_WAIT, exhausting the server’s max_connections limit (default 1024 for FAISS, 5000 for Milvus).
3. Readiness probe misconfiguration – The model pod reports Ready before it has successfully established a stable vector‑store client, allowing traffic to be routed to a pod that cannot yet query the store.
4. Timeout settings too low – The default vector_store.timeout_ms of 30000 ms (30 s) is insufficient when the store is under load from the rollout, causing the client to abort before the server can accept new connections.

This chain of events matches the community reports in the GitHub issue “Vector store timeout during rolling update” and the production incident where “grpc deadline exceeded” errors persisted for 2‑3 minutes.

Investigation and Debugging

Below is a reproducible debugging workflow that was used to isolate the problem.

1. Log inspection

2026-08-25T14:02:31.112Z model-pod-5c9f7c9d8f-9kzlm INFO  Starting inference request id=abc123
2026-08-25T14:02:31.115Z model-pod-5c9f7c9d8f-9kzlm ERROR VectorStoreError: connection refused - unable to establish TCP connection to faiss-service:8080
2026-08-25T14:02:31.120Z model-pod-5c9f7c9d8f-9kzlm WARN grpc deadline exceeded: context deadline exceeded while attempting to query vector store
2026-08-25T14:02:31.125Z kubelet INFO  Pod model-pod-5c9f7c9d8f-9kzlm terminating (reason: RollingUpdate)
2026-08-25T14:02:31.130Z faiss-service INFO  New connection from 10.2.3.45:54321 rejected: max connections reached

2. Connection‑pool metrics

Using kubectl exec on the vector‑store pod:

kubectl exec -it faiss-service-0 -- bash
# inside container
netstat -anp | grep :8080 | wc -l
# => 1024  (maxed out)

3. Verify readiness probe timing

kubectl get pod model-pod-5c9f7c9d8f-9kzlm -o yaml | grep readinessProbe -A4
...
readinessProbe:
  httpGet:
    path: /healthz
    port: 8081
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

The probe only checks the HTTP health endpoint, not the vector‑store client status.

4. Capture traffic during rollout

tcpdump -i eth0 port 8080 -w /tmp/faiss-rollout.pcap
# After rollout, analyze with Wireshark:
# Observe many SYN packets followed by RST from the Faiss server.

5. Compare configuration defaults vs. tuned values

Parameter	Default	Tuned
vector_store.timeout_ms	30000	60000
vector_store.max_pool_size	20	50
faiss.max_connections	1024	2048
readinessProbe.initialDelaySeconds	5	30

Resolution

The fix consists of three coordinated changes: graceful pod shutdown, connection‑pool scaling, and readiness‑probe enhancement.

1. Add a preStop hook to drain vector‑store sockets

Before (deployment snippet):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-model
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
      - name: model
        image: ghcr.io/facebookresearch/llama:2.0
        ports:
        - containerPort: 8080

After – add preStop that closes the gRPC client and waits for in‑flight requests:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-model
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
      - name: model
        image: ghcr.io/facebookresearch/llama:2.0
        ports:
        - containerPort: 8080
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh","-c","python /app/drain_vector_store.py && sleep 10"]

The drain_vector_store.py script closes the client pool and optionally sends a health‑check /drain request to the vector store.

2. Increase vector‑store connection limits

For FAISS (via faiss.conf) and Milvus (via milvus.yaml), raise the max connections:

# faiss.conf
max_connections = 2048

# milvus.yaml
etcd:
  max_conn: 5000

3. Extend readiness probe to validate vector‑store connectivity

Replace the simple HTTP health check with a script that attempts a lightweight vector‑store query.

readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      python -c "
import grpc, faiss_pb2_grpc, faiss_pb2
channel = grpc.insecure_channel('faiss-service:8080', options=[('grpc.keepalive_time_ms', 10000)])
try:
    stub = faiss_pb2_grpc.FaissStub(channel)
    stub.Ping(faiss_pb2.PingRequest())
    exit(0)
except Exception:
    exit(1)
      "
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

4. Adjust client‑side timeout and pool size

Update the model’s configuration (see deployment_config.md):

vector_store:
  timeout_ms: 60000          # increased from 30000
  max_pool_size: 50          # increased from 20
  retry:
    max_attempts: 3
    backoff_ms: 2000

Verification

After applying the changes, perform the following checks:

1. Rolling update test – trigger a rollout in a staging namespace and watch logs for the absence of timeout errors.
2. Connection metrics – query the vector‑store’s connection count before, during, and after rollout:

kubectl exec -it faiss-service-0 -- curl http://localhost:8080/metrics | grep grpc_server_connections
# Expected: peak < 1500 (well below 2048 limit)

Readiness probe health – ensure pods only become Ready after the vector‑store ping succeeds:

kubectl get pods -l app=llama-model -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'
# Expected output: "True" for all pods after ~30 s

Functional test – issue a sample RAG request and confirm the response contains retrieved passages.

curl -X POST http://llama-gateway/api/rag -d '{"query":"What is the return policy?"}' -H "Content-Type: application/json"
# Response should include "retrieved_documents": [...]

Operational Experience

During the initial investigation, the most misleading symptom was the rapid appearance of grpc deadline exceeded errors, which suggested a client‑side timeout rather than a server‑side connection‑pool exhaustion. Only after capturing a packet trace did we see a flood of SYN packets being reset by the vector store.

Another surprise was that the default Kubernetes maxUnavailable: 0 strategy, while preserving overall replica count, still allowed a momentary dip to zero available vector‑store connections because the vector store itself became a bottleneck.

Adding the preStop hook not only closed sockets cleanly but also gave the service mesh (e.g., Istio) time to drain in‑flight requests, eliminating the “CircuitBreakerOpenException” observed in the incident logs.

Best Practices and Prevention

• Graceful shutdown: Always define a preStop hook that closes external client pools and waits for pending RPCs.
• Readiness validation: Extend readiness probes to cover downstream dependencies, especially stateful services like vector stores.
• Capacity planning: Size max_connections on FAISS/Milvus to accommodate the peak connection count during rollouts (typically replicas × max_pool_size × surge_factor).
• Timeout tuning: Set vector_store.timeout_ms to at least twice the expected worst‑case latency under load.
• Observability: Export connection‑pool and circuit‑breaker metrics (e.g., grpc_server_connections, circuit_breaker_state) and alert on sudden spikes.
• Canary rollouts: Deploy a single new pod first, verify vector‑store health, then proceed with the full rolling update.

Related Questions

1. Why does the vector‑store timeout only during rolling updates?
   Because the update introduces a burst of simultaneous connection attempts while old pods still hold sockets, exceeding the store’s connection limit and causing timeouts.
2. How can I verify which cipher suite was negotiated for gRPC between the model and vector store?
   Run grpcurl -v -proto your.proto your-vector-store:8080 YourService/Method and inspect the TLS handshake details in the verbose output.
3. Can connection pooling affect gRPC handshake behavior?
   Yes. An exhausted pool forces the client to open new TCP connections, which may be rejected if the server’s max_connections is reached, leading to handshake failures.
4. Why does OpenSSL succeed while the application traffic fails?
   OpenSSL tests a single connection without pooling, whereas the application reuses a pool that can become saturated during rollout.
5. What Kubernetes settings help avoid zero‑available vector‑store connections?
   Configure maxSurge and maxUnavailable such that the total number of model pods never exceeds the vector store’s capacity, and use preStop hooks to drain connections.

Related Topic Hub: LLM Systems Troubleshooting Hub