TGI RAG context injection failure after disaster recovery failover

TGI RAG Context Injection Failure After Disaster Recovery Failover

Problem

During an automated disaster‑recovery (DR) failover to the secondary TGI cluster, the Retrieval‑Augmented Generation (RAG) pipeline stops appending the retrieved documents to the generation prompt. The model receives an empty context field, leading to truncated answers, loss of factual grounding, and a measurable drop in BLEU/ROUGE scores (≈ ‑30 %).

Observed symptoms:

  • API response payload contains "context": "" despite successful vector‑store lookup.
  • Log entry INFO rag_injector - Retrieved 3 docs, total size 12 KB followed by WARN rag_injector - No context injected into prompt.
  • Latency drops (from ~2 s to ~0.5 s) because the prompt length is reduced.
  • Monitoring alerts fire on tgi_rag_context_missing metric.

Root Cause

The DR failover replicates the vector store (e.g., FAISS or Milvus) to the secondary cluster but does not replicate the RAGContextCache service configuration. The primary cluster runs the rag_injector as a sidecar that reads the vector‑store results, serialises them to a protobuf, and injects the string into the prompt via the PromptBuilder API.

In the secondary environment:

  1. Network bandwidth between the secondary TGI pods and the replicated vector store is limited (<10 Mbps) causing the grpc stream to exceed the default grpc.max_receive_message_length (4 MB) timeout.
  2. The rag_injector sidecar falls back to a NOOP mode when it cannot receive the full payload within 5 s. The fallback is triggered by the RAG_INJECTOR_TIMEOUT_MS environment variable, which defaults to 5000.
  3. Because the sidecar silently disables injection, the downstream PromptBuilder receives an empty context string.

The root cause is therefore a combination of:

  • Insufficient bandwidth causing gRPC message truncation.
  • Missing explicit timeout handling and health‑check configuration for the sidecar in the DR manifest.
  • Lack of a fallback alert when the sidecar disables injection.

Debug

Step‑by‑step investigation that reproduced the failure.

1. Verify vector‑store connectivity

kubectl exec -it tgi-dr-pod-0 -- curl -s http://vector-store:8080/healthz
{"status":"ok","replicas":1}

2. Inspect rag_injector logs

2024-11-15T08:12:34.112Z INFO  rag_injector - Received request id=7f9c3a2b
2024-11-15T08:12:34.115Z INFO  rag_injector - Retrieved 3 docs, total size 12KB
2024-11-15T08:12:39.120Z WARN  rag_injector - Injection timeout (5000ms) exceeded, disabling injection for request id=7f9c3a2b
2024-11-15T08:12:39.121Z INFO  rag_injector - No context injected into prompt

3. Capture the gRPC traffic

kubectl exec -it tgi-dr-pod-0 -- tcpdump -i any -s 0 -w /tmp/rag.grpc.pcap port 50051
# After reproducing the request, download the pcap and inspect with wireshark:
tshark -r rag.grpc.pcap -Y "grpc" -T fields -e grpc.message_length

Result shows a message length of 6 MB, exceeding the default 4 MB limit.

4. Compare primary vs secondary sidecar configuration

Parameter Primary Cluster Secondary (DR) Cluster
RAG_INJECTOR_TIMEOUT_MS 10000 5000 (default)
GRPC_MAX_MESSAGE_LENGTH 8 MB 4 MB (default)
NETWORK_BANDWIDTH_LIMIT unlimited 10 Mbps (network policy)

Solution

Apply three coordinated changes: increase gRPC limits, adjust timeout, and add a health‑check guard that aborts the request if injection is disabled.

1. Update sidecar deployment manifest

Before (secondary manifest snippet):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi-dr
spec:
  template:
    spec:
      containers:
      - name: rag-injector
        image: myrepo/rag-injector:1.2.0
        env:
        - name: RAG_INJECTOR_TIMEOUT_MS
          value: "5000"
        - name: GRPC_MAX_MESSAGE_LENGTH
          value: "4194304"

After (patched manifest):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi-dr
spec:
  template:
    spec:
      containers:
      - name: rag-injector
        image: myrepo/rag-injector:1.2.0
        env:
        - name: RAG_INJECTOR_TIMEOUT_MS
          value: "15000"               # give extra time for low‑bandwidth links
        - name: GRPC_MAX_MESSAGE_LENGTH
          value: "16777216"            # 16 MB to accommodate larger payloads
        - name: INJECTION_FAILURE_MODE
          value: "FAIL"                # abort request instead of silent NOOP
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"

2. Adjust network policy to allow higher throughput for the DR cluster (optional but recommended)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dr-high-bandwidth
spec:
  podSelector:
    matchLabels:
      app: vector-store
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: rag-injector
    ports:
    - protocol: TCP
      port: 8080
  policyTypes:
  - Ingress
  # Remove the 10 Mbps egress limit in the DR namespace

3. Add a startup probe that validates the sidecar can receive a test payload

livenessProbe:
  exec:
    command: ["curl", "-s", "http://localhost:8080/healthz"]
  initialDelaySeconds: 5
  periodSeconds: 10

Verify

After applying the manifest, run the same request and confirm injection.

Log verification

2024-11-15T09:03:12.045Z INFO  rag_injector - Received request id=9a2d5f1c
2024-11-15T09:03:12.050Z INFO  rag_injector - Retrieved 3 docs, total size 12KB
2024-11-15T09:03:12.053Z INFO  rag_injector - Injected 12KB context into prompt for request id=9a2d5f1c

Prompt payload

{
  "prompt": "User: Explain the difference between ACID and BASE.\nContext: ...[retrieved docs]...",
  "max_new_tokens": 256
}

Metric validation

  • tgi_rag_context_missing drops to 0 across all pods.
  • BLEU/ROUGE scores return to baseline (within 5 % of pre‑failover values).

Prevent

  • Include the sidecar timeout and gRPC limits in the cluster‑wide ConfigMap so that DR manifests inherit the same values.
  • Enable a FAIL mode for injection failures; this forces the request to be rejected rather than silently producing context‑less output.
  • Set up a Prometheus alert on tgi_rag_context_missing with a for: 2m duration to catch regressions early.
  • Run a periodic synthetic RAG query (e.g., via a CronJob) that validates end‑to‑end context injection in both primary and DR clusters.
  • Document the required minimum network bandwidth (≥ 50 Mbps) for the vector‑store ↔︎ injector link; if bandwidth falls below this, trigger a failover to a higher‑capacity network path.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the injection work in the primary cluster but not after failover? The primary manifest sets a higher GRPC_MAX_MESSAGE_LENGTH and a longer timeout. The DR manifest inherits defaults, which are too low for the larger payloads combined with the throttled network link.
  2. Can I keep the sidecar in NOOP mode and still get correct answers? No. NOOP discards the retrieved context, so the model generates without grounding. The only safe fallback is to abort the request or return an explicit error to the caller.
  3. Is increasing the gRPC message size enough? It resolves the truncation but does not address the underlying bandwidth limitation. Both the message size and timeout must be increased, and the network policy should allow sufficient throughput.
  4. How do I test that the injection works before a real disaster? Deploy a RAGHealthCheck job that performs a vector‑store query, captures the injected prompt, and asserts that the context field length > 0. Integrate this job into the CI/CD pipeline for DR manifests.
  5. What should I monitor to detect similar failures in the future? Track tgi_rag_context_missing, grpc_message_too_large_total, and the sidecar’s injection_timeout_total counters. Alert when any of these exceed a threshold for more than one minute.