TGI RAG Context Injection Failure After Disaster Recovery Failover
Problem
During an automated disaster‑recovery (DR) failover to the secondary TGI cluster, the Retrieval‑Augmented Generation (RAG) pipeline stops appending the retrieved documents to the generation prompt. The model receives an empty context field, leading to truncated answers, loss of factual grounding, and a measurable drop in BLEU/ROUGE scores (≈ ‑30 %).
Observed symptoms:
- API response payload contains
"context": ""despite successful vector‑store lookup. - Log entry
INFO rag_injector - Retrieved 3 docs, total size 12 KBfollowed byWARN rag_injector - No context injected into prompt. - Latency drops (from ~2 s to ~0.5 s) because the prompt length is reduced.
- Monitoring alerts fire on
tgi_rag_context_missingmetric.
Root Cause
The DR failover replicates the vector store (e.g., FAISS or Milvus) to the secondary cluster but does not replicate the RAGContextCache service configuration. The primary cluster runs the rag_injector as a sidecar that reads the vector‑store results, serialises them to a protobuf, and injects the string into the prompt via the PromptBuilder API.
In the secondary environment:
- Network bandwidth between the secondary TGI pods and the replicated vector store is limited (<10 Mbps) causing the
grpcstream to exceed the defaultgrpc.max_receive_message_length(4 MB) timeout. - The
rag_injectorsidecar falls back to aNOOPmode when it cannot receive the full payload within5 s. The fallback is triggered by theRAG_INJECTOR_TIMEOUT_MSenvironment variable, which defaults to5000. - Because the sidecar silently disables injection, the downstream
PromptBuilderreceives an empty context string.
The root cause is therefore a combination of:
- Insufficient bandwidth causing gRPC message truncation.
- Missing explicit timeout handling and health‑check configuration for the sidecar in the DR manifest.
- Lack of a fallback alert when the sidecar disables injection.
Debug
Step‑by‑step investigation that reproduced the failure.
1. Verify vector‑store connectivity
kubectl exec -it tgi-dr-pod-0 -- curl -s http://vector-store:8080/healthz
{"status":"ok","replicas":1}
2. Inspect rag_injector logs
2024-11-15T08:12:34.112Z INFO rag_injector - Received request id=7f9c3a2b
2024-11-15T08:12:34.115Z INFO rag_injector - Retrieved 3 docs, total size 12KB
2024-11-15T08:12:39.120Z WARN rag_injector - Injection timeout (5000ms) exceeded, disabling injection for request id=7f9c3a2b
2024-11-15T08:12:39.121Z INFO rag_injector - No context injected into prompt
3. Capture the gRPC traffic
kubectl exec -it tgi-dr-pod-0 -- tcpdump -i any -s 0 -w /tmp/rag.grpc.pcap port 50051
# After reproducing the request, download the pcap and inspect with wireshark:
tshark -r rag.grpc.pcap -Y "grpc" -T fields -e grpc.message_length
Result shows a message length of 6 MB, exceeding the default 4 MB limit.
4. Compare primary vs secondary sidecar configuration
| Parameter | Primary Cluster | Secondary (DR) Cluster |
|---|---|---|
RAG_INJECTOR_TIMEOUT_MS |
10000 | 5000 (default) |
GRPC_MAX_MESSAGE_LENGTH |
8 MB | 4 MB (default) |
NETWORK_BANDWIDTH_LIMIT |
unlimited | 10 Mbps (network policy) |
Solution
Apply three coordinated changes: increase gRPC limits, adjust timeout, and add a health‑check guard that aborts the request if injection is disabled.
1. Update sidecar deployment manifest
Before (secondary manifest snippet):
apiVersion: apps/v1
kind: Deployment
metadata:
name: tgi-dr
spec:
template:
spec:
containers:
- name: rag-injector
image: myrepo/rag-injector:1.2.0
env:
- name: RAG_INJECTOR_TIMEOUT_MS
value: "5000"
- name: GRPC_MAX_MESSAGE_LENGTH
value: "4194304"
After (patched manifest):
apiVersion: apps/v1
kind: Deployment
metadata:
name: tgi-dr
spec:
template:
spec:
containers:
- name: rag-injector
image: myrepo/rag-injector:1.2.0
env:
- name: RAG_INJECTOR_TIMEOUT_MS
value: "15000" # give extra time for low‑bandwidth links
- name: GRPC_MAX_MESSAGE_LENGTH
value: "16777216" # 16 MB to accommodate larger payloads
- name: INJECTION_FAILURE_MODE
value: "FAIL" # abort request instead of silent NOOP
resources:
limits:
cpu: "500m"
memory: "256Mi"
2. Adjust network policy to allow higher throughput for the DR cluster (optional but recommended)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dr-high-bandwidth
spec:
podSelector:
matchLabels:
app: vector-store
ingress:
- from:
- podSelector:
matchLabels:
app: rag-injector
ports:
- protocol: TCP
port: 8080
policyTypes:
- Ingress
# Remove the 10 Mbps egress limit in the DR namespace
3. Add a startup probe that validates the sidecar can receive a test payload
livenessProbe:
exec:
command: ["curl", "-s", "http://localhost:8080/healthz"]
initialDelaySeconds: 5
periodSeconds: 10
Verify
After applying the manifest, run the same request and confirm injection.
Log verification
2024-11-15T09:03:12.045Z INFO rag_injector - Received request id=9a2d5f1c
2024-11-15T09:03:12.050Z INFO rag_injector - Retrieved 3 docs, total size 12KB
2024-11-15T09:03:12.053Z INFO rag_injector - Injected 12KB context into prompt for request id=9a2d5f1c
Prompt payload
{
"prompt": "User: Explain the difference between ACID and BASE.\nContext: ...[retrieved docs]...",
"max_new_tokens": 256
}
Metric validation
tgi_rag_context_missingdrops to0across all pods.- BLEU/ROUGE scores return to baseline (within 5 % of pre‑failover values).
Prevent
- Include the sidecar timeout and gRPC limits in the cluster‑wide
ConfigMapso that DR manifests inherit the same values. - Enable a
FAILmode for injection failures; this forces the request to be rejected rather than silently producing context‑less output. - Set up a
Prometheusalert ontgi_rag_context_missingwith afor: 2mduration to catch regressions early. - Run a periodic synthetic RAG query (e.g., via a CronJob) that validates end‑to‑end context injection in both primary and DR clusters.
- Document the required minimum network bandwidth (≥ 50 Mbps) for the vector‑store ↔︎ injector link; if bandwidth falls below this, trigger a failover to a higher‑capacity network path.
Related Topic Hub: Model Serving Troubleshooting Hub
FAQ
- Why does the injection work in the primary cluster but not after failover? The primary manifest sets a higher
GRPC_MAX_MESSAGE_LENGTHand a longer timeout. The DR manifest inherits defaults, which are too low for the larger payloads combined with the throttled network link. - Can I keep the sidecar in NOOP mode and still get correct answers? No. NOOP discards the retrieved context, so the model generates without grounding. The only safe fallback is to abort the request or return an explicit error to the caller.
- Is increasing the gRPC message size enough? It resolves the truncation but does not address the underlying bandwidth limitation. Both the message size and timeout must be increased, and the network policy should allow sufficient throughput.
- How do I test that the injection works before a real disaster? Deploy a
RAGHealthCheckjob that performs a vector‑store query, captures the injected prompt, and asserts that thecontextfield length > 0. Integrate this job into the CI/CD pipeline for DR manifests. - What should I monitor to detect similar failures in the future? Track
tgi_rag_context_missing,grpc_message_too_large_total, and the sidecar’sinjection_timeout_totalcounters. Alert when any of these exceed a threshold for more than one minute.