RAG Answer Extraction Empty After Model Evaluation In Kubernetes

In this article

Table of Contents

Problem Description

During automated model evaluation runs in a CI/CD pipeline on a Kubernetes cluster, the Retrieval‑Augmented Generation (RAG) microservice returns an answer field that is either empty or malformed. The API response typically looks like:

{
  "question": "What is the capital of France?",
  "answer": ""
}

Typical log excerpts from the RAG pod include:

2024-09-13T10:12:45.123Z ERROR  - Error: empty response from LLM
2024-09-13T10:12:45.124Z INFO   - Retrieved 3 documents, total tokens: 1024
2024-09-13T10:12:45.125Z WARN   - ReadTimeoutError: LLM request timed out after 30s
2024-09-13T10:12:45.126Z ERROR  - Failed to parse answer JSON: Unexpected token < in JSON at position 0

These symptoms manifest as failed evaluation metrics, flaky test results, and downstream alerts indicating missing answer payloads.

Root Cause Analysis

The failure is rarely a single misconfiguration; it is usually a cascade of resource, networking, and lifecycle issues that prevent the generation step from completing. The most common root causes, corroborated by the evidence package, are:

Pod restart or OOMKilled before generation: When the container exceeds its memory limit during batch embedding of retrieved documents, the kernel sends SIGKILL, and the process terminates before the LLM call returns. Kubernetes then restarts the pod, but the evaluation job proceeds with the stale request, yielding an empty answer. (Kubernetes Docs – Resource Requests & Limits)
Missing vector store or model checkpoint mount: ConfigMaps/Secrets that provide the vector DB connection string or the model checkpoint may fail to mount (e.g., due to a typo in the volume name). Retrieval succeeds because the service falls back to an in‑memory stub, but the generation step cannot locate the model weights, resulting in a silent failure. (Kubernetes Docs – ConfigMaps and Secrets)
Readiness probe misconfiguration: If the readiness probe returns HTTP 502 while the LLM endpoint is still warming up, traffic is routed to a pod that cannot forward the generation request. The client receives a 200 OK from the RAG service (because the retrieval path succeeded) but an empty answer. (Kubernetes Docs – Liveness and Readiness Probes)
Network policy or side‑car timeout: In clusters where egress to the external LLM API is restricted, the request times out (30 s default in many SDKs). The service logs a ReadTimeoutError and returns an empty answer. (Stack Overflow – Empty response from LLM in RAG when running inside k8s)

Investigation and Debugging Steps

Collect pod status and recent events
```
kubectl get pod -n rag-eval -l app=rag-service -o wide
kubectl describe pod <pod-name> -n rag-eval
```
Look for OOMKilled, CrashLoopBackOff, or readiness probe failures.
Inspect container logs around the failure timestamp
```
kubectl logs <pod-name> -n rag-eval --since=5m
```
Confirm presence of messages such as "Error: empty response from LLM" or "ReadTimeoutError".

Verify ConfigMap/Secret mounts

kubectl exec <pod-name> -n rag-eval -- ls /app/config
kubectl exec <pod-name> -n rag-eval -- cat /app/config/vector-db.yaml
kubectl exec <pod-name> -n rag-eval -- cat /app/config/model-checkpoint.path

If files are missing or empty, the mount definition is incorrect.

Check resource usage during evaluation
```
kubectl top pod <pod-name> -n rag-eval
kubectl logs <pod-name> -n rag-eval | grep -i "memory"
```
Spikes above the declared limits.memory indicate OOM risk.
Test LLM endpoint connectivity from inside the pod
```
kubectl exec -it <pod-name> -n rag-eval -- curl -s -o /dev/null -w "%{http_code}" $LLM_ENDPOINT
```
A non‑200 status or a timeout confirms network policy blockage.

Validate readiness probe behavior

kubectl get pod <pod-name> -n rag-eval -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
kubectl logs <pod-name> -n rag-eval --tail=20 | grep "readiness"

Ensure the probe only succeeds after the LLM client has successfully established a connection.

Resolution

1. Adjust Resource Requests & Limits

Increase memory limits to accommodate peak embedding load and add a requests.memory value to give the scheduler headroom.

# Before
resources:
  limits:
    cpu: "2"
    memory: "2Gi"
  requests:
    cpu: "1"
    memory: "1Gi"

# After
resources:
  limits:
    cpu: "4"
    memory: "6Gi"
  requests:
    cpu: "2"
    memory: "4Gi"

Adding a memorySwap limit is optional but can prevent OOMKilled when the host permits swap.

2. Correct ConfigMap/Secret Mounts

Ensure the volume name matches the mountPath and that the ConfigMap keys are correctly referenced.

# Before (missing key)
volumeMounts:
  - name: rag-config
    mountPath: /app/config
volumes:
  - name: rag-config
    configMap:
      name: rag-configmap

# After (explicit key mapping)
volumeMounts:
  - name: rag-config
    mountPath: /app/config
    readOnly: true
volumes:
  - name: rag-config
    configMap:
      name: rag-configmap
      items:
        - key: vector-db.yaml
          path: vector-db.yaml
        - key: model-checkpoint.path
          path: model-checkpoint.path

3. Refine Liveness/Readiness Probes

Delay readiness until the LLM client reports a successful health check.

# Before
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

# After (probe checks both retrieval and generation health)
readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - |
        curl -s http://localhost:8080/healthz | grep '"generation":true' && exit 0 || exit 1
  initialDelaySeconds: 15
  periodSeconds: 10

4. Open Egress to the LLM API or Deploy a Proxy Side‑car

If a NetworkPolicy blocks outbound traffic, add an allow rule or run a side‑car that tunnels the request.

# Example NetworkPolicy allowing egress to the LLM endpoint CIDR
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-llm-egress
  namespace: rag-eval
spec:
  podSelector:
    matchLabels:
      app: rag-service
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 34.120.0.0/16   # LLM provider CIDR
      ports:
        - protocol: TCP
          port: 443

5. Add Init Container to Wait for Vector DB Readiness

Deploy an init container that polls the vector store until it returns a healthy status.

# Init container snippet
initContainers:
  - name: wait-for-vector-db
    image: curlimages/curl:7.85.0
    command: ["sh", "-c"]
    args:
      - |
        until curl -s http://vector-db:8080/ready | grep "true"; do
          echo "Waiting for vector DB..."
          sleep 5
        done

Verification

Redeploy the corrected Helm chart or manifest and wait for the pod to become Ready.
Run a single evaluation request manually:

curl -X POST http://rag-service.rag-eval.svc.cluster.local/v1/query \
  -H "Content-Type: application/json" \
  -d '{"question":"What is the capital of France?"}'

Confirm the response contains a non‑empty answer field.
Check pod logs for the absence of the previous error messages.
Inspect the CI/CD job summary; evaluation metrics should now reflect successful answer generation.

Prevention and Best Practices

Resource safety margins: Allocate at least 150 % of the observed peak memory during batch embedding.
Config validation CI step: Use kubectl apply --dry-run=client and a custom script that verifies required ConfigMap keys are present.
Readiness probe that checks downstream dependencies: Include both vector store and LLM health checks.
Observability: Export metrics for retrieval count, generation latency, and LLM error codes; set alerts on generation_error_total > 0.
Network policy review: Periodically audit egress rules against external LLM provider IP ranges.
Helm chart version pinning: Lock the vector store and LLM client images to known good versions to avoid accidental breaking changes during upgrades.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does the RAG service return an empty answer only after a Helm upgrade?
The upgrade replaced the ConfigMap volume name, causing the model checkpoint to become unavailable. Retrieval succeeded because the vector store mount was unchanged, but generation failed silently.
How can I tell whether the empty answer is due to an OOMKilled pod or a network timeout?
Check kubectl describe pod for State: Terminated (Reason: OOMKilled). If the pod is Running, look for ReadTimeoutError in the logs and verify egress connectivity with curl $LLM_ENDPOINT.
Is increasing the readiness probe delay enough to fix the issue?
Only if the root cause is premature traffic routing. If the pod is OOMKilled or the ConfigMap is missing, the probe will never succeed regardless of delay.
Can side‑car containers help avoid empty answers?
Yes. A side‑car that proxies LLM requests can implement retries and expose its own health endpoint, allowing the main container’s readiness probe to depend on successful proxy initialization.
What monitoring alerts should I create to catch this early?
Alert on any of the following:
- Pod restarts > 2 in 5 minutes.
- Container OOMKilled events.
- Metric generation_error_total > 0.
- Readiness probe failures lasting > 30 seconds.