RAG prompt template rendering error during performance benchmark on Kubernetes

Problem

The retrieval‑augmented generation (RAG) service deployed on a Kubernetes cluster fails during the performance benchmark phase with a prompt template rendering error. The error appears only under load and aborts the benchmark run, causing the following observable symptoms:

• Benchmark client receives HTTP 500 responses with body {"error":"PromptTemplateError: variable 'context' is undefined"}.
• Pod logs contain stack traces from the Jinja2 renderer.
• CPU usage spikes to 90 % on the rag-worker pods, while memory remains stable.
• Other endpoints (e.g., health check, simple echo) continue to respond normally.

Root Cause

Under load the RAG service processes multiple concurrent requests that share a singleton PromptTemplateRenderer instance. The renderer caches the compiled Jinja2 template and re‑uses the same Context dictionary for each request. When the benchmark driver fires hundreds of parallel queries, the shared context is mutated by one request before another request finishes rendering, leading to missing keys such as context or retrieved_docs. The resulting jinja2.exceptions.UndefinedError propagates as a 500 error.

Two intertwined factors make the problem surface only during benchmarking:

1. Singleton pattern in the container image. The Python module creates the renderer at import time, so all worker threads share mutable state.
2. Kubernetes pod concurrency settings. The deployment uses replicas: 3 with resources.limits.cpu: "2" and requests.cpu: "1". The horizontal pod autoscaler (HPA) does not scale out because the benchmark runs for a short burst, so the three pods become the bottleneck.

Debug

Step‑by‑step investigation that isolated the issue:

1. Collect pod logs during benchmark.

kubectl logs -l app=rag-service -c rag-worker --since=5m > benchmark.log

Sample excerpt from benchmark.log:

2026-10-19T12:34:56.123Z ERROR rag_worker.renderer - PromptTemplateError: variable 'context' is undefined
Traceback (most recent call last):
  File "/app/renderer.py", line 87, in render
    rendered = self.template.render(context)
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1085, in render
    return self.environment.handle_exception(exc_info, True)
jinja2.exceptions.UndefinedError: 'context' is undefined

2. Inspect the renderer singleton. Attach to a running pod and dump the object’s state.

kubectl exec -it $(kubectl get pod -l app=rag-service -o name | head -n1) -- bash
python - <<'PY'
from rag.renderer import PromptTemplateRenderer
print(PromptTemplateRenderer._instance)  # shows shared instance
PY

Output shows a single instance with a mutable self.last_context attribute.

3. Reproduce locally with threading. Running the same template rendering in a multi‑threaded script reproduces the UndefinedError after ~150 concurrent threads.

python -m rag.tests.concurrent_render_test

4. Check Kubernetes resource limits. Verify that the pod CPU limit is not being throttled.

kubectl top pod -l app=rag-service
NAME                CPU(cores)   MEMORY(bytes)
rag-service-abc123  1900m        512Mi
rag-service-def456  1850m        508Mi
rag-service-ghi789  1920m        511Mi

CPU usage is near the limit, confirming contention.

Solution

The fix consists of two parts: make the renderer stateless per request and adjust the deployment to allow scaling under burst load.

Code changes

Before (singleton renderer with mutable state):

# renderer.py
from jinja2 import Environment, FileSystemLoader

class PromptTemplateRenderer:
    _instance = None

    def __new__(cls, *args, **kwargs):
        if not cls._instance:
            cls._instance = super().__new__(cls)
            env = Environment(loader=FileSystemLoader("/templates"))
            cls._instance.template = env.get_template("rag_prompt.jinja2")
            cls._instance.last_context = {}
        return cls._instance

    def render(self, context: dict) -> str:
        # Mutates shared state
        self.last_context = context
        return self.template.render(context)

After (stateless renderer, instantiated per request):

# renderer.py
from jinja2 import Environment, FileSystemLoader, StrictUndefined

class PromptTemplateRenderer:
    def __init__(self, template_name: str = "rag_prompt.jinja2"):
        env = Environment(
            loader=FileSystemLoader("/templates"),
            undefined=StrictUndefined,   # fail fast on missing keys
            autoescape=False,
        )
        self.template = env.get_template(template_name)

    def render(self, context: dict) -> str:
        # No shared mutable state
        return self.template.render(context)

Update the request handler to create a fresh renderer instance:

# handler.py
from .renderer import PromptTemplateRenderer

def handle_rag_request(request):
    payload = request.json()
    renderer = PromptTemplateRenderer()
    try:
        prompt = renderer.render({
            "question": payload["question"],
            "retrieved_docs": payload["docs"],
            "context": payload.get("context", ""),
        })
    except Exception as e:
        logger.error("PromptTemplateError: %s", e)
        raise
    # continue with LLM call...

Deployment changes

Increase burst capacity by allowing the HPA to scale out quickly and by adding a podDisruptionBudget to keep at least one replica healthy.

# rag-deployment.yaml (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rag-service
  template:
    metadata:
      labels:
        app: rag-service
    spec:
      containers:
      - name: rag-worker
        image: myregistry.com/rag-service:1.4.0
        resources:
          requests:
            cpu: "500m"
            memory: "256Mi"
          limits:
            cpu: "2"
            memory: "1Gi"
        env:
        - name: MAX_CONCURRENCY
          value: "50"   # limit per pod
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: rag-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rag-service
  minReplicas: 3
  maxReplicas: 12
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Pods
        value: 4
        periodSeconds: 60

Verify

After applying the changes, run the benchmark again and confirm that the error disappears.

1. Redeploy the updated image and HPA configuration:

kubectl apply -f rag-deployment.yaml
kubectl rollout status deployment/rag-service

2. Execute the benchmark script (e.g., bench/run.sh) and monitor logs:

kubectl logs -l app=rag-service -c rag-worker -f | grep PromptTemplateError

No output should appear.

3. Validate that the HPA scaled out during the burst:

kubectl get hpa rag-service-hpa -w
# Expected: replicas increase from 3 to 8‑10 during the test, then scale back.

4. Run a functional check on a single request to ensure the template renders correctly:

curl -X POST http://rag-service.default.svc.cluster.local/v1/generate \
  -H "Content-Type: application/json" \
  -d '{"question":"What is RAG?","docs":["doc1","doc2"]}' | jq .

Response should contain a generated_text field without error.

Prevent

Implement the following guardrails to avoid regression:

• Stateless components. Ensure any object that holds request‑specific data is instantiated per request or uses thread‑local storage.
• Strict template rendering. Use StrictUndefined in Jinja2 to catch missing variables early.
• Load testing in CI. Include a benchmark stage that simulates at least 200 concurrent requests and asserts zero 5xx responses.
• Resource alerts. Configure Prometheus alerts on container_cpu_usage_seconds_total and on the custom metric rag_prompt_render_errors_total (incremented in the exception handler).
• HPA tuning. Set maxReplicas and scaleUp policies to handle short‑lived spikes.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does the error appear only under load and not during normal traffic?
   Because the singleton renderer shares mutable state across concurrent requests. Under low traffic the race condition rarely manifests, but high concurrency creates overlapping renders that overwrite the shared context.
2. Can I keep a singleton renderer for performance reasons?
   Only if the renderer is truly immutable. Compile the template once (which is safe) but do not store request‑specific data on the instance. Pass a fresh context dictionary to render() each time.
3. Do I need to increase the pod CPU limit after fixing the code?
   Not necessarily. The stateless renderer reduces CPU spikes, but you may still want a modest headroom (e.g., 2 CPU) to accommodate bursty LLM inference workloads.
4. How can I detect similar race conditions in other services?
   Enable StrictUndefined or equivalent strict mode in templating engines, and add unit tests that run the rendering function in parallel threads, asserting no UndefinedError occurs.
5. Is there a way to make the HPA react faster to short bursts?
   Configure behavior.scaleUp with a low stabilizationWindowSeconds and a higher value per periodSeconds. Also consider using external metrics that reflect request latency or queue length for more immediate scaling signals.