ONNX Runtime RAG Inference Fails Due To Malformed Prompt Templates

Q: 4. Add explicit validation in the renderer

Modify the custom PromptRenderer to raise a clear error when required variables are missing.

In this article

Table of Contents

Problem – RAG inference fails because the prompt template is malformed

In a distributed micro‑services deployment the Retrieval‑Augmented Generation (RAG) service invokes an ONNX Runtime sidecar via gRPC. The sidecar builds a prompt by substituting variables (e.g., {{question}}, {{context}}) into a JSON‑encoded template before feeding the concatenated string to a transformer model.

Symptoms observed in production:

Model returns an empty string or a RuntimeError: Failed to format prompt – KeyError: 'context'.
Sidecar logs contain ValueError: Prompt variable '{question}' not found in input data or ONNXRuntimeError: Unexpected token '{{' in rendered prompt string.
Kubernetes pod restarts with KeyError: "context" during prompt rendering (see incident “Kubernetes deployment of ONNX Runtime sidecar observed repeated pod restarts”).
Tracing shows latency spikes coinciding with malformed prompts.

Root Cause – Why the placeholders are not replaced

The failure originates from three interacting factors:

Template syntax mismatch: The sidecar expects double‑brace placeholders ({{variable}}) as defined in the ONNX Runtime “Running Transformers with ONNX Runtime” tutorial. A recent schema change introduced single‑brace placeholders ({variable}) in the upstream retrieval service. Because the sidecar’s PromptRenderer uses str.format() on the raw string, the double braces are interpreted as literal characters, producing prompts such as Answer the following: {{question}}.
gRPC payload deserialization loss: In the incident “Microservice A sent a protobuf message with field ‘user_query’ but the sidecar’s deserializer dropped the field”, the protobuf definition was updated without regenerating the Python stubs. The field is silently omitted, so the renderer receives an empty dictionary and cannot substitute {{question}}.
Header compression side effect: HTTP/2 HPACK compression introduced an extra brace ({{{question}}}) on the wire, as reported in the “load‑balancer introduced HTTP/2 header compression” incident. The extra brace breaks the placeholder pattern recognized by the renderer, leading to the ONNXRuntimeError: Unexpected token '{{' in rendered prompt string.

Collectively these issues cause the prompt string to be malformed before it reaches the model, resulting in input‑shape mismatches and empty responses.

Debug – Investigation steps

Follow this systematic approach to isolate the failure:

Collect sidecar logs around the failing request.


kubectl logs -l app=rag-sidecar -c onnx-runtime --since=5m
...
2026-07-15T12:34:56Z ERROR PromptRenderer: Failed to render prompt – KeyError: 'context'
2026-07-15T12:34:56Z DEBUG Received payload: {"question":"How many users?"}
2026-07-15T12:34:56Z DEBUG Template: "Answer the following: {{question}} Context: {{context}}"

Inspect the gRPC request payload using grpcurl or a tracing interceptor.


grpcurl -plaintext -d '{"user_query":"How many users?"}' \
  localhost:50051 rag.RagService/Generate

Confirm that the field name matches the sidecar’s protobuf definition.

Validate the prompt template JSON schema used by the sidecar.


cat /etc/onnxruntime/prompt_template.json
{
  "template": "Answer the following: {{question}} Context: {{context}}",
  "variables": ["question", "context"]
}

If the variables array does not list a required placeholder, the renderer will raise a ValueError.

Reproduce the rendering locally with the same code path.


from onnxruntime import InferenceSession
from prompt_renderer import PromptRenderer   # custom module

template = "Answer the following: {{question}} Context: {{context}}"
renderer = PromptRenderer(template)

payload = {"question": "How many users?"}   # missing 'context'
print(renderer.render(payload))

Expected KeyError: 'context' matches the observed log.

Check HTTP/2 header compression by capturing the raw gRPC frames.


tcpdump -i any -s 0 -w grpc.pcap port 50051
# later, use wireshark to inspect HPACK encoded headers for extra braces

Solution – Fixing the prompt template handling

Apply the following changes in the order presented.

1. Align template syntax with renderer expectations

Update the upstream retrieval service to emit double‑brace placeholders, or modify the sidecar to accept single‑brace syntax.

Before (upstream service JSON):
{
  "template": "Answer the following: {question} Context: {context}"
}

After (aligned with ONNX Runtime docs):
{
  "template": "Answer the following: {{question}} Context: {{context}}"
}

Reference: ONNX Runtime “Running Transformers with ONNX Runtime” which explicitly uses double braces.

2. Regenerate protobuf stubs after schema change

Ensure the sidecar’s Python client matches the updated protobuf definition that includes user_query and context fields.


# In the retrieval service repository
protoc --python_out=. --grpc_python_out=. rag.proto

# Deploy updated sidecar image
docker build -t myregistry/onnxruntime-sidecar:v2 .
kubectl set image deployment/rag-sidecar onnx-runtime=myregistry/onnxruntime-sidecar:v2

3. Sanitize incoming headers to prevent brace corruption

Add a middleware in the gRPC server that decodes HPACK and removes stray braces.


import re
def clean_placeholders(message):
    cleaned = re.sub(r'\{{3,}', '{{', message)   # collapse {{{ to {{
    cleaned = re.sub(r'\}{3,}', '}}', cleaned)   # collapse }}} to }}
    return cleaned

Insert this step before the payload reaches PromptRenderer.

4. Add explicit validation in the renderer

Modify the custom PromptRenderer to raise a clear error when required variables are missing.

Before:

def render(self, data):
    return self.template.format(**data)

After:

class PromptRenderer:
    def __init__(self, template):
        self.template = template
        self.vars = re.findall(r'\{{2}(\w+)\}{2}', template)

    def render(self, data):
        missing = [v for v in self.vars if v not in data]
        if missing:
            raise ValueError(f"Prompt variable(s) {missing} not found in input data")
        return self.template.format(**data)

This mirrors the pattern described in the official ONNX Runtime Python API custom operators guide.

Verify – Confirming that the issue is resolved

Trigger a request with a full payload (question + context) and inspect the rendered prompt.


kubectl exec -it $(kubectl get pod -l app=rag-sidecar -o jsonpath="{.items[0].metadata.name}") -- \
  cat /tmp/rendered_prompt.txt
# Expected output:
Answer the following: How many users? Context: The system has 1,200 active users.

Check model inference output is non‑empty and matches expected answer.


curl -X POST http://rag-service/api/generate \
  -d '{"question":"How many users?"}' -H "Content-Type: application/json"
# Expected JSON response:
{"answer":"There are 1,200 active users."}

Monitor pod restarts and sidecar logs for a period (e.g., 30 minutes) to ensure no KeyError or ONNXRuntimeError entries appear.

Prevent – Best practices to avoid recurrence

Schema versioning: Store the prompt‑template JSON schema in a ConfigMap with a version label. Services must validate compatibility at startup.
Contract tests: Add an integration test that renders a template with a representative payload and asserts no placeholder remains.
gRPC interceptor logging: Log the raw protobuf message before deserialization; detect missing fields early.
Header sanitization library: Centralize HPACK cleanup in a shared library used by all sidecars.
Observability: Emit a custom metric prompt_render_success_total and set an alert when the failure rate exceeds 0.1%.

FAQ – Related questions

Why does the prompt rendering work locally but fail in the sidecar?
Local tests often bypass the gRPC deserialization layer and use the raw JSON template, so missing protobuf fields or header corruption are not reproduced.
Can I use Jinja2 instead of the built‑in formatter?
Yes, but you must ensure the template syntax matches the engine (e.g., {{ variable }} for Jinja2) and update the renderer accordingly.
What error is raised when a required placeholder is absent?
The custom renderer now raises ValueError: Prompt variable(s) ['context'] not found in input data, which is more actionable than the generic RuntimeError seen previously.
How do I verify that HPACK compression is not corrupting placeholders?
Capture a sample gRPC frame with tcpdump and inspect the header field values; ensure double braces appear exactly as {{ and }}.
Is there a way to fallback to a default prompt when variables are missing?
Implement a wrapper around PromptRenderer.render() that catches ValueError and substitutes a safe default string (e.g., “No context provided”).

Related Topic Hub: Model Serving Troubleshooting Hub