Problem: GPT‑4o RAG citations returned as plain text after model version upgrade
After upgrading the OpenAI model version from gpt‑4‑turbo to gpt‑4o in a Kubernetes‑based RAG microservice, downstream components started to fail. The service expects each citation to be a JSON array with the fields source, page, and confidence. Instead, the model now emits a free‑form plain‑text block such as:
Sources:
1. "OpenAI API Documentation", page 12
2. "Vector Search Best Practices", page 3
This format violates the contract of the /v1/rag/query endpoint, causing JSON parsing errors and breaking the citation enrichment pipeline.
Root Cause Analysis
The root cause is a combination of two factors introduced by the model upgrade:
- System Prompt Change: The default system prompt for
gpt‑4oincludes a higher‑level instruction to “provide citations in a human‑readable list”. The previous prompt explicitly forced JSON output viaoutput_format=json. - Model‑side Temperature & Sampling:
gpt‑4odefaults totemperature=0.7. At non‑zero temperature the model may deviate from strict token‑by‑token constraints, especially when the JSON schema is not reinforced with aresponse_formatparameter.
Because the Kubernetes deployment did not propagate the updated response_format field (the OpenAI SDK version used in the container still referenced the older API signature), the request sent to the OpenAI endpoint lacked the JSON schema hint, allowing the model to fall back to its internal “human‑readable” style.
Debugging Steps
1. Inspect the request payload sent from the pod
$ kubectl exec -it $(kubectl get pod -l app=rag-service -o jsonpath='{.items[0].metadata.name}') \
--namespace rag -- \
cat /var/log/rag-service/request.log | tail -n 20
{
"model": "gpt-4o",
"messages": [
{"role":"system","content":"You are a helpful assistant..."},
{"role":"user","content":"Explain the impact of vector similarity on retrieval."}
],
"temperature": 0.7
}
Notice the missing "response_format" field.
2. Verify the OpenAI SDK version inside the container
$ kubectl exec -it $(kubectl get pod -l app=rag-service -o jsonpath='{.items[0].metadata.name}') \
--namespace rag -- \
python -c "import openai, pkg_resources; print(pkg_resources.get_distribution('openai').version)"
0.27.0
Version 0.27.0 predates the response_format argument introduced in 0.28.0.
3. Capture the raw response from OpenAI
$ kubectl logs -l app=rag-service -c rag-container --since=5m | grep -A3 '"choices"'
"choices": [
{
"message": {
"role": "assistant",
"content": "Sources:\n1. \"OpenAI API Documentation\", page 12\n2. \"Vector Search Best Practices\", page 3"
}
}
]
The content field contains plain text, confirming the model ignored any JSON schema.
4. Reproduce locally with the same SDK version
>>> import openai
>>> openai.api_key = "sk-..."
>>> resp = openai.ChatCompletion.create(
... model="gpt-4o",
... messages=[{"role":"user","content":"Give a citation"}],
... temperature=0.7
... )
>>> resp.choices[0].message.content
'Sources:\n1. "OpenAI API Documentation", page 12'
The same behavior occurs outside Kubernetes, confirming it is not a cluster‑specific network issue.
Solution: Enforce JSON schema and lock temperature
1. Upgrade the OpenAI SDK in the container image
Update requirements.txt:
openai>=0.28.0
Rebuild and push the image:
docker build -t registry.example.com/rag-service:2024-11-18 .
docker push registry.example.com/rag-service:2024-11-18
2. Add a strict response_format to the request
Modified request generation (Python snippet):
def build_chat_payload(user_query: str) -> dict:
return {
"model": "gpt-4o",
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_query}
],
# Force deterministic output
"temperature": 0.0,
# JSON schema expected by downstream
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "citation_schema",
"strict": True,
"schema": {
"type": "object",
"properties": {
"citations": {
"type": "array",
"items": {
"type": "object",
"properties": {
"source": {"type": "string"},
"page": {"type": "integer"},
"confidence": {"type": "number"}
},
"required": ["source", "page", "confidence"]
}
}
},
"required": ["citations"]
}
}
}
}
3. Update the system prompt to remove any “human‑readable” instruction
SYSTEM_PROMPT = (
"You are a Retrieval‑Augmented Generation assistant. "
"When you reference a source, return ONLY a JSON object matching the schema: "
"{\"citations\": [{\"source\": string, \"page\": integer, \"confidence\": number}]}. "
"Do not add any explanatory text outside the JSON."
)
4. Deploy the updated manifest
apiVersion: apps/v1
kind: Deployment
metadata:
name: rag-service
namespace: rag
spec:
replicas: 3
selector:
matchLabels:
app: rag-service
template:
metadata:
labels:
app: rag-service
spec:
containers:
- name: rag-container
image: registry.example.com/rag-service:2024-11-18
env:
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: openai-secret
key: api-key
resources:
limits:
cpu: "500m"
memory: "512Mi"
Verification
1. Smoke test via curl
$ curl -X POST https://rag.example.com/v1/query \
-H "Content-Type: application/json" \
-d '{"prompt":"Explain vector similarity"}' | jq .
{
"answer": "...",
"citations": [
{
"source": "OpenAI API Documentation",
"page": 12,
"confidence": 0.96
},
{
"source": "Vector Search Best Practices",
"page": 3,
"confidence": 0.89
}
]
}
2. Log inspection
$ kubectl logs -l app=rag-service -c rag-container --since=1m | grep '"citations"'
"citations": [{"source":"OpenAI API Documentation","page":12,"confidence":0.96},{"source":"Vector Search Best Practices","page":3,"confidence":0.89}]
The logs now contain a well‑formed JSON array, confirming the fix.
3. Downstream consumer test
Run the integration test suite that parses the citations field. All tests pass:
$ pytest tests/test_citation_parsing.py -q
.. [100%]
2 passed in 0.31s
Prevention and Operational Guardrails
- Pin SDK version: Use a
requirements.txtlock file and CI checks to prevent accidental downgrade. - Schema validation middleware: Insert a lightweight JSON schema validator (e.g.,
jsonschema) after the OpenAI call; reject responses that do not match. - Alert on parsing failures: Configure a Prometheus alert on the
rag_citation_parsing_errors_totalcounter. - Immutable system prompt: Store the prompt in a ConfigMap and mount it read‑only; any change requires a rollout.
- Temperature lock: Enforce
temperature=0.0for all RAG calls in production to guarantee deterministic JSON output.
Related Topic Hub: LLM Systems Troubleshooting Hub
FAQ
- Why does setting
temperatureto 0 fix the citation format? A non‑zero temperature introduces stochastic token sampling, which can cause the model to stray from the exact JSON tokens required by the schema. Zero temperature forces deterministic, token‑by‑token generation. - Can I keep the human‑readable citation list for debugging? Yes. Wrap the JSON object inside a top‑level field, e.g.,
{"citations": [...], "debug": "human readable"}, and let downstream ignoredebug. Do not rely on free‑form text. - Do I need to upgrade the OpenAI API endpoint version? No. The
response_formatfield is supported on the existing/v1/chat/completionsendpoint; just ensure the SDK version is recent enough to expose the parameter. - What if I must use a higher temperature for creative answers? Separate the generation path: first request the answer with
temperature>0, then make a second deterministic request (temperature 0) that only asks for citations, merging the results downstream. - How can I monitor that citations remain valid JSON over time? Export a metric from the validation middleware, e.g.,
rag_citation_json_valid{status="true|false"}, and set a Grafana alert on the error rate exceeding 0.1%.