GPT-4o RAG citation plain text after model version upgrade in Kubernetes

Problem: GPT‑4o RAG citations returned as plain text after model version upgrade

After upgrading the OpenAI model version from gpt‑4‑turbo to gpt‑4o in a Kubernetes‑based RAG microservice, downstream components started to fail. The service expects each citation to be a JSON array with the fields source, page, and confidence. Instead, the model now emits a free‑form plain‑text block such as:


Sources:
1. "OpenAI API Documentation", page 12
2. "Vector Search Best Practices", page 3

This format violates the contract of the /v1/rag/query endpoint, causing JSON parsing errors and breaking the citation enrichment pipeline.

Root Cause Analysis

The root cause is a combination of two factors introduced by the model upgrade:

  1. System Prompt Change: The default system prompt for gpt‑4o includes a higher‑level instruction to “provide citations in a human‑readable list”. The previous prompt explicitly forced JSON output via output_format=json.
  2. Model‑side Temperature & Sampling: gpt‑4o defaults to temperature=0.7. At non‑zero temperature the model may deviate from strict token‑by‑token constraints, especially when the JSON schema is not reinforced with a response_format parameter.

Because the Kubernetes deployment did not propagate the updated response_format field (the OpenAI SDK version used in the container still referenced the older API signature), the request sent to the OpenAI endpoint lacked the JSON schema hint, allowing the model to fall back to its internal “human‑readable” style.

Debugging Steps

1. Inspect the request payload sent from the pod


$ kubectl exec -it $(kubectl get pod -l app=rag-service -o jsonpath='{.items[0].metadata.name}') \
  --namespace rag -- \
  cat /var/log/rag-service/request.log | tail -n 20
{
  "model": "gpt-4o",
  "messages": [
    {"role":"system","content":"You are a helpful assistant..."},
    {"role":"user","content":"Explain the impact of vector similarity on retrieval."}
  ],
  "temperature": 0.7
}

Notice the missing "response_format" field.

2. Verify the OpenAI SDK version inside the container


$ kubectl exec -it $(kubectl get pod -l app=rag-service -o jsonpath='{.items[0].metadata.name}') \
  --namespace rag -- \
  python -c "import openai, pkg_resources; print(pkg_resources.get_distribution('openai').version)"
0.27.0

Version 0.27.0 predates the response_format argument introduced in 0.28.0.

3. Capture the raw response from OpenAI


$ kubectl logs -l app=rag-service -c rag-container --since=5m | grep -A3 '"choices"'
"choices": [
  {
    "message": {
      "role": "assistant",
      "content": "Sources:\n1. \"OpenAI API Documentation\", page 12\n2. \"Vector Search Best Practices\", page 3"
    }
  }
]

The content field contains plain text, confirming the model ignored any JSON schema.

4. Reproduce locally with the same SDK version


>>> import openai
>>> openai.api_key = "sk-..."
>>> resp = openai.ChatCompletion.create(
...     model="gpt-4o",
...     messages=[{"role":"user","content":"Give a citation"}],
...     temperature=0.7
... )
>>> resp.choices[0].message.content
'Sources:\n1. "OpenAI API Documentation", page 12'

The same behavior occurs outside Kubernetes, confirming it is not a cluster‑specific network issue.

Solution: Enforce JSON schema and lock temperature

1. Upgrade the OpenAI SDK in the container image

Update requirements.txt:


openai>=0.28.0

Rebuild and push the image:


docker build -t registry.example.com/rag-service:2024-11-18 .
docker push registry.example.com/rag-service:2024-11-18

2. Add a strict response_format to the request

Modified request generation (Python snippet):


def build_chat_payload(user_query: str) -> dict:
    return {
        "model": "gpt-4o",
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_query}
        ],
        # Force deterministic output
        "temperature": 0.0,
        # JSON schema expected by downstream
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "citation_schema",
                "strict": True,
                "schema": {
                    "type": "object",
                    "properties": {
                        "citations": {
                            "type": "array",
                            "items": {
                                "type": "object",
                                "properties": {
                                    "source": {"type": "string"},
                                    "page": {"type": "integer"},
                                    "confidence": {"type": "number"}
                                },
                                "required": ["source", "page", "confidence"]
                            }
                        }
                    },
                    "required": ["citations"]
                }
            }
        }
    }

3. Update the system prompt to remove any “human‑readable” instruction


SYSTEM_PROMPT = (
    "You are a Retrieval‑Augmented Generation assistant. "
    "When you reference a source, return ONLY a JSON object matching the schema: "
    "{\"citations\": [{\"source\": string, \"page\": integer, \"confidence\": number}]}. "
    "Do not add any explanatory text outside the JSON."
)

4. Deploy the updated manifest


apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-service
  namespace: rag
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rag-service
  template:
    metadata:
      labels:
        app: rag-service
    spec:
      containers:
      - name: rag-container
        image: registry.example.com/rag-service:2024-11-18
        env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: api-key
        resources:
          limits:
            cpu: "500m"
            memory: "512Mi"

Verification

1. Smoke test via curl


$ curl -X POST https://rag.example.com/v1/query \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Explain vector similarity"}' | jq .
{
  "answer": "...",
  "citations": [
    {
      "source": "OpenAI API Documentation",
      "page": 12,
      "confidence": 0.96
    },
    {
      "source": "Vector Search Best Practices",
      "page": 3,
      "confidence": 0.89
    }
  ]
}

2. Log inspection


$ kubectl logs -l app=rag-service -c rag-container --since=1m | grep '"citations"'
"citations": [{"source":"OpenAI API Documentation","page":12,"confidence":0.96},{"source":"Vector Search Best Practices","page":3,"confidence":0.89}]

The logs now contain a well‑formed JSON array, confirming the fix.

3. Downstream consumer test

Run the integration test suite that parses the citations field. All tests pass:


$ pytest tests/test_citation_parsing.py -q
..                                                                  [100%]
2 passed in 0.31s

Prevention and Operational Guardrails

  • Pin SDK version: Use a requirements.txt lock file and CI checks to prevent accidental downgrade.
  • Schema validation middleware: Insert a lightweight JSON schema validator (e.g., jsonschema) after the OpenAI call; reject responses that do not match.
  • Alert on parsing failures: Configure a Prometheus alert on the rag_citation_parsing_errors_total counter.
  • Immutable system prompt: Store the prompt in a ConfigMap and mount it read‑only; any change requires a rollout.
  • Temperature lock: Enforce temperature=0.0 for all RAG calls in production to guarantee deterministic JSON output.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does setting temperature to 0 fix the citation format? A non‑zero temperature introduces stochastic token sampling, which can cause the model to stray from the exact JSON tokens required by the schema. Zero temperature forces deterministic, token‑by‑token generation.
  2. Can I keep the human‑readable citation list for debugging? Yes. Wrap the JSON object inside a top‑level field, e.g., {"citations": [...], "debug": "human readable"} , and let downstream ignore debug. Do not rely on free‑form text.
  3. Do I need to upgrade the OpenAI API endpoint version? No. The response_format field is supported on the existing /v1/chat/completions endpoint; just ensure the SDK version is recent enough to expose the parameter.
  4. What if I must use a higher temperature for creative answers? Separate the generation path: first request the answer with temperature>0, then make a second deterministic request (temperature 0) that only asks for citations, merging the results downstream.
  5. How can I monitor that citations remain valid JSON over time? Export a metric from the validation middleware, e.g., rag_citation_json_valid{status="true|false"}, and set a Grafana alert on the error rate exceeding 0.1%.