VLLM Citation Formatting Errors After RAG Prompt Augmentation

Q: Root Cause Analysis

vLLM inserts citation markers during the generate call when the use_citations=True flag is set (see API reference). The formatter builds a JSON block that looks like:

Q: Resolution 1. Adjust Token Budgets

Increase the model’s max_total_tokens to accommodate the longest expected answer plus the citation block. For a 8‑B model with a 4096 token context, a safe margin is:

In this article

Table of Contents

Problem Description – Inconsistent Citation Formatting in vLLM RAG Pipelines

In a production Kubernetes deployment that serves high‑throughput GPU‑accelerated inference, the vLLM engine is used to generate responses augmented with retrieved documents (RAG). The downstream reference validator expects a strict JSON payload that contains a citations array, each citation delimited by the marker syntax defined in the vLLM RAG guide. Engineers observed the following symptoms:

Generated responses sometimes omit the [CITATION] tags entirely.
When tags appear, they are truncated (e.g., [CITAT) or malformed JSON is emitted.
Downstream parsers raise JSONSchemaValidationError: 'citations' field is required.
The issue correlates with spikes in request rate (>2000 rps) and latency‑critical paths (<50 ms target).

Typical log excerpts:


ERROR vllm.core.output: CitationError – Missing citation marker after document retrieval.
Traceback (most recent call last):
  File ".../vllm/core/output.py", line 212, in format_citations
    raise TokenizerError("Unexpected token '[' while parsing citation block.")
TokenizerError: Unexpected token '[' while parsing citation block.
JSONSchemaValidationError: 'citations' field is required – caused by omitted or malformed citation tags in the model output.

Root Cause Analysis

vLLM inserts citation markers during the generate call when the use_citations=True flag is set (see API reference). The formatter builds a JSON block that looks like:


{
  "text": "... generated answer ...",
  "citations": [
    {"id": 1, "source": "doc_42"},
    {"id": 2, "source": "doc_87"}
  ]
}

Two interacting mechanisms cause the observed failures:

Token budget overflow – When the sum of prompt_tokens + max_new_tokens exceeds the model’s max_total_tokens, the internal output buffer truncates the tail of the generation. Because the citation JSON block is emitted after the main answer, it is the first victim of truncation. This matches the production incident on GKE where scaling to >2000 rps caused intermittent missing [CITATION] tags (Evidence: “Production cluster on GKE … request‑level token budget overflow”).
Streaming termination before delimiter – vLLM streams token chunks to the client (see Streaming Output guide). Under high load, the ingress controller timeout (30 s) or GPU OOM‑induced pod restarts cut the stream mid‑token, leaving an incomplete delimiter such as [CITAT. This aligns with the Reddit post reporting “citation formatting errors under high load”.

Both conditions violate the assumption in the RAG guide that the citation block will be fully emitted, leading to downstream JSON schema failures.

Investigation and Debugging Steps

1. Reproduce with Controlled Token Budget

Run a single request with explicit token limits to see where truncation occurs:

python - <<'PY'
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-8B")
params = SamplingParams(
    max_new_tokens=256,
    use_citations=True,
    citation_format="json",
    max_total_tokens=512  # intentionally low
)
prompt = "Summarize the following documents and cite them."
output = llm.generate([prompt], params=params)
print(output[0].text)
PY

Expected output ends with a well‑formed "citations": [...] block. If the block is missing, the token budget is the culprit.

2. Inspect Streaming Logs

Enable verbose streaming logs in the pod:

kubectl logs -f deployment/vllm-infer -c vllm \
  --since=5m | grep -i "stream"

Look for lines like:

2026-07-31T12:04:03.210Z INFO vllm.core.stream: Chunk sent – tokens=1024
2026-07-31T12:04:03.215Z WARN vllm.core.stream: Stream closed before delimiter

3. Verify Ingress Timeout

Check the ingress controller configuration:

kubectl get ingress vllm-api -o yaml | grep timeout

If timeout: 30s is set, it may cut off the citation block during high‑latency responses.

4. Examine GPU Memory Pressure

GPU OOM can cause pod restarts that flush incomplete streams. Monitor with:

kubectl top pod -l app=vllm -n inference
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

5. Review Configuration of `use_citations` and `citation_format`

Confirm that the API request includes the correct parameters (Evidence: API reference).

curl -X POST https://api.example.com/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "Explain quantum tunneling with sources.",
        "max_new_tokens": 512,
        "use_citations": true,
        "citation_format": "json"
      }'

Resolution

1. Adjust Token Budgets

Increase the model’s max_total_tokens to accommodate the longest expected answer plus the citation block. For a 8‑B model with a 4096 token context, a safe margin is:

Component	Typical Tokens
Prompt + retrieved docs	≈ 3000
Generated answer	≤ 800
Citation JSON block	≈ 200

Set max_total_tokens=4096 (or the model‑specific limit) and raise max_new_tokens accordingly.

2. Enforce a Dedicated Stop Token for Citations

Append a custom stop sequence that forces the model to emit the citation block before termination. Update the generation call:

params = SamplingParams(
    max_new_tokens=1024,
    use_citations=True,
    citation_format="json",
    stop=["\n\n"],  # ensures a blank line after citations
)

3. Increase Ingress Timeout and Enable Chunk Buffering

Modify the ingress spec to allow at least 60 seconds for the longest response and enable buffering to avoid mid‑stream truncation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vllm-api
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-buffering: "true"
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1/generate
        pathType: Prefix
        backend:
          service:
            name: vllm-service
            port:
              number: 80

4. Patch the Citation Formatter (Optional)

If upgrading vLLM is not immediate, apply the community‑provided fix from GitHub Issue #917 that guards against incomplete buffers:

# Original snippet (vllm/core/output.py)
def format_citations(tokens):
    # ... assumes full buffer
    return json.loads(tokens)

# Patched version
def format_citations(tokens):
    try:
        return json.loads(tokens)
    except json.JSONDecodeError:
        # Fallback: strip incomplete trailing token and retry
        cleaned = tokens.rstrip('[').rstrip()
        return json.loads(cleaned) if cleaned else {}

Verification

After applying the fixes, perform the following checks:

Functional test – Run a request that forces the maximum token budget and inspect the response:

curl -X POST https://api.example.com/v1/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Provide a detailed summary with citations.", "max_new_tokens":1024, "use_citations":true, "citation_format":"json"}'

Expected JSON fragment:

{
  "text": "...",
  "citations": [
    {"id":1,"source":"doc_12"},
    {"id":2,"source":"doc_34"}
  ]
}

Schema validation – Run the downstream validator:

python validate_citations.py --payload response.json

Should exit with code 0 and no JSONSchemaValidationError.

Load test – Use hey or locust to generate 2500 rps for 5 minutes and monitor logs for any CitationError messages. No new errors should appear.

Prevention and Best Practices

Reserve token budget for metadata – Always allocate at least 10 % of max_total_tokens for citation JSON.
Explicit stop tokens – Define a stop sequence that forces the model to finish the citation block before the stream ends.
Monitor token usage metrics – Export vllm_inference_total_tokens and set alerts when usage approaches the configured limit.
Graceful shutdown handling – Enable preStop hooks in the pod spec to flush streaming buffers before container termination.
Upgrade regularly – The citation formatter bug was fixed in vLLM 0.4.5 (see Issue #917). Keep the deployment on a supported version.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why do citations disappear only under high load?
High load increases request latency, causing the model to hit the max_total_tokens limit or the ingress timeout, both of which truncate the citation block.
Can I disable streaming to avoid truncation?
Disabling streaming removes the mid‑stream timeout issue but adds latency. A better approach is to increase the ingress timeout and ensure the token budget includes the citation payload.
What stop token should I use for JSON citation formatting?
A double newline (\n\n) or a custom sentinel like ---END-CITATIONS--- works because the formatter always emits a newline after the JSON block.
Do I need to modify the tokenizer for citation markers?
No. The built‑in tokenizer already knows the [CITATION] token. Issues arise only when the output is cut before the tokenizer can emit the closing delimiter.
How can I detect a partially flushed citation block in logs?
Look for warning messages such as vllm.core.stream: Stream closed before delimiter or truncated token patterns like [CITAT in the raw output.