Claude streaming RAG citations missing fields after schema change

1. Omitted the new fields because they were not mentioned in the prompt.
2. Continued to use the old delimiter, which downstream parsers treat as a syntax error.
3. Placed the citation block after the \\n\\n separator that signals the end of the answer, causing the streaming client to treat it as part of the answer body.

Because the streaming pipeline validates citations only after the full response is received, the malformed fragments were not caught until they reached the consumer, resulting in runtime failures.

Debug

Below are the steps taken to isolate the problem.

1. Log inspection

2026-10-09T12:04:13.842Z INFO  streaming_handler.go:112 "Received token: \"The\""
2026-10-09T12:04:13.845Z INFO  streaming_handler.go:112 "Received token: \"answer\""
2026-10-09T12:04:13.848Z INFO  streaming_handler.go:112 "Received token: \"...\""
2026-10-09T12:04:14.001Z INFO  streaming_handler.go:124 "Citation start detected"
2026-10-09T14:02:14.003Z WARN  citation_parser.go:45 "Missing fields: source_id,relevance_score"
2026-10-09T14:02:14.005Z WARN  citation_parser.go:48 "Delimiter ';' not allowed in schema v1.1"
2026-10-09T14:02:14.010Z ERROR streaming_handler.go:138 "Failed to serialize citations: json: unknown field \"source\""

2. Prompt template comparison

Provide the answer. Then list citations as:
[["title","url","source"]]; separate entries with ';'

Provide the answer. Then list citations as a JSON array:
[
  {"title": "...", "url": "...", "source_id": "...", "relevance_score": ...}
]
Separate entries with ','

3. Reproducing the failure locally

# Run Claude in streaming mode with the old template
anthropic-cli chat --model=claude-3-opus-20240229 \
  --stream \
  --system-prompt="$(cat prompts/v1.0_template.txt)" \
  --user-prompt="Explain the benefits of vector search."

The output shows citations without source_id and relevance_score, confirming the template mismatch.

Solution

Update the streaming prompt to reflect the new schema and enforce strict validation before emitting tokens.

1. Prompt template revision

Replace the old template with the following version (saved as prompts/v1.1_template.txt):

You are Claude, a helpful assistant. Answer the user query using the retrieved documents.
After the answer, output a JSON array named "citations" that conforms to schema v1.1:

{
  "citations": [
    {
      "title": "string",               // Document title
      "url": "string",                 // Permanent URL
      "source_id": "string",           // Unique identifier from the retrieval store
      "relevance_score": number        // Float between 0 and 1
    },
    ...
  ]
}

Do NOT emit any text outside the JSON object. Use commas to separate array items.

2. Server‑side validation hook

Add a lightweight validator that runs on each token batch. If a citation fragment violates the schema, abort streaming and return an error.

// citation_validator.go
package validator

import (
    "encoding/json"
    "errors"
)

type Citation struct {
    Title          string  `json:"title"`
    URL            string  `json:"url"`
    SourceID       string  `json:"source_id"`
    RelevanceScore float64 `json:"relevance_score"`
}

func ValidateCitationJSON(raw []byte) error {
    var wrapper struct {
        Citations []Citation `json:"citations"`
    }
    if err := json.Unmarshal(raw, &wrapper); err != nil {
        return err
    }
    for _, c := range wrapper.Citations {
        if c.Title == "" || c.URL == "" || c.SourceID == "" {
            return errors.New("missing required citation fields")
        }
        if c.RelevanceScore < 0 || c.RelevanceScore > 1 {
            return errors.New("relevance_score out of range")
        }
    }
    return nil
}

Integrate the validator in the streaming loop:

// streaming_handler.go (excerpt)
if strings.Contains(token, "\"citations\"") {
    // accumulate tokens until the closing brace
    buffer.WriteString(token)
    if strings.HasSuffix(token, "}") {
        if err := validator.ValidateCitationJSON(buffer.Bytes()); err != nil {
            log.Errorf("Citation validation failed: %v", err)
            stream.SendError("Citation format error")
            return
        }
        stream.Send(buffer.Bytes())
    }
    continue
}

3. Deployment changes

Apply the updated prompt and binary to the streaming service.

# Kubernetes rolling update
kubectl set env deployment/claude-streamer \
  PROMPT_FILE=/app/prompts/v1.1_template.txt \
  --record

kubectl rollout status deployment/claude-streamer

Verify

After redeployment, run the same query and inspect the streamed output.

2026-10-09T15:22:01.112Z INFO  streaming_handler.go:112 "Received token: \"The\""
...
2026-10-09T15:22:02.045Z INFO  streaming_handler.go:124 "Citation start detected"
2026-10-09T15:22:02.050Z INFO  citation_parser.go:30 "Citation JSON validated"
2026-10-09T15:22:02.052Z INFO  streaming_handler.go:138 "Sent citation block"

Sample final citation block:

{
  "citations": [
    {
      "title": "Vector Search in Production",
      "url": "https://example.com/vector-search",
      "source_id": "doc-42f7a9",
      "relevance_score": 0.93
    },
    {
      "title": "RAG Best Practices",
      "url": "https://example.com/rag-best-practices",
      "source_id": "doc-9b3c1d",
      "relevance_score": 0.87
    }
  ]
}

Validate with the schema validator:

$ go run validator_test.go
PASS: All citations conform to schema v1.1

Prevent

• Version‑locked prompts: Store prompt files with a semantic version tag and enforce a CI check that the prompt version matches the citation schema version declared in config.yaml.
• Schema‑driven generation: Use Claude’s function‑calling feature to request a structured JSON object instead of free‑form text, guaranteeing field presence.
• Streaming guardrails: Implement a timeout that aborts streaming if the citations block is not completed within n seconds after the answer delimiter.
• Automated regression tests: Add an integration test that sends a representative query to the streaming endpoint and asserts that the returned JSON validates against the current schema.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why do citations sometimes appear after the final token?
   Because the prompt does not explicitly tell Claude to emit the citation block before the end‑of‑stream token. Updating the prompt to place the JSON object before the final delimiter forces correct ordering.
2. Can I keep using the old delimiter ';' with schema v1.1?
   No. The schema definition for v1.1 specifies a JSON array where commas separate items. Using a semicolon produces invalid JSON and causes downstream parsers to fail.
3. How do I test that my streaming service correctly validates citations?
   Run the validator_test.go integration test as part of the CI pipeline. It feeds a mock response containing both valid and intentionally malformed citation blocks and asserts that only the valid ones pass.
4. Is there a way to make Claude generate citations automatically without a custom prompt?
   Yes. Claude supports function calls. Define a function schema matching the citation structure and invoke it from the system prompt. Claude will return a well‑formed JSON object that satisfies the schema without manual string manipulation.
5. What should I monitor to detect future citation format regressions?
   Create a Prometheus metric streaming_citation_validation_errors_total that increments on each validation failure. Alert when the rate exceeds a threshold (e.g., > 0.1 errors/min) over a 5‑minute window.