Grafana AI panel RAG citation format error after config change

Problem

After a recent configuration change in the staging environment, the Grafana AI panel that uses Retrieval‑Augmented Generation (RAG) began emitting malformed citation markers. Instead of the required markdown reference style [^1] with a matching footnote block, the output contains raw URLs, duplicated brackets, or missing footnote definitions. This breaks downstream rendering in dashboards and confuses users who rely on proper source attribution.

Root Cause

The RAG integration builds citations by concatenating a citation_template string with the metadata returned from the vector store. A configuration change introduced an extra newline character and altered the placeholder syntax from {{cite}} to {{ citation }}. The templating engine (Go text/template) treats the new placeholder as an undefined variable, causing the fallback path to emit the raw document URL without the markdown footnote wrapper. Additionally, the max_citations_per_response limit was increased from 3 to 10 without adjusting the citation_footer_template, resulting in the footer generation loop never closing the footnote block.

Debug

Step‑by‑step investigation performed in the staging cluster:

  1. Collect AI panel logs from the grafana-agent pod:

    
    kubectl -n monitoring logs -l app=grafana-agent -c ai-panel --since=5m
    2024-10-12T14:32:07Z INFO  rag: rendering response
    2024-10-12T14:32:07Z DEBUG rag: citation_template = "{{ citation }}"
    2024-10-12T14:32:07Z WARN  rag: undefined variable "citation" in template
    2024-10-12T14:32:07Z INFO  rag: generated citation: https://docs.example.com/api/v1
    2024-10-12T14:32:07Z INFO  rag: response length 842 chars
    
  2. Inspect the effective configuration stored in ConfigMap grafana-ai-config:

    
    kubectl -n monitoring get configmap grafana-ai-config -o yaml
    apiVersion: v1
    data:
      rag.yaml: |
        citation_template: "{{ citation }}"
        citation_footer_template: |
          {{ range $i, $c := .Citations }}
          [^{{add $i 1}}]: {{$c.URL}}
          {{ end }}
        max_citations_per_response: 10
    
  3. Compare with a known‑good configuration from the previous release (saved in Git):

    
    citation_template: "{{cite}}"
    citation_footer_template: |
      {{ range $i, $c := .Citations }}
      [^{{add $i 1}}]: {{$c.Title}} ({{$c.URL}})
      {{ end }}
    max_citations_per_response: 3
    
  4. Run a manual RAG request using the CLI tool grafana-ai-cli to isolate the templating step:

    
    grafana-ai-cli query \
      --prompt "Explain Prometheus remote write" \
      --debug
    [...]
    Generated citation: https://prometheus.io/docs/prometheus/latest/configuration/remote_write/
    

    The output lacks the [^1] marker and footnote block.

Solution

Restore the original templating variables and align the footer template with the new max_citations_per_response value. The fix consists of three parts:

1. Update ConfigMap

Replace the incorrect placeholders and adjust the footer loop to handle up to ten citations while preserving markdown syntax.


# Before (staging)
citation_template: "{{ citation }}"
citation_footer_template: |
  {{ range $i, $c := .Citations }}
  [^{{add $i 1}}]: {{$c.URL}}
  {{ end }}
max_citations_per_response: 10

# After (fixed)
citation_template: "{{cite}}"
citation_footer_template: |
  {{ range $i, $c := .Citations }}
  [^{{add $i 1}}]: {{if $c.Title}}{{$c.Title}} ({{$c.URL}}){{else}}{{$c.URL}}{{end}}
  {{ end }}
max_citations_per_response: 10

Apply the updated ConfigMap and trigger a rolling restart of the AI panel deployment:


kubectl -n monitoring apply -f - <

2. Patch the Go template helper

The helper add used to generate sequential footnote numbers was inadvertently removed during the refactor. Re‑introduce it in internal/rag/template_funcs.go:


// Before (missing)
func add(a, b int) int { return a + b }

// After (added back)
func add(a, b int) int { return a + b }

func init() {
    template.FuncMap["add"] = add
}

3. Validate vector store metadata

Ensure that each document stored in the vector database includes a Title field; otherwise the fallback to raw URL is used. A migration script was added to backfill missing titles:


#!/usr/bin/env python3
import json, sys
from vectordb import client

for doc in client.scan():
    if 'Title' not in doc.metadata:
        doc.metadata['Title'] = fetch_title_from_url(doc.metadata['URL'])
        client.update(doc.id, doc.metadata)

Verify

After applying the fixes, perform the following checks:

  1. Query the AI panel from the UI and inspect the rendered markdown:

    
    User prompt: "Summarize Grafana Loki architecture."
    AI response (excerpt):
    Loki stores logs in a series of chunks …[^1]
    
    [^1]: Loki Architecture Overview (https://grafana.com/docs/loki/latest/architecture/)
    
  2. Confirm that the pod logs no longer contain the undefined‑variable warning:

    
    2024-10-12T14:45:12Z INFO  rag: rendering response
    2024-10-12T14:45:12Z DEBUG rag: citation_template = "{{cite}}"
    2024-10-12T14:45:12Z INFO  rag: generated 2 citations
    
  3. Run a health‑check endpoint that validates citation rendering:

    
    curl -s http://grafana-ai-panel.monitoring.svc.cluster.local/healthz | jq .
    {
      "status":"ok",
      "citation_rendering":"valid"
    }
    

Prevent

To avoid recurrence:

  • Configuration linting: Add a CI step that parses rag.yaml with the Go template engine and fails on undefined variables.
  • Schema enforcement: Enforce a JSON schema for vector store documents that requires Title and URL fields.
  • Regression testing: Include an automated test that generates a sample RAG response and asserts the presence of correctly formatted markdown footnotes.
  • Change‑control checklist: Require a review of any modifications to citation_template or citation_footer_template to verify placeholder names and loop logic.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

  1. Why does the citation appear correctly in development but not in staging? The staging ConfigMap was overwritten during a Helm upgrade that introduced a typo in the placeholder name. Development uses a separate ConfigMap that retained the original template.
  2. Can I use a custom citation style (e.g., APA) with the AI panel? Yes. Define a new citation_template and adjust citation_footer_template to emit the desired format. Remember to update the add helper if you need sequential numbering.
  3. What happens if a document lacks a title? The fallback logic prints only the URL. After the fix, the template checks for $c.Title and prints the URL alone when the title is missing, preserving markdown syntax.
  4. Is there a limit to the number of citations per response? The default is 3 for performance reasons. Increasing max_citations_per_response is safe as long as the footer template can iterate over the larger slice and the UI can render the longer footnote list.
  5. How can I debug template rendering locally? Use the grafana-ai-cli --template-debug flag, which prints the intermediate data structure passed to the Go template engine.