Problem
After a recent configuration change in the staging environment, the Grafana AI panel that uses Retrieval‑Augmented Generation (RAG) began emitting malformed citation markers. Instead of the required markdown reference style [^1] with a matching footnote block, the output contains raw URLs, duplicated brackets, or missing footnote definitions. This breaks downstream rendering in dashboards and confuses users who rely on proper source attribution.
Root Cause
The RAG integration builds citations by concatenating a citation_template string with the metadata returned from the vector store. A configuration change introduced an extra newline character and altered the placeholder syntax from {{cite}} to {{ citation }}. The templating engine (Go text/template) treats the new placeholder as an undefined variable, causing the fallback path to emit the raw document URL without the markdown footnote wrapper. Additionally, the max_citations_per_response limit was increased from 3 to 10 without adjusting the citation_footer_template, resulting in the footer generation loop never closing the footnote block.
Debug
Step‑by‑step investigation performed in the staging cluster:
-
Collect AI panel logs from the
grafana-agentpod:kubectl -n monitoring logs -l app=grafana-agent -c ai-panel --since=5m 2024-10-12T14:32:07Z INFO rag: rendering response 2024-10-12T14:32:07Z DEBUG rag: citation_template = "{{ citation }}" 2024-10-12T14:32:07Z WARN rag: undefined variable "citation" in template 2024-10-12T14:32:07Z INFO rag: generated citation: https://docs.example.com/api/v1 2024-10-12T14:32:07Z INFO rag: response length 842 chars -
Inspect the effective configuration stored in ConfigMap
grafana-ai-config:kubectl -n monitoring get configmap grafana-ai-config -o yaml apiVersion: v1 data: rag.yaml: | citation_template: "{{ citation }}" citation_footer_template: | {{ range $i, $c := .Citations }} [^{{add $i 1}}]: {{$c.URL}} {{ end }} max_citations_per_response: 10 -
Compare with a known‑good configuration from the previous release (saved in Git):
citation_template: "{{cite}}" citation_footer_template: | {{ range $i, $c := .Citations }} [^{{add $i 1}}]: {{$c.Title}} ({{$c.URL}}) {{ end }} max_citations_per_response: 3 -
Run a manual RAG request using the CLI tool
grafana-ai-clito isolate the templating step:grafana-ai-cli query \ --prompt "Explain Prometheus remote write" \ --debug [...] Generated citation: https://prometheus.io/docs/prometheus/latest/configuration/remote_write/The output lacks the
[^1]marker and footnote block.
Solution
Restore the original templating variables and align the footer template with the new max_citations_per_response value. The fix consists of three parts:
1. Update ConfigMap
Replace the incorrect placeholders and adjust the footer loop to handle up to ten citations while preserving markdown syntax.
# Before (staging)
citation_template: "{{ citation }}"
citation_footer_template: |
{{ range $i, $c := .Citations }}
[^{{add $i 1}}]: {{$c.URL}}
{{ end }}
max_citations_per_response: 10
# After (fixed)
citation_template: "{{cite}}"
citation_footer_template: |
{{ range $i, $c := .Citations }}
[^{{add $i 1}}]: {{if $c.Title}}{{$c.Title}} ({{$c.URL}}){{else}}{{$c.URL}}{{end}}
{{ end }}
max_citations_per_response: 10
Apply the updated ConfigMap and trigger a rolling restart of the AI panel deployment:
kubectl -n monitoring apply -f - <
2. Patch the Go template helper
The helper add used to generate sequential footnote numbers was inadvertently removed during the refactor. Re‑introduce it in internal/rag/template_funcs.go:
// Before (missing)
func add(a, b int) int { return a + b }
// After (added back)
func add(a, b int) int { return a + b }
func init() {
template.FuncMap["add"] = add
}
3. Validate vector store metadata
Ensure that each document stored in the vector database includes a Title field; otherwise the fallback to raw URL is used. A migration script was added to backfill missing titles:
#!/usr/bin/env python3
import json, sys
from vectordb import client
for doc in client.scan():
if 'Title' not in doc.metadata:
doc.metadata['Title'] = fetch_title_from_url(doc.metadata['URL'])
client.update(doc.id, doc.metadata)
Verify
After applying the fixes, perform the following checks:
-
Query the AI panel from the UI and inspect the rendered markdown:
User prompt: "Summarize Grafana Loki architecture." AI response (excerpt): Loki stores logs in a series of chunks …[^1] [^1]: Loki Architecture Overview (https://grafana.com/docs/loki/latest/architecture/) -
Confirm that the pod logs no longer contain the undefined‑variable warning:
2024-10-12T14:45:12Z INFO rag: rendering response 2024-10-12T14:45:12Z DEBUG rag: citation_template = "{{cite}}" 2024-10-12T14:45:12Z INFO rag: generated 2 citations -
Run a health‑check endpoint that validates citation rendering:
curl -s http://grafana-ai-panel.monitoring.svc.cluster.local/healthz | jq . { "status":"ok", "citation_rendering":"valid" }
Prevent
To avoid recurrence:
- Configuration linting: Add a CI step that parses
rag.yamlwith the Go template engine and fails on undefined variables. - Schema enforcement: Enforce a JSON schema for vector store documents that requires
TitleandURLfields. - Regression testing: Include an automated test that generates a sample RAG response and asserts the presence of correctly formatted markdown footnotes.
- Change‑control checklist: Require a review of any modifications to
citation_templateorcitation_footer_templateto verify placeholder names and loop logic.
Related Topic Hub: Observability Troubleshooting Hub
FAQ
- Why does the citation appear correctly in development but not in staging? The staging ConfigMap was overwritten during a Helm upgrade that introduced a typo in the placeholder name. Development uses a separate ConfigMap that retained the original template.
- Can I use a custom citation style (e.g., APA) with the AI panel? Yes. Define a new
citation_templateand adjustcitation_footer_templateto emit the desired format. Remember to update theaddhelper if you need sequential numbering. - What happens if a document lacks a title? The fallback logic prints only the URL. After the fix, the template checks for
$c.Titleand prints the URL alone when the title is missing, preserving markdown syntax. - Is there a limit to the number of citations per response? The default is 3 for performance reasons. Increasing
max_citations_per_responseis safe as long as the footer template can iterate over the larger slice and the UI can render the longer footnote list. - How can I debug template rendering locally? Use the
grafana-ai-cli --template-debugflag, which prints the intermediate data structure passed to the Go template engine.