Grafana 10 RAG prompt rendering error due to undefined variables

Problem

Grafana 10’s new LLM integration allows panels to generate answers using Retrieval‑Augmented Generation (RAG). In our on‑premises deployment the RAG prompt template fails to compile, producing errors such as:

[2024-10-12 03:14:27] error   plugin.rag   Error rendering RAG prompt: variable "question" is undefined
[2024-10-12 03:14:27] error   plugin.rag   Template compilation failed: unexpected token "}}" at line 4

The failure prevents any panel that relies on the prompt template from returning a response, effectively breaking the LLM feature.

Root Cause

The RAG prompt engine uses Go’s text/template parser. During compilation Grafana substitutes variables supplied by the panel request (e.g., {{ .question }}, {{ .context }}). The observed error occurs when:

• One or more variables referenced in the template are not present in the PromptData struct passed to the engine.
• Custom datasource plugins inject additional fields but forget to register them in the PromptVariables map.
• The template contains stray delimiters (e.g., an extra }}) that cause the parser to treat a valid variable name as undefined.

In our environment the custom datasource my-llm-datasource added a metadata field but did not update the PromptVariables registration, leading to the “undefined variable” error for {{ .metadata }}. Additionally, a copy‑paste mistake introduced a stray closing brace in the shared prompt template.

Debug

1. Collect the failing request logs

$ journalctl -u grafana-server -f | grep "RAG prompt"
Oct 12 03:14:27 grafana-server plugin.rag: Error rendering RAG prompt: variable "question" is undefined
Oct 12 03:14:27 grafana-server plugin.rag: Template compilation failed: unexpected token "}}" at line 4

2. Inspect the compiled template payload

Enable verbose template logging in grafana.ini:

[plugin.rag]
log_template = true

After a restart, the log shows the raw template received from the panel:

2024-10-12T03:14:27Z plugin.rag: Received template:
{{ if .question }}
You are asked: "{{ .question }}"
{{ end }}
{{ .context }}
{{ .metadata }}
}}

3. Verify the variable map sent by the datasource

Intercept the HTTP request from the panel to the datasource using tcpdump and decode the JSON payload:

$ sudo tcpdump -i any -s 0 -w /tmp/grafana.pcap port 3000
$ tshark -r /tmp/grafana.pcap -Y http.request -T json

Extracted JSON snippet:

{
  "variables": {
    "question": "What is the SLA?",
    "context": "Service level agreements for XYZ."
    // Note: "metadata" missing
  }
}

4. Reproduce the failure locally

Run the template engine in a Go REPL:

package main

import (
    "bytes"
    "text/template"
    "log"
)

func main() {
    tmpl := `{{ if .question }}You are asked: "{{ .question }}"{{ end }}{{ .context }}{{ .metadata }}}`
    data := map[string]string{
        "question": "What is the SLA?",
        "context":  "Service level agreements for XYZ.",
    }
    t, err := template.New("rag").Parse(tmpl)
    if err != nil {
        log.Fatalf("compile error: %v", err)
    }
    var out bytes.Buffer
    if err := t.Execute(&out, data); err != nil {
        log.Fatalf("execution error: %v", err)
    }
}

Output:

2024/10/12 03:20:11 compile error: template: rag:1: unexpected "}" in operand

The error confirms both the missing variable and the stray brace.

Solution

1. Add missing variable registration in the datasource

Update my-llm-datasource to include metadata in the PromptVariables map.

Before (datasource.go)

func (d *Datasource) PromptVariables() map[string]interface{} {
    return map[string]interface{}{
        "question": d.Query.Question,
        "context":  d.Query.Context,
        // metadata omitted
    }
}

After

func (d *Datasource) PromptVariables() map[string]interface{} {
    return map[string]interface{}{
        "question":  d.Query.Question,
        "context":   d.Query.Context,
        "metadata":  d.Query.Metadata, // newly added
    }
}

2. Fix the stray delimiter in the shared prompt template

The template file /etc/grafana/rag_prompt.tmpl contained an extra closing brace.

Before

{{ if .question }}
You are asked: "{{ .question }}"
{{ end }}
{{ .context }}
{{ .metadata }}
}}

After

{{ if .question }}
You are asked: "{{ .question }}"
{{ end }}
{{ .context }}
{{ .metadata }}

3. Reload the datasource plugin

# Restart Grafana to pick up the updated plugin binary
systemctl restart grafana-server

4. Verify template compilation succeeds

With log_template = true enabled, the log now shows:

2024-10-12T03:25:10Z plugin.rag: Compiled template successfully
2024-10-12T03:25:10Z plugin.rag: Prompt rendered:
You are asked: "What is the SLA?"
Service level agreements for XYZ.
Additional metadata: {"source":"api"}

Verify

1. Open the affected dashboard panel and trigger a query.
2. Confirm the panel displays the generated answer instead of an error.
3. Check the Grafana logs for the absence of “undefined variable” or “unexpected token” messages.
4. Run a health‑check endpoint for the LLM integration:

curl -s http://localhost:3000/api/plugins/rag/health | jq .
{
  "status": "ok",
  "details": {
    "templateCompilation": "success"
  }
}

If the JSON reports "status":"ok", the fix is verified.

Prevent

• Schema validation: Add a unit test in the datasource repository that marshals a full PromptVariables map and asserts that every variable referenced in rag_prompt.tmpl exists.
• Template linting: Integrate golang.org/x/tools/go/analysis/passes/templatecheck into the CI pipeline to catch stray delimiters and undefined variables before deployment.
• Runtime guardrails: Enable log_template = true in staging environments and configure an alert on log patterns “Error rendering RAG prompt”.
• Version pinning: Keep Grafana 10.x patched to the latest minor release; later releases include stricter validation of prompt templates.
• Documentation: Maintain a canonical list of allowed variables in docs/rag_variables.md and reference it in plugin READMEs.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the error only appear after adding a new field to the datasource? The LLM engine compiles the template at request time using the variable map supplied by the datasource. Adding a field without exposing it in PromptVariables leaves the template referencing a name that does not exist, causing the “undefined variable” error.
2. Can I disable variable validation to ignore missing fields? No. The Go text/template engine treats missing keys as an execution error. Silencing it would mask genuine bugs and produce incomplete prompts.
3. Is there a way to see which variables a template expects? Use the template.Parse function with template.Option("missingkey=error") and log the parsed AST. Grafana’s debug mode also prints the raw template before compilation.
4. Do I need to restart Grafana after fixing the template file? Yes. Template files are read once during plugin initialization. A full plugin reload (or Grafana service restart) is required for changes to take effect.
5. How do I test the prompt rendering without a full panel? Execute the same Go snippet used in the debugging section, passing a JSON payload that mimics the datasource’s PromptVariables. This isolates the template engine from Grafana UI components.