Prometheus nightly batch aborts due to RAG prompt placeholder not substituted

Problem – Nightly RAG Batch Jobs Abort with Unsubstituted Prompt Placeholders

The nightly batch pipeline that drives large‑scale Retrieval‑Augmented Generation (RAG) for reporting is failing. Each run aborts with a non‑zero exit code and the generated CSV contains malformed or empty fields. The primary symptom is a rendering error where template variables such as {{user_query}}, {{context}}, or {{document_id}} are not substituted, producing prompts that the RAG API rejects.

Typical log excerpt (from Prometheus Logging Specification – “Template Rendering Errors”):

2024-03-15T02:12:03Z ERROR TemplateRenderError: missing variable 'user_query' in prompt template (job_id=4721)
2024-03-15T02:12:03Z ERROR TemplateRenderError: missing variable 'user_query' in prompt template (job_id=4722)
...
Batch terminated with exit code 1 after processing 4,800 jobs

Other observed errors include:

• KeyError: ‘document_id’
• Jinja2.UndefinedError: ‘retrieval_context’ is undefined
• SyntaxError: unexpected token ‘<‘ in generated prompt (empty placeholder rendered as raw markup)

Root Cause – Mismatch Between Template Definitions and Runtime Variable Supply

Prometheus uses Jinja2‑style templates for RAG prompts (Template Variables and Substitution). During batch execution the engine builds a context map from the job definition and passes it to the rendering engine. The failures trace back to three common root causes:

1. Schema drift – A recent schema change renamed the placeholder {{context}} to {{retrieval_context}}. The nightly job configuration was not updated, so the rendering engine receives no value for the original name, resulting in Jinja2.UndefinedError. (See incident on 2024‑02‑28.)
2. Model version upgrade – Upgrading the RAG model introduced a new required variable document_id. The batch job definition still supplies the old variable set, causing KeyError: 'document_id' across all prompts (2024‑01‑11 outage).
3. Batch‑mode variable loading bug – In batch mode the engine loads variables from a cached JSON file. A regression introduced in version v2.3.1 prevented the cache from being refreshed after a deployment, so placeholders remained undefined. This is documented in GitHub issue #3890.

All three scenarios violate the contract described in the official Prompt Template Guide, which requires that every placeholder referenced in the template be present in the rendering context.

Debug – Systematic Investigation Steps

1. Reproduce the failure locally

# Clone the nightly job definition
git clone https://github.com/company/ai-reporting.git
cd ai-reporting

# Run a single job in batch mode with verbose logging
PROMETHEUS_LOG_LEVEL=debug prometheus-rag-batch \
  --job-id 1 \
  --template ./templates/report_prompt.j2 \
  --variables ./samples/variables_20240315.json \
  --dry-run

Expected output: rendered prompt printed to stdout. Observed output shows placeholders unchanged:

Prompt:
"User asked: {{user_query}}
Context: {{retrieval_context}}
Document ID: {{document_id}}"

2. Inspect the variable payload

cat ./samples/variables_20240315.json | jq .

{
  "user_query": "What were the sales numbers for Q1?",
  "context": "Sales data for Q1 2024..."
  // note: missing "retrieval_context" and "document_id"
}

3. Compare template vs. expected variables (official spec)

Template Variable	Required by Template	Provided in Payload
{{user_query}}	Yes	Yes
{{retrieval_context}}	Yes (renamed from {{context}})	No
{{document_id}}	Yes (new in v2.3.1)	No

4. Check the batch job configuration version

grep -A2 "template_version" ./batch/nightly_job.yaml

template_version: "v2.2.0"
# The job still references the old schema

5. Review recent changelogs

Prometheus release notes for v2.3.0 introduced retrieval_context and made document_id mandatory (Nightly Batch Job Configuration). The nightly job had not been migrated.

Solution – Align Templates, Variable Payloads, and Engine Version

Step 1 – Update the prompt template to use the current variable names

Before:

templates/report_prompt.j2
User asked: {{user_query}}
Context: {{context}}
Document ID: {{document_id}}

After (aligned with v2.3.1 spec):

templates/report_prompt.j2
User asked: {{user_query}}
Context: {{retrieval_context}}
Document ID: {{document_id}}

Step 2 – Extend the variable generation script to include the new fields

Original snippet (Python):

payload = {
    "user_query": query,
    "context": fetch_context(query)
}

Updated snippet:

payload = {
    "user_query": query,
    "retrieval_context": fetch_context(query),
    "document_id": fetch_document_id(query)
}

Step 3 – Bump the nightly job definition to the latest template version

# batch/nightly_job.yaml
template_version: "v2.3.1"
template_path: "./templates/report_prompt.j2"
variables_path: "./generated/variables_{{date}}.json"

Step 4 – Clear the stale variable cache introduced by the regression

# Remove cached JSON used by batch mode
rm -rf /var/lib/prometheus/rag/cache/*.json
# Restart the batch daemon to force a fresh load
systemctl restart prometheus-rag-batch.service

Step 5 – Run a smoke test for a single job

PROMETHEUS_LOG_LEVEL=info prometheus-rag-batch \
  --job-id 42 \
  --dry-run

Expected log excerpt:

2024-08-25T01:45:12Z INFO Prompt rendered successfully for job_id=42

Verification – Confirm the Fix in Production

1. Health check: Verify that the batch service reports status: ready after restart.
2. Log inspection: Grep for any remaining TemplateRenderError entries.

   journalctl -u prometheus-rag-batch.service | grep TemplateRenderError || echo "No errors"

3. Sample output validation: Pull the first generated CSV row and ensure all required fields are populated.

   head -n 1 /data/rag_outputs/2024-08-25/report.csv

   Expected columns: user_query, retrieval_context, document_id, answer – none empty.

4. Metrics: Confirm that the rag_prompt_render_success_total counter increased and rag_prompt_render_failure_total is zero for the night.

   curl -s http://localhost:9090/metrics | grep rag_prompt_render_

Prevention – Guardrails to Avoid Future Placeholder Mismatches

• Schema validation step – Add a pre‑run CI job that parses the template with a dummy context and fails if any UndefinedError is raised.
• Version pinning – Store the required template version in the job manifest and enforce it via a Git hook that blocks commits when the version is out of sync with the engine.
• Automated variable diff – Generate a JSON schema from the template (prometheus-rag-template-schema --output schema.json) and compare it against the variable payload using jsonschema before batch execution.
• Monitoring alerts – Create a Prometheus alert on rag_prompt_render_failure_total with a threshold of >0 within a 5‑minute window.
• Cache invalidation policy – Schedule a daily purge of the rendering cache (cron: 0 3 * * *) to avoid stale variable maps after deployments.

FAQ – Common Follow‑Up Questions

• Why does the same template work in interactive mode but fail in batch?
  Batch mode loads variables from a persisted cache that may not be refreshed after schema changes. Interactive mode builds the context on‑the‑fly, so it sees the latest variables.
• How can I programmatically list all placeholders required by a template?
  Use the built‑in CLI: prometheus-rag-template-inspect --list-placeholders ./templates/report_prompt.j2. It outputs a JSON array of variable names.
• Is there a way to fallback to a default value when a placeholder is missing?
  Jinja2 supports the default filter: {{ retrieval_context | default('N/A') }}. However, for required fields the API will still reject empty strings, so the preferred approach is to guarantee the variable exists.
• Can I disable the cache if I’m troubleshooting?
  Set the environment variable PROMETHEUS_RAG_DISABLE_CACHE=1 before running the batch daemon. This forces a fresh load for every job.
• What metric should I watch to detect a regression in placeholder substitution?
  Monitor rag_prompt_render_failure_total and set an alert on any increase compared to the baseline of the previous night.

Related Topic Hub: Observability Troubleshooting Hub