TensorRT RAG Prompt Rendering Error After Air‑Gapped Deployment
Problem
In an on‑premise, air‑gapped environment, executing a Retrieval‑Augmented Generation (RAG) workflow with TensorRT raises a rendering exception when the prompt template is applied. The inference job aborts with a stack trace similar to:
Traceback (most recent call last):
File "/opt/tensorrt/rag_engine.py", line 212, in run
rendered = self.prompt_template.render(context)
File "/opt/tensorrt/template_engine.py", line 87, in render
raise TemplateRenderError("Missing required variable: {{retrieved_text}}")
TemplateRenderError: Missing required variable: {{retrieved_text}}
Symptoms observed:
- Inference latency spikes before failure.
- No network traffic to external services (as expected in an air‑gapped setup).
- Only the first request in a batch succeeds; subsequent requests hit the rendering error.
Root Cause
The failure originates from the template engine used by the RAG component. In a connected environment the engine lazily loads auxiliary Jinja2 extensions and a small JSON manifest (templates_manifest.json) that maps template identifiers to required variables. During the air‑gapped deployment the manifest file was omitted, and the Jinja2 package was installed without the optional jinja2‑sandbox extension. Consequently:
- Missing manifest: The engine cannot resolve the
{{retrieved_text}}placeholder because the manifest that declares it as a required variable is absent. - Incompatible Jinja2 version: The bundled TensorRT container ships with Jinja2 3.1.2, but the RAG code expects 2.11 APIs for sandboxed rendering. The API mismatch silently disables the sandbox, allowing untrusted template execution but breaking variable substitution.
- Lazy loading path: The code constructs the template path using
os.getenv("TEMPLATE_ROOT"). In the air‑gapped image this environment variable is unset, causing the engine to fall back to/tmp/templates, a directory that does not contain the pre‑loaded assets.
These three issues combine to produce the observed TemplateRenderError.
Debug & Investigation
Step‑by‑step diagnostics that reproduced the failure:
- Inspect container filesystem for template assets:
docker exec -it rag_container ls -R /opt/tensorrt/templatesOutput:
/opt/tensorrt/templates: default_prompt.j2The manifest file is missing.
- Check Jinja2 version:
python -c "import jinja2, sys; print(jinja2.__version__)"Output:
3.1.2The RAG code imports
jinja2.sandbox.SandboxedEnvironment, which was removed in 3.0. - Verify environment variable:
echo $TEMPLATE_ROOTOutput is empty, confirming the fallback path.
- Enable verbose template logging by setting
RAG_DEBUG=1and re‑run a request. The log shows:[RAG][DEBUG] Loading template from /tmp/templates/default_prompt.j2The file does not exist, leading to a fallback to an empty template string.
Solution
The fix consists of three coordinated changes:
1. Include the manifest and all template assets in the container image
Update the Dockerfile to copy templates/ and templates_manifest.json into the runtime path and set TEMPLATE_ROOT.
# Before
COPY rag_engine.py /opt/tensorrt/
# After
COPY rag_engine.py /opt/tensorrt/
COPY templates/ /opt/tensorrt/templates/
COPY templates_manifest.json /opt/tensorrt/
ENV TEMPLATE_ROOT=/opt/tensorrt/templates
2. Pin a compatible Jinja2 version and install the sandbox extension
Modify the requirements.txt used during build:
# Before
jinja2>=3.0
# After
jinja2==2.11.3
jinja2-sandbox==0.1.0
Rebuild the image so the RAG code can import jinja2.sandbox.SandboxedEnvironment without error.
3. Guard against missing environment variables in code
Patch template_engine.py to provide a sensible default and raise a clear error if the directory is empty.
# Before
template_path = os.getenv("TEMPLATE_ROOT") + "/" + template_name
# After
default_root = "/opt/tensorrt/templates"
template_root = os.getenv("TEMPLATE_ROOT", default_root)
template_path = os.path.join(template_root, template_name)
if not os.path.isdir(template_root):
raise RuntimeError(f"Template root not found: {template_root}")
4. Validate the manifest at startup
Add a sanity check in rag_engine.py:
# After
manifest_path = os.path.join(template_root, "templates_manifest.json")
if not os.path.isfile(manifest_path):
raise FileNotFoundError(f"Missing manifest: {manifest_path}")
with open(manifest_path) as f:
self.manifest = json.load(f)
Verification
After applying the three changes, the following validation steps confirm success:
- Container inspection:
docker exec -it rag_container ls -R /opt/tensorrt/templatesOutput includes
default_prompt.j2andtemplates_manifest.json. - Jinja2 version check:
python -c "import jinja2; print(jinja2.__version__)"Output:
2.11.3 - Run a single RAG request and capture logs:
curl -X POST http://localhost:8000/rag -d '{"query":"What is TensorRT?"}' -H "Content-Type: application/json"Log excerpt:
[RAG][INFO] Rendering template default_prompt.j2 [RAG][INFO] Rendered prompt length: 342 characters [TensorRT][INFO] Inference completed in 45 msNo
TemplateRenderErrorappears. - Batch test (10 parallel requests) – all succeed, latency remains stable.
Prevention & Operational Guardrails
- CI validation: Add a unit test that loads every template defined in
templates_manifest.jsonand asserts that required variables are present. - Image build checklist: Verify that
TEMPLATE_ROOTis set and that the manifest file exists before publishing the container. - Dependency lockfile: Keep a
requirements.lockwith exact versions of Jinja2 and its extensions to avoid accidental upgrades. - Runtime health endpoint: Expose
/health/templatesthat returns HTTP 200 only if the manifest and all template files are readable. - Monitoring: Alert on log pattern
TemplateRenderErroror on missing manifest detection at startup.
FAQ
- Why does the error appear only after moving to an air‑gapped server?
The manifest and Jinja2 sandbox extension are normally pulled from an internal PyPI mirror during CI builds. In the isolated environment those artifacts were not bundled, so the runtime lacked the required resources. - Can I keep Jinja2 3.x and still use sandboxed rendering?
Jinja2 3.x removedjinja2.sandbox. To use sandboxing you must stay on the 2.11 series or refactor the engine to usejinja2.Environmentwith custom filters and a strict undefined policy. - Is setting
TEMPLATE_ROOTenough to fix the issue?
It resolves the path problem but does not address the missing manifest or incompatible Jinja2 version. All three components must be aligned. - How can I test that the manifest matches the templates before deployment?
Run the providedvalidate_templates.pyscript which parses the manifest, loads each Jinja2 file, and asserts that all{{variable}}placeholders listed in the manifest are present. - What should I do if a new template is added after the image is built?
Update the manifest, rebuild the container, and redeploy. For truly dynamic templates consider mounting a read‑only volume with the template directory and adding a hot‑reload flag in the engine.
Related Topic Hub: Model Serving Troubleshooting Hub