RAG Answer Extraction Inconsistent Between Blue And Green Deployments

Q: Problem: Inconsistent RAG Answer Extraction Between Blue and Green Deployments

In a production RAG pipeline accelerated with NVIDIA TensorRT, the same user query yields different answer strings when routed to the blue deployment versus the green deployment. The discrepancy appears intermittently during traffic shifts and can cause downstream ranking failures, SLA breaches, and user‑visible errors.

Q: Root Cause Analysis Determinism Guarantees in TensorRT

According to the TensorRT Developer Guide – Engine Serialization and Deserialization, identical models can produce divergent results if the engines are built with different optimization profiles or precision flags (FP32 vs FP16 vs INT8). The serialization format stores the chosen profile, but any missing calibration cache forces a rebuild with default precision, breaking determinism.

Q: Blue‑Green Deployment Differences Identified Precision mode mismatch: The blue slot used an engine built with INT8 (calibrated cache present), while the green slot fell back to FP16 because the TRT_ALLOW_GPU_FALLBACK=1 environment variable was set only in green. CUDA/cuDNN version drift: The green environment ran on a newer cuDNN version (8.9) whereas blue still used 8.7, altering the softmax implementation as documented in the CUDA Toolkit Compatibility Matrix. This change manifested as slight numerical differences that shifted token ranking. Missing timing cache: The serialized engine file was copied to both slots, but the accompanying timing.cache was omitted for green. TensorRT regenerated the engine on‑fly with a different max_batch_size, leading to the “Binding dimension mismatch for input 0” warning. GPU architecture variance: CUDA_VISIBLE_DEVICES differed, causing green to run on a V100 while blue used an A100. The differing FP16 arithmetic units produced non‑identical rounding behavior, a known source of answer drift (NVIDIA Developer Forums thread). These factors collectively break the deterministic inference guarantee required for RAG answer extraction, explaining why identical inputs diverge across deployments. Investigation and Debugging Steps 1. Verify Engine Files and Metadata

# List engine files and timestamps ls -l /models/rag_t5/1/ -rw-r--r-- 1 triton triton 1.2G Jul 5 13:45 rag_t5_int8.trt -rw-r--r-- 1 triton triton 1.2G Jul 5 13:45 rag_t5_fp16.trt Check that both slots reference the same engine path in config.pbtxt:

Q: 2. Inspect Environment Variables

# In blue container env | grep TRT TRT_LOGGING_LEVEL=INFO TRT_ALLOW_GPU_FALLBACK=0 # In green container env | grep TRT TRT_LOGGING_LEVEL=INFO TRT_ALLOW_GPU_FALLBACK=1 3. Compare CUDA/cuDNN Versions # Blue nvidia-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 | +-----------------------------------------------------------------------------+ cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR #define CUDNN_MAJOR 8 #define CUDNN_MINOR 7 # Green nvidia-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 | +-----------------------------------------------------------------------------+ cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR #define CUDNN_MAJOR 8 #define CUDNN_MINOR 9 4. Capture Tensor Outputs Use TensorRT Runtime API calls to dump the raw logits before softmax.

In this article

Table of Contents

Problem: Inconsistent RAG Answer Extraction Between Blue and Green Deployments

In a production RAG pipeline accelerated with NVIDIA TensorRT, the same user query yields different answer strings when routed to the blue deployment versus the green deployment. The discrepancy appears intermittently during traffic shifts and can cause downstream ranking failures, SLA breaches, and user‑visible errors.

Typical symptoms observed in logs:


[2026-07-05 14:23:11] ERROR tritonserver: Model "rag_t5" version 1: Engine deserialization failed: "Failed to deserialize engine from file"
[2026-07-05 14:23:12] WARN  tritonserver: Binding dimension mismatch for input 0 (expected [1,128] got [1,64])
[2026-07-05 14:23:13] ERROR tritonserver: Unexpected NaN in output tensor (layer: softmax)
[2026-07-05 14:23:14] INFO  traffic‑router: Routing request id=7f9c to GREEN slot
[2026-07-05 14:23:14] INFO  rag_t5: Generated answer: "Paris is the capital of France."
[2026-07-05 14:23:15] INFO  traffic‑router: Routing request id=9ab2 to BLUE slot
[2026-07-05 14:23:15] INFO  rag_t5: Generated answer: "The capital of France is Paris."

Although the answers are semantically equivalent, the token probabilities differ enough that downstream ranking or confidence thresholds treat them as failures. The issue persists even after confirming that the model weights, tokenizer, and retrieval index are identical across slots.

Root Cause Analysis

Determinism Guarantees in TensorRT

According to the TensorRT Developer Guide – Engine Serialization and Deserialization, identical models can produce divergent results if the engines are built with different optimization profiles or precision flags (FP32 vs FP16 vs INT8). The serialization format stores the chosen profile, but any missing calibration cache forces a rebuild with default precision, breaking determinism.

Blue‑Green Deployment Differences Identified

Precision mode mismatch: The blue slot used an engine built with INT8 (calibrated cache present), while the green slot fell back to FP16 because the TRT_ALLOW_GPU_FALLBACK=1 environment variable was set only in green.
CUDA/cuDNN version drift: The green environment ran on a newer cuDNN version (8.9) whereas blue still used 8.7, altering the softmax implementation as documented in the CUDA Toolkit Compatibility Matrix. This change manifested as slight numerical differences that shifted token ranking.
Missing timing cache: The serialized engine file was copied to both slots, but the accompanying timing.cache was omitted for green. TensorRT regenerated the engine on‑fly with a different max_batch_size, leading to the “Binding dimension mismatch for input 0” warning.
GPU architecture variance: CUDA_VISIBLE_DEVICES differed, causing green to run on a V100 while blue used an A100. The differing FP16 arithmetic units produced non‑identical rounding behavior, a known source of answer drift (NVIDIA Developer Forums thread).

These factors collectively break the deterministic inference guarantee required for RAG answer extraction, explaining why identical inputs diverge across deployments.

Investigation and Debugging Steps

1. Verify Engine Files and Metadata


# List engine files and timestamps
ls -l /models/rag_t5/1/
-rw-r--r-- 1 triton triton  1.2G Jul  5 13:45 rag_t5_int8.trt
-rw-r--r-- 1 triton triton  1.2G Jul  5 13:45 rag_t5_fp16.trt

Check that both slots reference the same engine path in config.pbtxt:


name: "rag_t5"
platform: "tensorrt_plan"
max_batch_size: 1
default_model_filename: "rag_t5_int8.trt"

2. Inspect Environment Variables


# In blue container
env | grep TRT
TRT_LOGGING_LEVEL=INFO
TRT_ALLOW_GPU_FALLBACK=0

# In green container
env | grep TRT
TRT_LOGGING_LEVEL=INFO
TRT_ALLOW_GPU_FALLBACK=1

3. Compare CUDA/cuDNN Versions


# Blue
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 7

# Green
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 9

4. Capture Tensor Outputs

Use TensorRT Runtime API calls to dump the raw logits before softmax.


# C++ snippet
auto output = context->getBindingDimensions(output_index);
float *logits;
cudaMalloc(&logits, output_size * sizeof(float));
context->enqueueV2(bindings, stream, nullptr);
cudaMemcpy(logits, device_output, output_size * sizeof(float), cudaMemcpyDeviceToHost);
for (int i = 0; i < 10; ++i) {
    printf("%d: %.8f\n", i, logits[i]);
}

Differences in the first few logits between blue and green confirm non‑deterministic behavior.

5. Validate Engine Build Flags


# In each slot, query the engine builder settings
trtexec --loadEngine=/models/rag_t5/1/rag_t5_int8.trt --verbose | grep "Precision"
[INFO] Using INT8 precision
[INFO] Using FP16 precision

6. Check for Missing Timing Cache


# Expected location
ls /models/rag_t5/1/timing.cache
-rw-r--r-- 1 triton triton  256K Jul  5 13:44 timing.cache
# Green slot output
[2026-07-05 14:23:12] WARN  tritonserver: Timing cache not found, rebuilding engine.

Resolution: Achieving Deterministic RAG Inference Across Deployments

Step‑by‑Step Fix

Standardize Engine Build Configuration
- Generate a single engine with the desired precision (e.g., INT8) and store both the .trt file and its calibration cache in a shared artifact repository.
- Disable fallback to avoid accidental FP16 builds.
Synchronize CUDA/cuDNN Versions
- Update the green container Dockerfile to match the blue’s cuDNN 8.7 version.
- Pin the nvidia/cuda base image tag (e.g., nvidia/cuda:12.0.1-cudnn8-runtime-ubuntu22.04).
Export and Deploy the Timing Cache
- Copy timing.cache alongside the engine file to both slots.
- Set TRT_TIMING_CACHE_FILE=/models/rag_t5/1/timing.cache in both environments.
Align Environment Variables
- Remove TRT_ALLOW_GPU_FALLBACK or set it to 0 in both slots.
- Ensure CUDA_VISIBLE_DEVICES points to GPUs of the same architecture, or add TRT_FORCE_TENSORRT_GPU_ARCH=ampere if mixed GPUs are unavoidable.
Re‑deploy with Blue‑Green Consistency Checks
- Use Triton’s model versioning to load the same engine file path for both slots.
- Run a sanity test suite (see Validation section) before traffic cut‑over.

Before / After Comparison

Aspect	Before	After
Engine precision	Blue: INT8 (calibrated) Green: FP16 (fallback)	Both: INT8 (single calibrated engine)
CUDA/cuDNN	Blue: cuDNN 8.7 Green: cuDNN 8.9	Both: cuDNN 8.7
Timing cache	Blue: present Green: missing (rebuild)	Both: present
Environment vars	Blue: `TRT_ALLOW_GPU_FALLBACK=0` Green: `TRT_ALLOW_GPU_FALLBACK=1`	Both: `TRT_ALLOW_GPU_FALLBACK=0`

Verification: Confirming Deterministic Behavior

Run an identical query batch against both slots and compare raw logits.
Expect identical floating‑point values up to the last decimal place.
Check Triton logs for “Engine deserialization succeeded” and absence of “Unexpected NaN” warnings.

Automated health‑check endpoint:


curl -X POST http://blue.example.com/v2/models/rag_t5/infer -d @sample_request.json
curl -X POST http://green.example.com/v2/models/rag_t5/infer -d @sample_request.json
diff <(jq -r .outputs[0].data <(curl ...blue...)) <(jq -r .outputs[0].data <(curl ...green...))

Returns no differences.

Monitor the model_inference_success metric in Prometheus for both slots; they should report identical latency and success rates.

Prevention and Best Practices

Immutable Engine Artifacts: Store the serialized engine, calibration cache, and timing cache in a version‑controlled artifact store (e.g., S3 with SHA256 checksum) and reference them via immutable URLs in config.pbtxt.
Determinism Flag: Set TRT_FORCE_DETERMINISTIC=1 in both containers to enforce deterministic kernels where supported.
Unified Runtime Stack: Use a single base Docker image for all deployment slots to guarantee identical CUDA, cuDNN, and TensorRT versions.
Continuous Validation: Integrate a nightly “golden query” test that asserts byte‑identical model outputs across all slots.
Blue‑Green Guardrails: Before traffic switch, run a tritonserver --model-repository … --strict-model-config check that validates engine compatibility and prints warnings for any mismatched profiles.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the same query sometimes produce different token ordering even though the probabilities are close?
Because TensorRT’s FP16/INT8 kernels introduce rounding variance. When the softmax input differs by 1e‑5, the argmax can flip, causing a different token order. Enforcing identical precision eliminates this.
Can I safely enable TRT_ALLOW_GPU_FALLBACK in production?
It is useful for development but in production it can cause the engine to rebuild with a lower‑precision profile, breaking determinism. Disable it or set it consistently across all slots.
Do I need to rebuild the engine after a CUDA driver upgrade?
Yes. Engine serialization is tied to the TensorRT library version and the underlying driver. A driver upgrade may change kernel implementations (e.g., softmax), so rebuild with the same version to keep outputs stable.
How can I verify that the timing cache is being used?
Check Triton logs for “Loading timing cache from …” and confirm the file path matches the one you deployed. Absence of this message indicates a rebuild.
Is it possible to achieve deterministic inference on different GPU architectures?
Determinism across architectures is not guaranteed due to differing floating‑point units. The safest approach is to run all blue‑green slots on the same GPU family or enable TRT_FORCE_TENSORRT_GPU_ARCH to force a common code path.