img IX mining rig inside white and gray room

ONNX Runtime hybrid search scoring inconsistencies on GPU

by

Problem – HybridSearch Scoring Inconsistencies on GPU

In a development sandbox (Ubuntu 22.04, NVIDIA A10G, ONNX Runtime 1.18, PyTorch backend) a BERT‑based reranker that uses the HybridSearch operator returns different top‑k results when the model is executed on the CPU (FP32) versus the GPU (FP16 mixed‑precision). Typical symptoms include:

  • Top‑5 candidate order differs by up to three positions.
  • Score drift up to 0.0045 between CPU and GPU runs.
  • Intermittent NaN values in the similarity matrix when ORT_ENABLE_FP16=1 is set.
  • GPU runs occasionally abort with CUDA_ERROR_ILLEGAL_ADDRESS for batch sizes > 128.

Example log excerpt from a failing GPU run:

[2026-07-03 12:15:32] ERROR: Result mismatch: expected score 0.8234, got 0.8190
[2026-07-03 12:15:33] WARNING: Tensor shape mismatch between CPU and CUDA providers
[2026-07-03 12:15:34] CUDA_ERROR_ILLEGAL_ADDRESS: kernel HybridSearchReduction launched with invalid memory access

Root Cause Analysis

Precision handling in CUDA kernels

The CUDA Execution Provider (EP) uses FP16 kernels for matrix multiplication and dot‑product operations when ORT_ENABLE_FP16=1 is enabled (ONNX Runtime Mixed Precision guide). FP16 arithmetic introduces rounding errors that accumulate in large similarity matrices, especially when the model performs vector · vector dot‑products followed by a scalar bias addition inside HybridSearch. The official CUDA EP documentation notes that “precision‑dependent kernels may produce results that differ from FP32 within a tolerance defined by the underlying hardware”. In practice the tolerance is larger than the 0.001 – 0.005 range observed, causing ranking changes.

Custom CUDA kernel bug

Release notes for ONNX Runtime 1.18 (GitHub release) list a fix for an “uninitialized workspace buffer in the HybridSearch CUDA kernel” that could produce NaN values. In the sandbox the environment variable ORT_ENABLE_FP16=1 triggers that kernel path, leading to the observed NaNs.

TensorRT fallback side‑effect

When the TensorRT EP is present, the final reduction step of HybridSearch may silently fall back to the CPU provider if the TensorRT kernel cannot handle the mixed‑precision tensor shape. This hybrid execution path yields a mixture of FP16 (GPU) and FP32 (CPU) results, as reported in the GitHub issue “Hybrid search scoring mismatch between CPU and CUDA providers” (#15892). The mixed‑precision reduction therefore diverges from the pure‑GPU path.

Memory‑access violation for large batches

For batch sizes larger than 128 the CUDA kernel attempts to allocate a temporary buffer whose size exceeds the default workspace limit, resulting in CUDA_ERROR_ILLEGAL_ADDRESS. The CPU fallback then processes a different number of vectors, changing the ranking order.

Investigation and Debugging Steps

  1. Reproduce with deterministic seed – set torch.manual_seed(42) and ensure the same input tensors are fed to both providers.
  2. Collect provider‑specific logs – enable verbose logging for the CUDA EP: 
    export ORT_LOG_VERBOSITY=3
export ORT_CUDA_LOG_VERBOSITY=3
  3. Compare raw scores – dump the output of the HybridSearch operator for CPU and GPU: 
    # Python snippet
import onnxruntime as ort
sess_cpu = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
sess_gpu = ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider"])
scores_cpu = sess_cpu.run(None, {"input": input_tensor})[0]
scores_gpu = sess_gpu.run(None, {"input": input_tensor})[0]
print("CPU scores:", scores_cpu[:10])
print("GPU scores:", scores_gpu[:10])
  4. Inspect NaNs and out‑of‑range values – use numpy.isnan and numpy.isfinite on the GPU output.
  5. Force FP32 on GPU – disable mixed precision: 
    export ORT_ENABLE_FP16=0

    and rerun the inference. Matching scores indicate the issue is FP16‑related.

  6. Disable TensorRT EP – start the session without TensorRT: 
    sess_gpu = ort.InferenceSession(
    "model.onnx",
    providers=[("CUDAExecutionProvider", {"device_id": 0, "trt_engine_cache_enable": False})]
)

    Observe whether the fallback disappears.

  7. Reduce batch size – run with batch_size=64 to confirm the illegal address error is batch‑size dependent.
  8. Check workspace size – query the CUDA EP for the allocated workspace: 
    ort.get_device_allocator().GetCurrentMemoryInfo()

    Increase the limit if necessary:

    export ORT_CUDA_MAX_WORKSPACE_SIZE=4294967296  # 4 GiB

Resolution – Aligning CPU and GPU Scoring

Option 1: Disable FP16 for HybridSearch

When exact reproducibility is required, keep the CUDA EP in FP32 mode for the operators involved in similarity computation.

Before (mixed‑precision enabled):

export ORT_ENABLE_FP16=1
export ORT_CUDA_FP16_ENABLE=1

After (force FP32 on the relevant kernels):

# Disable global FP16
export ORT_ENABLE_FP16=0

# Alternatively, override per‑operator precision via session options
import onnxruntime as ort
so = ort.SessionOptions()
so.add_session_config_entry("session.enable_fp16", "0")
sess = ort.InferenceSession("model.onnx", sess_options=so, providers=["CUDAExecutionProvider"])

Why it works: The CUDA kernels fall back to the FP32 implementation, eliminating rounding‑induced drift and the uninitialized FP16 workspace bug.

Option 2: Pin the HybridSearch operator to the CPU EP

If GPU throughput is still desired for the bulk of the model, isolate the scoring step to the CPU:

import onnxruntime as ort
# Split the graph: run feature extraction on GPU, then HybridSearch on CPU
gpu_ep = ("CUDAExecutionProvider", {"device_id": 0})
cpu_ep = ("CPUExecutionProvider", {})
sess = ort.InferenceSession("model.onnx", providers=[gpu_ep, cpu_ep])
# Use execution provider mapping to force HybridSearch to CPU
sess.set_providers([gpu_ep, cpu_ep], ["CUDAExecutionProvider", "CPUExecutionProvider"])

This avoids the buggy FP16 kernel while preserving GPU acceleration for earlier layers.

Option 3: Upgrade to ONNX Runtime 1.19‑rc (if available)

The 1.19 release candidate contains a fix for the uninitialized workspace buffer (GitHub release notes). Upgrading eliminates the NaN issue without sacrificing FP16 speed.

Verification – Confirming Consistent Scores

  1. Run the same deterministic input through both providers after applying the chosen fix.
  2. Compute the maximum absolute difference: 
    import numpy as np
max_diff = np.max(np.abs(scores_cpu - scores_gpu))
print("Maximum absolute difference:", max_diff)

    Expected max_diff < 1e-6 for FP32 alignment.

  3. Validate top‑k ordering matches: 
    topk_cpu = np.argsort(-scores_cpu)[:5]
topk_gpu = np.argsort(-scores_gpu)[:5]
assert np.array_equal(topk_cpu, topk_gpu), "Ranking mismatch"
  4. Check for absence of NaNs: 
    assert not np.isnan(scores_gpu).any(), "NaN detected in GPU scores"
  5. Monitor runtime metrics (GPU utilization, memory usage) to ensure performance impact is acceptable.

Operational Experience – Lessons Learned

  • Misleading symptom: The initial error “Result mismatch” suggested a model bug, but the root cause was a precision‑specific CUDA kernel.
  • Incorrect assumption: Enabling FP16 always improves latency; in similarity‑heavy workloads the added rounding error can outweigh speed gains.
  • Production edge case: When batch size crosses the workspace threshold the kernel crashes with CUDA_ERROR_ILLEGAL_ADDRESS, silently falling back to CPU for the reduction step and producing divergent scores.
  • Community insight: Disabling TensorRT (as discussed in the GitHub issue #15892) prevented the silent CPU fallback that caused mixed‑precision mismatches.

Best Practices and Prevention

  • Enable ORT_CUDA_LOG_VERBOSITY=3 in staging to capture provider‑level warnings early.
  • Run a numerical‑tolerance test suite that asserts max(|score_cpu - score_gpu|) < 1e-4 for all exported models.
  • Pin critical operators (e.g., HybridSearch, cosine similarity) to the CPU EP when exact reproducibility is required.
  • When using FP16, set ORT_CUDA_FP16_ENABLE=1 only after confirming that the model’s similarity calculations are robust to rounding (e.g., by quantizing the reference implementation).
  • Allocate sufficient CUDA workspace via ORT_CUDA_MAX_WORKSPACE_SIZE to avoid out‑of‑memory or illegal address errors for large batches.
  • Keep ONNX Runtime up‑to‑date; the 1.18 release notes already fixed several GPU‑related bugs, and later patches address the FP16 workspace issue.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why do CPU (FP32) and GPU (FP16) runs produce different top‑k rankings?

FP16 kernels round intermediate results, especially dot‑product accumulations, causing small score deviations that can cross the decision threshold for ranking. The HybridSearch operator does not apply a tolerance layer, so the ordering can change.

Can I keep FP16 inference and still get deterministic scores?

Only if you add a post‑processing step that re‑quantizes the similarity scores to FP32 and applies a deterministic tie‑breaker. Otherwise, pinning the scoring operator to FP32 is the safest approach.

What does the log message “Result mismatch: expected score X, got Y” mean?

The CUDA EP performed a runtime check against a CPU reference (enabled when ORT_ENABLE_FP16=1). The mismatch indicates that the FP16 computation produced a value outside the provider’s tolerance.

How do I disable the TensorRT fallback for HybridSearch?

Either remove TensorRT from the provider list or set trt_engine_cache_enable=False in the CUDA EP configuration, as shown in the “Disable TensorRT EP” code snippet above.

Is there a way to detect illegal memory accesses before they crash?

Enable CUDA EP debug mode (ORT_CUDA_DEBUG=1) and monitor the CUDA_ERROR_ILLEGAL_ADDRESS warning. Reducing batch size or increasing ORT_CUDA_MAX_WORKSPACE_SIZE typically resolves the underlying buffer overflow.