ONNX Runtime RAG Reranker Timeout on Premises Server

Q: Investigation and Debugging Steps 1. Capture Runtime Logs

# Systemd journal for the RAG service journalctl -u rag-service -f | grep -i timeout Typical output:

Q: 3. Profile CPU Utilization

# Top‑like view of per‑process CPU usage pid=$(pgrep -f rag-service) htop -p $pid During a timeout, CPU usage often hits 100 % across all cores, confirming thread starvation.

Q: 4. Measure Memory Footprint per Batch

# Record RSS for each batch size for B in 16 32 48 64; do python run_batch.py --batch-size $B & wait $! echo "Batch $B: $(ps -o rss= -p $!) KB" done When RSS approaches the server’s RAM limit, the OS starts paging, which can be seen in vmstat output:

Q: 5. Confirm Execution Provider

print("Provider:", session.get_providers()) If the output is ['CPUExecutionProvider'] during a period when a GPU should be present, the fallback is active.

In this article

Table of Contents

Problem Description

The RAG (Retrieval‑Augmented Generation) pipeline uses an ONNX Runtime reranker model to score retrieved documents before generation. In a private data‑center deployment with limited CPU cores and memory, the reranker frequently exceeds the configured inference timeout during peak query load, returning errors such as:


Error: Inference session timed out after 30000 ms (RunOptions.timeout exceeded).
ORT runtime error: Execution failed – Timeout exceeded while waiting for kernel execution.
SessionOptions: timeout exceeded while waiting for kernel execution – possible thread starvation.

Typical symptoms include:

Latency spikes for batch sizes > 32 documents.
Timeout errors logged by the RAG service during high traffic windows.
Occasional “kernel execution stalled” messages in journalctl when CPU cores are saturated.

Root Cause Analysis

The timeout originates from three intertwined factors:

Thread‑pool starvation: ONNX Runtime creates an internal thread pool sized to the number of physical cores (see the Performance Tuning Guide). Under heavy concurrent queries, all threads become blocked on CPU‑bound kernels, leaving no free workers to start new runs. This matches the behavior reported in GitHub issue #13002 where “SessionOptions timeout not respected under heavy load”.
Batch size vs. memory pressure: Larger document batches increase tensor sizes quadratically. On a server with 8 GB RAM, batches of 64 cause the process to swap, as observed in the healthcare analytics incident (“ran out of RAM during large batch processing, triggering OS paging”). Swapping dramatically elongates kernel execution, causing the 30 s default timeout to be hit.
Execution provider fallback: When the GPU driver crashes, ONNX Runtime silently falls back to the CPU provider (see the Execution Providers documentation). The CPU provider is ~5× slower, which combined with the above factors pushes inference beyond the timeout, as described in the financial services firm incident.

In summary, the reranker times out because the runtime cannot schedule kernel execution quickly enough due to insufficient CPU threads, memory pressure, and occasional provider fallback.

Investigation and Debugging Steps

1. Capture Runtime Logs


# Systemd journal for the RAG service
journalctl -u rag-service -f | grep -i timeout

Typical output:


2026-06-22 14:03:12,451 ERROR [rag.reranker] Inference session timed out after 30000 ms (RunOptions.timeout exceeded)
2026-06-22 14:03:12,452 WARN  [onnxruntime] Execution failed – Timeout exceeded while waiting for kernel execution

2. Inspect ONNX Runtime Session Options

Verify that the timeout is set and that the thread pool size matches the hardware.


import onnxruntime as ort

session = ort.InferenceSession("reranker.onnx")
opts = session.get_session_options()
print("Timeout (ms):", opts.run_options.timeout)
print("Intra‑op threads:", opts.intra_op_num_threads)
print("Inter‑op threads:", opts.inter_op_num_threads)

Expected output (default values):


Timeout (ms): 30000
Intra‑op threads: 0   # 0 means use all physical cores
Inter‑op threads: 0

3. Profile CPU Utilization


# Top‑like view of per‑process CPU usage
pid=$(pgrep -f rag-service)
htop -p $pid

During a timeout, CPU usage often hits 100 % across all cores, confirming thread starvation.

4. Measure Memory Footprint per Batch


# Record RSS for each batch size
for B in 16 32 48 64; do
    python run_batch.py --batch-size $B &
    wait $!
    echo "Batch $B: $(ps -o rss= -p $!) KB"
done

When RSS approaches the server’s RAM limit, the OS starts paging, which can be seen in vmstat output:


procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0  819200 123456  20480 1024000   0    0    10    20  500 1000 85 10  5  0 0

5. Confirm Execution Provider


print("Provider:", session.get_providers())

If the output is ['CPUExecutionProvider'] during a period when a GPU should be present, the fallback is active.

Resolution

1. Tune Thread Pools

Explicitly limit intra‑op threads to avoid oversubscription and reserve cores for other services.


# Before (default)
session_options = ort.SessionOptions()
session_options.run_options.timeout = 30000  # 30 s

# After – reserve 2 cores for OS / other services
session_options = ort.SessionOptions()
session_options.intra_op_num_threads = max(1, os.cpu_count() - 2)
session_options.inter_op_num_threads = 1
session_options.run_options.timeout = 60000  # increase to 60 s for safety

2. Reduce Batch Size Dynamically

Implement a guard that caps the reranker batch size based on current load.


def adaptive_batch(docs, max_batch=32):
    # If CPU usage > 80 %, shrink batch
    cpu = psutil.cpu_percent(interval=0.5)
    if cpu > 80:
        return docs[:max_batch]
    return docs

3. Increase Available Memory or Enable Swap Management

Upgrade server RAM to at least 16 GB for batch sizes ≥ 64.
Configure vm.swappiness=10 to keep swapping minimal.
Use ulimit -v to prevent the process from exhausting system memory.

4. Pin Execution Provider

Force the GPU provider and fail fast if unavailable, avoiding silent CPU fallback.


session_options = ort.SessionOptions()
session_options.append_execution_provider("CUDAExecutionProvider")
# Optional: fallback to CPU only after explicit check
if not ort.get_available_providers().count("CUDAExecutionProvider"):
    raise RuntimeError("GPU provider not available")

5. Adjust Timeout for Peak Loads

Increase the timeout only for large batches, keeping the default for normal traffic.


run_options = ort.RunOptions()
run_options.timeout = 30000  # default 30 s

if len(batch) > 32:
    run_options.timeout = 60000  # 60 s for heavy batches
output = session.run(None, input_feed, run_options=run_options)

Verification

After applying the changes, perform the following checks:

Functional test: Run a batch of 64 documents during a simulated peak load and confirm no timeout error.


python run_batch.py --batch-size 64 --simulate-load

Latency metrics: Observe the reranker latency histogram in Prometheus. The 99th percentile should stay below the new timeout (e.g., 55 s for a 60 s timeout).
CPU/Memory usage: Verify that CPU utilization peaks at ~70 % and RSS stays well under RAM limits.
Provider confirmation: Log the selected provider at startup; ensure it remains CUDAExecutionProvider in normal operation.

Prevention and Best Practices

Capacity planning: Size the on‑prem server to handle the maximum expected batch size with a safety margin (≥ 2× RAM, ≥ 1.5× CPU cores).
Dynamic batching: Implement load‑aware batch sizing to prevent overload during traffic spikes.
Monitoring: Export onnxruntime_session_duration_seconds and onnxruntime_thread_pool_active_threads metrics; set alerts when latency exceeds 80 % of the timeout.
Graceful degradation: If the GPU provider becomes unavailable, return a clear error instead of silently falling back to CPU.
Configuration versioning: Store SessionOptions in a version‑controlled YAML file and validate against the schema at deployment time.

FAQ

Why does the timeout only happen with large batches?
Large batches increase tensor memory and kernel execution time. When combined with thread starvation, the runtime cannot finish within the default 30 s limit.
Can I disable the timeout entirely?
Setting run_options.timeout = 0 disables the watchdog, but this is unsafe in production because a hung kernel can block the entire process.
How do I know if the GPU provider has fallen back to CPU?
Check session.get_providers() at startup and log the result. Additionally, enable ORT_LOG_SEVERITY=1 to see provider selection messages in the logs.
Is increasing the timeout sufficient?
Only a temporary mitigation. Without addressing thread contention and memory pressure, larger timeouts will merely delay failure and increase resource waste.
What kernel execution errors indicate thread starvation?
Messages such as “ORT runtime error: Execution failed – Timeout exceeded while waiting for kernel execution” combined with 100 % CPU usage across all cores are classic signs.

Related Topic Hub: Model Serving Troubleshooting Hub

ONNX Runtime RAG reranker timeout during inference serving

ONNX Runtime RAG reranker timeout on premises server