2026-09-03 14:45:12,101 [ORT] INFO Inference completed in 124345 ms 2026-09-03 14:45:12,102 [ORT] INFO Output tensor shape: (1, 512, 768) Resource usage snapshot # CPU top -b -n 1 | grep onnxruntime PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12345 onnxrun 20 0 32G 24G 1.0G R 85.0 60.0 2:04.12 onnxruntime # GPU nvidia-smi --query-gpu=memory.used,memory.total --format=csv -i 0 GPU 0: 12400MiB / 24576MiB Programmatic health check import onnxruntime as ort, numpy as np session = ort.InferenceSession("large_transformer.onnx") run_opts = ort.RunOptions() run_opts.timeout = 900_000 dummy_input = np.random.randint(0, 30522, (1, 512), dtype=np.int64) outputs = session.run(None, {"input_ids": dummy_input}, run_options=run_opts) assert outputs[0].shape == (1, 512, 768) print("Validation passed") Operational Best Practices and Prevention Monitor run‑time metrics. Set up an alert when ort_run_duration_ms exceeds 80 % of the configured timeout. Capacity planning. Use the ONNX Runtime Performance Guide for large models to size RAM/VRAM and to decide on model partitioning (e.g., split the transformer into encoder/decoder sub‑graphs). Static configuration. Store all SessionOptions and RunOptions in a central JSON/YAML so that every deployment uses the same thread counts, optimization level, and timeout. Isolation per model instance. Run each large transformer in its own container or cgroup with dedicated CPU cores and memory limits to avoid noisy‑neighbor effects. Regular stress tests. Periodically run a synthetic workload that pushes the model to its worst‑case latency and confirm that the observed duration stays below the configured timeout. FAQ (Related Questions) Why does the timeout only appear on the on‑prem server and not in the cloud? The cloud runtimes usually provision larger instance types and automatically set higher run_options.timeout. On‑prem defaults are conservative (5 minutes) and the hardware may be shared, leading to the observed OOM/PCIe contention.

ONNX Runtime Inference Job Timeout On On-prem Server With Large Transformer

Q: 1. Capture the exact error and surrounding logs

2026-09-03 14:12:45,321 [ORT] ERROR OrtAllocateBuffer failed: out of memory 2026-09-03 14:12:45,322 [ORT] ERROR ORT_RUN_TIMEOUT: Inference session timed out after 300000 ms 2. Verify resource utilization during the run # CPU & memory top -b -n 1 | grep onnxruntime # GPU memory (if using CUDA EP) nvidia-smi --query-gpu=memory.used,memory.total --format=csv -i 0 Typical output that signals a problem:

Q: 6. Reproduce the issue with a minimal script and a timer

import time, onnxruntime as ort session = ort.InferenceSession("model.onnx") run_options = ort.RunOptions() run_options.timeout = 300_000 # 5 minutes (default) start = time.time() try: outputs = session.run(None, {"input": data}, run_options=run_options) except ort.OrtFail as e: print("Run failed:", e) print("Elapsed:", time.time() - start) Solution (Resolution) The fix consists of three coordinated changes:

Q: 1. Extend the run‑time limit

# Python example import onnxruntime as ort session_opts = ort.SessionOptions() run_opts = ort.RunOptions() run_opts.timeout = 900_000 # 15 minutes, adjust to expected latency session = ort.InferenceSession("large_transformer.onnx", sess_options=session_opts) outputs = session.run(None, {"input_ids": ids}, run_options=run_opts) For C API callers, use OrtRunOptionsSetRunLogVerbosityLevel together with the custom timeout flag (see the C API docs).

Q: 2. Tune execution provider resources SettingBeforeAfter (recommended) session_options.intra_op_num_threads0 (auto)Number of physical cores per socket (e.g., 24) session_options.inter_op_num_threads0 (auto)1–2 (to avoid oversubscription) session_options.graph_optimization_levelORT_ENABLE_BASICORT_ENABLE_EXTENDED ORT_DISABLE_MEMORY_PATTERN (env)unset1 (disable) # Python session_opts = ort.SessionOptions() session_opts.intra_op_num_threads = 24 session_opts.inter_op_num_threads = 2 session_opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_EXTENDED session_opts.enable_mem_pattern = False # equivalent to ORT_DISABLE_MEMORY_PATTERN 3. Apply hardware affinity and isolation

# Pin inference process to dedicated cores (Linux) taskset -c 0-23 ./run_inference.sh # For multi‑GPU nodes, set CUDA_DEVICE_MAX_CONNECTIONS to 1 to avoid deadlocks export CUDA_DEVICE_MAX_CONNECTIONS=1 Pinning eliminates the hyper‑thread scheduler pauses that caused the financial services firm's timeout.

Problem Description

On‑premises inference jobs that run large transformer models (e.g., BERT‑large, GPT‑2‑XL) with ONNX Runtime repeatedly fail with a timeout error such as:

ORT_RUN_TIMEOUT: Inference session timed out after 300000 ms
Error: Execution failed due to timeout – see OrtRunOptionsSetRunLogVerbosityLevel for details

Typical symptoms observed across multiple deployments include:

Job termination after exactly 5 minutes (the default run‑time limit).
Intermittent OOM messages in the system log (OrtAllocateBuffer failed: out of memory) followed by the same timeout.
GPU‑based runs on multi‑GPU nodes producing CUDA error: device‑side assert triggered – leads to session termination with timeout message.
CPU runs on heavily oversubscribed cores where the kernel scheduler pauses the inference thread, after which the runtime aborts with the same timeout.

Root Cause Analysis

The timeout originates from the interaction of three orthogonal factors:

Default run‑time limit. ONNX Runtime imposes a 5‑minute limit on OrtRunOptions unless the caller explicitly overrides it. This is documented in the C API reference for OrtRunOptions and echoed in the Python SessionOptions docs (session_options.execution_mode).
Resource contention. Large transformers require > 10 GB of RAM (or several GB of GPU memory) and tens of CPU cores for parallel attention kernels. On‑prem clusters often run multiple inference workers on the same node, leading to:
- Memory pressure that triggers the OS OOM killer (see the on‑prem CPU cluster incident where a 1.3 B‑parameter BERT timed out after 300 s because pages were reclaimed).
- PCIe saturation on A100‑based nodes when several instances share a GPU, producing the ORT_RUN_TIMEOUT observed in the NVIDIA A100 incident.
- Hyper‑thread oversubscription that stalls the kernel scheduler, as reported by the financial services firm running GPT‑2‑XL on a virtualized server.
Execution provider configuration. The default CPU execution provider uses a memory pattern allocator that can fragment large contiguous buffers. Disabling it (via ORT_DISABLE_MEMORY_PATTERN) or adjusting thread counts (session_options.intra_op_num_threads, session_options.inter_op_num_threads) changes the allocation pattern and can eliminate the implicit “timeout‑due‑to‑OOM” cascade.

In summary, the observed timeout is not a single bug but the manifestation of a default guard (5‑minute limit) being reached because the inference workload cannot complete within that window due to insufficient resources or sub‑optimal provider settings.

Investigation and Debugging Steps

Below is a reproducible debugging workflow that was used in the cited incidents.

1. Capture the exact error and surrounding logs

2026-09-03 14:12:45,321 [ORT] ERROR OrtAllocateBuffer failed: out of memory
2026-09-03 14:12:45,322 [ORT] ERROR ORT_RUN_TIMEOUT: Inference session timed out after 300000 ms

2. Verify resource utilization during the run

# CPU & memory
top -b -n 1 | grep onnxruntime

# GPU memory (if using CUDA EP)
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -i 0

Typical output that signals a problem:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
12345 onnxrun   20   0  32G   28G   1.2G R  95.0  70.0   4:59.12 onnxruntime
...
GPU 0:  24500MiB / 24576MiB

3. Check the effective timeout value

python - <<'PY'
import onnxruntime as ort
opts = ort.SessionOptions()
print("Default timeout (ms):", getattr(opts, "run_options", None))
PY

If the output is None, the default 5‑minute limit is in effect.

4. Inspect execution provider settings

python - <<'PY'
import onnxruntime as ort
opts = ort.SessionOptions()
print("Intra‑op threads:", opts.intra_op_num_threads)
print("Inter‑op threads:", opts.inter_op_num_threads)
print("Graph optimization level:", opts.graph_optimization_level)
PY

5. Perform a packet capture (GPU‑only scenario)

# Capture PCIe traffic between host and GPU
sudo tcpdump -i pcie0 -w pcie_capture.pcap

Look for sustained high‑throughput bursts that correlate with the timeout window.

6. Reproduce the issue with a minimal script and a timer

import time, onnxruntime as ort

session = ort.InferenceSession("model.onnx")
run_options = ort.RunOptions()
run_options.timeout = 300_000  # 5 minutes (default)

start = time.time()
try:
    outputs = session.run(None, {"input": data}, run_options=run_options)
except ort.OrtFail as e:
    print("Run failed:", e)
print("Elapsed:", time.time() - start)

Solution (Resolution)

The fix consists of three coordinated changes:

1. Extend the run‑time limit

# Python example
import onnxruntime as ort

session_opts = ort.SessionOptions()
run_opts = ort.RunOptions()
run_opts.timeout = 900_000  # 15 minutes, adjust to expected latency

session = ort.InferenceSession("large_transformer.onnx", sess_options=session_opts)
outputs = session.run(None, {"input_ids": ids}, run_options=run_opts)

For C API callers, use OrtRunOptionsSetRunLogVerbosityLevel together with the custom timeout flag (see the C API docs).

2. Tune execution provider resources

Setting	Before	After (recommended)
`session_options.intra_op_num_threads`	0 (auto)	Number of physical cores per socket (e.g., 24)
`session_options.inter_op_num_threads`	0 (auto)	1–2 (to avoid oversubscription)
`session_options.graph_optimization_level`	ORT_ENABLE_BASIC	ORT_ENABLE_EXTENDED
`ORT_DISABLE_MEMORY_PATTERN` (env)	unset	1 (disable)

# Python
session_opts = ort.SessionOptions()
session_opts.intra_op_num_threads = 24
session_opts.inter_op_num_threads = 2
session_opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_EXTENDED
session_opts.enable_mem_pattern = False   # equivalent to ORT_DISABLE_MEMORY_PATTERN

3. Apply hardware affinity and isolation

# Pin inference process to dedicated cores (Linux)
taskset -c 0-23 ./run_inference.sh

# For multi‑GPU nodes, set CUDA_DEVICE_MAX_CONNECTIONS to 1 to avoid deadlocks
export CUDA_DEVICE_MAX_CONNECTIONS=1

Pinning eliminates the hyper‑thread scheduler pauses that caused the financial services firm's timeout.

4. Ensure sufficient memory provisioning

Allocate at least 1.5× the model size in RAM (e.g., a 2 GB model → 3 GB free RAM).
On GPU, reserve enough VRAM for both the model and intermediate activation buffers (BERT‑large typically needs ~12 GB per instance).

Verification (Validation)

After applying the changes, verify the fix with the following checks:

Successful run log

2026-09-03 14:45:12,101 [ORT] INFO Inference completed in 124345 ms
2026-09-03 14:45:12,102 [ORT] INFO Output tensor shape: (1, 512, 768)

Resource usage snapshot

# CPU
top -b -n 1 | grep onnxruntime
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
12345 onnxrun   20   0  32G   24G   1.0G R  85.0  60.0   2:04.12 onnxruntime

# GPU
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -i 0
GPU 0:  12400MiB / 24576MiB

Programmatic health check

import onnxruntime as ort, numpy as np

session = ort.InferenceSession("large_transformer.onnx")
run_opts = ort.RunOptions()
run_opts.timeout = 900_000
dummy_input = np.random.randint(0, 30522, (1, 512), dtype=np.int64)
outputs = session.run(None, {"input_ids": dummy_input}, run_options=run_opts)
assert outputs[0].shape == (1, 512, 768)
print("Validation passed")

Operational Best Practices and Prevention

Monitor run‑time metrics. Set up an alert when ort_run_duration_ms exceeds 80 % of the configured timeout.
Capacity planning. Use the ONNX Runtime Performance Guide for large models to size RAM/VRAM and to decide on model partitioning (e.g., split the transformer into encoder/decoder sub‑graphs).
Static configuration. Store all SessionOptions and RunOptions in a central JSON/YAML so that every deployment uses the same thread counts, optimization level, and timeout.
Isolation per model instance. Run each large transformer in its own container or cgroup with dedicated CPU cores and memory limits to avoid noisy‑neighbor effects.
Regular stress tests. Periodically run a synthetic workload that pushes the model to its worst‑case latency and confirm that the observed duration stays below the configured timeout.

FAQ (Related Questions)

Why does the timeout only appear on the on‑prem server and not in the cloud?
The cloud runtimes usually provision larger instance types and automatically set higher run_options.timeout. On‑prem defaults are conservative (5 minutes) and the hardware may be shared, leading to the observed OOM/PCIe contention.
Can I disable the timeout entirely?
Setting run_options.timeout = 0 disables the guard, but it is unsafe because runaway inference can hang the process. Instead, increase the limit to a value that comfortably exceeds the 99‑th percentile latency.
How do I know which execution provider is actually being used?
Inspect the session’s provider list:
```
session.get_providers()   # e.g., ['CPUExecutionProvider'] or ['CUDAExecutionProvider']
```
Does disabling the memory pattern allocator affect performance?
Disabling ORT_DISABLE_MEMORY_PATTERN can increase allocation overhead but prevents fragmentation that leads to OOM‑induced timeouts for very large models. Benchmark both configurations on representative payloads before deciding.
What if I still see occasional timeouts after applying all fixes?
Collect a core dump (use gcore or cuda-gdb) and run it through the ONNX Runtime debugger. Often the residual cases are caused by transient OS scheduling spikes or background batch jobs that temporarily steal CPU/GPU bandwidth.

Related Topic Hub: Model Serving Troubleshooting Hub