Problem
In a production Kubernetes cluster, a Docker container that runs ONNX Runtime (ORT) to serve inference requests starts failing with a Job completion timeout error. The same model runs within the expected latency when the container is started without CPU limits, but exceeds the request deadline when the pod is constrained by a CPU quota (e.g., cpu: "0.5"). The symptom manifests as HTTP 504 responses from the inference service and log entries such as:
2024-11-15 10:23:41.112 INFO [InferenceWorker] Inference request received (id=7f2c3a)
2024-11-15 10:23:41.115 DEBUG [OrtSession] Running model: resnet50.onnx
2024-11-15 10:23:46.120 ERROR [InferenceWorker] Inference timed out after 5000ms (id=7f2c3a)
2024-11-15 10:23:46.121 INFO [Kubelet] Pod sandbox changed: containerd://abcd1234 (reason: DeadlineExceeded)
The issue is reproducible across multiple pods when the CPU limit is set below a certain threshold, regardless of the underlying hardware.
Root Cause
ONNX Runtime creates a thread pool whose size is derived from the number of logical CPUs reported by the host at process start. Inside a Docker container, the cpuset and cgroup limits are applied after the process is launched. When the container is started with a reduced CPU quota (e.g., cpu: "0.5"), the Linux kernel still reports the full host CPU count to the process, causing ORT to spawn more worker threads than the cgroup permits.
These excess threads are immediately throttled by the CFS (Completely Fair Scheduler) quota enforcement. The scheduler enforces a hard limit on CPU time per period (default 100 ms). If the aggregate demand of the thread pool exceeds the allocated quota, the kernel repeatedly puts threads to sleep, dramatically increasing latency. Because ORT’s default inference timeout (5 s in the example) does not account for this throttling, the request hits the timeout before the model finishes.
In addition, ORT’s session_options.intra_op_num_threads and session_options.inter_op_num_threads default to 0 (auto), which resolves to std::thread::hardware_concurrency(). Under cgroup constraints this value is misleading, leading to the over‑provisioned thread pool.
Debug
Below is a step‑by‑step debugging workflow that isolates the CPU‑quota interaction.
1. Verify cgroup limits inside the container
# Inside the failing pod
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us # 50000 (0.5 CPU)
cat /sys/fs/cgroup/cpu/cpu.cfs_period_us # 100000
cat /sys/fs/cgroup/cpuset/cpuset.cpus # 0-3 (host reports 4 CPUs)
2. Check what ORT thinks the CPU count is
# Python snippet executed in the container
import multiprocessing, onnxruntime as ort
print("OS reported CPUs:", multiprocessing.cpu_count())
print("ORT intra_op threads:", ort.SessionOptions().intra_op_num_threads)
Typical output when the container is limited to 0.5 CPU:
OS reported CPUs: 4
ORT intra_op threads: 0
Because intra_op_num_threads is 0, ORT will later resolve it to 4.
3. Observe thread scheduling throttling
# Install perf (or use top -H)
perf stat -e cpu-clock,task-clock,context-switches,cpu-migrations -p $(pgrep -f onnxruntime) sleep 5
Typical output shows a high cpu-clock value but a low task-clock, indicating that threads are being throttled.
4. Reproduce with explicit thread count
# Modify inference server to force 1 intra‑op thread
import onnxruntime as ort
opts = ort.SessionOptions()
opts.intra_op_num_threads = 1
opts.inter_op_num_threads = 1
session = ort.InferenceSession("model.onnx", sess_options=opts)
After redeploying with the above change, the same request completes in ~200 ms, well within the timeout.
Solution
The fix consists of two complementary actions:
- Explicitly bound ONNX Runtime thread pools to the cgroup CPU limit. Set
intra_op_num_threadsandinter_op_num_threadsto the number of CPUs allocated to the container. This can be derived from thecpu.cfs_quota_us / cpu.cfs_period_uscalculation at runtime. - Configure the container runtime to expose the reduced CPU count to the process. Recent Docker and containerd versions support the
--cpusflag combined with--cpuset-cpusor thecpu.cfs_quota_uscgroup to limit/proc/self/statusVmPeakvalues. Using the--cpu-quotaflag alone is insufficient for ORT auto‑detection.
Before: Default configuration (no thread limits)
# inference_server.py (original)
import onnxruntime as ort
session = ort.InferenceSession("model.onnx") # defaults
After: Explicit thread limits derived from cgroup
# inference_server.py (patched)
import os
import onnxruntime as ort
def get_cgroup_cpu_quota():
try:
with open("/sys/fs/cgroup/cpu/cpu.cfs_quota_us") as f:
quota = int(f.read())
with open("/sys/fs/cgroup/cpu/cpu.cfs_period_us") as f:
period = int(f.read())
if quota > 0 and period > 0:
return max(1, quota // period)
except Exception:
pass
return os.cpu_count() # fallback
cpu_limit = get_cgroup_cpu_quota()
opts = ort.SessionOptions()
opts.intra_op_num_threads = cpu_limit
opts.inter_op_num_threads = cpu_limit
session = ort.InferenceSession("model.onnx", sess_options=opts)
print(f"ORT configured with {cpu_limit} intra‑op threads")
Deploy the updated container with the same CPU limit (cpu: "0.5"). The server now logs:
ORT configured with 1 intra‑op threads
2024-11-15 10:45:12.003 INFO [InferenceWorker] Inference completed in 182ms (id=9a4d1b)
Verify
After applying the fix, perform the following checks:
- Confirm the thread count inside the container:
python -c "import onnxruntime as ort; print(ort.get_available_providers())"
# No direct API for thread count, but we can inspect the log line printed above.
- Run a load test (e.g.,
heyorwrk) against the service with the same timeout settings and verify latency:
hey -z 30s -c 10 -q 5 http://service:8080/predict
Expected 95th‑percentile latency < 500 ms, no timeout errors.
- Check cgroup throttling metrics to ensure they are near zero:
cat /sys/fs/cgroup/cpu/cpu.stat
# Sample output after fix
nr_periods 1000
nr_throttled 0
throttled_time 0
Prevent
- Always set ORT thread counts explicitly when containers run under CPU quotas. Use the cgroup‑derived calculation shown above or expose the limit via an environment variable (e.g.,
ORT_NUM_THREADS). - Enable container runtime to propagate CPU limits to the process namespace. For Docker 20.10+ use
--cpustogether with--cpu-period/--cpu-quotaor enablecgroup v2unified hierarchy, which makescpu.cfs_quota_usvisible at process start. - Monitor
cpu.throttled.timeandcpu.throttled.periodsvia Prometheus node exporter or cAdvisor. Alert when throttling exceeds 5 % of total runtime. - Include a health‑check endpoint that runs a single inference and returns the latency; configure Kubernetes liveness/readiness probes to fail if latency exceeds a safe margin.
- Document the relationship between CPU limits and ORT thread pools in the service runbook to avoid future regressions when scaling pods.
FAQ
- Why does the inference succeed when I run the container locally without CPU limits?
Because the host reports the full CPU count, allowing ORT to create a thread pool that matches the physical cores. No throttling occurs, so the default 5 s timeout is never reached. - Can I rely on
os.cpu_count()inside the container?
No.os.cpu_count()returns the number of logical CPUs visible to the process, which under cgroup limits may still be the host count. Use the cgroup files (cpu.cfs_quota_usandcpu.cfs_period_us) to compute the effective limit. - Is increasing the inference timeout a viable workaround?
It masks the underlying throttling and can lead to SLA violations under load. Properly sizing the thread pool to the allocated CPU is the recommended fix. - Do I need to adjust memory limits as well?
Memory limits do not affect ORT thread scheduling, but insufficient memory can cause out‑of‑memory kills. Keep memory limits comfortably above the model’s peak consumption plus a safety margin for the thread pool. - How does this issue differ on Windows containers?
Windows uses a different job‑object based CPU quota mechanism; ORT reads theNUMBER_OF_PROCESSORSenvironment variable, which Windows updates correctly when a job object limits CPU. The problem described here is specific to Linux cgroup v1/v2 behavior.
Related Topic Hub: Model Serving Troubleshooting Hub