Problem – ONNX Runtime model loading timeout during pod startup
In a Kubernetes‑based microservice, each replica loads a dedicated ONNX model at container start‑up using onnxruntime.InferenceSession. The pod repeatedly fails to become Ready because the model loading step exceeds the default ONNXRuntime timeout, causing the container to restart:
2024-11-20T08:12:34.567Z stdout F 2024-11-20 08:12:34,123 [ERROR] onnxruntime.capi._pybind_state:
Failed to load model '/models/predictor.onnx' (timeout after 30000 ms)
2024-11-20T08:12:34.568Z stderr F 2024-11-20 08:12:34,124 [INFO] Container health check failed: model not loaded
The symptom manifests as:
- Pod stays in
Init:0/1orCrashLoopBackOffstate. - Readiness probe never succeeds.
- CPU usage spikes briefly during the timeout period, then drops.
Root Cause – Interaction of model size, storage latency, and ONNX Runtime initialization
ONNX Runtime allocates a native thread pool and attempts to memory‑map the model file. The default load timeout (30 s) is enforced by the C++ API. In the observed environment:
- The model file is ~1.2 GB, stored on a Read‑Only Container File System (overlayfs) that is backed by a slow network‑mounted Persistent Volume (NFS).
- During pod start‑up the node’s
kubeletperforms anInitContainerthat copies the model from a remote object store into the PV. The copy can take >30 s under load. - ONNX Runtime begins loading the model before the copy finishes, hitting the 30 s timeout and aborting.
Thus the timeout is not a bug in ONNX Runtime itself but a race between the file becoming fully available and the runtime’s load attempt.
Debug – Step‑by‑step investigation
- Inspect pod events and init container logs to confirm copy duration.
kubectl describe pod predictor-7f9c8d9f5c-kwz9l
kubectl logs predictor-7f9c8d9f5c-kwz9l -c init-copy
Typical init log:
2024-11-20 08:11:50 INFO Starting copy of predictor.onnx (1.2 GB) from s3://models/predictor.onnx
2024-11-20 08:12:35 INFO Copy completed in 45 seconds
Notice the copy exceeds the 30 s window.
- Verify file availability at the moment the main container starts.
kubectl exec -it predictor-7f9c8d9f5c-kwz9l -- ls -l /models/predictor.onnx
If the file size is reported as 0 or the command hangs, the model is still being written.
- Check ONNX Runtime timeout configuration.
import onnxruntime as ort
session_options = ort.SessionOptions()
print(session_options.session_config_entries.get('session.load_model_timeout'))
# Output: None (defaults to 30000 ms)
- Capture a strace of the loading process.
kubectl exec -it predictor-7f9c8d9f5c-kwz9l -- strace -f -e trace=open,read,stat -p $(pgrep -f onnxruntime)
Typical excerpt showing repeated stat failures until the file appears:
stat("/models/predictor.onnx", 0x1c0) = -1 ENOENT (No such file or directory)
stat("/models/predictor.onnx", 0x1c0) = 0 {st_mode=S_IFREG|0644, st_size=0, ...}
stat("/models/predictor.onnx", 0x1c0) = 0 {st_mode=S_IFREG|0644, st_size=1288490189, ...}
Solution – Ensure model availability before ONNX Runtime starts
Approach 1 – Serialize model copy and inference server start
Move the copy logic into an InitContainer that blocks until the file is fully written, then start the main container.
Before (single container with in‑process download):
# app.py
import boto3, onnxruntime as ort
s3 = boto3.client('s3')
s3.download_file('my-bucket', 'predictor.onnx', '/models/predictor.onnx')
session = ort.InferenceSession('/models/predictor.onnx')
After (InitContainer + ready‑check):
# init-copy.yaml
apiVersion: v1
kind: Pod
metadata:
name: predictor
spec:
initContainers:
- name: init-copy
image: amazon/aws-cli
command: ["sh", "-c"]
args:
- |
aws s3 cp s3://my-bucket/predictor.onnx /models/predictor.onnx \
&& echo "copy complete"
volumeMounts:
- name: model-vol
mountPath: /models
containers:
- name: predictor
image: myrepo/predictor:latest
command: ["python", "app.py"]
volumeMounts:
- name: model-vol
mountPath: /models
volumes:
- name: model-vol
persistentVolumeClaim:
claimName: model-pvc
Approach 2 – Increase ONNX Runtime load timeout
If the copy cannot be moved to an init phase (e.g., model is refreshed at runtime), raise the timeout:
# app.py
import onnxruntime as ort
session_options = ort.SessionOptions()
session_options.session_config_entries['session.load_model_timeout'] = '60000' # 60 s
session = ort.InferenceSession('/models/predictor.onnx', sess_options=session_options)
Note: This only masks the symptom; the underlying I/O latency remains.
Approach 3 – Use a faster storage class
Switch from NFS to a block‑level PVC (e.g., gp2 on AWS EKS) or pre‑warm a emptyDir with the model via a sidecar that streams the file directly from the object store into memory‑mapped storage.
Verify – Confirm that the pod becomes Ready without timeout
kubectl apply -f init-copy.yaml
kubectl rollout status deployment/predictor
kubectl get pod -l app=predictor -o wide
Expected output:
deployment "predictor" successfully rolled out
NAME READY STATUS RESTARTS AGE
predictor-7f9c8d9f5c-kwz9l 1/1 Running 0 2m30s
Check the application log for a successful load message:
2024-11-20 08:13:02 INFO Model loaded successfully (size=1.2 GB, load_time=12.4 ms)
Prevent – Operational guardrails to avoid recurrence
- Pod startup ordering: Always place heavyweight I/O (model download, extraction) in
initContainersso Kubernetes treats them as a prerequisite. - Health probe design: Use a
execprobe that checks file existence and size before the inference server starts. - Resource sizing: Allocate sufficient
cpuandephemeral-storageto avoid throttling during copy. - Monitoring: Emit a custom metric (
model_copy_duration_seconds) from the init container and set an alert if it exceeds a threshold (e.g., 20 s). - Versioned model PVCs: Keep each model version on its own PVC and mount it read‑only; this eliminates copy time at start‑up.
FAQ
- Why does the timeout only happen after a rolling update?
During a rolling update the new pod starts before the old pod has finished copying the model to the shared PVC, causing a race condition. Existing pods already have the model locally, so they start instantly. - Can I disable the timeout entirely?
ONNX Runtime does not expose a “disable” flag; the timeout can only be set to a higher value. Setting it to0reverts to the default 30 s. - Is the timeout affected by the number of threads in the session?
No. The timeout is measured from the momentInferenceSessionis constructed until the native model loader returns. Thread pool size influences inference latency, not load time. - How do I debug a timeout when the model is stored on a local SSD?
Capturestraceorperflogs to see if the loader is blocked onmmapor CPU‑bound parsing. For SSDs, timeouts usually indicate corruption or insufficient file permissions. - Will using
onnxruntime-gpuchange the timeout behavior?
The GPU package uses the same C++ loader, so the timeout semantics are identical. However, GPU initialization may add additional latency; consider increasing the timeout accordingly.
Related Topic Hub: Model Serving Troubleshooting Hub