ONNX Runtime model loading timeout during Kubernetes pod startup

Problem – ONNX Runtime model loading timeout during pod startup

In a Kubernetes‑based microservice, each replica loads a dedicated ONNX model at container start‑up using onnxruntime.InferenceSession. The pod repeatedly fails to become Ready because the model loading step exceeds the default ONNXRuntime timeout, causing the container to restart:


2024-11-20T08:12:34.567Z stdout F 2024-11-20 08:12:34,123 [ERROR] onnxruntime.capi._pybind_state: 
    Failed to load model '/models/predictor.onnx' (timeout after 30000 ms)
2024-11-20T08:12:34.568Z stderr F 2024-11-20 08:12:34,124 [INFO]  Container health check failed: model not loaded

The symptom manifests as:

  • Pod stays in Init:0/1 or CrashLoopBackOff state.
  • Readiness probe never succeeds.
  • CPU usage spikes briefly during the timeout period, then drops.

Root Cause – Interaction of model size, storage latency, and ONNX Runtime initialization

ONNX Runtime allocates a native thread pool and attempts to memory‑map the model file. The default load timeout (30 s) is enforced by the C++ API. In the observed environment:

  • The model file is ~1.2 GB, stored on a Read‑Only Container File System (overlayfs) that is backed by a slow network‑mounted Persistent Volume (NFS).
  • During pod start‑up the node’s kubelet performs an InitContainer that copies the model from a remote object store into the PV. The copy can take >30 s under load.
  • ONNX Runtime begins loading the model before the copy finishes, hitting the 30 s timeout and aborting.

Thus the timeout is not a bug in ONNX Runtime itself but a race between the file becoming fully available and the runtime’s load attempt.

Debug – Step‑by‑step investigation

  1. Inspect pod events and init container logs to confirm copy duration.

kubectl describe pod predictor-7f9c8d9f5c-kwz9l
kubectl logs predictor-7f9c8d9f5c-kwz9l -c init-copy

Typical init log:


2024-11-20 08:11:50 INFO  Starting copy of predictor.onnx (1.2 GB) from s3://models/predictor.onnx
2024-11-20 08:12:35 INFO  Copy completed in 45 seconds

Notice the copy exceeds the 30 s window.

  1. Verify file availability at the moment the main container starts.

kubectl exec -it predictor-7f9c8d9f5c-kwz9l -- ls -l /models/predictor.onnx

If the file size is reported as 0 or the command hangs, the model is still being written.

  1. Check ONNX Runtime timeout configuration.

import onnxruntime as ort
session_options = ort.SessionOptions()
print(session_options.session_config_entries.get('session.load_model_timeout'))
# Output: None (defaults to 30000 ms)
  1. Capture a strace of the loading process.

kubectl exec -it predictor-7f9c8d9f5c-kwz9l -- strace -f -e trace=open,read,stat -p $(pgrep -f onnxruntime)

Typical excerpt showing repeated stat failures until the file appears:


stat("/models/predictor.onnx", 0x1c0) = -1 ENOENT (No such file or directory)
stat("/models/predictor.onnx", 0x1c0) = 0 {st_mode=S_IFREG|0644, st_size=0, ...}
stat("/models/predictor.onnx", 0x1c0) = 0 {st_mode=S_IFREG|0644, st_size=1288490189, ...}

Solution – Ensure model availability before ONNX Runtime starts

Approach 1 – Serialize model copy and inference server start

Move the copy logic into an InitContainer that blocks until the file is fully written, then start the main container.

Before (single container with in‑process download):


# app.py
import boto3, onnxruntime as ort
s3 = boto3.client('s3')
s3.download_file('my-bucket', 'predictor.onnx', '/models/predictor.onnx')
session = ort.InferenceSession('/models/predictor.onnx')

After (InitContainer + ready‑check):


# init-copy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: predictor
spec:
  initContainers:
  - name: init-copy
    image: amazon/aws-cli
    command: ["sh", "-c"]
    args:
      - |
        aws s3 cp s3://my-bucket/predictor.onnx /models/predictor.onnx \
        && echo "copy complete"
    volumeMounts:
    - name: model-vol
      mountPath: /models
  containers:
  - name: predictor
    image: myrepo/predictor:latest
    command: ["python", "app.py"]
    volumeMounts:
    - name: model-vol
      mountPath: /models
  volumes:
  - name: model-vol
    persistentVolumeClaim:
      claimName: model-pvc

Approach 2 – Increase ONNX Runtime load timeout

If the copy cannot be moved to an init phase (e.g., model is refreshed at runtime), raise the timeout:


# app.py
import onnxruntime as ort
session_options = ort.SessionOptions()
session_options.session_config_entries['session.load_model_timeout'] = '60000'  # 60 s
session = ort.InferenceSession('/models/predictor.onnx', sess_options=session_options)

Note: This only masks the symptom; the underlying I/O latency remains.

Approach 3 – Use a faster storage class

Switch from NFS to a block‑level PVC (e.g., gp2 on AWS EKS) or pre‑warm a emptyDir with the model via a sidecar that streams the file directly from the object store into memory‑mapped storage.

Verify – Confirm that the pod becomes Ready without timeout


kubectl apply -f init-copy.yaml
kubectl rollout status deployment/predictor
kubectl get pod -l app=predictor -o wide

Expected output:


deployment "predictor" successfully rolled out
NAME                                 READY   STATUS    RESTARTS   AGE
predictor-7f9c8d9f5c-kwz9l            1/1     Running   0          2m30s

Check the application log for a successful load message:


2024-11-20 08:13:02 INFO  Model loaded successfully (size=1.2 GB, load_time=12.4 ms)

Prevent – Operational guardrails to avoid recurrence

  • Pod startup ordering: Always place heavyweight I/O (model download, extraction) in initContainers so Kubernetes treats them as a prerequisite.
  • Health probe design: Use a exec probe that checks file existence and size before the inference server starts.
  • Resource sizing: Allocate sufficient cpu and ephemeral-storage to avoid throttling during copy.
  • Monitoring: Emit a custom metric (model_copy_duration_seconds) from the init container and set an alert if it exceeds a threshold (e.g., 20 s).
  • Versioned model PVCs: Keep each model version on its own PVC and mount it read‑only; this eliminates copy time at start‑up.

FAQ

  1. Why does the timeout only happen after a rolling update?
    During a rolling update the new pod starts before the old pod has finished copying the model to the shared PVC, causing a race condition. Existing pods already have the model locally, so they start instantly.
  2. Can I disable the timeout entirely?
    ONNX Runtime does not expose a “disable” flag; the timeout can only be set to a higher value. Setting it to 0 reverts to the default 30 s.
  3. Is the timeout affected by the number of threads in the session?
    No. The timeout is measured from the moment InferenceSession is constructed until the native model loader returns. Thread pool size influences inference latency, not load time.
  4. How do I debug a timeout when the model is stored on a local SSD?
    Capture strace or perf logs to see if the loader is blocked on mmap or CPU‑bound parsing. For SSDs, timeouts usually indicate corruption or insufficient file permissions.
  5. Will using onnxruntime-gpu change the timeout behavior?
    The GPU package uses the same C++ loader, so the timeout semantics are identical. However, GPU initialization may add additional latency; consider increasing the timeout accordingly.

Related Topic Hub: Model Serving Troubleshooting Hub

Related Articles