ONNX Runtime model load timeout after S3 API throttling in Kubernetes

Problem

In a production Kubernetes deployment the ONNX Runtime inference service intermittently fails to load models stored in cloud object storage (e.g., Amazon S3, Google Cloud Storage). The failure manifests as a model‑load timeout followed by inference request timeouts. The issue appears only after a period of sustained traffic and disappears after a pod restart.

Observed Symptoms

• ONNX Runtime logs show ModelLoadError: timeout while downloading model from S3.
• Pod logs contain repeated 503 Service Unavailable errors from the S3 SDK.
• CPU and memory usage remain normal; the container is not OOM‑killed.
• Metrics: onnxruntime_model_load_duration_seconds spikes to >30 s.
• After a pod restart the model loads successfully and inference latency returns to baseline.

Sample Log Excerpts

2026-09-22T12:34:56.123Z INFO  ModelLoader - Starting download of s3://ml-models/v1/model.onnx
2026-09-22T12:35:06.789Z ERROR ModelLoader - ModelLoadError: timeout after 10s while fetching from S3
2026-09-22T12:35:06.790Z WARN  S3Client - Received 429 TooManyRequests from s3.amazonaws.com
2026-09-22T12:35:07.001Z INFO  HealthCheck - Inference unavailable: model not loaded

Root Cause

The ONNX Runtime container downloads the model directly from S3 at startup using the AWS SDK. During traffic spikes the Kubernetes pod makes many concurrent model‑load attempts (e.g., during rolling updates or when the pod restarts after a crash). The cloud provider enforces API request throttling (HTTP 429) when the request rate exceeds the allocated quota. Because the SDK does not retry with exponential backoff by default for GetObject calls, the download aborts after the client‑side timeout, leaving the runtime in a failed state.

Key contributing factors:

1. Unbounded parallel downloads: Each pod attempts to download the full model file (~200 MiB) on every start, regardless of whether the model is already cached on the node.
2. Insufficient S3 request quota: The default bucket request rate limit (≈3,500 GET requests / second per prefix) is exceeded during coordinated rollouts.
3. Missing retry logic: The SDK call is wrapped in a single GetObject request without retry policy, so a transient 429 error results in immediate failure.

Debug

The following steps were used to confirm the root cause.

1. Inspect S3 request metrics

aws cloudwatch get-metric-statistics \
    --namespace AWS/S3 \
    --metric-name 4xxErrors \
    --dimensions Name=BucketName,Value=ml-models Name=StorageType,Value=AllStorageTypes \
    --statistics Sum \
    --period 60 \
    --start-time 2026-09-22T12:00:00Z \
    --end-time 2026-09-22T13:00:00Z

Result: spikes of 4xxErrors aligning with pod restarts.

2. Capture SDK retry configuration

import boto3
from botocore.config import Config

cfg = Config(retries={'max_attempts': 3, 'mode': 'standard'})
s3 = boto3.client('s3', config=cfg)

The existing code used the default config, which sets max_attempts to 1 for GetObject.

3. Verify pod concurrency during rollout

kubectl get pods -n ml-inference -l app=onnx-runtime -o wide
kubectl describe deployment onnx-runtime -n ml-inference

During a rolling update 5 pods started simultaneously, each triggering a model download.

4. Network trace (optional)

kubectl exec -n ml-inference onnx-runtime-abc123 -- tcpdump -i any -w /tmp/capture.pcap host s3.amazonaws.com

Analysis of the pcap showed multiple concurrent GET /ml-models/v1/model.onnx requests with overlapping TCP retransmissions.

Solution

Two complementary mitigations were applied:

1. Add exponential backoff and retry to the S3 download code

Wrap the model fetch in a retry loop with jitter. This handles transient 429 responses without aborting the startup.

Before

def download_model(bucket, key, dest_path):
    s3 = boto3.client('s3')
    s3.download_file(bucket, key, dest_path)  # single attempt, default timeout

After

import time
import random
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

RETRY_MAX = 5
BASE_DELAY = 1  # seconds

def download_model(bucket, key, dest_path):
    cfg = Config(
        retries={'max_attempts': RETRY_MAX, 'mode': 'standard'},
        connect_timeout=5,
        read_timeout=30,
    )
    s3 = boto3.client('s3', config=cfg)

    for attempt in range(1, RETRY_MAX + 1):
        try:
            s3.download_file(bucket, key, dest_path)
            return
        except ClientError as e:
            if e.response['Error']['Code'] in ('Throttling', 'TooManyRequests'):
                delay = BASE_DELAY * (2 ** (attempt - 1)) + random.uniform(0, 0.5)
                print(f"Retry {attempt}/{RETRY_MAX} after {delay:.2f}s due to {e.response['Error']['Code']}")
                time.sleep(delay)
                continue
            raise
    raise RuntimeError("Failed to download model after retries")

2. Cache the model on the node to avoid repeated downloads

Introduce an initContainer that pulls the model once to a shared emptyDir volume. Subsequent containers mount the same volume, eliminating S3 traffic on pod restarts.

Before (single container download)

containers:
- name: onnx-runtime
  image: myrepo/onnx-runtime:latest
  env:
    - name: MODEL_S3_URI
      value: s3://ml-models/v1/model.onnx

After (initContainer cache)

volumes:
- name: model-cache
  emptyDir: {}

initContainers:
- name: model-fetcher
  image: amazon/aws-cli:2.13.0
  command: ["sh", "-c"]
  args:
    - |
      aws s3 cp s3://ml-models/v1/model.onnx /model/model.onnx \
        --no-progress --retry-mode standard --max-attempts 5
  envFrom:
    - secretRef:
        name: aws-credentials
  volumeMounts:
    - name: model-cache
      mountPath: /model

containers:
- name: onnx-runtime
  image: myrepo/onnx-runtime:latest
  env:
    - name: MODEL_PATH
      value: /model/model.onnx
  volumeMounts:
    - name: model-cache
      mountPath: /model

3. Increase S3 request quota (optional)

Request a higher request rate limit from the cloud provider or enable request rate optimization by using a random prefix in the object key (e.g., v1/ab12/model.onnx).

Verify

After applying the changes, perform the following checks.

1. Observe pod startup logs

2026-09-22T14:02:01.112Z INFO  ModelLoader - Model found in /model/model.onnx, skipping download
2026-09-22T14:02:01.115Z INFO  HealthCheck - Inference ready

2. Confirm no 429 errors in CloudWatch

aws cloudwatch get-metric-statistics \
    --namespace AWS/S3 \
    --metric-name 4xxErrors \
    --statistics Sum \
    --period 60 \
    --start-time 2026-09-22T14:00:00Z \
    --end-time 2026-09-22T15:00:00Z

Result: Sum = 0 for the monitored period.

3. Validate inference latency

curl -s http://onnx-runtime.ml.svc.cluster.local/predict -d '{"input": [1,2,3]}' | jq .

Latency returns to ~15 ms per request, matching pre‑incident baseline.

4. Check retry metrics

kubectl logs -l app=onnx-runtime -c onnx-runtime | grep "Retry"

Only occasional retries observed during a controlled load test, confirming backoff works.

Prevent

• Cache models locally: Use init containers or sidecars to pull artifacts once per node.
• Enable SDK retries: Configure max_attempts and mode=standard for all S3 interactions.
• Apply exponential backoff with jitter to avoid thundering‑herd effects during rollouts.
• Monitor S3 throttling metrics (4xx errors, ThrottlingException) and set alerts.
• Rate‑limit pod restarts: Use maxSurge and maxUnavailable settings in the Deployment to stagger rollouts.
• Use bucket prefixes or request‑rate‑optimized storage classes when high concurrency is expected.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the model load succeed after a pod restart but fail during a rolling update?

   The restart triggers a single download, while a rolling update may start multiple pods simultaneously, exceeding the S3 request rate limit and causing 429 throttling.

2. Can I rely on the default AWS SDK retry policy?

   No. For GetObject the default max_attempts is 1. Explicitly set a higher retry count and use mode=standard to enable exponential backoff.

3. Is increasing the S3 request quota always safe?

   It mitigates throttling but does not address the underlying thundering‑herd pattern. Combining quota increase with caching and backoff yields a more robust solution.

4. How do I verify that the init container has successfully cached the model?

   Check the init container exit code (should be 0) and list the file inside the shared volume: kubectl exec -- ls -l /model/model.onnx.

5. What if the model size is >1 GiB and does not fit in an emptyDir?

   Use a hostPath volume backed by a node‑local SSD or a persistent volume with sufficient capacity, and mount it read‑only for the inference container.