ONNX Runtime authentication errors after secret rotation in multi‑region replication

Problem

After a scheduled secret rotation, ONNX Runtime instances deployed in several cloud regions start failing to load model artifacts. The runtime logs contain authentication errors, and the model loading step hangs or aborts. This outage appears only after the rotation completes, despite the secret manager entries being updated correctly.

Root Cause

ONNX Runtime uses the Ort::SessionOptions AddExternalFile API to fetch model files from a private object store (e.g., Amazon S3, Azure Blob, GCS). The credentials are obtained from a cloud‑native secret manager (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) via a side‑car credential‑provider process. The rotation workflow replaces the secret value but does not invalidate the cached credentials held by the provider. Because each region runs its own provider process, the following sequence occurs:

• Rotation updates the secret value in the manager.
• The provider process continues to serve the old, now‑revoked token until it receives a SIGHUP or restarts.
• ONNX Runtime instances, which keep a long‑lived HTTP connection pool to the object store, reuse the stale token, resulting in 403 Forbidden or 401 Unauthorized responses.
• Only after a manual restart of the provider (or the entire pod) do the new credentials propagate, restoring access.

The root cause is therefore a missing cache‑invalidation mechanism for secret‑derived credentials in the multi‑region deployment.

Debug

Below is a typical log excerpt from an ONNX Runtime pod after rotation:

2026-09-30T12:15:42.317Z [ERROR] OrtSession::LoadModelFromFile: Failed to open model file s3://ml-models/us-east-1/model.onnx
2026-09-30T12:15:42.317Z [ERROR] S3Client::GetObject: HTTP/1.1 403 Forbidden
2026-09-30T12:15:42.317Z [INFO]  RequestId: 3F2A1B7C5D6E7F8G9H0I
2026-09-30T12:15:42.317Z [DEBUG] CredentialProvider: Using cached token expiring at 2026-09-30T12:10:00Z

Key observations:

• The token expiration timestamp (12:10:00Z) is earlier than the current time, indicating a stale token.
• No reload of the secret is attempted after the initial load.
• The same error repeats across all regions, confirming a systemic caching issue.

Diagnostic steps performed:

1. Inspect the provider process:

$ ps -ef | grep credential-provider
root     12457 12455  0 12:00 ?        00:00:00 /usr/local/bin/credential-provider --region us-east-1

2. Check the provider’s internal cache file (if any):

$ cat /var/run/credential-provider/cache.json
{
  "access_key_id": "AKIA************",
  "secret_access_key": "**************",
  "session_token": "FQoGZXIvYXdzE...",
  "expiration": "2026-09-30T12:10:00Z"
}

3. Force a token refresh manually:

$ kill -SIGHUP 12457   # sends reload signal
$ journalctl -u credential-provider -n 20
...
2026-09-30T12:16:05.123Z INFO Refreshing credentials from Secrets Manager
2026-09-30T12:16:05.456Z INFO New token expires at 2026-09-30T18:00:00Z

4. Verify that ONNX Runtime picks up the new token:

$ curl -v -H "Authorization: Bearer $(cat /var/run/credential-provider/cache.json | jq -r .session_token)" \
  https://s3.us-east-1.amazonaws.com/ml-models/us-east-1/model.onnx -o /dev/null
*   Trying 52.216.0.1...
* Connected to s3.us-east-1.amazonaws.com (52.216.0.1) port 443 (#0)
> GET /ml-models/us-east-1/model.onnx HTTP/1.1
> Host: s3.us-east-1.amazonaws.com
> Authorization: Bearer AQoDYXdzE...
> User-Agent: curl/7.88.1
> Accept: */*
< HTTP/1.1 200 OK
...

After the manual reload, the model loads successfully, confirming the cache‑invalidation hypothesis.

Solution

The fix consists of two parts:

1. Update the credential‑provider to watch the secret manager for changes and automatically refresh the cache. This can be achieved by enabling the native SecretsManagerEventBridge integration (AWS) or KeyVaultEventGrid (Azure) and configuring the provider to listen for SecretRotation events.
2. Modify the ONNX Runtime deployment to restart the provider process on rotation without affecting the model serving containers. This is done by adding a sidecar container that watches the secret version and sends SIGHUP to the provider when a new version is detected.

Below are the relevant configuration changes.

Before (static provider, no event handling)

apiVersion: v1
kind: Pod
metadata:
  name: onnx-runtime-us-east-1
spec:
  containers:
  - name: onnx-runtime
    image: myrepo/onnx-runtime:2.1
    env:
    - name: MODEL_URI
      value: s3://ml-models/us-east-1/model.onnx
  - name: credential-provider
    image: myrepo/credential-provider:1.0
    args: ["--region", "us-east-1"]
    env:
    - name: SECRET_ID
      value: arn:aws:secretsmanager:us-east-1:123456789012:secret:ml-credentials

After (event‑driven refresh and sidecar watcher)

apiVersion: v1
kind: Pod
metadata:
  name: onnx-runtime-us-east-1
spec:
  containers:
  - name: onnx-runtime
    image: myrepo/onnx-runtime:2.1
    env:
    - name: MODEL_URI
      value: s3://ml-models/us-east-1/model.onnx
  - name: credential-provider
    image: myrepo/credential-provider:1.2
    args: ["--region", "us-east-1", "--enable-event-listener"]
    env:
    - name: SECRET_ID
      value: arn:aws:secretsmanager:us-east-1:123456789012:secret:ml-credentials
    - name: EVENT_BUS_ARN
      value: arn:aws:events:us-east-1:123456789012:event-bus/default
  - name: secret-watcher
    image: bitnami/kubectl:1.27
    command: ["/bin/sh", "-c"]
    args:
      - |
        while true; do
          VERSION=$(aws secretsmanager describe-secret --secret-id $SECRET_ID \
            --query 'VersionIdsToStages' --output text | awk '{print $1}')
          if [ "$VERSION" != "$(cat /tmp/last_version 2>/dev/null)" ]; then
            echo $VERSION > /tmp/last_version
            kill -SIGHUP $(pidof credential-provider)
          fi
          sleep 30
        done
    env:
    - name: SECRET_ID
      value: arn:aws:secretsmanager:us-east-1:123456789012:secret:ml-credentials

Key changes:

• The provider now runs with --enable-event-listener, subscribing to the EventBridge rule that fires on secret rotation.
• A lightweight secret-watcher sidecar polls the secret version and sends SIGHUP to the provider when a change is detected, ensuring a refresh even if the event delivery is delayed.
• No changes are required in the ONNX Runtime container; it continues to read the token from the shared cache file.

Verify

After applying the updated manifest, perform the following checks:

1. Confirm the provider receives rotation events:

$ kubectl logs onnx-runtime-us-east-1 -c credential-provider | grep "Received rotation event"
2026-09-30T13:00:02.112Z INFO Received rotation event for secret ml-credentials
2026-09-30T13:00:02.415Z INFO Refreshing credentials from Secrets Manager

2. Validate that the cache file reflects the new expiration:

$ cat /var/run/credential-provider/cache.json | jq .expiration
"2026-09-30T19:00:00Z"

3. Run a model load test:

$ python - <<'PY'
import onnxruntime as ort
sess = ort.InferenceSession("s3://ml-models/us-east-1/model.onnx")
print("Model loaded, inputs:", sess.get_inputs())
PY
Model loaded, inputs: [Name: input_1, Shape: [1, 3, 224, 224], Type: float]

4. Check that no authentication errors appear in the runtime logs for at least one full rotation interval.

Prevent

To avoid recurrence, adopt the following guardrails:

• Enable automatic credential refresh in the provider library. Ensure the --enable-event-listener flag is part of every deployment.
• Instrument health checks that verify the provider can reach the secret manager. Example endpoint: /healthz?type=credential returning HTTP 200 only when a fresh token (< 5 min to expiry) is cached.
• Set up alerts on authentication error spikes. Use a log‑based metric that counts 403 Forbidden responses from the object store with a label source=onnx-runtime.
• Document the rotation schedule and ensure the sidecar watcher runs with a short poll interval (≤ 30 s) in case event delivery is delayed.
• Run integration tests that simulate secret rotation. The test suite should rotate a dummy secret, trigger the watcher, and assert that a model load succeeds without pod restarts.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the issue appear only after rotation and not during initial deployment?
   During the initial start, the provider fetches a fresh token and caches it. Rotation replaces the secret but the provider continues to serve the cached token until it receives a reload signal, causing the failure.
2. Can I rely solely on the provider’s built‑in event listener without a sidecar?
   The built‑in listener works if the event bus delivers events reliably. Adding a sidecar watchdog provides a safety net for missed events and is recommended for production multi‑region setups.
3. What if my object store uses IAM roles instead of static credentials?
   When using IAM roles (e.g., EC2 instance profile or EKS service account), the provider should request temporary credentials from the metadata service. The same caching issue applies; ensure the provider refreshes tokens based on the Expiration field.
4. How do I test the rotation flow in a staging environment?
   Create a test secret, configure the provider to point to it, rotate the secret manually via the cloud console or CLI, and verify that the provider logs a refresh and that ONNX Runtime can still load the model.
5. Is there a performance impact of refreshing credentials on every request?
   No. The provider refreshes only when the token is near expiration or when a rotation event is received. Regular inference requests continue to use the cached token, preserving low latency.