ONNX Runtime Token Acquisition Timeout After Azure AD OAuth2 Config Change

Q: Problem – ONNX Runtime Fails to Load Model After Azure AD OAuth2 Config Change

In a staging environment the ONNX Runtime instance is configured to download models from an Azure Machine Learning Model Registry that is protected by Azure AD OAuth2 client‑credentials flow. After a recent change to the Azure AD application (secret rotation, redirect‑URI update, or role modification) model loading started to fail with the following symptoms:

Q: Debug – Systematic Investigation Steps 1. Verify the exact HTTP response from Azure AD

curl -X POST \ -d "client_id=YOUR_CLIENT_ID" \ -d "client_secret=YOUR_CLIENT_SECRET" \ -d "grant_type=client_credentials" \ -d "scope=https%3A%2F%2Fml.azure.com%2F.default" \ https://login.microsoftonline.com/YOUR_TENANT_ID/oauth2/v2.0/token -v Expected output (successful case):

Q: 3. Check network connectivity from the pod/container

# Verify DNS resolution nslookup login.microsoftonline.com # Test TCP connectivity (default 443) nc -zv login.microsoftonline.com 443 # Capture the outbound request (requires tcpdump) tcpdump -i eth0 host login.microsoftonline.com and port 443 -w token.pcap If the capture shows no outbound SYN packets, a network policy or firewall is blocking the request.

Q: 4. Review Azure AD app settings Client secret validity – confirm the secret has not expired. Redirect URIs – ensure the URI used by ONNX Runtime (usually urn:ietf:wg:oauth:2.0:oob) is still registered. API permissions – verify the service principal has the “Model Registry Reader” role (Model Registry docs). Token lifetime – if reduced, consider enabling token caching refresh logic or increasing the timeout in the provider. 5. Examine ONNX Runtime logs for retry behavior

2024-09-04 12:15:32.487 INFO OrtSessionOptions: Starting token acquisition (attempt 1) 2024-09-04 12:15:32.492 WARN OrtSessionOptions: Token request failed with 400 invalid_client 2024-09-04 12:15:32.497 INFO OrtSessionOptions: Retrying token acquisition (attempt 2) ... 2024-09-04 12:15:47.501 ERROR OrtSession: Model load failed due to token acquisition timeout The log shows repeated retries until the provider’s internal timeout (default 20 seconds) is reached.

Q: Scenario B: Redirect URI Change

Re‑add the original redirect URI (urn:ietf:wg:oauth:2.0:oob) to the Azure AD app’s “Redirect URIs” list. No code change is required; the provider will succeed once Azure AD accepts the request.

Q: Scenario D: Network Policy Blocking Token Endpoint

Modify the Kubernetes NetworkPolicy to allow egress to login.microsoftonline.com on port 443:

In this article

Table of Contents

Problem – ONNX Runtime Fails to Load Model After Azure AD OAuth2 Config Change

In a staging environment the ONNX Runtime instance is configured to download models from an Azure Machine Learning Model Registry that is protected by Azure AD OAuth2 client‑credentials flow. After a recent change to the Azure AD application (secret rotation, redirect‑URI update, or role modification) model loading started to fail with the following symptoms:


2024-09-04 12:15:32.487 INFO  OrtSessionOptions: Authentication provider returned error code 401
2024-09-04 12:15:32.489 ERROR OrtSession: Model load failed due to token acquisition timeout
Traceback (most recent call last):
  File ".../onnxruntime/__init__.py", line 124, in load_model
    session = InferenceSession(model_path, sess_options)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplementedError: OAuth2 flow interrupted

Other log entries observed in the same run:


Failed to acquire token: request timed out

The error matches the common_errors entry “Error: "Failed to acquire token: request timed out" – appears in ONNX Runtime logs when the HTTP call to https://login.microsoftonline.com/.../oauth2/v2.0/token exceeds the configured timeout.

Root Cause – Why the Token Acquisition Timeout Occurs

ONNX Runtime delegates token acquisition to an authentication provider that performs a standard OAuth2 client‑credentials request (Azure AD docs). The provider expects a successful HTTP 200 response within a default 15‑second window. The timeout can be triggered by any of the following conditions, all of which have been reported in real incidents:

Invalid client secret or client_id after a secret rotation – Azure AD returns 400 invalid_client, the provider retries until the timeout expires (GitHub issue #12457).
Redirect URI mismatch after updating the Azure AD app – Azure AD rejects the request with 400 invalid_client, leading to repeated retries (real incident #2).
Insufficient permissions – the service principal lost the “Model Registry Reader” role, Azure AD returns 403 and the provider treats it as a transient failure (real incident #4).
Network connectivity blockage – e.g., a Kubernetes network policy prevented outbound traffic to the token endpoint, causing the HTTP client to wait until its socket timeout (real incident #3).
Token lifetime reduction – Azure AD app configured a 5‑minute token lifetime; ONNX Runtime cached the token past expiration and attempted a refresh that timed out (real incident #1).

In all cases the provider logs “Failed to acquire token: request timed out” because the underlying requests call never received a successful response within the allotted time.

Debug – Systematic Investigation Steps

1. Verify the exact HTTP response from Azure AD


curl -X POST \
  -d "client_id=YOUR_CLIENT_ID" \
  -d "client_secret=YOUR_CLIENT_SECRET" \
  -d "grant_type=client_credentials" \
  -d "scope=https%3A%2F%2Fml.azure.com%2F.default" \
  https://login.microsoftonline.com/YOUR_TENANT_ID/oauth2/v2.0/token -v

Expected output (successful case):


< HTTP/2 200
{
  "token_type":"Bearer",
  "expires_in":3599,
  "access_token":"eyJ0eXAiOiJKV1QiLCJ..."
}

If you see 400 with invalid_client or 403, the problem is in the Azure AD app configuration.

2. Inspect ONNX Runtime authentication provider configuration

Python example using the built‑in provider:


import onnxruntime as ort

sess_options = ort.SessionOptions()
sess_options.authentication_provider = ort.AuthProvider(
    authority="https://login.microsoftonline.com/YOUR_TENANT_ID",
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    scope="https://ml.azure.com/.default"
)

ort.InferenceSession("model.onnx", sess_options)

If a custom provider is used (C++ API OrtSessionOptionsSetAuthenticationProvider), ensure the callback respects the 15‑second deadline (C++ docs).

3. Check network connectivity from the pod/container


# Verify DNS resolution
nslookup login.microsoftonline.com

# Test TCP connectivity (default 443)
nc -zv login.microsoftonline.com 443

# Capture the outbound request (requires tcpdump)
tcpdump -i eth0 host login.microsoftonline.com and port 443 -w token.pcap

If the capture shows no outbound SYN packets, a network policy or firewall is blocking the request.

4. Review Azure AD app settings

Client secret validity – confirm the secret has not expired.
Redirect URIs – ensure the URI used by ONNX Runtime (usually urn:ietf:wg:oauth:2.0:oob) is still registered.
API permissions – verify the service principal has the “Model Registry Reader” role (Model Registry docs).
Token lifetime – if reduced, consider enabling token caching refresh logic or increasing the timeout in the provider.

5. Examine ONNX Runtime logs for retry behavior


2024-09-04 12:15:32.487 INFO  OrtSessionOptions: Starting token acquisition (attempt 1)
2024-09-04 12:15:32.492 WARN  OrtSessionOptions: Token request failed with 400 invalid_client
2024-09-04 12:15:32.497 INFO  OrtSessionOptions: Retrying token acquisition (attempt 2)
...
2024-09-04 12:15:47.501 ERROR OrtSession: Model load failed due to token acquisition timeout

The log shows repeated retries until the provider’s internal timeout (default 20 seconds) is reached.

Solution – Fixing the Token Acquisition Timeout

Scenario A: Secret Rotation or Client Credential Mismatch

Before (invalid secret):


client_secret = "old_secret_123"

After (updated secret):


client_secret = "new_secret_ABCdef456"

Update the secret in the deployment configuration (e.g., Kubernetes Secret or Azure Key Vault reference) and redeploy the pod.

Scenario B: Redirect URI Change

Re‑add the original redirect URI (urn:ietf:wg:oauth:2.0:oob) to the Azure AD app’s “Redirect URIs” list. No code change is required; the provider will succeed once Azure AD accepts the request.

Scenario C: Missing Role Assignment

Grant the service principal the “Model Registry Reader” role:


az role assignment create \
  --assignee $SERVICE_PRINCIPAL_ID \
  --role "Model Registry Reader" \
  --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE

Scenario D: Network Policy Blocking Token Endpoint

Modify the Kubernetes NetworkPolicy to allow egress to login.microsoftonline.com on port 443:


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-azure-ad-token
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 13.107.0.0/16   # Azure AD IP range (example)
    ports:
    - protocol: TCP
      port: 443

Scenario E: Token Lifetime Reduction

Increase the provider’s timeout and enable proactive token refresh:


import onnxruntime as ort
import time

class RefreshingAuthProvider(ort.AuthProvider):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._token = None
        self._expiry = 0

    def get_token(self):
        now = int(time.time())
        # Refresh if less than 60 seconds to expiry
        if self._token is None or self._expiry - now < 60:
            self._token, self._expiry = self._request_token()
        return self._token

    def _request_token(self):
        # Same curl/post logic as earlier, but with a 30‑second timeout
        ...

sess_options = ort.SessionOptions()
sess_options.authentication_provider = RefreshingAuthProvider(
    authority="https://login.microsoftonline.com/YOUR_TENANT_ID",
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    scope="https://ml.azure.com/.default",
    timeout_seconds=30   # custom timeout
)

Deploy the updated code and verify that token refresh occurs before expiration.

Verify – Confirming the Issue Is Resolved

Run a model load operation and watch the logs for a successful token acquisition:


2024-09-04 13:02:11.123 INFO  OrtSessionOptions: Token acquisition succeeded (expires_in=3599)
2024-09-04 13:02:11.130 INFO  OrtSession: Model loaded successfully

Use az account get-access-token with the same client_id/secret to ensure Azure AD returns a token:


az account get-access-token \
  --resource https://ml.azure.com/ \
  --client-id YOUR_CLIENT_ID \
  --client-secret YOUR_CLIENT_SECRET

The command should output a JSON object with accessToken and expiresOn.

Execute an inference request against the loaded model to confirm end‑to‑end functionality.

Prevent – Guardrails and Operational Best Practices

Monitoring: Emit a custom metric (e.g., onnxruntime_token_acquisition_latency_seconds) and alert if latency exceeds 5 seconds or if the error “Failed to acquire token” appears more than once in a 5‑minute window.
Secret Rotation Automation: Store client secrets in Azure Key Vault and use Managed Identity or Secret Rotation policies to update the runtime without redeploy.
Role Auditing: Periodically run az role assignment list --assignee $SP_ID and compare against a baseline to detect accidental permission loss.
Network Policy Review: Include egress rules for Azure AD token endpoints in a baseline network policy template.
Token Lifetime Awareness: If a short token lifetime is required, implement a proactive refresh strategy as shown in Scenario E.

FAQ – Common Follow‑Up Questions

Q1: Why does the token acquisition succeed locally but time out in the Kubernetes pod?

A: The pod is likely subject to a network policy or outbound firewall that blocks traffic to login.microsoftonline.com. Verify egress rules and capture traffic with tcpdump.

Q2: After rotating the client secret, why does ONNX Runtime still use the old secret for a few minutes?

A: ONNX Runtime caches the token until its expires_in value. If the token lifetime is short, the cache may hold an expired token, causing a timeout. Restart the process or implement a custom provider that discards cached tokens on secret change.

Q3: How can I increase the HTTP timeout for the built‑in authentication provider?

A: The Python API does not expose a direct timeout parameter, but you can wrap the provider with a custom implementation (see Scenario E) that sets a larger timeout on the underlying requests.post call.

Q4: Is there a way to see which Azure AD scopes are being requested by ONNX Runtime?

A: The provider uses the scope argument supplied during SessionOptions construction. Verify the value matches the registry’s required scope (e.g., https://ml.azure.com/.default).

Q5: What should I do if Azure AD returns a 429 “Too Many Requests” during token acquisition?

A: Azure AD throttles token endpoint calls. Implement exponential back‑off in a custom token provider and consider caching the token for its full lifetime to reduce request frequency.

Related Topic Hub: Model Serving Troubleshooting Hub