Amd ryzen 6000 series processor on circuit board background

AMD GPU token refresh failure during disaster recovery

by

Problem – Token Refresh Failure During Disaster Recovery on AMD GPU Systems

During a planned disaster‑recovery (DR) drill, AI inference workloads that rely on AMD Instinct GPUs stopped abruptly after the failover to the backup cluster. The ROCm runtime emitted the following error:

TokenRefreshError: failed to refresh access token – ROCm runtime aborts with exit code 1

Additional symptoms observed across the fleet:

  • Kernel log entry: [rocm] auth token renewal failed, error code 0x80070005.
  • Container logs: [amd-gpu] token_refresh: timeout after 30s while contacting auth server
  • Kubernetes events: FailedMount: token file not found in /var/run/secrets/...
  • Inference pods remain in CrashLoopBackOff state despite the GPU being healthy.

The failure prevents any AI workload from executing on the secondary site, breaking the service‑level agreement for high‑availability inference.

Root Cause – Why the Token Refresh Fails After Failover

AMD’s ROCm security model stores an authentication token that grants the driver and libraries access to protected resources such as encrypted model blobs or secure model registries. The token lifecycle is described in the ROCm Documentation – Security and Authentication and the Instinct GPU Architecture Guide – Secure Execution and Token Management.

During a normal run the driver:

  1. Loads a cached token from /var/run/rocm/token (or the path set by ROCM_TOKEN_DIR).
  2. Monitors the token’s expiry timestamp.
  3. When the token is within the renewal window, it contacts the configured authentication service (OAuth, Azure Key Vault, etc.) to obtain a fresh token.

In a DR scenario the following conditions commonly break this flow:

  • Missing service‑account credentials on the backup host. The backup node cannot reach the original Key Vault endpoint, leading to amdgpu: token acquisition failed: -EACCES in the kernel log (see the financial‑services incident).
  • Token cache directory not replicated. The token file is absent on the secondary host, causing a cache miss and immediate abort (FailedMount: token file not found from the Kubernetes event).
  • Driver flag --auth-token-refresh=off inherited from the primary node. ROCm 6.0.0 release notes state that the flag disables automatic renewal and is unsuitable for DR clusters.
  • Network egress restrictions. The backup subnet cannot reach the OAuth endpoint, resulting in a timeout (token_refresh: timeout after 30s while contacting auth server).

The combination of a missing credential source and an unreplicated token cache forces the driver to abort before any GPU work can be scheduled.

Debug – Investigation Steps

Below is a reproducible debugging workflow that was used to isolate the failure on a Kubernetes‑managed AI platform.

  1. Check driver logs for token errors.
# journalctl -u rocm-driver -b | grep -i token
[  12.345678] [rocm] auth token renewal failed, error code 0x80070005.
[  12.347890] [amd-gpu] token_refresh: timeout after 30s while contacting auth server
  1. Validate token cache presence.
# ls -l /var/run/rocm/token
ls: cannot access '/var/run/rocm/token': No such file or directory
  1. Inspect environment variables used by ROCm.
# env | grep ROCM
ROCM_PATH=/opt/rocm
ROCM_TOKEN_DIR=/var/run/rocm/token
ROCM_AUTH_MODE=oauth

If ROCM_TOKEN_DIR points to a non‑existent directory, the driver will not attempt a refresh.

  1. Verify connectivity to the auth service.
# curl -v https://mykeyvault.vault.azure.net/.well-known/openid-configuration
*   Trying 10.1.2.3:443...
* connect to 10.1.2.3 port 443 failed: Connection timed out

A timeout here matches the “token_refresh: timeout after 30s” log entry.

  1. Check driver flags in the container runtime.
# docker inspect my-rocmlib | grep auth
"ROCM_DRIVER_FLAGS": "--auth-token-refresh=off"

The flag disables renewal, which is appropriate only for static, single‑node deployments.

Solution – Restoring Token Refresh Capability

The fix consists of three coordinated actions:

1. Replicate the token cache directory and seed it with a valid bootstrap token.

On the primary site, export the current token:

# cat /var/run/rocm/token > /tmp/rocm_token.backup

Copy it to the backup host (e.g., via rsync over a secure channel):

# rsync -avz /tmp/rocm_token.backup backup-node:/var/run/rocm/token
# chmod 600 /var/run/rocm/token

2. Provide the backup node with a service‑account key that can request new tokens.

For Azure Key Vault, create a dedicated service principal and store its credentials in a Kubernetes secret:

# kubectl create secret generic dr-keyvault-cred \
    --from-literal=client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
    --from-literal=client_secret=YOUR_SECRET \
    --namespace=ai-workloads

Mount the secret into the pod and point ROCm to it:

env:
  - name: AZURE_CLIENT_ID
    valueFrom:
      secretKeyRef:
        name: dr-keyvault-cred
        key: client_id
  - name: AZURE_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: dr-keyvault-cred
        key: client_secret
  - name: ROCM_TOKEN_DIR
    value: /var/run/rocm/token

3. Enable automatic token renewal.

Remove the disabling flag and explicitly enable renewal via driver configuration (ROCm 6.0.0 introduced --auth-token-refresh=on).

Before:

{
  "driver_flags": "--auth-token-refresh=off"
}

After:

{
  "driver_flags": "--auth-token-refresh=on"
}

If the driver is launched via rocm-docker, update the entrypoint script:

#!/bin/bash
exec /opt/rocm/bin/rocminfo --auth-token-refresh=on "$@"

4. Ensure network egress to the auth endpoint.

Add a firewall rule or VPC route on the DR subnet:

# iptables -I OUTPUT -p tcp -d mykeyvault.vault.azure.net --dport 443 -j ACCEPT

Verify – Confirming the Fix Works

  1. Restart the affected pods or services.
  2. Check that the token file exists and is readable.
  3. Observe driver logs for successful renewal.
# journalctl -u rocm-driver -f
[  45.123456] [rocm] auth token renewal succeeded, new expiry: 2026-08-01T12:00:00Z
[  45.124789] [amd-gpu] token_refresh: completed in 1.2s

Run a simple inference job to ensure the GPU is usable:

# python -c "import torch; print(torch.cuda.is_available())"
True

Metrics from Prometheus should now show gpu_inference_success_total increasing on the backup cluster.

Prevent – Operational Guardrails for Future DR Events

Preventive Action Implementation Detail Monitoring/Alert
Synchronize token cache directory across primary and DR sites Use rsync or a shared NFS mount; run nightly job Alert if /var/run/rocm/token missing on any node
Store service‑account credentials in a secret manager with DR replication Azure Key Vault geo‑replication or HashiCorp Vault HA Alert on secret read failures (HTTP 403/404)
Enforce driver flag --auth-token-refresh=on in CI pipelines Lint Dockerfile / helm chart for prohibited flags Fail build if prohibited flag detected
Validate network egress to auth endpoints during DR drills Run curl -s -o /dev/null -w "%{http_code}" https://mykeyvault... as a pre‑flight check Alert on non‑200 HTTP code during node provisioning

Integrate the above checks into your automated DR test suite to catch regressions before a real failover.

FAQ – Common Follow‑Up Questions

  1. Why does the token refresh succeed on the primary cluster but fail on the backup?

    The primary cluster has the service‑account key and a populated token cache, while the backup lacks both. The driver therefore cannot contact the auth server or locate a valid token.

  2. Can I disable token caching entirely?

    Disabling the cache (ROCM_TOKEN_CACHE=0) forces the driver to request a new token on every startup, which adds latency and increases load on the auth service. It is not recommended for production DR scenarios.

  3. What ROCm version introduced the --auth-token-refresh flag?

    The flag was added in ROCm 6.0.0, as noted in the ROCm 6.0.0 release notes under “Token Refresh and Failover Support”. Upgrading to at least this version is required for reliable DR token renewal.

  4. How do I troubleshoot a persistent “token acquisition failed: -EACCES” error?

    Check the IAM role bindings for the service account on the backup cluster, verify that the secret containing the credentials is mounted, and ensure the driver process runs with sufficient permissions to read the token file.

  5. Is it safe to copy the token file from the primary to the backup?

    Yes, provided the token is still within its validity window and the backup node has the same client identity (service principal). Refresh the token after the copy to avoid using a near‑expiry token.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub