yellow and white 10 card

Qwen token refresh failure causing auth errors after hybrid cloud migration

by

Problem – Symptom Overview

After migrating the Qwen inference workload to a hybrid cloud topology (on‑premises servers + public‑cloud VMs), API calls to the Qwen service began returning HTTP 401/403 errors. The errors are intermittent on the cloud side but consistent on‑premises. Example response payloads:

{
  "error": "invalid_token",
  "error_description": "The access token expired or is malformed."
}

Log excerpts from the Qwen gateway show a recurring pattern:

2026-10-11T14:23:07.842Z WARN  token_manager.go:112] Token refresh failed: oauth2: token expired
2026-10-11T14:23:07.845Z ERROR request_handler.go:57] Authentication error for request ID 7f9c3d: invalid_token

Impact:

  • Client applications experience request failures, causing downstream job retries and SLA breaches.
  • Metrics show a spike in qwen_api_auth_failure_total from 0 → 150/min within minutes of the migration.
  • On‑premises nodes exhaust their token cache, leading to a cascade of 401 responses.

Root Cause – Why the Refresh Fails

The Qwen service obtains short‑lived OAuth2 access tokens from a shared Auth Service. Tokens are cached locally on each inference node and refreshed automatically 30 seconds before expiry. The hybrid deployment introduced two subtle changes:

  1. Network address translation (NAT) on the public cloud subnet. The Auth Service sees the source IP of cloud nodes as the NAT gateway IP, while on‑premises nodes appear with their true IPs.
  2. Clock drift between environments. Cloud VMs were provisioned with NTP disabled, resulting in a systematic 45‑second forward skew.

The token refresh logic builds the client_id claim from the node’s IP address. When the NAT‑translated IP does not match the one stored in the original token, the Auth Service rejects the refresh request with invalid_grant. Additionally, the forward clock skew causes the client to believe the token is still valid while the server already considers it expired, leading to the “token expired” log entry.

Debugging – Investigation Process

The following steps reproduced the failure and pinpointed the cause:

1. Correlate logs from Qwen nodes and Auth Service

# On a cloud node
journalctl -u qwen-inference -f | grep "Token refresh"

# On Auth Service
journalctl -u auth-service -f | grep "grant_type=refresh_token"

Observed mismatch in client_ip field of the refresh request.

2. Verify system clocks

# On a cloud VM
timedatectl status

# Sample output
Local time: Thu 2026-10-12 03:12:45 UTC
Universal time: Thu 2026-10-12 03:12:45 UTC
RTC time: n/a
Time zone: UTC (UTC, +0000)
System clock synchronized: no
NTP service: active

Clock synchronization was disabled.

3. Capture the refresh HTTP request

# Using tcpdump on the cloud node
sudo tcpdump -i eth0 -s 0 -w refresh.pcap port 443

# Decode with tshark
tshark -r refresh.pcap -Y http.request -T fields -e http.host -e http.request.full_uri

The client_ip query parameter resolved to the NAT gateway address (e.g., 34.215.12.7) instead of the node’s private IP.

4. Compare token cache files

# On‑premises node
cat /var/lib/qwen/token_cache.json
{
  "access_token":"eyJhbGciOi...","expires_at":1697126400,"client_ip":"10.2.5.23"
}

# Cloud node
cat /var/lib/qwen/token_cache.json
{
  "access_token":"eyJhbGciOi...","expires_at":1697126400,"client_ip":"34.215.12.7"
}

The differing client_ip explains the Auth Service rejection.

Solution – Fixing the Refresh Path

Two independent changes resolve the issue:

1. Enforce NTP synchronization on all cloud instances

Update the cloud VM image to enable chrony and enforce a drift < 5 seconds.

# Before (cloud VM image)
# /etc/chrony.conf – NTP disabled
pool 0.pool.ntp.org iburst
# No "allow" directive

# After
# /etc/chrony.conf
pool pool.ntp.org iburst
allow 10.0.0.0/8
driftfile /var/lib/chrony/drift

Restart the service:

systemctl restart chronyd
timedatectl set-ntp true

2. Modify Qwen token manager to use a stable identifier instead of the source IP

Patch token_manager.go to replace the client_ip claim with a configurable node_id that is identical across environments.

// Before
func buildRefreshRequest(token *oauth2.Token) *http.Request {
    req, _ := http.NewRequest("POST", authEndpoint, nil)
    q := req.URL.Query()
    q.Add("client_id", getLocalIP())
    q.Add("refresh_token", token.RefreshToken)
    req.URL.RawQuery = q.Encode()
    return req
}

// After
func buildRefreshRequest(token *oauth2.Token) *http.Request {
    req, _ := http.NewRequest("POST", authEndpoint, nil)
    q := req.URL.Query()
    // nodeID is read from /etc/qwen/node_id (same value on‑prem and cloud)
    q.Add("client_id", getNodeID())
    q.Add("refresh_token", token.RefreshToken)
    req.URL.RawQuery = q.Encode()
    return req
}

Deploy the updated binary and create a consistent node_id file on every node:

# /etc/qwen/node_id
node-001

3. Refresh the token cache

Delete stale cache files to force a fresh authentication flow.

rm -f /var/lib/qwen/token_cache.json
systemctl restart qwen-inference

Verification – Confirming the Fix

  1. Check that NTP is synchronized on cloud VMs.
# Expected output
System clock synchronized: yes
NTP service: active
  1. Inspect the refreshed token cache; client_id should now contain the static node_id.
# Sample token_cache.json after fix
{
  "access_token":"eyJhbGciOi...","expires_at":1697126400,"client_id":"node-001"
}
  1. Trigger a refresh request and verify Auth Service logs a successful grant.
# Auth Service log snippet
2026-10-12T03:15:22Z INFO token_service.go:78] Refresh token granted for client_id=node-001
  1. Run a load test against the Qwen endpoint and confirm qwen_api_auth_failure_total returns to zero.
# Prometheus query
rate(qwen_api_auth_failure_total[5m]) == 0

Prevention – Guardrails for Future Hybrid Deployments

  • Clock drift monitoring: Export node_clock_offset_seconds from chrony and alert if >5 s.
  • Static node identifiers: Enforce the presence of /etc/qwen/node_id via configuration management; reject deployments missing it.
  • Auth Service request validation: Log mismatched client_id values and emit a metric auth_mismatch_total for early detection.
  • Network topology awareness: When NAT is present, configure the client to send a header (e.g., X-Real-IP) that the Auth Service can ignore for token refresh.
  • Automated token cache health check: A cron job that verifies the cached token is still valid (via introspection endpoint) and forces a refresh if expiry is within 60 seconds.

FAQ – Common Follow‑up Questions

  1. Why did the on‑premises nodes continue working while cloud nodes failed?
    On‑premises nodes used their true IP addresses, which matched the client_id stored in the original token. Cloud nodes were NAT‑translated, causing a mismatch.
  2. Can I keep using the IP address as the client identifier?
    Only if you guarantee a one‑to‑one mapping between the IP seen by the Auth Service and the node’s cache. In NAT or load‑balanced environments, a static identifier is safer.
  3. Do I need to rotate existing tokens after the fix?
    Yes. Deleting the stale token_cache.json forces a fresh login, ensuring the new client_id is embedded.
  4. How do I detect clock drift before it causes token failures?
    Export the chrony_tracking metric (or run chronyc tracking) and set an alert threshold of 5 seconds.
  5. Is there a way to make the Auth Service ignore the client_id during refresh?
    Modifying the Auth Service to treat client_id as optional for refresh grants is possible, but it weakens the binding between token and node. Prefer a stable identifier instead.

Related Topic Hub: LLM Systems Troubleshooting Hub