OAuth2 Token Request Timeout In Kubernetes Pods After Docker Restart

Q: 2. Update OAuth2 Client Configuration

Change the redirect_uri from a pod‑specific address to a stable service DNS name (e.g., https://auth-callback.ai.svc.cluster.local/callback). This avoids reliance on pod IPs.

In this article

Table of Contents

Problem – OAuth2 Token Request Timeout After Docker Restart

After a Docker daemon restart on a node, pods that host the AI inference service begin to fail during the OAuth2 token acquisition phase. The service logs show repeated timeouts when contacting the authorization server, and the authentication flow aborts, causing inference requests to return 401 Unauthorized.

Typical log excerpt from the inference container:


2026-09-26T14:32:10.123Z ERROR oauth2_client - Token request to https://auth.example.com/oauth2/token timed out after 30s
2026-09-26T14:32:10.124Z WARN auth_middleware - Authentication failed: token acquisition timeout
2026-09-26T14:32:10.125Z INFO request_handler - Responding with 401 Unauthorized

The issue appears only after the Docker daemon on the node is restarted; a fresh pod deployment on a healthy node works as expected.

Root Cause – Network Namespace Reset and Stale iptables Rules

Kubernetes creates a separate network namespace for each pod. When the Docker daemon restarts, the underlying docker0 bridge is recreated, but existing iptables NAT rules that reference the old bridge IP are not flushed. Pods that were already running retain their original eth0 IP address, yet outbound traffic is NAT‑ed to a non‑existent bridge, causing packets to be dropped silently. The OAuth2 client library interprets the lack of a response as a timeout.

Compounding the issue, the OAuth2 client is configured with a redirect_uri that includes the pod’s IP address (e.g., http://10.244.1.23/callback). After the Docker restart, the pod IP remains the same, but the underlying network path is broken, so the authorization server cannot reach the redirect endpoint, leading to mismatched redirect URI errors in the server logs.

Debug – Step‑by‑Step Investigation

Confirm the timeout originates from the network, not the auth server. Use curl from inside the pod:
```
kubectl exec -it ai-inference-5d9f8c9d9f-abcde -- curl -v -m 5 https://auth.example.com/oauth2/token
```
Expected output: HTTP 200 or 400 response.
Observed output: curl: (28) Operation timed out after 5000 milliseconds

Check pod IP and node bridge IP.


POD_IP=$(kubectl get pod ai-inference-5d9f8c9d9f-abcde -o jsonpath='{.status.podIP}')
BRIDGE_IP=$(ip -4 addr show docker0 | awk '/inet/ {print $2}' | cut -d/ -f1)
echo "Pod IP: $POD_IP, Bridge IP: $BRIDGE_IP"

If BRIDGE_IP differs from the one recorded before the Docker restart, the NAT mapping is stale.

Inspect iptables NAT table for the pod’s source IP.
```
iptables -t nat -S | grep 10.244.1.23
```
Typical stale rule:
```
-A POSTROUTING -s 10.244.1.23/32 -j MASQUERADE
```
If the rule references the old docker0 interface, it will never match after the bridge is recreated.
Validate redirect URI handling on the auth server. Look at the auth server logs for the same request ID:
```
2026-09-26T14:32:09.987Z WARN oauth2 - Redirect URI mismatch: expected http://10.244.1.23/callback, received http://10.244.1.24/callback
```
This confirms the pod IP is still being used in the redirect while the underlying network cannot reach it.

Solution – Refresh Network Namespace and Align Redirect URIs

The fix consists of two parts: (1) force a network namespace reset for affected pods, and (2) modify the OAuth2 client to use a stable, service‑level redirect URI instead of the pod IP.

1. Force Pod Recreation (Network Reset)

Delete the affected pods so that kubelet recreates them with a fresh network namespace that matches the current docker0 bridge.


kubectl delete pod -l app=ai-inference

Kubernetes will schedule new pods; the new iptables rules will reference the correct bridge.

2. Update OAuth2 Client Configuration

Change the redirect_uri from a pod‑specific address to a stable service DNS name (e.g., https://auth-callback.ai.svc.cluster.local/callback). This avoids reliance on pod IPs.

Before (go‑oauth2 client snippet):


config := &oauth2.Config{
    ClientID:     "ai-service",
    ClientSecret: "********",
    Endpoint: oauth2.Endpoint{
        TokenURL: "https://auth.example.com/oauth2/token",
        AuthURL:  "https://auth.example.com/oauth2/authorize",
    },
    RedirectURL: fmt.Sprintf("http://%s/callback", podIP), // podIP injected at runtime
    Scopes:      []string{"inference.read"},
}

After (service‑level redirect):


config := &oauth2.Config{
    ClientID:     "ai-service",
    ClientSecret: "********",
    Endpoint: oauth2.Endpoint{
        TokenURL: "https://auth.example.com/oauth2/token",
        AuthURL:  "https://auth.example.com/oauth2/authorize",
    },
    RedirectURL: "https://auth-callback.ai.svc.cluster.local/callback",
    Scopes:      []string{"inference.read"},
}

Deploy the updated container image:


kubectl set image deployment/ai-inference ai-inference=registry.example.com/ai-inference:v2.3.1

Verify – Confirm the Issue Is Resolved

Check token acquisition succeeds.


kubectl exec -it $(kubectl get pod -l app=ai-inference -o jsonpath='{.items[0].metadata.name}') -- curl -s -o /dev/null -w "%{http_code}" https://auth.example.com/oauth2/token

Expected HTTP status: 200 (or 400 if request parameters are invalid, but not a timeout).

Inspect pod logs for absence of timeout messages.
```
kubectl logs -l app=ai-inference | grep "Token request"
```
There should be no lines containing timed out.

Validate redirect URI acceptance on the auth server.


grep "Redirect URI mismatch" /var/log/auth-server/auth.log || echo "No mismatch"

Output should be No mismatch.

Run an end‑to‑end inference request.


curl -X POST https://ai.example.com/v1/predict -H "Authorization: Bearer $(cat token.txt)" -d '{"input":"test"}'

Response should be 200 OK with a prediction payload.

Prevent – Guardrails to Avoid Recurrence

Use service‑level redirect URIs. Never embed pod IPs in OAuth2 configuration; rely on stable DNS names.
Automate network health checks. Deploy a DaemonSet that periodically verifies outbound connectivity from each pod to the auth server (e.g., curl -s -o /dev/null -w "%{http_code}" https://auth.example.com/oauth2/token) and raises an alert on repeated timeouts.
Refresh iptables on Docker daemon restart. Add a systemd ExecStartPost hook that runs iptables -F && iptables -t nat -F and then restarts kubelet, ensuring pods are recreated with a clean NAT table.
Pin Docker version. The issue was observed with Docker 24.0.5; upgrading to Docker 24.0.7 includes a fix that automatically flushes stale NAT rules on daemon restart.
Enable pod disruption budgets. Prevent mass pod restarts during Docker daemon maintenance, reducing the window where stale network state can affect services.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does the token request succeed after a fresh deployment but fail after a Docker restart?
The Docker restart recreates the docker0 bridge, invalidating existing iptables MASQUERADE rules. Existing pods retain their IPs but lose a valid outbound path, causing silent packet drops that manifest as timeouts.
Can I keep the pod‑specific redirect URI and still avoid the issue?
It is technically possible by flushing iptables and recreating the pod’s network namespace on every Docker restart, but this adds operational complexity. Using a stable service DNS name is the recommended approach.
Is there a way to detect stale iptables rules automatically?
Yes. A simple script can compare the docker0 interface IP with the source IP in iptables -t nat -S. Mismatches can trigger a pod restart or a daemonset‑wide network reset.
Do Kubernetes network policies interfere with this fix?
Network policies operate at the pod‑level and are unaffected by the bridge IP change. However, ensure that the policy allows egress to auth.example.com:443 for the new pod IPs.
Will upgrading to container‑d eliminate the problem?
Container‑d manages its own CNI plugins and does not rely on the Docker bridge, so the specific NAT‑rule issue does not occur. Migrating to container‑d is a viable long‑term mitigation.