Kafka client token refresh failure inside Docker after container restart

Problem

After a Docker container running a Kafka client is restarted, the client repeatedly fails to authenticate with the broker. The failure manifests as token‑refresh errors and results in Authentication failed: token expired messages. New connections cannot be established until the container is recreated from scratch.

Root Cause

The Kafka client is configured to use SASL/OAUTHBEARER authentication. The OAuth token is obtained from an external token service and cached in memory by the client library. When the container stops, the in‑memory cache disappears, but the client process is often started by a process manager (e.g., supervisord) that re‑executes the Java process without re‑initialising the OAuthBearerTokenRefreshCallback. Consequently, the client attempts to reuse the last token value stored in the java.security.auth.login.config file or a stale environment variable. Because the token is already expired, the broker rejects the authentication request, and the client never triggers a fresh token fetch.

Additional contributing factors include:

• Missing KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL environment variable after restart (overridden by a .env file that is not re‑sourced).
• Docker’s default restart: on-failure policy restarts the container without clearing the mounted volume that holds a stale token file.
• The client’s token.refresh.ms setting is larger than the token’s actual TTL, so the refresh thread never fires before the token expires.

Debug

Below is a typical log excerpt from a restarted container:

2023-09-12 10:15:23,456 ERROR org.apache.kafka.clients.NetworkClient - Authentication failed: token expired
2023-09-12 10:15:23,459 WARN  org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerLoginModule - Token refresh failed: java.io.IOException: Token endpoint returned 401
2023-09-12 10:15:23,460 INFO  org.apache.kafka.clients.consumer.KafkaConsumer - Closing consumer due to authentication error

Key debugging steps that revealed the cause:

1. Inspect container environment:

docker exec -it kafka-client sh -c 'env | grep KAFKA_SASL_OAUTHBEARER'

Output showed the token endpoint variable missing after the restart.

2. Check token cache file (if any):

docker exec -it kafka-client cat /var/run/kafka/token.cache

The file contained a JWT with an exp timestamp that was already in the past.

3. Verify token refresh interval:

grep token.refresh.ms /etc/kafka/client.properties
# token.refresh.ms=3600000

The token TTL from the OAuth provider was 15 minutes, so the client would wait an hour before attempting a refresh.

4. Capture network traffic to the token endpoint:

docker exec -it kafka-client tcpdump -i eth0 -nn -s 0 -w /tmp/token.pcap host token.service.local

No HTTP request was observed, confirming that the client never attempted a new token fetch.

Solution

The fix consists of three coordinated changes:

1. Ensure the token endpoint environment variable is always set

Update the Docker Compose (or Kubernetes) definition to inject the variable at container start and to source the .env file each time.

# docker-compose.yml (before)
environment:
  - KAFKA_SASL_MECHANISM=OAUTHBEARER
  # KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL missing

# docker-compose.yml (after)
environment:
  - KAFKA_SASL_MECHANISM=OAUTHBEARER
  - KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL=https://token.service.local/oauth2/token
  - KAFKA_SASL_OAUTHBEARER_CLIENT_ID=my-client
  - KAFKA_SASL_OAUTHBEARER_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}

2. Align token.refresh.ms with the actual token TTL

Configure the client to refresh well before the token expires (e.g., 5 minutes before). Add the property to the client configuration file.

# client.properties (before)
token.refresh.ms=3600000

# client.properties (after)
token.refresh.ms=600000   # 10 minutes
token.expiry.buffer.ms=300000  # 5 minutes buffer

3. Persist a fresh token on container start

Introduce an entrypoint script that obtains a token using curl and writes it to the cache location before launching the Java process.

#!/bin/sh
# entrypoint.sh

set -e

TOKEN_ENDPOINT="${KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL}"
CLIENT_ID="${KAFKA_SASL_OAUTHBEARER_CLIENT_ID}"
CLIENT_SECRET="${KAFKA_SASL_OAUTHBEARER_CLIENT_SECRET}"
CACHE_FILE="/var/run/kafka/token.cache"

# Fetch a fresh token
TOKEN=$(curl -s -X POST "$TOKEN_ENDPOINT" \
  -u "$CLIENT_ID:$CLIENT_SECRET" \
  -d 'grant_type=client_credentials' | jq -r .access_token)

if [ -z "$TOKEN" ]; then
  echo "Failed to obtain OAuth token"
  exit 1
fi

# Write token in the format expected by the client library
cat > "$CACHE_FILE" <

# Dockerfile (before)
CMD ["java", "-jar", "/app/kafka-client.jar"]

# Dockerfile (after)
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["java", "-jar", "/app/kafka-client.jar"]

docker logs kafka-client | grep -i "Authentication succeeded"
# Expected: 2023-09-12 10:16:05,123 INFO  org.apache.kafka.clients.NetworkClient - Authentication succeeded

docker exec -it kafka-client cat /var/run/kafka/token.cache | jq .exp
# Expected: a future Unix timestamp (now + 900 seconds)

docker exec -it kafka-client kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group my-consumer-group
# Expected: No "UNKNOWN" state, lag values reasonable

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD curl -s http://localhost:8080/health | grep -q "token_valid"