Kafka producer OAuth2 token acquisition failure after client secret rotation

Problem

After a scheduled rotation of the OAuth2 client secret used by Kafka producers and consumers, the applications began failing to acquire access tokens from the authorization server. The resulting authentication errors caused the following observable symptoms:

• Kafka client logs contain org.apache.kafka.common.errors.SaslAuthenticationException with messages such as Failed to acquire OAuth token: 401 Unauthorized.
• Producer metrics show record-send-rate = 0 and record-error-rate spiking.
• Consumer groups report lag increasing continuously, leading to data loss in downstream processing.
• Health‑check endpoints for the streaming services return HTTP 500.

Sample log excerpt (producer)

2026-09-26 10:12:34,567 ERROR [producer-1] o.a.k.c.s.o.OAuthBearerLoginModule: Failed to acquire OAuth token: 401 Unauthorized
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed: token acquisition error
    at org.apache.kafka.common.security.oauthbearer.internals.secured.OAuthBearerLoginModule.acquireToken(OAuthBearerLoginModule.java:215)
    at org.apache.kafka.common.security.oauthbearer.internals.secured.OAuthBearerLoginModule.login(OAuthBearerLoginModule.java:124)
Caused by: java.io.IOException: HTTP 401 Unauthorized
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:2563)
    ... 12 more

Root Cause

The OAuth2 client secret is stored in a Kubernetes Secret that is mounted as a volume into the producer/consumer pods. During the rotation, the secret name remained the same but the underlying data changed. The Java KafkaOAuthBearerLoginModule caches the secret value at class initialization. Because the pod process was not restarted, the module continued to read the stale secret from its in‑memory cache, while the authorization server expected the new secret. Consequently, every token request was rejected with 401 Unauthorized.

Key contributing factors:

• Secret rotation performed without a rolling restart of the Kafka client pods.
• OAuth login module does not watch the mounted file for changes; it only reads the secret at JVM startup.
• Lack of a health‑check that validates token acquisition before traffic is accepted.

Debug

Step‑by‑step investigation that reproduced the failure and identified the missing restart:

1. Confirm secret mismatch by comparing the value mounted in the pod with the value stored in the authorization server.

# Inside the producer pod
cat /etc/kafka/oauth/client_secret
# Output (old secret)
c2VjcmV0X2xhc3Rfcm90YXRl

# Retrieve the new secret from the secret store
kubectl get secret kafka-oauth-client -o jsonpath="{.data.client_secret}" | base64 -d
# Output (new secret)
c2VjcmV0X25ld19yb3RhdGU=

2. Manual token request using the secret from the pod to verify the server response.

curl -s -X POST https://auth.example.com/oauth2/token \
  -d "grant_type=client_credentials" \
  -d "client_id=streaming-producer" \
  -d "client_secret=$(cat /etc/kafka/oauth/client_secret)" \
  -d "scope=kafka:write" -D -

HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
  "error":"invalid_client",
  "error_description":"Client authentication failed"
}

3. Check the JVM logs for the OAuth module initialization to see when the secret was read.

2026-09-26 09:58:12,001 INFO  [OAuthLoginModule] Loaded client secret from /etc/kafka/oauth/client_secret (cached)

4. Verify that a pod restart resolves the issue (performed on a staging replica).

kubectl rollout restart deployment/kafka-producer
# Wait for pod to become ready
kubectl logs -f deployment/kafka-producer -c producer | grep "OAuth token"

2026-09-26 10:45:02,112 INFO  [OAuthLoginModule] Acquired OAuth token, expires in 3599 seconds

Solution

Implement a two‑pronged fix:

1. Ensure pods are restarted after secret rotation

Update the CI/CD pipeline that performs the secret rotation to trigger a rolling restart of all deployments that consume the secret.

# Example Helm hook to restart deployments after secret update
apiVersion: batch/v1
kind: Job
metadata:
  name: rotate-oauth-secret
  annotations:
    "helm.sh/hook": post-upgrade
spec:
  template:
    spec:
      containers:
      - name: rotate
        image: bitnami/kubectl:latest
        command:
        - /bin/sh
        - -c
        - |
          kubectl rollout restart deployment/kafka-producer
          kubectl rollout restart deployment/kafka-consumer
      restartPolicy: OnFailure

2. Make the OAuth login module reload the secret on demand

Switch to the org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginModule implementation that supports a token.refresh.ms configuration and set oauth.client.secret.file to the mounted path. The module will re‑read the file on each token refresh.

Before (static secret)

properties.put("sasl.mechanism", "OAUTHBEARER");
properties.put("security.protocol", "SASL_SSL");
properties.put("sasl.jaas.config",
    "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required " +
    "clientId=\"streaming-producer\" " +
    "clientSecret=\"c2VjcmV0X2xhc3Rfcm90YXRl\" " +
    "scope=\"kafka:write\";");

After (file‑based secret, auto‑reload)

properties.put("sasl.mechanism", "OAUTHBEARER");
properties.put("security.protocol", "SASL_SSL");
properties.put("sasl.jaas.config",
    "org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginModule required " +
    "clientId=\"streaming-producer\" " +
    "clientSecretFile=\"/etc/kafka/oauth/client_secret\" " +
    "scope=\"kafka:write\" " +
    "tokenRefreshMs=\"300000\";"); // refresh every 5 min

With clientSecretFile, the login module reads the secret each time it needs a new token, eliminating the need for a pod restart after rotation.

Verify

After applying the changes, perform the following checks:

1. Log verification – ensure the first token acquisition succeeds.

2026-09-26 11:02:15,043 INFO  [OAuthLoginModule] Loaded client secret from /etc/kafka/oauth/client_secret
2026-09-26 11:02:15,112 INFO  [OAuthLoginModule] Acquired OAuth token, expires in 3599 seconds

2. Metrics – confirm record-send-rate returns to expected values and consumer lag stabilizes.

kafka.producer.record-send-rate  1250.0
kafka.consumer.lag               0

3. Health endpoint – a GET to /healthz should now return HTTP 200.

curl -s -o /dev/null -w "%{http_code}" http://producer-service/healthz
200

Prevent

Adopt the following guardrails to avoid recurrence:

• File‑based secret handling: Prefer clientSecretFile over embedding the secret in the JAAS config.
• Automated rolling restarts: Attach a post‑upgrade Helm hook or Argo CD sync hook that restarts dependent deployments after any secret change.
• Health‑check token validation: Implement a readiness probe that performs a lightweight token request; fail the pod if the request does not succeed within 5 seconds.
• Observability: Add a Prometheus gauge oauth_token_acquisition_success that increments on successful token fetches and alerts on consecutive failures.
• Secret versioning: Store the secret version as a label on the deployment and enforce that the pod’s label matches the secret’s metadata.resourceVersion before marking the pod ready.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why does the token acquisition fail only after secret rotation? The OAuth login module caches the secret at JVM startup. When the underlying file changes, the cached value remains stale, causing the authorization server to reject the request.
2. Can I avoid restarting pods by using a sidecar? Yes. A sidecar that watches the secret file and triggers a SIGHUP to the Java process can force the login module to reload the secret without a full pod restart.
3. What if the authorization server returns 500 instead of 401? A 500 indicates a server‑side issue (e.g., rate limiting or temporary outage). In that case, the client should implement exponential back‑off and expose a metric for “token acquisition failures” to alert operators.
4. Is it safe to store the client secret in a plain file inside the container? The file should be mounted read‑only from a Kubernetes Secret with appropriate RBAC restrictions. Use a runtime encryption solution (e.g., HashiCorp Vault Agent) for higher security requirements.
5. How often should the token be refreshed? Align the refresh interval with the token’s expires_in value, typically 1 hour. Setting tokenRefreshMs to 5 minutes provides a safety margin and ensures the secret is re‑read frequently.