Prometheus scrape 401 token limit exceeded during canary rollout

Problem – 401 Token Limit Exceeded During Canary Scrape

During a staged rollout of a new microservice, Prometheus began reporting scrape failures with the following error:

error scraping target "http://service-canary:9090/metrics": unexpected status code 401 Unauthorized (token limit exceeded)

Additional log entries from the Prometheus server process show the token acquisition step failing:

2024-08-12T14:23:07Z level=error msg="failed to fetch bearer token from /var/run/secrets/kubernetes.io/serviceaccount/token: token limit exceeded"

These 401 responses prevent the canary pods from exposing any metrics, causing gaps in the monitoring dashboard and potentially hiding regressions.

Root Cause – Why the Token Limit Was Hit

Authentication flow for scrape targets

Prometheus uses the bearer_token or bearer_token_file fields in a scrape_config to attach a JWT to each HTTP request (Prometheus docs – bearer_token). In many Kubernetes deployments the token is obtained from the pod’s service account and, when OIDC is enabled, the token is refreshed against the configured issuer (Authentication and Authorization – OAuth2/OIDC).

Per‑client token request limits

The JWT issuer used for the canary service enforces a strict rate limit on token issuance per client ID. In the production incident the issuer allowed only 100 token requests per minute per client. When the canary rollout created dozens of new pods, each pod triggered a token fetch because the side‑car token‑injector (or the Prometheus scrape itself) did not cache the token long enough. The rapid burst of token requests exceeded the issuer’s quota, and the issuer responded with HTTP 401 and the body “token limit exceeded”.

Similar behavior is documented in the Prometheus remote‑write token limits page (Remote write – token authentication and limits) and has been reported in several community issues:

• GitHub #10234 – “401 token limit exceeded when scraping OIDC‑protected targets during canary rollout”.
• GitHub #11567 – discussion of token cache size limits.
• Stack Overflow 74492123 – “Prometheus scrape returns 401 token limit exceeded after deploying new service”.

Debug – Investigation Steps

1. Verify the exact error and its source

# Inspect Prometheus server logs (journalctl or container logs)
journalctl -u prometheus -n 100 | grep "token limit"

Expected output contains the “token limit exceeded” string shown above.

2. Confirm the token acquisition path

# Show the scrape configuration for the canary service
kubectl -n monitoring get prometheus prometheus -o yaml | grep -A4 "job_name: service-canary"

Typical snippet:

scrape_configs:
- job_name: service-canary
  scheme: http
  static_configs:
  - targets: ["service-canary:9090"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  oauth2:
    client_id: prometheus
    client_secret_file: /etc/prometheus/oauth2/client_secret
    token_url: https://auth.example.com/oauth2/token
    scopes: ["metrics.read"]

3. Check the token issuer’s rate‑limit metrics

# Query the issuer’s Prometheus endpoint (if exposed)
curl -s http://auth.example.com/metrics | grep token_requests_total

Look for a spike coinciding with the rollout time.

4. Capture a failing HTTP request

# Use tcpdump on the Prometheus pod to capture the request/response
kubectl -n monitoring exec -it prometheus-0 -- tcpdump -i any -w /tmp/canary.pcap host service-canary

Open the pcap in Wireshark and verify the 401 response body contains “token limit exceeded”.

5. Validate token cache behavior

# List files in the token cache directory (used by the side‑car injector)
kubectl -n monitoring exec -it prometheus-0 -- ls -l /var/run/secrets/kubernetes.io/serviceaccount/

If the directory contains many short‑lived token files, the injector is likely not reusing tokens.

Solution – Fixing the Token‑Limit Failure

Approach A – Increase the issuer’s per‑client quota

If the issuer is under your control, raise the limit for the prometheus client ID (e.g., from 100 req/min to 500 req/min). This resolves the immediate burst during rollouts but does not address the underlying token churn.

Approach B – Enable long‑lived token caching in Prometheus

Modify the scrape_config to use a static bearer token that is refreshed only when the token expires, instead of fetching a new token for each scrape.

Before:

scrape_configs:
- job_name: service-canary
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  oauth2:
    client_id: prometheus
    client_secret_file: /etc/prometheus/oauth2/client_secret
    token_url: https://auth.example.com/oauth2/token
    scopes: ["metrics.read"]

After (static token with periodic rotation via a side‑car script):

scrape_configs:
- job_name: service-canary
  bearer_token_file: /etc/prometheus/static-token/token.jwt
  # oauth2 block removed – token is now managed externally

Deploy a side‑car that runs a cronjob or watch loop to refresh the token every 30 minutes and write it to /etc/prometheus/static-token/token.jwt. This reduces token requests to one per rotation period.

Approach C – Reduce the number of concurrent token fetches

When using the Prometheus Operator, set spec.scrapeInterval for the canary job to a higher value (e.g., from 15s to 60s) during rollout, or use relabel_configs to temporarily drop the canary targets until the token cache warms up.

# Example relabel to disable canary during rollout
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_canary]
  regex: "true"
  action: drop

Approach D – Adjust the side‑car token injector cache size

In environments using a token‑injector side‑car (e.g., istio-proxy or custom injector), increase the cache capacity:

# ConfigMap for injector
apiVersion: v1
kind: ConfigMap
metadata:
  name: token-injector-config
data:
  cacheSize: "500"

After updating the ConfigMap, restart the injector pods.

Verify – Confirming the Fix Works

1. Observe scrape health

# Query Prometheus for up metric of the canary job
curl -s 'http://prometheus:9090/api/v1/query?query=up{job="service-canary"}' | jq .

Expected result: "value": [, "1"] for each canary pod.

2. Check token request rate

curl -s http://auth.example.com/metrics | grep token_requests_total

The rate should stay well below the issuer’s limit (e.g., <10 req/min).

3. Review Prometheus logs for residual 401s

journalctl -u prometheus -f | grep "401"

No new “token limit exceeded” lines should appear.

4. Validate token rotation

# Verify token file timestamp changes only at the expected interval
stat -c %y /etc/prometheus/static-token/token.jwt

Prevent – Operational Guardrails

• Metric‑driven alerts: Create an alert on scrape_duration_seconds{job="service-canary"} > 0.5 or on up{job="service-canary"} == 0 to catch future token‑limit spikes early.
• Rate‑limit monitoring: Export the issuer’s token_requests_total metric to a separate Prometheus instance and set a threshold alert at 80 % of the allowed quota.
• Canary rollout policy: Limit the number of simultaneous canary pods (e.g., maxSurge: 1 in the Deployment) when the target uses OIDC‑protected metrics.
• Token cache sizing: Align the injector’s cache size with the expected maximum number of concurrent pods per rollout.
• Documentation lock‑step: Keep the Prometheus bearer_token_file usage in sync with the OIDC client configuration; avoid mixing static and dynamic token sources.

FAQ – Common Follow‑Up Questions

1. Why does the 401 appear only after the canary pods start? The canary pods trigger a burst of token fetches because each new pod’s side‑car requests a fresh JWT. The issuer’s per‑client limit is hit only when the aggregate request rate exceeds the quota.
2. Can I use the same static token for all scrape targets? Yes, provided the token’s scopes cover all required metrics and the token’s lifetime is long enough. Rotate it periodically to avoid expiration.
3. How do I know if the token cache is being evicted too aggressively? Inspect the injector’s cache metrics (e.g., token_injector_cache_evictions_total) or enable debug logging in the injector to see cache hit/miss ratios.
4. Is there a way to tell Prometheus which token to reuse for a given target? Prometheus itself does not cache tokens; it reads the file on each request. Implementing a side‑car that writes a stable token file is the recommended pattern.
5. What should I do if the issuer does not allow increasing the quota? Reduce the scrape concurrency (increase scrape_interval), batch canary pods, and ensure a shared token cache is used so that only one token request is made per rotation period.

Related Topic Hub: Observability Troubleshooting Hub