Prometheus Token Limit Exceeded In Cloud Managed Service

Q: Root Cause Analysis

Managed Prometheus services enforce a token quota that limits the number of authentication tokens that can be requested or used per minute per project/tenant. The quota protects the backend from token‑generation storms and from exhausting internal rate‑limiting resources.

Q: Investigation and Debugging 1. Identify the failing component

# Check Prometheus logs for remote_write errors journalctl -u prometheus -f | grep "token limit" 2. Verify the exact HTTP response # Use curl with the same endpoint and credentials curl -v -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus/api/v1/write # Expected snippet of response < HTTP/1.1 429 Too Many Requests < Content-Type: application/json < Retry-After: 60 < {"error":"token limit exceeded"} 3. Inspect token usage metrics (if exposed) # Example Prometheus query for token request rate rate(prometheus_remote_write_token_requests_total[1m]) 4. Correlate with high‑cardinality metric creation # Find series creation spikes increase(prometheus_tsdb_head_series_created_total[5m]) > 50000 5. Review remote_write configuration # Current remote_write block (before) remote_write: - url: "https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus/api/v1/write" bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token queue_config: capacity: 25000 max_shards: 200 min_shards: 1 remote_timeout: 30s 6. Check token TTL in the provider's IAM policy (if configurable) Many managed services issue short‑lived tokens (e.g., 5 minutes). Frequent token rotation can be observed by monitoring the token_renewal_success_total metric.

In this article

Table of Contents

Problem Description

Several teams observed remote‑write failures from their Prometheus instances that are managed by a cloud provider (Google Cloud Managed Service for Prometheus, AWS Managed Service for Prometheus, Azure Monitor for containers). The failure manifests as HTTP 429 responses with a clear message that the token quota has been exhausted.

Typical log entries include:

error="token limit exceeded" msg="remote write failed" component=remote_write
rpc error: code = ResourceExhausted desc = token limit exceeded while sending samples
failed to send samples: token limit exceeded (max 100000 tokens per minute)
alertmanager webhook error: token limit exceeded -- status=429

Impact ranges from missing metrics in dashboards to delayed or missing alerts, and in extreme cases a cascade of scrape timeouts across the whole cluster.

Root Cause Analysis

Managed Prometheus services enforce a token quota that limits the number of authentication tokens that can be requested or used per minute per project/tenant. The quota protects the backend from token‑generation storms and from exhausting internal rate‑limiting resources.

Google Cloud Managed Service for Prometheus defines a default limit of 100 000 token requests per minute per project.
Cortex/Thanos managed limits apply a similar token request ceiling, configurable only via a support ticket.
The Prometheus remote_write documentation notes that each remote write request includes an authentication token that may be refreshed on demand; excessive refreshes or a burst of new series can quickly consume the quota.

In the incidents referenced:

August 2023 AWS Managed Service outage – a sudden influx of ~1 M new series caused token request spikes that exceeded the per‑tenant limit, resulting in 429 errors across multiple customers.
February 2024 Google Cloud incident – misconfigured scrape jobs generated >10 k unique label combinations per second, each triggering a token renewal, which saturated the per‑project token quota.
Azure Monitor 2023 incident – a CI/CD pipeline pushed a large batch of metrics in a single run, exhausting the service‑wide token pool and delaying alert delivery.

The core reason is that the token limit is tied to the rate of token acquisition, not the number of samples sent. High‑cardinality metrics, frequent token refreshes (e.g., short token TTL), and aggressive remote_write queue settings all increase token acquisition frequency.

Investigation and Debugging

1. Identify the failing component

# Check Prometheus logs for remote_write errors
journalctl -u prometheus -f | grep "token limit"

2. Verify the exact HTTP response

# Use curl with the same endpoint and credentials
curl -v -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus/api/v1/write
# Expected snippet of response
< HTTP/1.1 429 Too Many Requests
< Content-Type: application/json
< Retry-After: 60
< {"error":"token limit exceeded"}

3. Inspect token usage metrics (if exposed)

# Example Prometheus query for token request rate
rate(prometheus_remote_write_token_requests_total[1m])

4. Correlate with high‑cardinality metric creation

# Find series creation spikes
increase(prometheus_tsdb_head_series_created_total[5m]) > 50000

5. Review remote_write configuration

# Current remote_write block (before)
remote_write:
  - url: "https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus/api/v1/write"
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    queue_config:
      capacity: 25000
      max_shards: 200
      min_shards: 1
    remote_timeout: 30s

6. Check token TTL in the provider’s IAM policy (if configurable)

Many managed services issue short‑lived tokens (e.g., 5 minutes). Frequent token rotation can be observed by monitoring the token_renewal_success_total metric.

Resolution

Approach A – Reduce token acquisition frequency

Increase the token TTL (if the provider allows) so that a single token can be reused for a longer period.
Configure remote_write to reuse the same token instead of requesting a new one per batch.

Before:

# remote_write with default token refresh (every 5 min)
remote_write:
  - url: "https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus/api/v1/write"
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    refresh_interval: 5m   # implicit default

After (Google Cloud example):

# Extend token lifetime to 30 min via IAM policy (support ticket) and set refresh interval accordingly
remote_write:
  - url: "https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus/api/v1/write"
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    refresh_interval: 25m

Approach B – Lower token request rate by throttling remote_write

Reduce queue_config.max_shards and increase queue_config.min_shards to limit parallel write attempts.
Enable backoff and retry limits to avoid a burst of token requests during a spike.

Before:

queue_config:
  max_shards: 200
  min_shards: 1
  capacity: 25000

After:

queue_config:
  max_shards: 50          # limit parallel writes
  min_shards: 5           # keep a baseline of workers
  capacity: 15000
  max_samples_per_send: 5000
  backoff_min: 100ms
  backoff_max: 5s

Approach C – Reduce high‑cardinality metric generation

Apply relabel_configs to drop unnecessary labels.
Introduce metric_relabel_configs to aggregate or drop low‑value series.
Use scrape_interval that matches the true data change rate.

Example relabel to drop a dynamic label:

# Drop the pod_uid label which creates a unique series per pod restart
metric_relabel_configs:
  - source_labels: [__name__, pod_uid]
    regex: .+;.*
    action: labeldrop
    target_label: pod_uid

Approach D – Request a higher quota

If the workload legitimately requires more token requests (e.g., a large multi‑tenant SaaS platform), open a support ticket with the cloud provider to raise the token quota. Provide metrics from prometheus_remote_write_token_requests_total and evidence of sustained need.

Validation

Confirm that no new “token limit exceeded” lines appear in the Prometheus logs for at least 30 minutes.
Run a remote_write health check:

# Use the built‑in Prometheus remote_write health endpoint
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.scrapeUrl|contains("remote_write"))'

Successful response should show health: "up" and no error field.

Query the token request rate to ensure it stays well below the quota threshold (e.g., < 70 % of limit).

# Example Grafana panel query
rate(prometheus_remote_write_token_requests_total[1m]) < 70000

Verify that downstream dashboards (Grafana, Cloud Monitoring) receive the expected new series without gaps.

Prevention and Best Practices

Monitor token request metrics (prometheus_remote_write_token_requests_total, token_renewal_success_total) and set alerts when the rate exceeds 80 % of the provisioned quota.
Enforce label cardinality limits at the scrape level; use max_label_names and max_label_value_length where supported.
Batch remote_write calls by tuning queue_config.max_samples_per_send and capacity to avoid excessive parallelism.
Leverage token caching in sidecar components (Thanos, Cortex) to minimize refresh calls.
Plan for bursts by requesting a higher token quota ahead of major releases or migrations that introduce many new series.
Document and review scrape configurations regularly; run promtool check config and promtool test rules in CI to catch accidental high‑cardinality label additions.

FAQ

Why does the token limit error appear only after a deployment?
A new deployment often introduces additional labels (e.g., version, commit SHA) that increase series cardinality. Each new series may trigger a token refresh, pushing the token request rate over the quota.
Can I disable token renewal and use a static token?
Managed services issue short‑lived tokens for security. Some providers allow a longer TTL via IAM policy, but a completely static token is not supported.
How do I differentiate between token quota exhaustion and sample rate throttling?
Token quota errors return HTTP 429 with the message “token limit exceeded”. Sample throttling typically returns 429 with “rate limit exceeded” or “ingestion limit reached”. Checking the response body and the Prometheus metric prometheus_remote_write_token_requests_total clarifies the cause.
Is increasing remote_write.queue_config.capacity a good fix?
Increasing capacity alone does not reduce token requests; it may actually increase parallelism and worsen the problem. Adjust max_shards and backoff settings instead.
What should I do if the provider cannot raise my token quota?
Implement aggressive label reduction, shard your workloads across multiple managed Prometheus instances, or switch to a self‑hosted Cortex/Thanos stack where you control the token limits.

Related Topic Hub: Observability Troubleshooting Hub

Prometheus remote read 429 error during cloud API rate limit

Prometheus token limit exceeded in cloud managed service