Prometheus Remote Read 429 Error During Cloud API Rate Limit

Q: Root Cause Analysis Interaction between Prometheus and cloud‑provider APIs

Prometheus batches samples according to remote_write.queue_config and sends them over HTTPS. Managed services enforce rate limits expressed as “requests per minute per project” (e.g., GCP Monitoring 5 000 write requests/min). When the batch size spikes, the number of HTTP POSTs exceeds the quota, and the endpoint returns 429 Too Many Requests. If the service is already saturated or undergoing a regional outage, it may return 503 Service Unavailable, causing context deadline exceeded errors.

Q: Why the default configuration fails under bursty load Insufficient queue capacity. The default max_samples_per_send is 5000 and capacity is 25000. A burst of 5 000 instances can generate > 50 000 samples per scrape, overflowing the queue. No exponential backoff for 429. Prometheus treats 429 as a permanent failure unless remote_write.retry_on_http_5xx is set; it does not apply the queue_config.max_shards backoff logic for 429, leading to immediate sample drops. Missing fallback for remote_read. During a provider outage (e.g., AWS MSP 503 in Jan 2024), Prometheus does not automatically fallback to its local TSDB, causing partial query results. Investigation and Debugging Log inspection

2024-08-14T12:03:45Z prometheus[12345]: level=error caller=remote_write.go:312 msg="remote write failed: HTTP status 429 Too Many Requests" err="samples dropped, retry backoff triggered" 2024-08-14T12:04:02Z prometheus[12345]: level=error caller=remote_write.go:312 msg="remote write failed: HTTP status 503 Service Unavailable" err="context deadline exceeded" 2024-08-14T12:07:18Z prometheus[12345]: level=error caller=remote_read.go:215 msg="remote read request failed: 503 Service Unavailable" Metrics to confirm throttling Prometheus exposes internal queue metrics:

Q: Packet capture (optional)

Running tcpdump -i eth0 -s 0 -w /tmp/prometheus.pcap host monitoring.googleapis.com during a burst shows a rapid series of POST requests followed by HTTP 429 responses.

Q: Configuration validation

Inspect prometheus.yml for the remote write block:

Q: Solution – Tuning remote_write, adding backoff, and implementing fallback for remote_read 1. Increase queue capacity and batch size

remote_write: - url: "https://monitoring.googleapis.com/v1/projects/my-project/timeSeries" queue_config: capacity: 100000 # double the default max_shards: 500 # more parallel workers min_shards: 5 max_samples_per_send: 15000 # larger payload per request remote_timeout: 30s retry_on_http_5xx: true # Enable exponential backoff for 429 via remote_write.retry_on_http_5xx (treated as 5xx) Rationale: A larger queue absorbs burst spikes; higher max_shards spreads load; bigger batches reduce request count, staying under per‑minute quotas.

Q: 2. Enable explicit backoff for 429

Prometheus does not have a dedicated 429 flag, but setting retry_on_http_5xx: true makes the client treat 429 as retryable. Additionally, use remote_write.backoff_min and remote_write.backoff_max (available from v2.45) to control exponential backoff:

Q: 4. Configure remote_read fallback to local TSDB

Wrap the remote read block with a read_recent flag and a local query timeout. When the remote endpoint returns 503, Prometheus will fall back to its own storage for the recent range.

Q: 5. Deploy a sidecar rate‑limiter (optional)

For environments with strict quotas, place an envoy or nginx sidecar that enforces rate_limit on outgoing POSTs. This smooths bursts by queuing requests locally before they hit the provider.

In this article

Table of Contents

Problem – Prometheus remote read/write 429/503 errors during cloud‑API rate‑limit bursts

During large‑scale LLM benchmarking runs, a fleet of GPU instances pushes telemetry (inference latency, GPU utilisation, evaluation accuracy) to a central Prometheus server. The server forwards these samples to a managed remote storage service (Google Cloud Monitoring, AWS Managed Service for Prometheus, Azure Monitor) via remote_write and queries historic data via remote_read.

When the batch size exceeds a few thousand instances, the cloud provider enforces per‑minute request quotas. Prometheus logs begin to show:


level=error ts=2024-08-14T12:03:45.678Z caller=remote_write.go:312 msg="remote write failed: HTTP status 429 Too Many Requests" err="samples dropped, retry backoff triggered"
level=error ts=2024-08-14T12:04:02.112Z caller=remote_write.go:312 msg="remote write failed: HTTP status 503 Service Unavailable" err="context deadline exceeded"
level=error ts=2024-08-14T12:07:18.443Z caller=remote_read.go:215 msg="remote read request failed: 503 Service Unavailable"

Consequences:

Gaps in time‑series for inference latency (up to 12 % of runs in Q4 2023 incident).
Dashboard charts show “no data” for GPU utilisation and throughput during the throttling window.
Down‑stream alerting rules miss SLA breaches because the data never reaches the remote store.

Root Cause Analysis

Interaction between Prometheus and cloud‑provider APIs

Prometheus batches samples according to remote_write.queue_config and sends them over HTTPS. Managed services enforce rate limits expressed as “requests per minute per project” (e.g., GCP Monitoring 5 000 write requests/min). When the batch size spikes, the number of HTTP POSTs exceeds the quota, and the endpoint returns 429 Too Many Requests. If the service is already saturated or undergoing a regional outage, it may return 503 Service Unavailable, causing context deadline exceeded errors.

Official documentation states that remote_write will drop samples when the queue overflows (remote_write docs) and that retry_on_http_5xx must be enabled to retry transient 5xx errors (write_relabel_configs).

Why the default configuration fails under bursty load

Insufficient queue capacity. The default max_samples_per_send is 5000 and capacity is 25000. A burst of 5 000 instances can generate > 50 000 samples per scrape, overflowing the queue.
No exponential backoff for 429. Prometheus treats 429 as a permanent failure unless remote_write.retry_on_http_5xx is set; it does not apply the queue_config.max_shards backoff logic for 429, leading to immediate sample drops.
Missing fallback for remote_read. During a provider outage (e.g., AWS MSP 503 in Jan 2024), Prometheus does not automatically fallback to its local TSDB, causing partial query results.

Investigation and Debugging

Log inspection


2024-08-14T12:03:45Z prometheus[12345]: level=error caller=remote_write.go:312 msg="remote write failed: HTTP status 429 Too Many Requests" err="samples dropped, retry backoff triggered"
2024-08-14T12:04:02Z prometheus[12345]: level=error caller=remote_write.go:312 msg="remote write failed: HTTP status 503 Service Unavailable" err="context deadline exceeded"
2024-08-14T12:07:18Z prometheus[12345]: level=error caller=remote_read.go:215 msg="remote read request failed: 503 Service Unavailable"

Metrics to confirm throttling

Prometheus exposes internal queue metrics:


# HELP prometheus_remote_write_queue_samples_current Number of samples currently in the remote write queue.
# TYPE prometheus_remote_write_queue_samples_current gauge
prometheus_remote_write_queue_samples_current{remote_name="gcp-monitoring"} 31234

# HELP prometheus_remote_write_queue_capacity Maximum number of samples the queue can hold.
# TYPE prometheus_remote_write_queue_capacity gauge
prometheus_remote_write_queue_capacity{remote_name="gcp-monitoring"} 25000

Values consistently above capacity indicate that the remote endpoint cannot keep up.

Packet capture (optional)

Running tcpdump -i eth0 -s 0 -w /tmp/prometheus.pcap host monitoring.googleapis.com during a burst shows a rapid series of POST requests followed by HTTP 429 responses.

Configuration validation

Inspect prometheus.yml for the remote write block:


remote_write:
  - url: "https://monitoring.googleapis.com/v1/projects/my-project/timeSeries"
    queue_config:
      capacity: 25000
      max_shards: 200
      min_shards: 1
      max_samples_per_send: 5000
    remote_timeout: 30s
    retry_on_http_5xx: false

Solution – Tuning remote_write, adding backoff, and implementing fallback for remote_read

1. Increase queue capacity and batch size


remote_write:
  - url: "https://monitoring.googleapis.com/v1/projects/my-project/timeSeries"
    queue_config:
      capacity: 100000        # double the default
      max_shards: 500         # more parallel workers
      min_shards: 5
      max_samples_per_send: 15000   # larger payload per request
    remote_timeout: 30s
    retry_on_http_5xx: true
    # Enable exponential backoff for 429 via remote_write.retry_on_http_5xx (treated as 5xx)

Rationale: A larger queue absorbs burst spikes; higher max_shards spreads load; bigger batches reduce request count, staying under per‑minute quotas.

2. Enable explicit backoff for 429

Prometheus does not have a dedicated 429 flag, but setting retry_on_http_5xx: true makes the client treat 429 as retryable. Additionally, use remote_write.backoff_min and remote_write.backoff_max (available from v2.45) to control exponential backoff:


remote_write:
  - url: "..."
    backoff_min: 1s
    backoff_max: 30s
    retry_on_http_5xx: true

3. Apply write relabeling to reduce sample volume

If certain metrics are not needed for post‑run analysis, drop them before remote write:


remote_write:
  - url: "..."
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop

4. Configure remote_read fallback to local TSDB

Wrap the remote read block with a read_recent flag and a local query timeout. When the remote endpoint returns 503, Prometheus will fall back to its own storage for the recent range.


remote_read:
  - url: "https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-abc123/api/v1/query"
    read_recent: true
    required_matchers:
      - '{job="benchmark-runner"}'
    remote_timeout: 30s
    # Enable retry for 5xx
    retry_on_http_5xx: true

5. Deploy a sidecar rate‑limiter (optional)

For environments with strict quotas, place an envoy or nginx sidecar that enforces rate_limit on outgoing POSTs. This smooths bursts by queuing requests locally before they hit the provider.

6. Automation – Dynamic quota awareness

Integrate a script that queries the provider’s quota API (e.g., gcloud monitoring quotas list) and adjusts max_samples_per_send via the Prometheus config reload API (POST /-/reload) when the remaining quota falls below a threshold.

Verification – Confirming that the fix works

Metrics after change


# HELP prometheus_remote_write_queue_samples_current Number of samples currently in the remote write queue.
# TYPE prometheus_remote_write_queue_samples_current gauge
prometheus_remote_write_queue_samples_current{remote_name="gcp-monitoring"} 0

# HELP prometheus_remote_write_failed_total Total number of remote write failures.
# TYPE prometheus_remote_write_failed_total counter
prometheus_remote_write_failed_total{remote_name="gcp-monitoring"} 0

Log sample


2024-08-14T13:02:11Z prometheus[12345]: level=info caller=remote_write.go:210 msg="remote write succeeded" samples_sent=14800 duration=0.342s
2024-08-14T13:05:04Z prometheus[12345]: level=info caller=remote_read.go:150 msg="remote read succeeded" query="sum(rate(inference_latency_seconds[1m]))"

Functional test

Run a synthetic benchmark that spawns 6 000 GPU instances for 10 minutes. Verify that the dashboard shows continuous latency and utilisation curves with no gaps. Compare against the previous run where gaps were present.

Prevention – Operational guardrails

Monitoring: Alert on prometheus_remote_write_failed_total > 0 and on queue utilization > 80 %.
Quota tracking: Export provider quota metrics (e.g., gcp_monitoring_write_requests_quota_used) to Prometheus and set alerts when > 90 % of the per‑minute limit is consumed.
Capacity planning: Use historic scrape rates to calculate expected samples_per_second and size queue_config.capacity accordingly (rule of thumb: capacity ≥ 2 × peak_samples_per_window).
Graceful degradation: Deploy a secondary remote storage (e.g., Cortex) and configure remote_write with multiple endpoints; Prometheus will attempt each in order.
Rolling configuration reloads: Automate /-/reload after any quota change to avoid manual restarts.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does Prometheus drop samples instead of retrying when it receives 429?
By default retry_on_http_5xx is false, and 429 is not considered a 5xx. Enabling the flag makes the client treat 429 as retryable, and the exponential backoff settings control the retry cadence.
Can I distinguish between a true rate‑limit (429) and a temporary service outage (503) in logs?
Yes. 429 messages contain “Too Many Requests” and are often accompanied by “samples dropped, retry backoff triggered”. 503 messages include “Service Unavailable” and usually a “context deadline exceeded” error.
How do I prevent remote read gaps during a provider outage?
Set read_recent: true in the remote_read block and enable retry_on_http_5xx. This forces Prometheus to fall back to its local TSDB for recent data while the remote store recovers.
Is it safe to increase max_samples_per_send to very large values?
Increasing the batch size reduces request count but also enlarges payload size, which may hit HTTP request size limits (often 10 MiB). Test the maximum payload accepted by your provider and keep the batch size below that threshold.
What is the recommended way to handle multiple cloud providers in the same Prometheus instance?
Configure separate remote_write blocks per provider, each with its own queue settings and backoff parameters. Use write_relabel_configs to route only the needed metric families to each endpoint.