Prometheus query latency spikes with high-cardinality metrics

Problem Description

During peak ingestion windows of a real‑time LLM inference pipeline, Prometheus query latency exceeds the 5‑second Service‑Level Objective (SLO). The most common symptom is a timeout error returned by the HTTP API:

error: query timeout after 5s

Dashboard panels that aggregate inference_latency_seconds by endpoint and model start rendering in 10‑12 seconds. The issue correlates with the ingestion of high‑cardinality series such as:

• inference_latency_seconds{node_id="gpu-01", gpu_uuid="GPU-ABC123", model_version="v2.4", endpoint="/chat", request_id="req‑7f3b..."}
• gpu_memory_usage_bytes{node="node-12", gpu="0", pod="inference‑svc-7f8c9"}

Metrics are scraped every 15 seconds from each GPU node, resulting in a rapid growth of series count (observed >2 M unique series) and delayed TSDB compaction cycles.

Root Cause Analysis

The latency spikes stem from two tightly coupled factors:

1. High‑cardinality label sets – The Prometheus “High cardinality metrics” guidance explains that each unique combination of label values creates a separate time series. In our case, the request_id label (unique per inference request) and per‑GPU identifiers explode the series count. The query engine must scan millions of samples for each aggregation, quickly hitting the default query.max-samples limit (see the warning “exceeded maximum samples per query (5000000)”).
2. Compaction back‑pressure – The TSDB storage layer periodically compacts 2‑hour blocks into larger 2‑day blocks. The TSDB “Compaction and retention” chapter notes that compaction pauses can last seconds when the write rate overwhelms disk I/O. Logs such as:

prometheus_tsdb_compaction_worker: compaction failed: context deadline exceeded
failed to load block for 1680940800000: mmap failed: No space left on device

indicate that compaction workers are throttled, leaving many small blocks on disk. Queries must read across a larger number of blocks, increasing seek latency and CPU overhead.

Community reports (GitHub #10484, #12412) and real incident post‑mortems (large LLM service March 2024, Uber engineering blog) confirm the same pattern: high‑cardinality ingestion + compaction stalls → query latency >5 s.

Investigation and Debugging Steps

1. Quantify Cardinality

# Count total series
prometheus_tsdb_head_series{job="prometheus"}  # e.g. 2,145,873
# Count series per metric
prometheus_tsdb_series{metric="inference_latency_seconds"}  # e.g. 1,876,342

Use the prometheus_tsdb_head_series and prometheus_tsdb_series metrics to confirm the explosion.

2. Identify Expensive Queries

# Example dashboard query
sum by (endpoint, model) (rate(inference_latency_seconds_sum[5m]))
# Histogram quantile query
histogram_quantile(0.95, sum by (le, model_id, region) (rate(inference_latency_seconds_bucket[5m])))

Run promtool query --debug (or enable --log.level=debug) to see the number of series scanned. The debug output typically shows “scanning X samples across Y series”.

3. Observe Compaction Activity

# Show compaction queue length
prometheus_tsdb_compaction_queue_length
# Show compaction duration histogram
prometheus_tsdb_compaction_duration_seconds_bucket

A rising queue length (>10) and a high 99th‑percentile duration (>5 s) indicate a bottleneck.

4. Check Storage Pressure

df -h /var/lib/prometheus
iostat -dx 5 3

Look for “No space left on device” errors or sustained high I/O wait (%iowait > 30%).

5. Verify Configuration Limits

# prometheus.yml excerpt
query:
  max-concurrency: 20
  timeout: 30s
  max-samples: 5000000

If max-samples is hit, Prometheus aborts the query with the “exceeded maximum samples per query” warning.

Resolution

1. Reduce Label Cardinality

Remove or replace per‑request labels. Instead of request_id, expose a request_type or aggregate at the endpoint level.

# Before (high cardinality)
inference_latency_seconds{endpoint="/chat", model="gpt-4", request_id="req-7f3b..."}

# After (low cardinality)
inference_latency_seconds{endpoint="/chat", model="gpt-4"}

Instrument the application to compute per‑request latency locally and expose only aggregated histograms.

2. Adjust Histogram Buckets

Use coarse‑grained buckets to limit series count:

# Before
inference_latency_seconds_bucket{le="0.001", le="0.005", le="0.01", le="0.025", le="0.05", le="0.1", le="0.25", le="0.5", le="1", le="+Inf"}

# After (merged)
inference_latency_seconds_bucket{le="0.01", le="0.1", le="1", le="+Inf"}

3. Tune TSDB Compaction

Increase the number of compaction workers and adjust block sizes:

# prometheus.yml excerpt
storage:
  tsdb:
    min-block-duration: 2h
    max-block-duration: 6h
    retention: 30d
    compaction:
      max-concurrency: 4   # default is 1

On systems with SSDs, raise max-concurrency to 4–8 to keep the compaction queue short.

4. Allocate Dedicated Disk Resources

Mount a separate high‑throughput SSD for /var/lib/prometheus and enable write‑back caching:

mount -o noatime,discard /dev/nvme1n1 /var/lib/prometheus

5. Raise Query Limits (temporary)

While refactoring labels, increase query.max-samples to avoid premature aborts:

query:
  max-samples: 10000000

Do not treat this as a permanent fix; it merely prevents timeouts while the underlying cardinality is reduced.

Validation

1. Series Count – Verify that total series dropped below 500 k:

prometheus_tsdb_head_series{job="prometheus"}  # expected ~450,000

2. Query Latency – Re‑run the previously slow dashboard query and measure response time:

time curl -g 'http://prometheus:9090/api/v1/query?query=sum%20by%20(endpoint)%20(rate(inference_latency_seconds_sum[5m]))'
# Expected response < 1s

3. Compaction Health – Ensure compaction queue length stays at 0‑1 and 99th‑percentile duration < 2 s:

prometheus_tsdb_compaction_queue_length  # 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="2"}  # > 0.99

4. Alerting – Confirm that alerts for query timeout and compaction failed no longer fire for a full ingestion cycle (≥ 30 min).

Operational Experience & Lessons Learned

• Misleading Symptom: Initial suspicion fell on network latency because the scrape interval is short (15 s). Log analysis later revealed the true bottleneck was TSDB compaction.
• Common Incorrect Assumption: “Adding a label improves granularity without cost.” In high‑throughput AI pipelines, per‑request identifiers are effectively unique, turning a single metric into millions of series.
• Edge Case in Production: During GPU node scaling events, the number of gpu_uuid labels doubled, briefly pushing series count over 3 M and causing a temporary compaction backlog (as seen in the Shopify 2023 incident).
• Lesson: Instrumentation should be designed with “cardinality budgeting” in mind. A rule of thumb is to keep per‑metric series < 500 k for a single Prometheus instance.

Best Practices and Prevention

• Audit label sets regularly; enforce a max-label-values policy per metric.
• Prefer *_total counters with rate() over per‑request histograms.
• Enable prometheus_tsdb_compaction_queue_length alerts and set a threshold (e.g., >5) to catch compaction stalls early.
• Separate high‑cardinality ingestion into a dedicated remote write endpoint (e.g., Cortex, Thanos) if retention of per‑request data is required.
• Schedule periodic “cardinality health checks” using queries like:

count by (__name__) (label_replace({__name__!=""}, "label", "$1", ".*", ""))

• Allocate SSD storage with sufficient IOPS (>10 k) for the TSDB directory.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the latency only spike during peak ingestion? During peaks, the write rate exceeds the compaction throughput, causing many small blocks to accumulate. Queries must read across these blocks, increasing I/O and CPU cost.
2. Can I keep the request_id label for debugging? Export it to a separate logging system (e.g., Loki) and remove it from Prometheus. If you need occasional per‑request analysis, use a short‑term remote write target with higher cardinality limits.
3. How do I know if compaction is the bottleneck? Monitor prometheus_tsdb_compaction_queue_length and prometheus_tsdb_compaction_duration_seconds_bucket. A growing queue and high 99th‑percentile duration correlate with query latency spikes.
4. Is increasing query.max-concurrency a safe fix? It helps when many users run queries simultaneously, but it does not address the root cause of high cardinality. Use it only as a temporary measure.
5. What is a practical limit for series count per Prometheus instance? The official documentation suggests staying below 5 M total series; in practice, most large deployments aim for < 500 k to maintain sub‑second query latency.