Grafana Dashboard Query Latency Spikes Over 8000ms With PromQL Aggregations

Q: 1. Verify metric cardinality

# Count series for the problematic metric promtool tsdb list --label=model_version | wc -l # Example output 150342 If the count is >100 k, the metric is a prime suspect.

Q: 2. Inspect PromQL execution plan

Enable the debug=true flag on the Prometheus HTTP endpoint and request the query with debug=true:

Q: 3. Check query queue metrics

# Prometheus internal metrics curl -s http://prometheus:9090/metrics | grep 'prometheus_engine_query_concurrency' # Example output prometheus_engine_query_concurrency{state="max"} 20 prometheus_engine_query_concurrency{state="current"} 30 The current value exceeding max confirms queue saturation.

Q: 1. Relabel high‑cardinality dimensions

Move model_version and region from metric labels to separate series or use label_replace to truncate them.

In this article

Table of Contents

Problem Description

Grafana dashboards that visualize real‑time model inference latency and GPU utilization are experiencing intermittent rendering latency spikes that exceed 8000 ms. The symptoms observed in the UI and logs are:

Panel data request timed out after 8000 ms (Grafana UI).
Grafana logs contain datasource query timeout and Failed to query datasource: context deadline exceeded.
Prometheus logs show query timeout after 2s, query: sum by (model_id) (rate(inference_requests_total[5m])) and query concurrency limit exceeded, current: 30, max: 20.
During spikes the Prometheus query queue is saturated, leading to HTTP 504 responses to Grafana.
Dashboard rendering timeouts cascade into alerting delays and reduced visibility of GPU utilization across distributed ML serving pods.

Root Cause Analysis

The latency spikes are the result of a combination of three interacting factors:

High‑cardinality label explosion – Recent deployments added model_version, region, and dynamic pod labels to model_inference_duration_seconds and gpu_utilization_percent. Each new model version creates a separate time series, inflating the series count from ~15 k to >150 k per Prometheus instance. This matches the Prometheus high‑cardinality best practices warning.
Complex multi‑variable PromQL aggregations – Dashboards use expressions such as:
```
histogram_quantile(0.95,
  sum by (model_id, pod, region) (
    rate(inference_latency_bucket[5m])
  )
)
```
These queries force Prometheus to materialize a large intermediate matrix before the sum by reduction, amplifying CPU and memory pressure.
Query queue saturation – With dozens of panels refreshing every 15 s, the query.max-concurrency limit (default 20) is exceeded. Prometheus logs from issue #12458 describe exactly this behavior: “query concurrency limit exceeded, current: 30, max: 20”. The queue overflow leads to the 2 s query timeout and the downstream 8 s Grafana timeout.

In short, the system is hitting the label cardinality explosion + aggregation cost + query concurrency limit triangle, which manifests as the observed 8000 ms latency spikes.

Investigation and Debugging

The following step‑by‑step investigation reproduces the findings from the incident at a large AI platform (2023‑Q4) and the community reports.

1. Verify metric cardinality

# Count series for the problematic metric
promtool tsdb list --label=model_version | wc -l
# Example output
150342

If the count is >100 k, the metric is a prime suspect.

2. Inspect PromQL execution plan

Enable the debug=true flag on the Prometheus HTTP endpoint and request the query with debug=true:

curl -g 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95,sum by (model_id, pod, region) (rate(inference_latency_bucket[5m])))&debug=true'

The response includes stats showing the number of series fetched and the time spent in each stage. Typical output during a spike:

{
  "status":"success",
  "data":{
    "resultType":"vector",
    "result":[...],
    "stats":{
      "samplesFetched":124578,
      "samplesTotal":124578,
      "executionTimeMs":7325
    }
  }
}

3. Check query queue metrics

# Prometheus internal metrics
curl -s http://prometheus:9090/metrics | grep 'prometheus_engine_query_concurrency'
# Example output
prometheus_engine_query_concurrency{state="max"} 20
prometheus_engine_query_concurrency{state="current"} 30

The current value exceeding max confirms queue saturation.

4. Review Grafana datasource timeout settings

Grafana’s rendering timeout defaults to 30 s, but panel‑level timeout can be overridden. The UI error “Panel data request timed out after 8000ms” indicates a custom datasource.timeout of 8 s (see Prometheus Data Source Configuration).

5. Correlate with scaling events

Cross‑reference Kubernetes events:

kubectl get pods -n ml-serving -l app=model-inference -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.creationTimestamp}{"\n"}{end}'

New pod creations coincide with spikes, confirming the dynamic label addition.

Resolution

The fix consists of three coordinated actions: reducing label cardinality, simplifying PromQL, and increasing query concurrency limits.

1. Relabel high‑cardinality dimensions

Move model_version and region from metric labels to separate series or use label_replace to truncate them.

Before (instrumentation code):

// Go client example
prometheus.MustRegister(prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "model_inference_duration_seconds",
        Help: "Inference latency",
        Buckets: prometheus.ExponentialBuckets(0.001, 2, 15),
    },
    []string{"model_id", "model_version", "region", "pod"},
))

After (reduced labels):

// Remove model_version and region from the histogram
prometheus.MustRegister(prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "model_inference_duration_seconds",
        Help: "Inference latency",
        Buckets: prometheus.ExponentialBuckets(0.001, 2, 15),
    },
    []string{"model_id", "pod"},
))

For existing deployments, add a relabel_configs block in the Prometheus scrape job to drop the heavy labels:

scrape_configs:
  - job_name: 'ml-inference'
    static_configs:
      - targets: ['ml-inference-service:9090']
    relabel_configs:
      - source_labels: [__name__, model_version]
        regex: 'model_inference_duration_seconds;.*'
        action: labeldrop
      - source_labels: [__name__, region]
        regex: 'model_inference_duration_seconds;.*'
        action: labeldrop

2. Rewrite aggregations with pre‑aggregation

Introduce recording rules that materialize the heavy sum by once per scrape interval.

# recording_rules.yml
groups:
  - name: inference_aggregations
    interval: 30s
    rules:
      - record: job:model_inference_latency:rate5m
        expr: sum by (model_id, pod) (rate(model_inference_duration_seconds_bucket[5m]))

Dashboard panels can now query the pre‑aggregated series:

histogram_quantile(0.95, job:model_inference_latency:rate5m)

3. Increase Prometheus query concurrency

Adjust the Prometheus server flags (or Helm values) to raise the limit:

# prometheus.yaml
global:
  query:
    max_concurrency: 40
    timeout: 30s

After applying the change, restart Prometheus and verify:

curl -s http://prometheus:9090/metrics | grep 'prometheus_engine_query_concurrency'
# Expected output
prometheus_engine_query_concurrency{state="max"} 40
prometheus_engine_query_concurrency{state="current"} 12

4. Extend Grafana panel timeout (optional)

If occasional spikes remain, increase the panel timeout to give queries a larger window:

# grafana.ini
[panels]
timeout = 15s

Validation

After applying the above changes, perform the following checks:

Re‑run a representative dashboard query and record response time:

time curl -g 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95,job:model_inference_latency:rate5m)'

Expected real time < 2 s.

Monitor Prometheus query queue metrics for at least 30 min:
```
curl -s http://prometheus:9090/metrics | grep 'prometheus_engine_query_concurrency'
```
Current should stay below the new max (e.g., 12 < 40).
Verify Grafana UI no longer shows the 8000 ms timeout banner on the affected panels.
Check that GPU utilization panels still reflect accurate values by comparing a sample of raw metric values from Prometheus with the dashboard display.

Best Practices and Prevention

Area	Recommendation
Metric design	Limit label cardinality; avoid per‑version or per‑region labels on high‑frequency metrics. Use `label_replace` or drop labels at scrape time.
PromQL	Prefer recording rules for heavy aggregations. Keep `rate()` windows short and avoid nested aggregations on raw high‑cardinality series.
Prometheus configuration	Set `query.max-concurrency` based on expected panel refresh rate and number of concurrent users. Tune `query.timeout` to match SLA.
Grafana	Adjust datasource timeout only after confirming backend can meet the SLA. Use `rendering_timeout` setting for image rendering pipelines.
Observability	Alert on `prometheus_engine_query_concurrency{state="current"} > 0.8 * max_concurrency` and on sudden spikes in `prometheus_engine_query_duration_seconds`.

Grafana dashboard query latency spikes over 8000ms with PromQL aggregations