Grafana Panel Duplicate Data After RAG Chunk Overlap In A/B Test

Q: Investigation and Debugging 1. Verify overlapping time ranges in panel queries

# Inspect the JSON model of panel 12 (variant A) curl -s -H "Authorization: Bearer $TOKEN" \ https://grafana.example.com/api/dashboards/uid/AB_test_dashboard \ | jq '.dashboard.panels[] | select(.id==12) | .targets' Typical output showing overlap:

Q: 2. Check chunk alignment warnings from the data source

journalctl -u grafana-server | grep "chunk alignment" Sample log entry:

Q: 3. Reproduce duplication with a direct data‑source query

# Prometheus HTTP API with overlapping windows curl -g 'http://prometheus.example.com/api/v1/query_range?query=rate(http_requests_total[1m])&start=1695037200&end=1695037500&step=30s&overlap=5m' Observe that the same series appears twice in the JSON result array.

Q: Resolution 1. Align query steps with data‑source chunk boundaries

Update the dashboard JSON to use a step that is a multiple of the underlying chunk size (e.g., 30 s for Prometheus 2‑minute chunks, 15 s for Loki 5‑minute chunks).

Q: 2. Remove or reduce overlap for A/B variants

Set overlap to 0s for the variant that is not the primary rollout, or use a feature‑flag guard that disables the second variant until the first fully stabilises.

In this article

Table of Contents

Problem: Grafana panels show duplicated or missing data during A/B testing of dashboard versions

During a staged rollout of two dashboard variants (A and B) using Grafana’s feature‑flag based A/B testing, operators observed the following symptoms:

Prometheus‑based time‑series panels display each metric twice, inflating KPI values by ~100%.
Loki log panels contain duplicate log entries for the overlapping interval and occasional gaps where the second variant filters out the duplicated rows.
Grafana UI shows the warning “panel data contains duplicate points”.

Server logs contain messages such as:


[2024-09-18T12:34:56Z] error fetching data: overlapping time range (query: 2024-09-18T12:00:00Z/2024-09-18T12:05:00Z)
[2024-09-18T12:35:01Z] duplicate series name detected: cpu_usage_total{job="app"}
[2024-09-18T12:35:03Z] request failed: chunk alignment error (datasource: Loki)

The impact includes inflated alert thresholds, misleading dashboards for stakeholders, and increased load on the data sources due to redundant queries.

Root Cause Analysis

The duplication originates from two interacting misconfigurations:

Overlapping query windows (RAG – “Read‑After‑Write” chunk overlap) – Grafana’s Query Options allow an explicit overlap parameter. When the A/B feature flag enables both variants simultaneously, each variant’s panel inherits the same step and interval settings, but the auto‑interval algorithm generates a 5‑minute overlap to guarantee continuity. This is documented in the official “Query Options and Time Range Overlap handling” page.
Chunk alignment mismatch – Prometheus and Loki store data in fixed‑size time chunks (e.g., 2‑minute chunks for Prometheus, 5‑minute chunks for Loki). If the query step does not align with the chunk boundaries, the data source returns the same chunk to both overlapping windows. The Grafana data source configuration docs (Data source configuration for chunked time series) warn that mismatched step values cause “chunk alignment error”.

When both variants query the same metric with overlapping windows, Grafana merges the results without de‑duplication because the series identifiers are identical. The transformation pipeline therefore produces duplicate timestamps, triggering the UI warning and inflating the visualized values.

Investigation and Debugging

1. Verify overlapping time ranges in panel queries


# Inspect the JSON model of panel 12 (variant A)
curl -s -H "Authorization: Bearer $TOKEN" \
  https://grafana.example.com/api/dashboards/uid/AB_test_dashboard \
  | jq '.dashboard.panels[] | select(.id==12) | .targets'

Typical output showing overlap:


[
  {
    "refId": "A",
    "expr": "rate(http_requests_total[1m])",
    "interval": "",
    "legendFormat": "{{handler}}",
    "datasource": "Prometheus",
    "queryType": "range",
    "range": {
      "from": "now-5m",
      "to": "now"
    },
    "overlap": "5m"
  }
]

2. Check chunk alignment warnings from the data source


journalctl -u grafana-server | grep "chunk alignment"

Sample log entry:


Sep 18 12:35:03 grafana-server grafana[12345]: request failed: chunk alignment error (datasource: Loki, query: {start:1695037200, end:1695037500})

3. Reproduce duplication with a direct data‑source query


# Prometheus HTTP API with overlapping windows
curl -g 'http://prometheus.example.com/api/v1/query_range?query=rate(http_requests_total[1m])&start=1695037200&end=1695037500&step=30s&overlap=5m'

Observe that the same series appears twice in the JSON result array.

4. Validate transformation pipeline

Grafana’s “Duplicate series removal” transformation can be toggled per panel (Panel data transformations). In the incident, the transformation was disabled, so duplicates propagated to the UI.

Resolution

1. Align query steps with data‑source chunk boundaries

Update the dashboard JSON to use a step that is a multiple of the underlying chunk size (e.g., 30 s for Prometheus 2‑minute chunks, 15 s for Loki 5‑minute chunks).


// Before (misaligned step)
"step": "45s"

// After (aligned step)
"step": "30s"

2. Remove or reduce overlap for A/B variants

Set overlap to 0s for the variant that is not the primary rollout, or use a feature‑flag guard that disables the second variant until the first fully stabilises.


// Before (5‑minute overlap)
"overlap": "5m"

// After (no overlap)
"overlap": "0s"

3. Enable “Duplicate series removal” transformation

In the panel editor, add the transformation “Remove duplicate series” and configure it to keep the first occurrence.


// Transformation JSON snippet
{
  "type": "duplicateSeries",
  "options": {
    "mode": "keepFirst"
  }
}

4. Adjust A/B feature flag rollout logic

Ensure that only one variant is active for a given time range. Example using Grafana’s feature‑management API:


curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://grafana.example.com/api/featureflags/ab_test_dashboard \
  -d '{"enabled": true, "variant": "A"}'

When the rollout is complete, switch the variant to “B” and disable “A”. This prevents simultaneous overlapping queries.

Verification

Panel visual inspection – The “duplicate series name detected” warning disappears and the KPI values return to expected levels.
Log validation – Search Grafana server logs for the previous error strings; they should no longer appear:
```
grep "overlapping time range" /var/log/grafana/grafana.log
# No output expected
```
Metric comparison – Query Prometheus directly for the same interval without overlap and compare the sum of values:
```
# Direct query (no overlap)
curl -g 'http://prometheus.example.com/api/v1/query?query=sum(rate(http_requests_total[1m]))'
```
The result should match the Grafana panel’s displayed total.
Alert sanity check – Verify that alerts based on the affected panels fire at the same rate as before the rollout.

Prevention and Best Practices

Synchronise step and chunk size – Always configure step as a divisor of the data source’s chunk interval. Reference the “Data source configuration for chunked time series” documentation.
Explicitly set overlap to zero for A/B variants – Overlap is useful for single‑variant dashboards to avoid gaps, but it should be disabled when multiple variants query the same metric concurrently.
Enable duplicate‑series removal as a default transformation for panels that aggregate high‑frequency data.
Automated tests for A/B rollouts – Include a CI step that renders both variants against a synthetic time range and asserts that the total number of series equals the expected count.
Monitoring alerts – Create a Grafana alert on the server log pattern “duplicate series name detected” to catch regressions early.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does disabling overlap only fix the duplication for Prometheus but not for Loki?
Loki stores logs in larger 5‑minute chunks; even with zero overlap, a misaligned step can cause the same chunk to be returned twice. Align the step with Loki’s chunk size and enable the “Remove duplicate series” transformation.
Can I keep overlap and still avoid duplicates?
Yes, by enabling the “Duplicate series removal” transformation and ensuring that each variant uses a distinct refId or adds a unique label (e.g., variant="A") to the query. This forces Grafana to treat the series as separate.
What is the recommended size for the overlap window?
The official docs suggest a small overlap (e.g., 30 s) only when the data source does not guarantee point continuity across query boundaries. For A/B testing, set overlap to 0s to prevent cross‑variant duplication.
How do I know if my custom datasource suffers from chunk mis‑alignment?
Inspect the datasource’s max_chunk_age or equivalent configuration (Prometheus: --storage.tsdb.min-block-duration, Loki: chunk_target_size) and verify that the panel’s step is a divisor of that interval. Mismatches generate the “request failed: chunk alignment error” log entry.
Is there a way to automatically reconcile duplicate points after they appear?
Grafana’s transformation pipeline can be configured to “Aggregate by time” with a function like avg or sum, which collapses duplicate timestamps. However, fixing the root cause (overlap and step alignment) is preferred to avoid unnecessary data processing.