Grafana dashboard time-series and log data misalignment issue

Problem – Misaligned Time‑Series and Log Visualizations

In a Docker‑compose sandbox the Grafana dashboard shows a noticeable gap between Prometheus metrics and Loki logs. The time‑series panel displays metric points at the expected timestamps, while the log panel appears shifted (commonly 5 minutes, 1 hour, or a few seconds). The visual gap makes correlation impossible and leads to false alerts.

Typical symptoms observed in the UI and logs:

• Metrics line chart aligns with the X‑axis, logs start later or earlier.
• Grafana panel warning: “Data source query returned non‑monotonic timestamps”.
• Loki container logs contain “Failed to parse timestamp” or “Invalid timestamp format for field __timestamp__”.
• Prometheus logs show “timestamp out of bounds” for some points.

Root Cause – Inconsistent Timestamp Handling Across Data Sources

The underlying issue is a mismatch between how timestamps are generated, stored, and interpreted by Prometheus and Loki:

Component	Expected Format	Observed Format	Impact
Prometheus remote write	Unix epoch seconds (float) in UTC	UTC (correct)	Baseline reference
Loki ingestion	RFC3339 or nanosecond epoch (UTC)	Local TZ (e.g., America/New_York) or seconds instead of nanoseconds	Log timestamps shifted relative to metrics

Evidence from the official docs:

• Grafana Loki datasource “Timestamp handling” requires RFC3339 or epoch‑nanosecond (Loki docs).
• Prometheus remote write expects timestamps in seconds (Prometheus docs).

Real‑world incidents confirm the pattern:

• Docker‑compose sandbox where host OS used UTC but containers ran with TZ=America/New_York, causing a 5‑hour visual gap (Docker‑compose sandbox incident).
• Mocked exporter emitting timestamps in seconds while Loki expected nanoseconds, leading to a 1‑second lag (Mocked Prometheus exporter incident).

When Grafana assembles a multimodal panel, it aligns series based on the time and timezone fields in the dashboard JSON model (Dashboard model docs). If one datasource supplies timestamps with a different epoch resolution or timezone, the panel cannot synchronize the series, producing the misalignment.

Investigation – Step‑by‑Step Debugging

1. Verify container clocks

docker exec prometheus date -u
docker exec loki date -u

Expected output: identical UTC timestamps. Any drift (e.g., 2 seconds) points to host vs container clock differences.

2. Inspect Prometheus metric timestamps

curl -s http://localhost:9090/api/v1/query?query=up | jq '.data.result[0].value'
# Example output: ["1697856000.123","1"]

The first element is the epoch seconds with millisecond fraction – correct format.

3. Inspect Loki log timestamps

docker exec loki loki-query -query '{job="app"}' -limit 5
# Sample output:
# {"stream":{"job":"app"},"values":[["2023-10-20T07:00:00.000Z","log line"]]}

If the timestamp appears as 2023-10-20T02:00:00-05:00 the log was stored with a local offset.

4. Check Grafana dashboard JSON

{
  "time": {"from":"now-6h","to":"now"},
  "timezone": "browser",
  "panels":[
    {"type":"timeseries","datasource":"Prometheus",...},
    {"type":"logs","datasource":"Loki",...}
  ]
}

Ensure timezone is set to UTC or consistent across panels.

5. Review Loki configuration for timestamp parsing

# loki-config.yaml
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        period: 24h
      # missing 'max_age' or 'timestamp_format' leads to truncation

6. Search for known community patterns

• GitHub issue #4562 – logs shifted by 1 h when Prometheus uses UTC.
• GitHub issue #37891 – multimodal dashboard misalignment due to differing timestamp formats.
• Stack Overflow 73584221 – logs 5 minutes ahead of metrics.

Solution – Align Timestamp Generation and Ingestion

1. Enforce UTC in all containers

Add the environment variable TZ=UTC to both Prometheus and Loki services in docker‑compose.yml:

services:
  prometheus:
    image: prom/prometheus:latest
    environment:
      - TZ=UTC
    ...

  loki:
    image: grafana/loki:2.9.1
    environment:
      - TZ=UTC
    ...

2. Normalize Loki timestamp parsing

Explicitly set timestamp_format to RFC3339Nano (or epoch_nanoseconds) in the Loki pipeline configuration:

# loki-config.yaml (excerpt)
scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
    pipeline_stages:
      - regex:
          expression: '^(?P<ts>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d+Z) (?P<msg>.*)$'
      - timestamp:
          source: ts
          format: RFC3339Nano

3. Adjust Prometheus exporter to emit nanosecond epoch if Loki expects it

For custom exporters, ensure the WriteRequest uses TimestampMs in milliseconds or seconds as required. Example Go exporter fix:

func (e *Exporter) Collect(ch chan<- prometheus.Metric) {
    // Convert time.Now() to Unix seconds (float64) – Prometheus default
    ts := float64(time.Now().UnixNano()) / 1e9
    metric := prometheus.NewMetricWithTimestamp(ts, prometheus.NewDesc(...), prometheus.GaugeValue, value)
    ch <- metric
}

4. Set Grafana dashboard timezone to UTC

In the dashboard settings, change Timezone from “browser” to “UTC”. This forces both panels to interpret timestamps identically.

5. Re‑deploy with corrected configurations

# Rebuild and restart containers
docker-compose down
docker-compose up -d --build

Verification – Confirm Alignment

1. Open the affected dashboard. The metric line and log entries should now share the same X‑axis points.
2. Run a Loki query with explicit time range matching a metric point:

{job="app"} |= "error" | logfmt | __timestamp__ >= 1697856000 && __timestamp__ <= 1697856060

3. Check Grafana panel warning area – the “non‑monotonic timestamps” message must disappear.
4. Inspect container logs for parsing errors; there should be none:

docker logs loki | grep "Failed to parse timestamp"
# (no output)

5. Validate that the dashboard’s time range selector shows consistent UTC times across panels.

Prevention – Guardrails and Monitoring

• Enforce UTC at the platform level. Use a base Docker image that sets TZ=UTC and lock it in CI pipelines.
• Schema validation. Add a Grafana alert that triggers when a panel reports “non‑monotonic timestamps”.
• Log ingestion health checks. Periodically query Loki for __timestamp__ format compliance and alert on parsing failures.
• Prometheus remote‑write audit. Enable remote_write metrics (remote_write_failed_samples_total) to detect out‑of‑bounds timestamps.
• Dashboard JSON linting. Run a CI step that checks the timezone field is set to UTC for multimodal dashboards.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the misalignment appear only in Grafana and not in raw Prometheus/Loki queries?
   Grafana normalizes timestamps based on the dashboard’s timezone. Raw queries return raw epoch values, so the offset is invisible until the UI attempts to align them.
2. Can I keep the container timezones local and still have aligned dashboards?
   Yes, but you must convert all timestamps to UTC before ingestion (e.g., using timestamp stage in Loki pipeline) and ensure Prometheus remote‑write also sends UTC.
3. What is the correct timestamp format for Loki when using LogQL?
   Loki accepts RFC3339 (with nanosecond precision) or epoch nanoseconds. See the official Loki docs for the timestamp stage configuration.
4. How do I detect clock drift between containers?
   Run docker exec <container> date -u on each service and compare against the host’s date -u. Any drift > 1 second should be investigated (e.g., NTP sync).
5. Why do I still see “Data source query returned non‑monotonic timestamps” after fixing timezone?
   The error can also be caused by duplicate or out‑of‑order data points. Ensure that the exporter does not emit stale samples and that Loki’s max_age is configured to retain recent logs.