Prometheus embedding dimension mismatch during blue-green deployment

Problem Description

During a blue‑green deployment of Prometheus, the transition from the active (blue) cluster to the inactive (green) cluster fails. Operators observe the following symptoms:

• Prometheus reload logs contain
  

  error loading config: duplicate series with different label dimensions

• Remote‑write pipelines reject samples with errors such as
  

  remote write failed: series with mismatched dimensions (expected 128, got 256)

• Alert rules that depend on latency or embedding metrics stop firing, producing alerts like
  

  alert evaluation error: vector selector must contain exactly one label matcher for each dimension

• Scrape targets emit “samples with different label cardinality than previous version” warnings.

The impact is two‑fold: monitoring gaps appear for critical ML‑model embedding metrics, and alerting pipelines become silent, risking SLA violations.

Root Cause Analysis

Prometheus treats each unique combination of metric name and label set as a distinct time series. During a blue‑green rollout the metric schema (i.e., the set of labels attached to a metric) must remain identical across both environments. The official configuration documentation states that “label sets are validated on reload; mismatched dimensions cause series duplication errors.”

In the incidents referenced:

• The fintech firm added a new label model_version to latency metrics only in the green environment, causing “duplicate series with different label dimensions” errors.
• The SaaS provider upgraded a custom exporter that emitted 256‑dimensional vectors instead of the previous 128‑dimensional model_embedding series, leading to remote‑write rejections (see GitHub issue #12456).
• The e‑commerce site introduced an embedding_type label in the green cluster, breaking alert rule evaluation (GitHub issue #10873).

These cases share a common pattern: the blue and green clusters expose metrics with divergent label sets or embedding vector dimensions. When Prometheus reloads the configuration, it detects that the same metric name now maps to multiple distinct label dimensions, which violates the series uniqueness guarantee and triggers the errors observed.

Investigation and Debugging

1. Examine Prometheus reload logs

2024-07-07T12:34:56Z level=error msg="error loading config: duplicate series with different label dimensions"
2024-07-07T12:34:56Z level=error msg="failed to ingest samples: metric 'model_embedding' has inconsistent label set across instances"

2. Identify the offending metric and label differences

Use the promtool check config command to pinpoint mismatches.

$ promtool check config /etc/prometheus/prometheus.yml
checking /etc/prometheus/prometheus.yml
  FAILED: duplicate series with different label dimensions for metric 'model_embedding'
    - blue cluster labels: {job="model_exporter",instance="10.0.1.5:9100",embedding_dim="128"}
    - green cluster labels: {job="model_exporter",instance="10.0.2.5:9100",embedding_dim="256"}

3. Inspect the exporter output

Capture a sample from each environment with curl and compare.

# Blue (old) exporter
$ curl http://10.0.1.5:9100/metrics | grep model_embedding
model_embedding{job="model_exporter",instance="10.0.1.5:9100",embedding_dim="128"} 0.12

# Green (new) exporter
$ curl http://10.0.2.5:9100/metrics | grep model_embedding
model_embedding{job="model_exporter",instance="10.0.2.5:9100",embedding_dim="256",embedding_type="dense"} 0.09

4. Verify remote‑write behavior

Inspect Cortex logs (or any remote storage) for dimension rejections.

2024-07-07T12:35:02Z level=error msg="remote write failed: series with mismatched dimensions (expected 128, got 256)" tenant=tenant-a

5. Check alert rule definitions

Alert expressions often assume a fixed label set. Example from alerts.yml:

- alert: ModelEmbeddingLatencyHigh
  expr: avg_over_time(model_embedding[5m]) > 0.2
  for: 2m
  labels:
    severity: critical

If model_embedding suddenly has an extra embedding_type label, the selector becomes ambiguous and the rule evaluation fails (as documented in the alerting rules guide).

Resolution

Step 1 – Align metric schemas across blue and green

Update the exporter in the blue environment to emit the new label set, or revert the green exporter to the old schema. The chosen approach depends on whether the new label is required for future analysis.

Before (blue exporter, old schema)

# model_exporter v1.3
metric_name: model_embedding
labels:
  - job: model_exporter
  - instance: ${HOSTNAME}:9100
  - embedding_dim: "128"

After (blue exporter, updated schema)

# model_exporter v1.4 (aligned with green)
metric_name: model_embedding
labels:
  - job: model_exporter
  - instance: ${HOSTNAME}:9100
  - embedding_dim: "256"
  - embedding_type: "dense"

Alternatively, if the new dimensions are optional, guard them behind a conditional flag so that both clusters can emit the same label set during the rollout.

Step 2 – Adjust Prometheus scrape config

Ensure the relabel_configs block does not drop or add labels inconsistently.

scrape_configs:
  - job_name: "model_exporter"
    static_configs:
      - targets: ["10.0.1.5:9100", "10.0.2.5:9100"]
    relabel_configs:
      # Preserve embedding_type if present, otherwise set a default
      - source_labels: [__meta_consul_tags]
        regex: .*embedding_type_(.+).*
        target_label: embedding_type
        replacement: "$1"
      - source_labels: [embedding_type]
        regex: ""
        target_label: embedding_type
        replacement: "unknown"

Step 3 – Reload Prometheus with validation

$ promtool check config /etc/prometheus/prometheus.yml
checking /etc/prometheus/prometheus.yml
  SUCCESS: no duplicate series with different label dimensions

$ kill -HUP $(pidof prometheus)   # or use the HTTP reload endpoint

Step 4 – Verify remote‑write compatibility

If using Cortex or Thanos, ensure the remote‑write endpoint is configured to accept the new vector dimension.

remote_write:
  - url: "https://cortex.example.com/api/v1/push"
    write_relabel_configs:
      - source_labels: [embedding_dim]
        regex: "256"
        action: keep

Validation

1. Confirm that Prometheus reload logs no longer contain dimension errors.
2. Query the metric to ensure a single series per instance:

$ curl -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=count by (instance,embedding_dim,embedding_type) (model_embedding)'
{"status":"success","data":{"resultType":"vector","result":[
  {"metric":{"instance":"10.0.1.5:9100","embedding_dim":"256","embedding_type":"dense"},"value":[1699046400,"1"]},
  {"metric":{"instance":"10.0.2.5:9100","embedding_dim":"256","embedding_type":"dense"},"value":[1699046400,"1"]}
]}}

Check that alert rules fire as expected:

$ curl -G 'http://localhost:9090/api/v1/alerts'
{"status":"success","data":{"alerts":[ ... "ModelEmbeddingLatencyHigh" ... ]}}

Inspect remote‑write logs for the absence of “mismatched dimensions” errors.

Prevention and Best Practices

• Schema versioning: Tag exporters with a schema_version label and enforce that all scrape targets share the same version during a rollout.
• Automated config validation: Integrate promtool check config into CI pipelines; fail the pipeline if duplicate series are detected.
• Blue‑green contract test: Before switching traffic, run a temporary job that scrapes both environments and diffs the label sets for every metric.
• Remote‑write compatibility matrix: Document expected vector dimensions per metric and verify them against the remote storage’s schema (see the remote‑write docs).
• Alert rule resilience: Use ignoring(label) or on(label) modifiers in alert expressions to tolerate optional labels when feasible.
• Monitoring for dimension drift: Create a Prometheus rule that alerts on sudden changes in label cardinality:

  
  - alert: MetricLabelCardinalityDrift
    expr: changes(count by (metric_name, __name__) (rate(prometheus_tsdb_head_samples_appended[5m])) ) > 0
    for: 1m
    labels:
      severity: warning
  

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the error appear only after the blue‑green switch and not during a normal reload?
   Because the blue environment still emits the old label set. When the green environment starts serving the new schema, Prometheus sees two different label sets for the same metric name during the same reload, triggering the duplicate‑series check.
2. Can I use ignore or on in alert rules to bypass the mismatch?
   Yes, but only for optional labels. If the dimension change is structural (e.g., vector size), the underlying series still diverge, and remote‑write will reject them. Use label‑agnostic selectors only after ensuring the series themselves are compatible.
3. How do I safely test a new exporter version without affecting production alerts?
   Deploy the new exporter to a separate namespace, scrape it with a temporary Prometheus instance, and run a diff script that compares label sets against the production exporter. Only promote when the diff is empty or when you have updated all dependent alert rules.
4. What is the recommended way to roll back if a blue‑green deployment introduces a dimension mismatch?
   Rollback the exporter binary or configuration to the previous version, then reload Prometheus. Because the metric schema reverts, the duplicate‑series error disappears and alerts resume.
5. Is there a limit on the number of dimensions (labels) a metric can have?
   Prometheus does not enforce a hard limit on label count, but each unique label combination creates a separate time series, increasing memory and storage usage. The remote‑write documentation warns that mismatched dimensions (e.g., 128 vs 256) are rejected by many back‑ends, so keep dimensions consistent across deployments.