Prometheus Query Error 400 Bad Request After Changing Label Selectors

Q: 2. Rename dash‑containing label to a valid identifier

Change the exporter to emit model_type instead of model-type:

Q: 4. Correct function argument types

If a rate() call was missing a range vector, add the interval:

In this article

Table of Contents

Problem: Prometheus query returns 400 Bad Request after changing label selectors

After a recent change to the AI model evaluation exporter, dashboard panels that query model_inference_latency_seconds, model_accuracy, and model_tokens_total started failing with HTTP 400 responses from the Prometheus query API. The error payload typically looks like:

{
  "status":"error",
  "errorType":"bad_data",
  "error":"parse error at char 27: unexpected character '\\' in label matcher"
}

Other observed messages include:

error="bad_data", errorType="bad_data", message="invalid regex in label matcher: error parsing regexp: missing closing ]"
error="bad_data", errorType="bad_data", message="argument to function 'rate' must be a range vector"
error="bad_data", errorType="bad_data", message="invalid label name 'model-version' (must match [a-zA-Z_][a-zA-Z0-9_]*)"

These errors prevent Grafana panels from rendering and break alert rules that depend on the same queries.

Root Cause Analysis

The 400 Bad Request responses are generated by the Prometheus query parser when it encounters malformed label selectors or invalid function arguments. Three concrete changes introduced the failures:

New required label model_version added to model_inference_latency_seconds – existing selectors omitted the label, causing binary expressions that require matching label sets to become unparsable.
Label name changed to include a dash (model-type) – PromQL label names must match [a-zA-Z_][a-zA-Z0-9_]* (see Prometheus docs). The dash is interpreted as a subtraction operator, leading to a parse error.
Regex selector with unescaped forward slash – the metric model_tokens_total{endpoint="/predict"} was queried with endpoint=~"/predict" without escaping the slash, producing an invalid regular expression (regex matching docs).

In each case the parser cannot construct a valid abstract syntax tree, so it returns HTTP 400 as documented in the Prometheus API error handling page.

Investigation and Debugging Steps

Follow these steps to isolate the offending selector:

Inspect the failing query via the HTTP API:

curl -G 'http://prometheus.example.com/api/v1/query' \
  --data-urlencode 'query=model_inference_latency_seconds{job="ai-service"}'

Response contains the parse error shown above.

Enable query logging (if not already) in prometheus.yml:
```
global:
  scrape_interval: 15s
  external_labels:
    monitor: "ai-monitor"

log_level: debug
query_log_file: /var/log/prometheus/query.log
```
After reloading Prometheus, the query log entry shows the exact character offset where parsing failed.

Validate label names against the PromQL grammar using a small Go script or promtool check rules:

# Example using promtool
cat > test.rules <<EOF
record: job:model_inference_latency_seconds
expr: model_inference_latency_seconds{model-type="v1"}
EOF

promtool check rules test.rules
# Output: error: invalid label name 'model-type'

Test regex selectors locally with grep -P or python -c to ensure they compile:

python - <<'PY'
import re
re.compile(r'/predict')
PY
# No exception – but when the slash is not escaped in PromQL it becomes an invalid RE

Review recent exporter changes (git diff) for added/renamed labels:

git diff HEAD~1 HEAD -- exporters/ai_service_exporter.go

Look for lines like:

metric.WithLabelValues(modelVersion, modelType).Set(value)

Resolution

Apply the following fixes to the affected queries and exporter configuration.

1. Update selectors to include the new required label

Before:

model_inference_latency_seconds{job="ai-service"}

After:

model_inference_latency_seconds{job="ai-service",model_version=~".*"}

Adding a wildcard matcher for model_version satisfies the selector requirement without hard‑coding a specific version.

2. Rename dash‑containing label to a valid identifier

Change the exporter to emit model_type instead of model-type:

// Before
metric.WithLabelValues(modelType).Set(value)

// After
metric.WithLabelValues(strings.ReplaceAll(modelType, "-", "_")).Set(value)

Corresponding query update:

model_accuracy{model_type="v1"}

3. Escape special characters in regex selectors

Forward slash must be escaped in the PromQL regex syntax:

# Before (invalid)
model_tokens_total{endpoint=~"/predict"}

# After (valid)
model_tokens_total{endpoint=~"\\/predict"}

4. Correct function argument types

If a rate() call was missing a range vector, add the interval:

# Before (invalid)
rate(model_eval_histogram_bucket[5m])

# After (valid)
rate(model_eval_histogram_bucket[5m])

Note that the original error often arises when the range vector is omitted entirely, e.g. rate(model_eval_histogram_bucket).

Verification

After applying the changes, confirm that queries succeed and dashboards render:

Run the API query again:

curl -G 'http://prometheus.example.com/api/v1/query' \
  --data-urlencode 'query=model_inference_latency_seconds{job="ai-service",model_version=~".*"}'

Expected response:

{
  "status":"success",
  "data":{
    "resultType":"vector",
    "result":[ ... ]
  }
}

Check Grafana panel status – the “No data” or “Error” banner should disappear.
Trigger an alert rule that uses the fixed query and verify that the alert fires (or resolves) as expected.
Inspect /var/log/prometheus/query.log for the absence of parse errors.

Operational Experience and Prevention

Key observations from the incident:

Missing required label in binary expressions often surfaces only after a new label is added to the exporter. Prometheus does not automatically propagate the label to existing selectors.
Label naming conventions are easy to overlook when using hyphens for readability; they clash with the subtraction operator in PromQL.
Regex escaping is a frequent source of “invalid regex” errors, especially with path‑like label values that contain slashes or dots.
Function argument validation is strict – range vectors must be present for rate(), increase(), etc. A missing interval yields the “argument to function 'rate' must be a range vector” error.

To avoid recurrence:

Enforce a linting step for PromQL expressions (e.g., promtool test rules or promql-lint) in CI pipelines.
Document label naming policies and require snake_case for all custom labels.
Include integration tests that exercise regex selectors with representative label values.
Version‑pin exporter code and run automated diff checks on metric families after each release.

Best Practices and Guardrails

Practice	Why it helps
Use explicit label matchers for all required labels	Prevents silent selector mismatches when new labels are added.
Validate label names against the PromQL grammar	Avoids parse errors caused by illegal characters.
Escape regex metacharacters in label values	Ensures the regex engine receives a syntactically correct pattern.
Run `promtool check rules` on PRs	Catches syntax and type errors before they reach production.
Monitor `/api/v1/query` error rate	Early detection of systemic query failures.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does the query work in the Prometheus UI but fail via the API? The UI often auto‑quotes label values and escapes regex characters for you. Direct API calls require the exact PromQL string, exposing any syntax issues.
Can I use a dash in a label name if I escape it? No. PromQL label names must follow the identifier regex [a-zA-Z_][a-zA-Z0-9_]*. Dashes are not allowed, even with escaping.
How do I find which metric introduced a new required label? Compare the exporter’s metric family definitions before and after the change (e.g., git diff on the exporter source) or query the __name__ series with label_names() to list all labels for a metric.
What is the proper way to match a literal forward slash in a regex selector? Escape it with a backslash: endpoint=~"\\/predict". Double backslashes are needed because the string literal is parsed by PromQL before the regex engine.
Why does histogram_quantile sometimes return a parse error after a label change? The function expects the le label to be present and numeric (or the string "+Inf"). If the exporter emits a malformed le label (e.g., missing quotes), the selector becomes invalid, leading to a 400 error.

Problem: Prometheus query returns 400 Bad Request after changing label selectors