MLflow tracking 400 Bad Request after JSON schema validation failure

MLflow Tracking — 400 Bad Request After JSON Schema Validation Failure

Problem

When logging experiment metadata (e.g., mlflow.log_param, mlflow.log_metric, or mlflow.set_tag) to a managed MLflow tracking endpoint (Azure ML, AWS SageMaker, GCP AI Platform), the client receives:

HTTP/1.1 400 Bad Request
Content-Type: application/json
{
  "error_code": "INVALID_PARAMETER_VALUE",
  "message": "JSON schema validation failed: request payload does not conform to the expected schema"
}

The failure is intermittent at first glance, but persists for a specific set of runs. No authentication errors appear, and the endpoint URL is reachable (HTTPS 200 on health‑check).

Root Cause

MLflow’s tracking server validates every incoming JSON payload against a strict OpenAPI schema. The schema enforces:

• String length limits (e.g., run_name ≤ 255 characters)
• Allowed characters for keys (ASCII alphanumerics, _, -)
• Data‑type constraints (numeric values must be JSON numbers, not strings)
• Maximum number of tags/parameters per request (default 100)

In the observed failures, one of the following violations occurred:

1. Oversized tag value: a tag containing a serialized JSON object (> 500 bytes) exceeded the maxLength of 500 characters defined for tag.value.
2. Non‑ASCII characters in keys: experiment names generated from user IDs included Unicode emojis, which the schema rejects because ^[A-Za-z0-9_-]+$ is enforced.
3. Numeric values sent as strings: a custom logging wrapper called log_metric_float performed str(value) before passing to mlflow.log_metric, causing the payload field value to be a JSON string instead of a number.

Because the tracking server aborts the request before persisting any data, the client sees only the generic 400 response without a stack trace.

Debug & Investigation

Step‑by‑step reproduction of the failure:

1. Enable MLFLOW_TRACKING_DEBUG on the client to capture request/response payloads.
2. Inspect the failing request body in the debug logs.
3. Validate the JSON against the official schema using jsonschema locally.

Example debug session:

export MLFLOW_TRACKING_DEBUG=1
python train.py
...
2026-10-31 12:45:02,317 - mlflow.tracking.client - DEBUG - POST /api/2.0/mlflow/runs/log-metric
Payload:
{
  "run_id": "12345abcde",
  "key": "accuracy",
  "value": "0.9234",
  "timestamp": 1698765902000,
  "step": 1
}
2026-10-31 12:45:02,322 - mlflow.tracking.client - ERROR - Request failed (status=400)
Response:
{
  "error_code": "INVALID_PARAMETER_VALUE",
  "message": "JSON schema validation failed: 'value' is not of type 'number'"
}

Validating the payload locally:

$ python - <<'PY'
import json, jsonschema, pathlib
schema = json.load(open('mlflow-openapi-schema.json'))['components']['schemas']['LogMetricRequest']
payload = {
  "run_id": "12345abcde",
  "key": "accuracy",
  "value": "0.9234",   # <-- string, should be number
  "timestamp": 1698765902000,
  "step": 1
}
try:
    jsonschema.validate(payload, schema)
except jsonschema.ValidationError as e:
    print(e)
PY
'value' is not of type 'number'

Similar steps revealed the oversized tag:

2026-10-31 13:02:11,104 - mlflow.tracking.client - DEBUG - POST /api/2.0/mlflow/runs/set-tag
Payload:
{
  "run_id": "67890fghij",
  "key": "model_config",
  "value": "{\"layers\":[{\"type\":\"dense\",\"units\":1024}, ... 800 more chars ...}"
}
Response:
{
  "error_code": "INVALID_PARAMETER_VALUE",
  "message": "JSON schema validation failed: 'value' exceeds maximum length of 500"
}

Solution

Fixes are applied at three levels: client code, logging wrappers, and optional server configuration (if the managed service permits schema overrides).

1. Ensure numeric metrics are passed as numbers

Replace any str() conversion before mlflow.log_metric:

Before:

def log_metric_float(run_id, key, value, step=0):
    # Incorrect: value becomes a string
    mlflow.log_metric(key, str(value), step=step)

After:

def log_metric_float(run_id, key, value, step=0):
    # Correct: keep as float
    mlflow.log_metric(key, float(value), step=step)

2. Truncate or compress large tag values

For tags that need to store structured data, serialize to a compact representation (e.g., base64‑encoded gzip) and enforce the 500‑character limit.

Before:

mlflow.set_tag("model_config", json.dumps(config_dict))

After:

import json, gzip, base64

def compress_tag(value: dict) -> str:
    json_bytes = json.dumps(value).encode('utf-8')
    gz = gzip.compress(json_bytes)
    return base64.b64encode(gz).decode('ascii')[:500]  # enforce limit

mlflow.set_tag("model_config", compress_tag(config_dict))

3. Sanitize experiment/run names and tag keys

Strip or replace non‑ASCII characters before calling MLflow APIs:

import re

def sanitize_key(key: str) -> str:
    # Keep only alphanumerics, underscore, hyphen
    return re.sub(r'[^A-Za-z0-9_-]', '_', key)

run_name = sanitize_key(user_provided_name)
mlflow.start_run(run_name=run_name)

4. (Optional) Increase server limits

If the managed service exposes a configuration knob (e.g., mlflow.tracking.max_tag_length), raise it after confirming compliance with organizational policies. This step varies by provider and may require a support ticket.

Verification

After applying the fixes, repeat the logging steps and confirm that the server returns HTTP 200.

2026-10-31 14:10:05,212 - mlflow.tracking.client - DEBUG - POST /api/2.0/mlflow/runs/log-metric
Payload:
{
  "run_id": "12345abcde",
  "key": "accuracy",
  "value": 0.9234,
  "timestamp": 1698765902000,
  "step": 1
}
2026-10-31 14:10:05,218 - mlflow.tracking.client - INFO - Metric logged successfully

Additional validation steps:

• Run mlflow experiments list and verify the new run appears.
• Query the tracking server via REST: curl -s -H "Authorization: Bearer $TOKEN" $MLFLOW_ENDPOINT/api/2.0/mlflow/runs/get?run_id=12345abcde and inspect the tags and metrics fields.
• Check monitoring dashboards for a drop in 400‑error rate.

Prevention

Implement guardrails to catch schema violations before they reach the server:

Guardrail	Implementation
Schema‑aware client wrapper	Wrap MLflow calls in a helper that runs jsonschema.validate against the local schema file.
Input sanitization library	Centralize sanitize_key and compress_tag functions in a shared utils module.
CI linting for experiment metadata	Add a unit test that generates a synthetic run with max‑size tags/parameters and asserts validation passes.
Alert on 4xx spikes	Configure monitoring (Prometheus/CloudWatch) to fire an alert when mlflow_http_4xx_total exceeds a threshold.

Document the accepted schema limits in the engineering handbook and include them in code review checklists.

FAQ

1. Why does the error only appear for some runs?
   The offending payload fields (e.g., a long tag or a Unicode experiment name) are generated conditionally based on user input or model configuration. Runs that do not hit those code paths produce compliant JSON, so the server responds with 200.
2. Can I disable JSON schema validation on the managed tracking server?
   Managed MLflow services typically enforce the schema for data integrity. Disabling it is not supported; the recommended approach is to fix the client payload.
3. How do I find the exact schema version used by the service?
   Query the OpenAPI spec: curl $MLFLOW_ENDPOINT/api/2.0/openapi.json. The components.schemas section contains the definitions for LogMetricRequest, SetTagRequest, etc.
4. Is there a way to automatically truncate tag values?
   Yes. Implement a wrapper around mlflow.set_tag that checks len(value) and truncates or compresses the string before sending.
5. What monitoring metric should I watch to detect future schema violations?
   Track mlflow_http_4xx_total{status="400"} (or the provider‑specific equivalent). A sudden increase indicates new payload violations.

Related Topic Hub: Model Serving Troubleshooting Hub