photo of computer cables

ChromaDB query API 400 Bad Request after client library version mismatch

by

Problem: 400 Bad Request from ChromaDB Query API in an Air‑Gapped Deployment

In an isolated internal network the client library attempts to invoke /api/v1/query on a ChromaDB server. The server returns HTTP 400 with a body similar to:


{
  "detail": "Invalid request payload: expected object with key 'queries', got null"
}

Symptoms observed:

  • Python client raises requests.exceptions.HTTPError: 400 Client Error
  • Server logs show a JSON parsing error
  • Only the client version that was built against a newer ChromaDB SDK reproduces the failure; older clients succeed.

Root Cause: Client‑Server Version Mismatch Leading to Malformed JSON Payload

ChromaDB introduced a breaking change in v0.5.0 of the Python client:

  • Previous payload format: {"queries": [...]}
  • New payload format (v0.5.0+): {"queries": [...], "metadata": {...}}

The server running version 0.4.3 still expects the legacy schema. When a newer client (e.g., 0.5.2) builds the request, it automatically adds the metadata field and serialises None values as null. The server’s JSON schema validator rejects the payload, resulting in a 400 response.

In an air‑gapped environment the client cannot be upgraded to match the server, nor can the server be patched without breaking other services. The mismatch therefore manifests as a “function calling format error”.

Debug: Reproducing and Isolating the Issue

1. Capture the HTTP request

Enable http.client debugging on the client:


import http.client as http_client
http_client.HTTPConnection.debuglevel = 1
import logging, sys
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)

Sample output (truncated):


send: 'POST /api/v1/query HTTP/1.1\r\nHost: chromadb.internal\r\nUser-Agent: python-requests/2.31.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 123\r\n\r\n'
send: b'{"queries":[{"embedding":[0.12,0.34,...]}],"metadata":null}'

2. Server log inspection

On the ChromaDB node, inspect journalctl -u chromadb (or the Docker logs if containerised):


2026-10-25 14:03:12,874 ERROR  chromadb.server.handlers.query - Invalid request payload: field 'metadata' is not allowed
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/chromadb/server/handlers.py", line 87, in handle_query
    schema.validate(request_json)
  File ".../jsonschema/validators.py", line 123, in validate
    raise ValidationError(message)
jsonschema.exceptions.ValidationError: Additional properties are not allowed ('metadata' was unexpected)

3. Version verification


# Client side
import chromadb
print(chromadb.__version__)   # 0.5.2

# Server side (inside container)
cat /app/version.txt          # 0.4.3

Solution: Align Request Format with Server Expectations

Two practical paths exist in an air‑gapped environment:

  1. Pin the client to the legacy API contract (recommended when server upgrade is impossible).
  2. Patch the server to accept the newer schema (requires rebuilding the server binary with back‑ported changes).

Option 1 – Pinning the client

Install the last client version that matches the server’s schema (0.4.3) and explicitly disable the new metadata field.


# Uninstall current client
pip uninstall chromadb -y

# Install legacy version
pip install "chromadb==0.4.3"

# Verify version
python -c "import chromadb, sys; print(chromadb.__version__)"

If the project must stay on a newer client for other features, monkey‑patch the request builder:


# monkey_patch.py
import chromadb
from chromadb.api import Client

_original_query = Client._prepare_query_payload

def _legacy_query(self, *args, **kwargs):
    payload = _original_query(self, *args, **kwargs)
    # Remove the 'metadata' key introduced in v0.5.0
    payload.pop('metadata', None)
    return payload

Client._prepare_query_payload = _legacy_query

Import the patch before creating the client:


import monkey_patch  # noqa: F401
import chromadb

client = chromadb.Client(...)
results = client.query(embeddings=[...])

Option 2 – Server patch

Rebuild the server with a relaxed JSON schema that marks metadata as optional:


# In chromadb/server/schemas.py
QUERY_SCHEMA = {
    "type": "object",
    "properties": {
        "queries": {"type": "array"},
        "metadata": {"type": ["object", "null"]}  # make optional
    },
    "required": ["queries"],
    "additionalProperties": False
}

Recompile and redeploy the container:


docker build -t internal/chromadb:0.4.3-patched .
docker tag internal/chromadb:0.4.3-patched internal/chromadb:latest
docker compose up -d chromadb

After the patch, the server accepts both legacy and newer payloads, eliminating the version lock‑step.

Verify: Confirming Successful Query Execution

Run a simple query and inspect the HTTP status and response body.


import chromadb, json, requests

client = chromadb.Client(...)
resp = client._session.post(
    "http://chromadb.internal/api/v1/query",
    json=client._prepare_query_payload(embeddings=[[0.1, 0.2, 0.3]])
)

print("Status:", resp.status_code)
print("Body:", json.dumps(resp.json(), indent=2))

Expected output after fix:


Status: 200
Body: {
  "results": [
    {
      "id": "doc-123",
      "score": 0.987
    }
  ]
}

Additional validation steps:

  • Check chromadb_server_requests_total{status="200"} metric in Prometheus.
  • Confirm no 400 entries appear in /var/log/chromadb/access.log for the past 5 minutes.
  • Run the integration test suite that exercises client.query() against the isolated instance.

Prevent: Guardrails for Future Deployments

  • Version pinning policy: Enforce matching chromadb client and server versions via a BOM (Bill of Materials) file checked into source control.
  • Contract testing: Use Pact or similar consumer‑driven contract tests to verify request/response schemas before promotion.
  • Schema version endpoint: Deploy a lightweight /api/v1/version endpoint that returns both server and supported client schema versions; automate compatibility checks in CI.
  • Air‑gap CI pipeline: Mirror the exact client wheel used in production into the internal artifact repository; reject builds that attempt to pull newer wheels from the public index.
  • Monitoring: Alert on any HTTP 400 responses from /api/v1/query with a threshold of >5 per minute.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Q1: Why does the same client work against a local ChromaDB instance but fail in the air‑gapped environment?

A1: The local instance is likely running the newer server version that matches the client. The isolated server remains on an older release, so the request payload includes fields the server does not recognise.

Q2: Can I disable the new metadata field without downgrading the entire client library?

A2: Yes. Import a monkey‑patch that removes the field from the generated payload (see the _legacy_query example). This approach isolates the change to the query path only.

Q3: Is there a way to make the server automatically ignore unknown fields?

A3: The server’s JSON schema uses "additionalProperties": false. Modifying the schema to true or adding the new field as optional restores forward compatibility, but requires rebuilding the server binary.

Q4: How do I detect version mismatches before they cause runtime failures?

A4: Implement a health‑check that queries /api/v1/version and compares the returned schema_version against the client’s expected version. Fail the startup if they diverge.

Q5: Will enabling allow_unknown_fields in the client’s JSON encoder help?

A5: The client already serialises unknown fields; the failure originates from the server’s strict schema validation. Adjusting the client encoder does not affect the server’s rejection.