ChromaDB filter expression parse error after blue‑green deployment switch

ChromaDB Filter Expression Parse Error After Blue‑Green Deployment Switch

Problem (Symptoms and Impact)

During a staged blue‑green rollout of the vector‑search microservice, traffic was gradually shifted from the blue version (v1.3.0) to the green version (v1.4.0). After ~30 % of requests were routed to the green pods, the service began returning HTTP 400 responses with the following payload:

{
  "error": "FilterExpressionParseError",
  "message": "Failed to parse filter expression: unexpected token 'OR' at position 27"
}

Only queries that contained nested logical operators (e.g., ((category='A' AND tag='x') OR (category='B'))) failed. Simple equality filters continued to work. The error manifested in:

  • Increased 5xx‑to‑4xx conversion rate in the API gateway.
  • Spike in latency as callers retried.
  • Alert “ChromaDB filter parse failures” firing on the monitoring dashboard.

Root Cause Analysis

The green deployment introduced a new chroma client library version (0.4.2) that switched the default filter parser from LegacyParser to FastParser. The FastParser enforces strict token ordering and does not tolerate whitespace variations that the legacy parser silently ignored.

Two concrete mis‑configurations triggered the parse error:

  1. Environment‑specific query builder: The green code path builds filter strings using String.format with a trailing space before each logical operator. Example:
    String filter = String.format("%s AND %s ", conditionA, conditionB);

    The extra space after conditionB results in a token sequence ... AND tag='x' (space before OR) that FastParser rejects.

  2. Missing parser compatibility flag: The previous version set CHROMA_PARSER=legacy via an environment variable. The green deployment omitted this variable, causing the client to default to FastParser without fallback.

Because the blue pods still used the legacy parser, the error only appeared after traffic switched to green, matching the observed symptom timeline.

Debug and Investigation Steps

The following checklist reproduces the investigation performed in production:

  1. Correlate logs with traffic shift:
    journalctl -u vector-search.service -f | grep FilterExpressionParseError

    Sample log excerpt:

    2026-11-10T14:23:41.112Z WARN filter_parser - FilterExpressionParseError: unexpected token 'OR' at position 27 (filter="category='A' AND tag='x' OR category='B'")
  2. Identify client version:
    kubectl exec -it $(kubectl get pod -l app=vector-search,env=green -o jsonpath='{.items[0].metadata.name}') -- cat /app/requirements.txt | grep chroma

    Output:

    chroma==0.4.2
  3. Reproduce locally using the same filter string:
    python - <<'PY'
    import chroma
    client = chroma.Client()
    try:
        client.query(filter="category='A' AND tag='x' OR category='B'")
    except Exception as e:
        print(e)
    PY

    Result:

    FilterExpressionParseError: unexpected token 'OR' at position 27
  4. Compare parser behavior between versions:
    # Legacy parser (0.3.9)
    client = chroma.Client(parser='legacy')
    client.query(filter="category='A' AND tag='x' OR category='B'")
    # succeeds
  5. Inspect environment variables in the green pod:
    kubectl exec -it $(kubectl get pod -l app=vector-search,env=green -o jsonpath='{.items[0].metadata.name}') -- printenv | grep CHROMA

    Output shows the variable is missing.

Solution (Code and Configuration Changes)

The fix consists of two parts: restore the legacy parser compatibility flag and clean up the filter string construction to avoid trailing whitespace.

Before (problematic code)

// src/main/java/com/example/query/FilterBuilder.java
public String buildFilter(Map<String, String> conditions) {
    StringBuilder sb = new StringBuilder();
    for (Map.Entry<String, String> entry : conditions.entrySet()) {
        sb.append(entry.getKey())
          .append("='")
          .append(entry.getValue())
          .append("' AND "); // always appends AND, even for the last entry
    }
    // Result contains trailing " AND "
    return sb.toString();
}

After (robust builder)

// src/main/java/com/example/query/FilterBuilder.java
public String buildFilter(Map<String, String> conditions) {
    List<String> parts = new ArrayList<>();
    for (Map.Entry<String, String> entry : conditions.entrySet()) {
        parts.add(entry.getKey() + "='" + entry.getValue() + "'");
    }
    // Join with a single space around the operator; no trailing whitespace
    return String.join(" AND ", parts);
}

Additionally, add the missing environment variable to the green deployment manifest:

# deployment-green.yaml (excerpt)
spec:
  containers:
    - name: vector-search
      image: registry.example.com/vector-search:1.4.0
      env:
        - name: CHROMA_PARSER
          value: "legacy"   # <-- added
        - name: CHROMA_ENDPOINT
          value: "http://chromadb:8000"

After applying the manifest, the green pods start with the correct parser mode and generate well‑formed filter strings.

Verification (How to Confirm the Fix)

  1. Smoke test the endpoint against a green pod:
    curl -s -X POST https://green.api.example.com/search \
          -H "Content-Type: application/json" \
          -d '{"filter":"category=\"A\" AND tag=\"x\" OR category=\"B\""}'

    Expected response: HTTP 200 with a non‑empty result set.

  2. Check logs for absence of parse errors:
    kubectl logs -l app=vector-search,env=green | grep FilterExpressionParseError

    No output indicates the parser no longer rejects the expression.

  3. Validate metrics (Prometheus query):
    sum(rate(http_requests_total{service="vector-search",code=~"4.."}[5m]))

    The rate should drop back to baseline (< 0.01 rps).

  4. Run integration test suite that exercises nested filters across all supported operators. All tests must pass.

Prevention (Best Practices and Guardrails)

  • Pin client library versions in CI/CD pipelines and enforce semantic version compatibility checks.
  • Explicitly set parser mode via environment variable or client configuration rather than relying on library defaults.
  • Centralize filter construction in a shared utility that guarantees whitespace‑free output; add unit tests for edge‑case expressions.
  • Include parser‑specific health checks in the service readiness probe:
    #!/bin/sh
    if chroma-client --ping --parser=${CHROMA_PARSER:-legacy}; then exit 0; else exit 1; fi
  • Blue‑green rollout validation: before shifting traffic, run a synthetic workload that exercises complex filters against the target version and assert zero parse errors.

FAQ

  1. Why did the error appear only after the traffic shift?
    The blue version used the legacy parser (default in chroma==0.3.x) which tolerated extra whitespace. The green version upgraded to chroma==0.4.2 where the default parser became stricter, exposing the malformed filter strings generated by the new code path.
  2. Can I keep using the fast parser without changing the code?
    Yes, but you must ensure filter strings are built without trailing spaces and conform to the strict token grammar. Refactoring the builder as shown eliminates the whitespace issue, after which the fast parser can be safely enabled.
  3. Is there a way to detect parser incompatibility automatically?
    Enable the client’s parser_version metric and set an alert when it switches from legacy to fast without an accompanying configuration change. This catches unintended upgrades during deployments.
  4. Do other ChromaDB operations (e.g., upserts) suffer from the same parser change?
    No. The parser is only invoked for metadata filter strings in query calls. Upserts and collection management use a separate JSON schema validator.
  5. What version of the client library restores the previous behavior?
    Downgrading to chroma==0.3.9 or explicitly setting CHROMA_PARSER=legacy with any newer version will revert to the tolerant parsing mode.

Related Topic Hub: Vector Databases Troubleshooting Hub