ChromaDB Stop Sequence Trigger Failure In Development Sandbox

Q: Problem – Stop Sequence Trigger Failure in Development Sandbox

Engineers using ChromaDB in a Docker‑based development sandbox report that the stop_sequences parameter supplied to query or generation calls is ignored. The symptom manifests as:

Q: 3. Missing CHROMA_STOP_SEQUENCE Environment Variable

When the sandbox runs without the CHROMA_STOP_SEQUENCE variable, the server does not pre‑load stop token IDs for fast lookup. The API then validates the incoming stop_sequences field and raises:

Q: Investigation and Debugging Steps Check sandbox environment variables. Run: docker exec -it chroma_sandbox env | grep CHROMA_ Expected output should include: CHROMA_TRUNCATE_ENABLED=true CHROMA_STOP_SEQUENCE=__END__ If CHROMA_TRUNCATE_ENABLED is false or CHROMA_STOP_SEQUENCE is missing, the stop logic is disabled. Verify tokenizer versions. On the client side: python -c "import sentencepiece as sp; print(sp.__version__)" On the server side (inside the container): docker exec -it chroma_sandbox pip show sentencepiece | grep Version Both should report the same version (e.g., 0.1.96). Mismatches indicate a vocabulary divergence. Inspect API request payload. Capture a failing request with tcpdump or curl -v: curl -X POST http://localhost:8000/v1/generate \ -H "Content-Type: application/json" \ -d '{"prompt":"Hello world","stop_sequences":["__END__"]}' -v Confirm that stop_sequences is a JSON array of strings. A missing field will trigger the ValueError mentioned earlier. Review container logs for truncation warnings. Use journalctl or docker logs: docker logs chroma_sandbox 2>&1 | grep -i truncation Look for the “WARN: Truncation disabled” line. Reproduce race condition. Run two parallel generation calls: #!/usr/bin/env bash curl -s -X POST http://localhost:8000/v1/generate -d '{"prompt":"A","stop_sequences":["__END__"]}' & curl -s -X POST http://localhost:8000/v1/generate -d '{"prompt":"B","stop_sequences":["__END__"]}' & wait If intermittent StopSequenceError appears, the buffer race is active. Resolution – Making Stop Sequences Reliable 1. Enable Truncation

Update the sandbox .env or Docker Compose file to set CHROMA_TRUNCATE_ENABLED=true:

Q: 3. Define the Stop Token Explicitly

Set the CHROMA_STOP_SEQUENCE variable to match the token string used in stop_sequences:

In this article

Table of Contents

Problem – Stop Sequence Trigger Failure in Development Sandbox

Engineers using ChromaDB in a Docker‑based development sandbox report that the stop_sequences parameter supplied to query or generation calls is ignored. The symptom manifests as:

Unbounded text generation that eventually exhausts container memory (OOM).

Log entries such as:

ERROR: StopSequenceError – Expected stop token not found in generated output.
WARN: Truncation disabled – possible OOM; set CHROMA_TRUNCATE_ENABLED=true to enforce stop sequences.
sequence_truncation_failed: token_id 12345 not in vocabulary

API responses that continue past the intended stop token, e.g.:

{
  "generated_text": "The quick brown fox jumps over the lazy dog. ... (hundreds of extra tokens)"
}

These failures appear only in the sandbox environment; production deployments respect stop_sequences as documented in the ChromaDB Retrieval API reference.

Root Cause Analysis

1. Truncation Flag Disabled

The sandbox configuration often disables automatic token truncation via the environment variable CHROMA_TRUNCATE_ENABLED. When set to false, the server skips the stop‑token buffer check, leading to the “WARN: Truncation disabled” message observed in the incident of 2024‑03‑15 (Docker Compose with a mounted volume).

2. Tokenizer Vocabulary Mismatch

ChromaDB relies on the server‑side tokenizer to map stop strings to token IDs. If the client uses a custom SentencePiece tokenizer that differs from the server’s version, the stop token ID may be absent from the server’s vocab, producing the log:

sequence_truncation_failed: token_id 12345 not in vocabulary

This mismatch was the primary factor in the 2024‑06‑02 CI pipeline failure (GitHub issue #1382).

3. Missing `CHROMA_STOP_SEQUENCE` Environment Variable

When the sandbox runs without the CHROMA_STOP_SEQUENCE variable, the server does not pre‑load stop token IDs for fast lookup. The API then validates the incoming stop_sequences field and raises:

ValueError – `stop_sequences` must be a list of strings; received NoneType in sandbox config.

This aligns with the Stack Overflow discussion (question 822345) that highlighted the missing env var.

4. Race Condition in Shared Buffer

Concurrent queries sharing the same stop‑sequence buffer can corrupt the internal state, resulting in intermittent StopSequenceError exceptions (incident 2024‑07‑19). The root cause is a lack of synchronization primitives around the buffer in the sandbox’s lightweight runtime.

Investigation and Debugging Steps

Check sandbox environment variables. Run:
```
docker exec -it chroma_sandbox env | grep CHROMA_
```
Expected output should include:
```
CHROMA_TRUNCATE_ENABLED=true
CHROMA_STOP_SEQUENCE=__END__
```
If CHROMA_TRUNCATE_ENABLED is false or CHROMA_STOP_SEQUENCE is missing, the stop logic is disabled.
Verify tokenizer versions. On the client side:
```
python -c "import sentencepiece as sp; print(sp.__version__)"
```
On the server side (inside the container):
```
docker exec -it chroma_sandbox pip show sentencepiece | grep Version
```
Both should report the same version (e.g., 0.1.96). Mismatches indicate a vocabulary divergence.
Inspect API request payload. Capture a failing request with tcpdump or curl -v:
```
curl -X POST http://localhost:8000/v1/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Hello world","stop_sequences":["__END__"]}' -v
```
Confirm that stop_sequences is a JSON array of strings. A missing field will trigger the ValueError mentioned earlier.
Review container logs for truncation warnings. Use journalctl or docker logs:
```
docker logs chroma_sandbox 2>&1 | grep -i truncation
```
Look for the “WARN: Truncation disabled” line.

Reproduce race condition. Run two parallel generation calls:

#!/usr/bin/env bash
curl -s -X POST http://localhost:8000/v1/generate -d '{"prompt":"A","stop_sequences":["__END__"]}' &
curl -s -X POST http://localhost:8000/v1/generate -d '{"prompt":"B","stop_sequences":["__END__"]}' &
wait

If intermittent StopSequenceError appears, the buffer race is active.

Resolution – Making Stop Sequences Reliable

1. Enable Truncation

Update the sandbox .env or Docker Compose file to set CHROMA_TRUNCATE_ENABLED=true:

# Before (docker-compose.yml)
environment:
  - CHROMA_TRUNCATE_ENABLED=false
  - CHROMA_STOP_SEQUENCE=__END__

# After
environment:
  - CHROMA_TRUNCATE_ENABLED=true
  - CHROMA_STOP_SEQUENCE=__END__

Restart the sandbox: docker compose up -d --build.

2. Align Tokenizer Versions

Ensure both client and server use the same tokenizer model file and library version. A typical fix is to mount the same SentencePiece model into the container and pin the library version:

# docker-compose.yml snippet
volumes:
  - ./tokenizer.model:/app/tokenizer.model:ro
environment:
  - SENTENCEPIECE_MODEL_PATH=/app/tokenizer.model
  - SENTENCEPIECE_VERSION=0.1.96

On the client side, install the exact version:

pip install sentencepiece==0.1.96

3. Define the Stop Token Explicitly

Set the CHROMA_STOP_SEQUENCE variable to match the token string used in stop_sequences:

# .env
CHROMA_STOP_SEQUENCE=__END__

If multiple stop strings are needed, separate them with commas and let the server split them at startup.

4. Serialize Access to Stop‑Sequence Buffer

Apply a lightweight lock around the buffer in the sandbox runtime. The upstream fix merged in PR #1382 adds a mutex; upgrade to v0.3.5 or later:

pip install chromadb==0.3.5

Alternatively, enforce single‑client access in CI by serializing test steps.

Verification – Confirming the Fix

Run a generation request that includes the stop token:

curl -X POST http://localhost:8000/v1/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Once upon a time","stop_sequences":["__END__"]}'

Expected output:

{
  "generated_text": "Once upon a time..."
}

The text should terminate before the stop token appears.

Check logs for absence of truncation warnings:
```
docker logs chroma_sandbox 2>&1 | grep -i truncation
```
No “WARN: Truncation disabled” lines should appear.
Validate token IDs:
```
curl http://localhost:8000/v1/tokenize -d '{"text":"__END__"}'
```
Response should contain a valid token_id that exists in the server’s vocab.
Run concurrent generation stress test (as in the Debug step) and confirm no StopSequenceError exceptions are logged.

Prevention – Guardrails for Future Sandboxes

Configuration Templates: Store a canonical .env.example with CHROMA_TRUNCATE_ENABLED=true and CHROMA_STOP_SEQUENCE defined. CI should fail if these variables differ.
Version Pinning: Use a requirements.txt that pins sentencepiece and chromadb to the same versions across client and server.
Automated Health Checks: Add a liveness probe that queries /v1/tokenize for the stop token and fails the pod if the token is missing.
Monitoring: Create a Grafana alert on the sequence_truncation_failed log pattern or on memory usage spikes that correlate with generation endpoints.
Isolation: In shared development environments, allocate a dedicated sandbox per developer to avoid race conditions in the stop‑sequence buffer.

FAQ – Common Follow‑Up Questions

Why does the stop sequence work in production but not in the sandbox?
Production containers enable CHROMA_TRUNCATE_ENABLED by default and ship with a pre‑built tokenizer model, whereas sandbox images often disable truncation for faster iteration and may omit the CHROMA_STOP_SEQUENCE variable.
How can I verify which token ID corresponds to my stop string?
Use the /v1/tokenize endpoint:
```
curl -X POST http://localhost:8000/v1/tokenize -d '{"text":"__END__"}'
```
The response JSON contains the token_id. Ensure this ID appears in the server’s vocabulary via chromadb token-list (if available).
Can a custom tokenizer be used safely in the sandbox?
Yes, but the same model file and library version must be mounted into the container, and CHROMA_STOP_SEQUENCE must be set to a string that exists in that model’s vocab. Mismatched versions cause the “sequence_truncation_failed” error.
What is the impact of disabling CHROMA_TRUNCATE_ENABLED?
Disabling truncation bypasses the stop‑token check, allowing the generation engine to run until its internal max token limit, which can quickly exhaust memory and trigger OOM kills, as seen in the 2024‑03‑15 incident.
How do I protect against race conditions when multiple developers share a sandbox?
Either upgrade to chromadb>=0.3.5 (which adds mutex protection) or enforce per‑developer sandbox instances via separate Docker Compose projects or Kubernetes namespaces.

Related Topic Hub: Vector Databases Troubleshooting Hub

ChromaDB stop sequence ignored during disaster recovery

ChromaDB stop sequence trigger failure in development sandbox