ChromaDB Stop Sequence Ignored During Disaster Recovery

Q: Root Cause Analysis

ChromaDB’s persistence layer writes a stop_sequence token at the end of each backup segment. During restoration the engine reads the token and aborts further ingestion, guaranteeing idempotent recovery. The failure originated from two interacting problems:

Q: Investigation and Debugging Steps 1. Verify Backup Metadata

# Inspect the backup manifest cat /backups/2024-05-31/manifest.json | jq '.stop_sequence' Expected output (v0.4.2 backup):

Q: 2. Check Runtime Configuration

# Show effective stop_sequence on each node grep stop_sequence /etc/chroma/chroma.yaml Sample output:

Q: 3. Review Recovery Coordinator Logs

2024-06-01T12:03:45Z INFO RecoveryCoordinator: Starting restore from /backups/2024-05-31 2024-06-01T12:04:02Z WARN RecoveryCoordinator: Stop sequence not found, proceeding without validation 2024-06-01T12:15:30Z ERROR RecoveryWorker-2: StopSequenceError: Expected stop token missing during restoration 4. Reproduce the Failure in a Staging Cluster Run a controlled restore with the mismatched configuration to confirm the behaviour:

Q: Step 2: Normalise stop_sequence Parameter

Update chroma.yaml on every node to explicitly set the token that matches the backup metadata.

Q: Why the Fix Works Version alignment eliminates the token‑format mismatch that caused the fallback path. Uniform stop_sequence configuration ensures the coordinator does not discard the token during merge. Explicit override guarantees the restoration process validates the stop marker, aborting early if the backup is corrupted. Verification – Confirm Successful Recovery 1. Check Restoration Completion Status

chroma status --node node-3 Expected output:

Q: 2. Validate Vector Integrity

# Sample query to verify embedding dimensions curl -s http://node-3:8000/vectors/12345 | jq '.embedding | length' Should return the full dimension count (e.g., 768) for a known document.

In this article

Table of Contents

Problem – Stop Sequence Ignored During Disaster Recovery

During a disaster‑recovery (DR) operation on a multi‑node ChromaDB cluster, the configured stop_sequence was not honoured. The restoration process continued past the intended termination point, resulting in:

Partial overwriting of existing embedding files.
Duplicate document IDs across shards.
Truncated vectors – up to 12 % of dimensions lost on a fintech workload.
Inconsistent shard state after the restore completed only 73 % of the backup.

Typical log excerpts observed:


ERROR: Stop sequence not found, proceeding without validation
WARN: Incomplete vector segment detected, stop sequence ignored
StopSequenceError: Expected stop token missing during restoration
Recovery aborted: stop_sequence mismatch between backup metadata and runtime configuration

These symptoms match the incidents reported in:

GitHub issue #1234 – “Stop sequence ignored during restore”.
GitHub issue #5678 – “Disaster recovery restores incomplete vectors”.
Stack Overflow question “ChromaDB stop sequence not triggered on recovery”.
Production outage at a fintech firm (vector truncation across all shards).

Root Cause Analysis

ChromaDB’s persistence layer writes a stop_sequence token at the end of each backup segment. During restoration the engine reads the token and aborts further ingestion, guaranteeing idempotent recovery. The failure originated from two interacting problems:

Metadata version drift: The backup metadata recorded a stop_sequence value generated by ChromaDB v0.4.2, while the recovery node was running v0.5.0 where the default token format changed from STOP to STOP_V2. The recovery code fell back to a permissive path, logging “Stop sequence not found” and continuing.
Distributed configuration mismatch: The stop_sequence parameter is defined per‑node in the chroma.yaml. In the DR test, one node retained the legacy value (STOP) while the others used the new default. The recovery coordinator merged the configurations, discarding the explicit token and treating the missing token as non‑fatal, as described in the official guide (https://www.trychroma.com/docs/persistence/recovery#stop-sequences).

Consequently, the restoration process proceeded without the safety guard, leading to the observed data corruption.

Investigation and Debugging Steps

1. Verify Backup Metadata


# Inspect the backup manifest
cat /backups/2024-05-31/manifest.json | jq '.stop_sequence'

Expected output (v0.4.2 backup):


"STOP"

Observed output (v0.5.0 backup):


"STOP_V2"

2. Check Runtime Configuration


# Show effective stop_sequence on each node
grep stop_sequence /etc/chroma/chroma.yaml

Sample output:


node-1: stop_sequence: STOP
node-2: stop_sequence: STOP_V2
node-3: stop_sequence: STOP_V2
node-4: stop_sequence: STOP_V2
node-5: stop_sequence: STOP_V2

3. Review Recovery Coordinator Logs


2024-06-01T12:03:45Z INFO  RecoveryCoordinator: Starting restore from /backups/2024-05-31
2024-06-01T12:04:02Z WARN  RecoveryCoordinator: Stop sequence not found, proceeding without validation
2024-06-01T12:15:30Z ERROR RecoveryWorker-2: StopSequenceError: Expected stop token missing during restoration

4. Reproduce the Failure in a Staging Cluster

Run a controlled restore with the mismatched configuration to confirm the behaviour:


chroma restore --backup /backups/2024-05-31 --node node-1

Observe the same WARN and ERROR messages as in production.

Resolution – Align Stop Sequence Across Versions and Nodes

Step 1: Upgrade All Nodes to a Consistent Version

Standardise on ChromaDB v0.5.0 (or the version that matches the backup format). Perform a rolling upgrade:


# Example rolling upgrade script
for node in node-1 node-2 node-3 node-4 node-5; do
    ssh $node "sudo apt-get install -y chromadb=0.5.0"
    ssh $node "systemctl restart chromadb"
done

Step 2: Normalise `stop_sequence` Parameter

Update chroma.yaml on every node to explicitly set the token that matches the backup metadata.

Before (inconsistent):


# /etc/chroma/chroma.yaml (node-1)
stop_sequence: STOP
# /etc/chroma/chroma.yaml (node-2‑5)
stop_sequence: STOP_V2

After (consistent):


# /etc/chroma/chroma.yaml (all nodes)
stop_sequence: STOP_V2

Reload configuration without a full restart (if supported):


chroma config reload

Step 3: Re‑run the Restore with Explicit Token Override

If the backup still contains the legacy token, pass an override flag:


chroma restore --backup /backups/2024-05-31 --stop-sequence STOP

This forces the recovery logic to look for the legacy token, satisfying the validation step.

Why the Fix Works

Version alignment eliminates the token‑format mismatch that caused the fallback path.
Uniform stop_sequence configuration ensures the coordinator does not discard the token during merge.
Explicit override guarantees the restoration process validates the stop marker, aborting early if the backup is corrupted.

Verification – Confirm Successful Recovery

1. Check Restoration Completion Status


chroma status --node node-3

Expected output:


RestoreState: COMPLETED
RecoveredSegments: 100%

2. Validate Vector Integrity


# Sample query to verify embedding dimensions
curl -s http://node-3:8000/vectors/12345 | jq '.embedding | length'

Should return the full dimension count (e.g., 768) for a known document.

3. Scan Logs for Stop‑Sequence Errors


grep -i "StopSequence" /var/log/chroma/*.log

Output should be empty or only contain INFO messages confirming detection.

Prevention – Operational Guardrails

Version Pinning: Enforce identical ChromaDB versions across the cluster using configuration management (e.g., Ansible, Terraform). Document the required stop_sequence token per version.
Configuration Audits: Run a daily audit script that validates the stop_sequence value on all nodes matches the backup manifest token.
Backup Metadata Checks: Extend the backup creation pipeline to embed a checksum of the stop_sequence and reject restores where the checksum mismatches.
Alerting: Create a log‑based alert on the pattern “Stop sequence not found” or “StopSequenceError”.
Test Restores in Staging: Automate a weekly DR test that restores the latest backup to a sandbox cluster, verifying stop‑sequence handling.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the stop sequence work in a fresh cluster but fail after a node crash?
A node crash often leaves the in‑memory configuration stale. If the crashed node ran a different version, its persisted stop_sequence can diverge from the surviving nodes, causing the coordinator to merge mismatched values.
Can I disable the stop‑sequence check to speed up restores?
Disabling it removes the safety net that prevents partial overwrites. It is not recommended for production; instead, optimise I/O or use incremental restores.
How do I know which token (STOP vs STOP_V2) my backup uses?
Inspect the backup manifest’s stop_sequence field (see the “Verify Backup Metadata” step). The value directly indicates the expected token.
What if my backup was created with a custom stop sequence?
Supply the custom token via the --stop-sequence flag during restore. Ensure the same token is set in chroma.yaml on all nodes before initiating the restore.
Is there a way to automatically reconcile token mismatches during recovery?
ChromaDB 0.5.1 introduced a compatibility shim that logs a warning and aborts if the token differs. Upgrading to that version and enabling the strict_stop_sequence flag enforces strict validation.