curl -s http://weaviate-replica-1:8080/v1/.well-known/ready # Expected output {"status":"healthy"} Replication lag metric # Prometheus UI weaviate_replication_lag_seconds{node="replica-1"} 0.004 weaviate_replication_lag_seconds{node="replica-2"} 0.006 Write consistency test # Insert a test object with consistency level QUORUM curl -X POST http://weaviate-primary:8080/v1/objects \ -H "Content-Type: application/json" \ -d '{"class":"Test","properties":{"name":"sync-check"}}' \ -H "X-Consistency-Level: QUORUM" # Query from a replica curl -s http://weaviate-replica-1:8080/v1/objects/Test/sync-check | jq '.properties.name' # Expected: "sync-check" Prevention & Best Practices Enforce configuration drift detection – use a GitOps tool (ArgoCD, Flux) to validate that WEAVIATE_REPLICATION_FACTOR is identical in all pod specs before a rollout. Schema version gating – integrate the --wait-for-replication flag into CI/CD pipelines so a new schema cannot be applied until all replicas acknowledge the change. Canary traffic shifting strategy – route a small percentage of reads/writes to the canary and monitor weaviate_replication_lag_seconds for at least 5 minutes before increasing traffic (see Canary Deployments documentation). Network reliability checks – configure service‑mesh health probes (e.g., Istio) to abort a rollout if TCP health checks between primary and replicas fail. Alerting thresholds – set replication_lag_seconds > 0.01 as a warning and > 0.03 as a critical alert, and include the replica’s node_state in the alert payload. Related Topic Hub: Vector Databases Troubleshooting Hub

Weaviate Replica Node Sync Lag After Canary Deployment

Q: Problem Description

A production Weaviate cluster exhibited a growing replica synchronization lag after a canary deployment that introduced a new Docker image and an updated vector index schema. The primary node continued to accept writes, but replica nodes reported the following errors in their logs:

Q: 1. Verify replication factor consistency

# From any pod kubectl exec -it weaviate-primary-0 -- printenv WEAVIATE_REPLICATION_FACTOR kubectl exec -it weaviate-replica-1 -- printenv WEAVIATE_REPLICATION_FACTOR Expected output (both): 3. Observed output: primary 3, replica 2.

Q: 2. Inspect schema version across nodes

curl -s http://weaviate-primary:8080/v1/schema | jq '.meta.version' curl -s http://weaviate-replica-1:8080/v1/schema | jq '.meta.version' If the versions differ, the primary is ahead of the replica.

Q: 3. Examine replication lag metrics

# Prometheus query weaviate_replication_lag_seconds{node="replica-1"} Typical alerting rule (from official docs):

Q: 4. Capture network health between primary and replicas

kubectl exec -it weaviate-primary-0 -- ss -tnp | grep 8080 kubectl exec -it weaviate-replica-1 -- ss -tnp | grep 8080 # Optional packet capture kubectl exec -it weaviate-primary-0 -- tcpdump -i eth0 port 8080 -c 20 -w /tmp/primary.pcap Look for retransmissions, TCP RSTs, or prolonged SYN‑ACK delays.

Q: 5. Review replica logs for mutation errors

kubectl logs weaviate-replica-1 -c weaviate | grep -E "replica_sync|index_update_failed|context deadline" 6. Check the health endpoint for node state curl -s http://weaviate-replica-1:8080/v1/.well-known/ready # Expected: {"status":"healthy"} # Observed: {"status":"degraded","reason":"replication lag > 10s"} Resolution Three corrective actions were applied in the order below. Each step is illustrated with before/after configuration snippets.

Q: 2. Propagate the new schema to all nodes

Use the weaviate-schema CLI to push the schema with --wait-for-replication so the primary blocks until quorum is satisfied.

Problem Description

A production Weaviate cluster exhibited a growing replica synchronization lag after a canary deployment that introduced a new Docker image and an updated vector index schema. The primary node continued to accept writes, but replica nodes reported the following errors in their logs:


2024-07-31T14:22:13Z replica_sync_timeout: replica did not acknowledge write within configured timeout.
2024-07-31T14:22:15Z write_consistency_error: quorum not reached
2024-07-31T14:22:18Z index_update_failed: replica lag exceeds threshold (lag=32µs)
2024-07-31T14:22:20Z node_state=DEGRADED – replication lag > 10s
2024-07-31T14:22:22Z failed to apply mutation on replica: context deadline exceeded

The symptom manifested as:

Stale search results on read‑only services that query replica nodes.
Occasional 504 Gateway Timeout responses from API gateways routing to replicas.
Monitoring alerts from the Weaviate replication lag metric crossing the replication_lag_seconds > 30µs threshold.

Root Cause Analysis

The canary rollout introduced three interacting problems that together broke the replication pipeline:

Mismatched replication factor – the canary image set WEAVIATE_REPLICATION_FACTOR=3 while the existing replica pods still used 2 (see Cluster Configuration and Node Roles). This caused the primary to write to a third shard that no replica was configured to store, triggering write_consistency_error: quorum not reached (GitHub Issue #2124).
Schema drift on the primary only – the new schema version (vector index configuration) was applied to the primary before the canary traffic shift. Replicas kept the old schema, so incoming mutations could not be deserialized on them, resulting in index_update_failed errors (Community Forum thread “Replication lag spikes during traffic shift”).
Transient network partition – during the traffic shift the service mesh temporarily dropped TCP keep‑alives between the primary and one replica, causing the replica’s write‑ahead log to fill. The primary continued to accept writes, but the replica could not pull the pending entries, leading to the observed replica_sync_timeout and a lag that only cleared after a manual restart (real incident “network partition … resulted in delayed index updates”).

Collectively these failures broke the Weaviate replication and consistency model, which assumes:

All nodes share the same replication_factor configuration.
Schema versions are identical cluster‑wide before writes are accepted.
Network connectivity remains stable for the duration of a write‑propagation window.

Investigation & Debugging Steps

The following checklist reproduces the diagnostic path that led to the root cause:

1. Verify replication factor consistency


# From any pod
kubectl exec -it weaviate-primary-0 -- printenv WEAVIATE_REPLICATION_FACTOR
kubectl exec -it weaviate-replica-1 -- printenv WEAVIATE_REPLICATION_FACTOR

Expected output (both): 3. Observed output: primary 3, replica 2.

2. Inspect schema version across nodes


curl -s http://weaviate-primary:8080/v1/schema | jq '.meta.version'
curl -s http://weaviate-replica-1:8080/v1/schema | jq '.meta.version'

If the versions differ, the primary is ahead of the replica.

3. Examine replication lag metrics


# Prometheus query
weaviate_replication_lag_seconds{node="replica-1"}

Typical alerting rule (from official docs):


ALERT ReplicationLagHigh
  IF weaviate_replication_lag_seconds > 0.03
  FOR 2m

4. Capture network health between primary and replicas


kubectl exec -it weaviate-primary-0 -- ss -tnp | grep 8080
kubectl exec -it weaviate-replica-1 -- ss -tnp | grep 8080
# Optional packet capture
kubectl exec -it weaviate-primary-0 -- tcpdump -i eth0 port 8080 -c 20 -w /tmp/primary.pcap

Look for retransmissions, TCP RSTs, or prolonged SYN‑ACK delays.

5. Review replica logs for mutation errors


kubectl logs weaviate-replica-1 -c weaviate | grep -E "replica_sync|index_update_failed|context deadline"

6. Check the health endpoint for node state


curl -s http://weaviate-replica-1:8080/v1/.well-known/ready
# Expected: {"status":"healthy"}
# Observed: {"status":"degraded","reason":"replication lag > 10s"}

Resolution

Three corrective actions were applied in the order below. Each step is illustrated with before/after configuration snippets.

1. Align replication factor across all pods

Before (canary pod):


# deployment-canary.yaml
env:
  - name: WEAVIATE_REPLICATION_FACTOR
    value: "3"

After (replica pods):


# deployment-replica.yaml (updated)
env:
  - name: WEAVIATE_REPLICATION_FACTOR
    value: "3"

Apply the change and rollout the replica deployment:


kubectl rollout restart deployment/weaviate-replica

2. Propagate the new schema to all nodes

Use the weaviate-schema CLI to push the schema with --wait-for-replication so the primary blocks until quorum is satisfied.


weaviate-schema apply ./schema.json --wait-for-replication

Alternatively, trigger a cluster‑wide re‑index:


curl -X POST http://weaviate-primary:8080/v1/cluster/reindex

3. Mitigate the transient network partition

Adjust the write‑propagation timeout and enable exponential back‑off in the primary’s configuration:


# weaviate.yaml (primary)
replication:
  write_timeout: "15s"
  backoff_initial: "500ms"
  backoff_max: "5s"

Restart the primary pod to pick up the new settings.

4. Force a catch‑up on lagging replicas (optional)

If a replica remains behind after the above steps, trigger a manual sync:


curl -X POST http://weaviate-replica-1:8080/v1/cluster/sync

Verification

Confirm that the cluster is healthy and replication lag is within acceptable bounds:

Health endpoint


curl -s http://weaviate-replica-1:8080/v1/.well-known/ready
# Expected output
{"status":"healthy"}

Replication lag metric


# Prometheus UI
weaviate_replication_lag_seconds{node="replica-1"}   0.004
weaviate_replication_lag_seconds{node="replica-2"}   0.006

Write consistency test


# Insert a test object with consistency level QUORUM
curl -X POST http://weaviate-primary:8080/v1/objects \
  -H "Content-Type: application/json" \
  -d '{"class":"Test","properties":{"name":"sync-check"}}' \
  -H "X-Consistency-Level: QUORUM"

# Query from a replica
curl -s http://weaviate-replica-1:8080/v1/objects/Test/sync-check | jq '.properties.name'
# Expected: "sync-check"

Prevention & Best Practices

Enforce configuration drift detection – use a GitOps tool (ArgoCD, Flux) to validate that WEAVIATE_REPLICATION_FACTOR is identical in all pod specs before a rollout.
Schema version gating – integrate the --wait-for-replication flag into CI/CD pipelines so a new schema cannot be applied until all replicas acknowledge the change.
Canary traffic shifting strategy – route a small percentage of reads/writes to the canary and monitor weaviate_replication_lag_seconds for at least 5 minutes before increasing traffic (see Canary Deployments documentation).
Network reliability checks – configure service‑mesh health probes (e.g., Istio) to abort a rollout if TCP health checks between primary and replicas fail.
Alerting thresholds – set replication_lag_seconds > 0.01 as a warning and > 0.03 as a critical alert, and include the replica’s node_state in the alert payload.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the primary accept writes while replicas report “replica_sync_timeout”?
The primary uses its local write‑ahead log and does not block on replication unless the configured write_consistency level requires quorum. A mismatched replication_factor or network issue prevents the replica from acknowledging the write, leading to the timeout.
Can I safely increase write_timeout to hide the symptom?
Increasing the timeout only masks the underlying problem. It may cause higher latency for client requests and does not resolve schema drift or configuration mismatch, which will eventually cause data inconsistency.
How do I know if a schema change has been applied to all nodes?
Query /v1/schema on each node and compare the .meta.version field. A cluster‑wide equality guarantees that subsequent writes can be deserialized on every replica.
Is there a way to automate replica catch‑up after a rollout?
Yes. Configure the weaviate_cluster_sync_interval (default 30s) and ensure the primary’s write_timeout is generous enough for the slowest replica. The built‑in sync loop will pull pending entries once network health is restored.
What monitoring dashboards should I add for future canary deployments?
Add panels for:
- weaviate_replication_lag_seconds per node.
- Counts of replica_sync_timeout and write_consistency_error log entries.
- Service‑mesh TCP health‑check success rates between primary and replicas.
These give early visibility before user‑facing latency spikes appear.