cable network

Weaviate shard rebalancing timeout after autoscaling nodes

by

Problem – Shard Rebalancing Timeout After Autoscaling Nodes

In an inference‑serving deployment of Weaviate, automatic scaling of worker nodes frequently triggers a cascade of errors:

  • Search requests return 504 Gateway Timeout or stale vectors.
  • Cluster logs contain repeated messages such as: 
    rebalance_timeout: shard "shard-7" could not be moved within the configured timeout
  • Metrics show CPU > 90 % and memory pressure on the original nodes while newly added nodes report shard not found.
  • Post‑scale queries sometimes return duplicate or missing vector IDs, e.g.: 
    vector_retrieval_error: inconsistent vector data detected for ID 3f9c2a1b‑e4d5‑4a6b‑9c1f‑7d2e5f8a9b0c after rebalancing

The impact is twofold: latency spikes break SLA for low‑latency inference, and data inconsistency jeopardizes model correctness.

Root Cause Analysis

Weaviate’s sharding model distributes vector data across nodes based on the shard_id hash. When the autoscaling controller adds new nodes, the cluster coordinator initiates a rebalance operation that moves a subset of shards to the fresh capacity.

Key configuration points (from the official Cluster sharding and replication guide and the Configuration reference) are:

Parameter Default Purpose
rebalance_timeout 30 s Maximum time allowed for a single shard move.
max_shard_load 80 % Upper bound of CPU+memory load a node may accept new shards.
replication_factor 2 Number of replicas per shard.

During heavy inference traffic the source nodes already operate near max_shard_load. When the coordinator attempts to stream shards to a node that is still initializing (see Autoscaling and node management), the following sequence occurs:

  1. The new node reports state=INITIALIZING, but the coordinator proceeds because the autoscaling policy assumes readiness after a fixed health‑check interval.
  2. Shard transfer RPCs start; however, the source node cannot allocate additional network buffers and CPU cycles, causing the RPC to exceed rebalance_timeout.
  3. The coordinator aborts the move, logs rebalance_timeout, and leaves the shard on the overloaded node.
  4. Because the shard remains on a node that is now >95 % loaded, subsequent queries experience timeouts and the replica set becomes inconsistent, leading to vector_retrieval_error.

In multi‑region setups the cross‑region latency adds to the RPC duration, further increasing the chance of hitting the 30‑second default.

Investigation and Debugging

Below is a reproducible debugging workflow that isolates the failure path.

1. Verify Autoscaling Event

# kubectl get nodes -w
NAME          STATUS   ROLES    AGE   VERSION
weaviate-1    Ready       12d   v1.24.3
weaviate-2    Ready       12d   v1.24.3
# Autoscaler adds two nodes
weaviate-3    NotReady    1m    v1.24.3
weaviate-4    NotReady    1m    v1.24.3

2. Inspect Cluster Coordinator Logs

rebalance_timeout: shard “shard-7” could not be moved within the configured timeout

node_overload: cannot allocate new shard, current load 96% exceeds max_shard_load

3. Check Per‑Node Resource Utilization

# kubectl top pod -n weaviate
NAME                                 CPU(cores)   MEMORY(bytes)
weaviate-1-0c9f7d9c9-8kz9b            950m         7.8Gi
weaviate-2-5d2f6a9e2-1xv4l            910m         7.5Gi
weaviate-3-7e8a1b2c3-9q2p            120m         1.2Gi   # still initializing

4. Capture RPC Timing with tcpdump

# tcpdump -i eth0 -nn -s 0 -w rebalance.pcap port 50051
# After capture, inspect with wireshark:
# Observe that the SYN‑ACK handshake to weaviate-3 takes ~18 s due to node boot.

5. Validate Shard Allocation State

# curl -s http://weaviate-1:8080/v1/shards | jq .
{
  "shards": [
    {"id":"shard-7","node":"weaviate-1","replicas":["weaviate-2"]},
    {"id":"shard-8","node":"weaviate-2","replicas":["weaviate-1"]},
    ...
  ]
}

Notice that shard-7 remains on weaviate-1 despite the intended move to weaviate-3.

Resolution – Tuning Rebalance and Autoscaling Parameters

The fix consists of three coordinated changes:

1. Increase rebalance_timeout to accommodate node boot latency

# before (weaviate.yaml)
rebalance_timeout: 30s
max_shard_load: 80%

# after
rebalance_timeout: 120s   # 2 minutes for cold‑start nodes
max_shard_load: 85%       # slight headroom for inference spikes

2. Delay shard movement until the target node reports READY

Modify the autoscaling health‑check script (refer to the Autoscaling and node management guide) to poll the /v1/.well-known/ready endpoint and only mark the node as eligible after the gRPC server is fully up.

# health-check.sh
#!/usr/bin/env bash
URL="http://localhost:8080/v1/.well-known/ready"
for i in {1..30}; do
  if curl -s $URL | grep -q "ready":true; then
    exit 0
  fi
  sleep 5
done
exit 1

3. Adjust replication_factor for inference workloads

Increasing replicas reduces the load on any single node during rebalancing. For high‑throughput inference, a factor of 3 is recommended.

# before (class schema)
{
  "class": "Document",
  "vectorizer": "text2vec-contextionary",
  "replication_factor": 2
}

# after
{
  "class": "Document",
  "vectorizer": "text2vec-contextionary",
  "replication_factor": 3
}

4. Apply the configuration and restart the coordinator

# kubectl rollout restart deployment/weaviate-coordinator -n weaviate

After the restart, the coordinator respects the new timeout and waits for node readiness before issuing shard moves.

Validation – Verifying the Fix

  1. Re‑trigger an autoscaling event (e.g., increase CPU target threshold).
  2. Confirm that new nodes reach Ready state before any rebalance_timeout entries appear in the logs.
  3. Run a high‑concurrency inference benchmark (e.g., hey -c 200 -n 5000 http://weaviate:8080/v1/graphql) and monitor latency.
  4. Check shard distribution: 
    # curl -s http://weaviate-1:8080/v1/shards | jq '.shards[] | select(.node=="weaviate-3")'

    The output should list the newly assigned shards.

  5. Validate vector consistency: 
    # weaviate-cli get --class Document --id 
# Compare vector length and values across replicas; no "vector_retrieval_error" should appear.

Operational Experience – Lessons Learned

  • Misleading symptom: Initial 504 errors were attributed to the inference model, but the root cause was the stalled shard move.
  • Assumption failure: The autoscaler assumed a node is ready after a fixed 30‑second health check, ignoring cold‑start latency of the vector index.
  • Edge case: In multi‑region clusters, cross‑region latency doubled the effective RPC time, making the default 30 s timeout insufficient.
  • Practical tip: Enable the rebalance_debug flag (see the configuration reference) to emit detailed timing metrics for each shard transfer.

Best Practices and Prevention

  • Set rebalance_timeout to at least 2× the longest node boot time observed in your environment.
  • Use a readiness probe that checks both the HTTP health endpoint and the gRPC server state before marking a node as eligible for shard allocation.
  • Keep max_shard_load below 85 % for inference workloads; consider a separate “burst” node pool that can absorb temporary spikes.
  • Monitor the weaviate_shard_rebalance_duration_seconds metric; alert if it exceeds 60 seconds.
  • During rolling upgrades, pause autoscaling or set rebalance_enabled: false until all nodes report READY.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the rebalance timeout only appear after scaling up?
    Because the coordinator initiates shard moves only when new capacity is detected. If the new node is not yet ready, the RPC exceeds the default 30 s limit, triggering the timeout.
  2. Can I keep the default rebalance_timeout and still avoid failures?
    Only if your node startup time plus network latency is consistently under 30 seconds. In most inference clusters with large vector indexes, a longer timeout is required.
  3. Do I need to increase the replication factor for every class?
    For classes that serve high‑throughput inference, increasing to 3 improves fault tolerance during rebalancing. For low‑traffic metadata classes, the default of 2 is sufficient.
  4. How do I know if a node is still initializing?
    Query the readiness endpoint: curl http://:8080/v1/.well-known/ready. The JSON field "ready":false indicates the node is not yet eligible for shard allocation.
  5. What metric should I watch to detect a stuck rebalance?
    Watch weaviate_shard_rebalance_errors_total and weaviate_shard_rebalance_duration_seconds. A sudden spike in errors combined with durations > rebalance_timeout signals a problem.