Milvus ReplicaSet scaling failure during high concurrency benchmarking

Problem – Milvus ReplicaSet Scaling Failure Under High‑Concurrency Benchmark

During a distributed performance benchmark of Milvus 2.x, the team attempted to increase the replicaCount of the QueryNode (and optionally the Proxy) to meet a target query throughput of ~10 k QPS. The scaling operation consistently failed:

• New QueryNode pods remained in Pending or entered CrashLoopBackOff.
• Existing pods began emitting gRPC errors such as:

  rpc error: code = Unavailable desc = connection refused
  etcdserver: request timed out
  Failed to sync replica set: context deadline exceeded
  panic: runtime error: invalid memory address or nil pointer dereference

• Overall benchmark throughput dropped sharply, and latency spikes appeared.

This article walks through the root cause analysis, systematic debugging steps, a concrete resolution, validation procedures, and preventive measures.

Root Cause – Interaction of Resource Limits, etcd quorum, and Proxy gRPC Saturation

Milvus consists of three core components that are each deployed as a Kubernetes Deployment with a ReplicaSet:

• Proxy – front‑end gRPC gateway that multiplexes client connections.
• QueryNode – executes vector search; each replica loads the full index into memory.
• DataNode – handles data ingestion and persistence.

According to the Milvus 2.x Kubernetes Deployment Guide, scaling a component requires:

• Adjusting replicaCount in the Helm chart.
• Ensuring sufficient resources.limits (CPU, memory) for the workload.
• Maintaining etcd quorum (default 3 members) for metadata coordination.

Three intertwined failures were observed in the benchmark environment (see the “Real incidents” evidence):

1. Insufficient memory limits – Each QueryNode loads a large IVF‑PQ index (~12 GB). The default resources.limits.memory of 8 GiB caused OOM kills when the replica count doubled, leading to Kubelet failed to start container for milvus‑querynode: OOMKilled.
2. etcd leader election stalls – Adding eight QueryNode pods increased the number of simultaneous writes to etcd (metadata updates for node registration). With the default etcd pod CPU request (100 m) and no dedicated etcd node pool, the etcd leader could not keep up, producing etcdserver: request timed out across all services.
3. Proxy gRPC connection pool exhaustion – The Proxy’s grpc.maxConcurrentStreams default (100) and its internal connection pool were sized for a modest number of back‑ends. When new QueryNode pods appeared, the Proxy attempted to open new streams for each, saturating the pool and returning rpc error: code = Unavailable desc = connection refused for inbound client requests.

These conditions collectively prevented the ReplicaSet controller from reaching the desired replica count, manifesting as the “Failed to sync replica set: context deadline exceeded” error.

Debug – Systematic Investigation Steps

1. Verify Helm values and pod status

helm get values milvus -n milvus | grep replicaCount
kubectl get pods -n milvus -l app=milvus-querynode
kubectl describe pod <new‑querynode‑pod> -n milvus

Typical output showing pending state:

Name:         milvus-querynode-6f7c9b8d5c-abcde
Namespace:    milvus
Status:       Pending
Reason:       Unschedulable
Message:      0/10 nodes are available: 10 Insufficient memory.

2. Inspect OOM events and container logs

kubectl logs milvus-querynode-6f7c9b8d5c-abcde -n milvus --previous
kubectl get events -n milvus --field-selector reason=OOMKilled

Sample log snippet:

2024-03-12T10:15:32Z FATAL: out of memory: kill process 12345 (milvus) score 987 or sacrifice child

3. Check etcd health

kubectl exec -it milvus-etcd-0 -n milvus -- etcdctl endpoint health
kubectl exec -it milvus-etcd-0 -n milvus -- etcdctl endpoint status --write-out=table

Output indicating timeouts:

https://milvus-etcd-0.milvus.svc:2379 is unhealthy: request timed out

4. Examine Proxy gRPC metrics

Milvus exposes Prometheus metrics at /metrics. Query the relevant series:

curl -s http://milvus-proxy:9091/metrics | grep grpc_server_handled_total
curl -s http://milvus-proxy:9091/metrics | grep grpc_server_streams_active

Metrics showed a spike in active streams reaching the configured limit:

# HELP grpc_server_streams_active Number of active gRPC streams.
grpc_server_streams_active{service="proxy"} 102

5. Review Helm chart defaults (official docs)

The Milvus 2.x Kubernetes Deployment Guide lists the default values:

Component	Default replicaCount	Default memory limit
QueryNode	3	8Gi
Proxy	2	4Gi
etcd	3	2Gi

Solution – Adjust Resources, Etcd Sizing, and Proxy gRPC Settings

1. Increase QueryNode memory limits to accommodate the full index

# values.yaml (before)
queryNode:
  resources:
    limits:
      memory: 8Gi
    requests:
      memory: 6Gi

# values.yaml (after)
queryNode:
  resources:
    limits:
      memory: 16Gi   # enough for 12 Gi index + overhead
    requests:
      memory: 12Gi

2. Expand etcd CPU and enable a dedicated node pool

# values.yaml (etcd section before)
etcd:
  resources:
    limits:
      cpu: "0.5"
    requests:
      cpu: "0.2"

# after – allocate more CPU and pin to etcd‑node pool
etcd:
  nodeSelector:
    nodepool: etcd
  resources:
    limits:
      cpu: "2"
    requests:
      cpu: "1"

Deploy a separate node pool with SSD‑backed storage to avoid I/O contention.

3. Raise Proxy gRPC concurrency limits

# values.yaml (proxy section before)
proxy:
  grpc:
    maxConcurrentStreams: 100

# after – increase to match expected back‑end count
proxy:
  grpc:
    maxConcurrentStreams: 500
  resources:
    limits:
      memory: 8Gi
    requests:
      memory: 6Gi

4. Enforce pod anti‑affinity for QueryNode replicas

# values.yaml (queryNode anti‑affinity)
queryNode:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values: ["milvus-querynode"]
          topologyKey: "kubernetes.io/hostname"

This prevents the scheduler from placing multiple QueryNode pods on the same physical host, avoiding network saturation and “context deadline exceeded” errors.

5. Apply the updated Helm chart

helm upgrade milvus milvus/milvus -n milvus -f values.yaml

Watch the rollout:

kubectl rollout status deployment/milvus-querynode -n milvus
kubectl get pods -n milvus -l app=milvus-querynode

Verify – Confirm That Scaling Now Succeeds

• Replica count: kubectl get rs -n milvus -l app=milvus-querynode should show the desired DESIRED replicas matching AVAILABLE.
• Memory usage: kubectl top pod -n milvus -l app=milvus-querynode should report < 80 % of the 16 Gi limit.
• etcd health: Re‑run etcdctl endpoint health – all members return “healthy”.
• Proxy gRPC streams: Verify grpc_server_streams_active stays well below the new limit (e.g., 200 vs 500).
• Benchmark throughput: Rerun the query load (10 k QPS) and observe latency < 5 ms and error rate < 0.1 %.

Prevent – Operational Guardrails for Future Scaling

• Capacity planning: Use the Milvus Performance Tuning Documentation to size CPU/memory per index size. Allocate at least 1.5× the index memory per QueryNode.
• Etcd sizing checklist: Ensure etcd CPU requests ≥ 1 core per member and enable dedicated node pools for high‑throughput workloads.
• Proxy tuning: Set grpc.maxConcurrentStreams to replicaCount × 50 as a rule of thumb, and monitor the grpc_server_streams_active metric.
• Pod anti‑affinity policies: Enforce host‑level anti‑affinity for all Milvus back‑end components to avoid single‑point network bottlenecks.
• Automated health checks: Add liveness/readiness probes that verify gRPC health (e.g., grpc_health_probe) and etcd health endpoints; configure alerts on OOMKilled events and etcd request timeouts.

FAQ – Common Follow‑Up Questions

1. Why does scaling work in dev but fail in the benchmark cluster?
   Because the dev cluster runs with a small index (≈2 Gi) and ample resources, while the benchmark loads a 12 Gi index on each QueryNode. The default memory limits and etcd CPU are insufficient for the larger workload.
2. Can I keep the default grpc.maxConcurrentStreams and still scale?
   Only if you also increase the Proxy replica count proportionally. The limit is per Proxy instance, so adding more Proxy pods spreads the stream load.
3. Do I need to restart the entire Milvus deployment after changing etcd resources?
   No. Updating the etcd Deployment with a rolling restart (via helm upgrade) is sufficient; the cluster will re‑elect a leader without full downtime.
4. How do I know the appropriate memory limit for a QueryNode?
   Measure the size of the loaded index (e.g., via du -sh /var/lib/milvus/indexes) and add 30‑40 % headroom for working set, cache, and Go runtime overhead.
5. What alert thresholds should I set for etcd timeouts?
   Trigger an alert when etcd_server_leader_changes_seen_total increments rapidly or when etcd_server_failed_requests_total exceeds 5 per minute.

Related Topic Hub: Vector Databases Troubleshooting Hub