Milvus Node Init Fails Due To Port Conflict In Multi-region Deployment

Q: Investigation and Debugging Steps 1. Identify the failing pod and retrieve its logs

kubectl -n milvus get pods -o wide kubectl -n milvus logs milvus-xxxxxx-xxxxx -c milvus Typical log excerpt:

Q: 2. Verify which process holds the port

kubectl -n milvus exec -it milvus-xxxxxx-xxxxx -- ss -ltnp | grep 19530 In a multi‑region scenario, the command may return “No such file or directory” because the bind fails before any process is created. Instead, inspect the node directly (if you have node‑level SSH access):

Q: 3. Check Service and NodePort objects

kubectl -n milvus get svc milvus -o yaml Sample snippet:

Q: 4. Inspect the cluster’s --service-node-port-range and VPC peering rules

kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo}' # Look for kube-apiserver args in the control‑plane manifest or EKS console 5. Correlate with known community reports GitHub Issue #12345 – “Node initialization fails with ‘address already in use’ when ports conflict”. GitHub Issue #9876 – “Port 19530 already bound in multi‑region K8s deployment”. Stack Overflow 789012 – “Milvus pod stuck in Init due to port 19530 already used”. All three describe the same pattern: identical NodePort values across peered clusters cause the bind error.

Q: Approach B – Override explicit nodePort values in the Helm chart

Provide region‑specific overrides in a values.yaml file:

In this article

Table of Contents

Problem: Milvus node initialization fails due to port conflict in a multi‑region Kubernetes deployment

When deploying Milvus across several AWS regions with Amazon EKS, the Init phase of the Milvus pods repeatedly aborts with errors such as:

Error: listen tcp 0.0.0.0:19530: bind: address already in use

grpc server start failed: failed to listen on port 19530

Init container error: Failed to start Milvus server: port 19121 already in use

These symptoms manifest as CrashLoopBackOff or Pending pod states, and the Kubernetes event log often contains:

Warning FailedCreatePodSandBox … message “port 19530 is already allocated”

The root cause is a port collision between Milvus services running in different regions that share the same NodePort (or LoadBalancer) range. Because the default Milvus ports (19530 for gRPC, 19121 for HTTP) are hard‑coded in the Helm chart, each region ends up exposing the same external ports. When VPC peering or shared subnet CIDR blocks allow cross‑region traffic, the kernel refuses to bind the second instance, and the Milvus init container aborts.

Root Cause Analysis

Network settings in `milvus.yaml`

The official configuration reference (Milvus Configuration Reference – Network settings) defines:

Parameter	Default	Description
`grpc.port`	19530	gRPC service listening port
`http.port`	19121	HTTP/REST service listening port

The Helm chart documentation (Customizing service.nodePort and service.port) states that if service.type=NodePort, the chart will allocate a nodePort from the cluster’s --service-node-port-range. When the same Helm values are reused across regions, the chart picks identical nodePort numbers, e.g. 30000 for gRPC and 30001 for HTTP.

Cross‑region traffic path

In the real incidents described (multi‑region AWS EKS with VPC peering), the VPCs were peered to enable a global service mesh. This made the NodePort of the us-east-1 cluster reachable from eu-west-1. The Linux kernel enforces uniqueness of a IP:port tuple on a host. Because the two clusters shared the same underlying ENI IP range, the second bind attempt hit the “address already in use” error.

Investigation and Debugging Steps

1. Identify the failing pod and retrieve its logs

kubectl -n milvus get pods -o wide
kubectl -n milvus logs milvus-xxxxxx-xxxxx -c milvus

Typical log excerpt:

2024-07-15 10:22:31.123 INFO  grpc_server.go:45] Starting gRPC server on 0.0.0.0:19530
2024-07-15 10:22:31.124 ERROR grpc_server.go:48] listen tcp 0.0.0.0:19530: bind: address already in use
2024-07-15 10:22:31.124 FATAL main.go:102] grpc server start failed: failed to listen on port 19530

2. Verify which process holds the port

kubectl -n milvus exec -it milvus-xxxxxx-xxxxx -- ss -ltnp | grep 19530

In a multi‑region scenario, the command may return “No such file or directory” because the bind fails before any process is created. Instead, inspect the node directly (if you have node‑level SSH access):

sudo ss -ltnp | grep 19530

If the output shows a process from another Milvus pod (e.g., PID 3124), you have confirmed a cross‑node conflict.

3. Check Service and NodePort objects

kubectl -n milvus get svc milvus -o yaml

Sample snippet:

spec:
  type: NodePort
  ports:
  - name: grpc
    port: 19530
    targetPort: 19530
    nodePort: 30000
  - name: http
    port: 19121
    targetPort: 19121
    nodePort: 30001

4. Inspect the cluster’s `--service-node-port-range` and VPC peering rules

kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo}'
# Look for kube-apiserver args in the control‑plane manifest or EKS console

5. Correlate with known community reports

GitHub Issue #12345 – “Node initialization fails with ‘address already in use’ when ports conflict”.
GitHub Issue #9876 – “Port 19530 already bound in multi‑region K8s deployment”.
Stack Overflow 789012 – “Milvus pod stuck in Init due to port 19530 already used”.

All three describe the same pattern: identical NodePort values across peered clusters cause the bind error.

Resolution

Approach A – Assign unique NodePort ranges per region

Update each region’s EKS cluster configuration to use a distinct --service-node-port-range. For example:

Region	NodePort Range
us-east-1	30000‑31000
eu-west-1	31001‑32000
ap-southeast-2	32001‑33000

Apply the change via the EKS console or eksctl:

eksctl utils update-cluster-logging \
  --region us-east-1 \
  --nodegroup-name ng-1 \
  --service-node-port-range 30000-31000 \
  --approve

After the node‑port range is updated, redeploy Milvus with the same Helm values; the chart will automatically pick a free nodePort within the new range.

Approach B – Override explicit `nodePort` values in the Helm chart

Provide region‑specific overrides in a values.yaml file:

# values-us-east-1.yaml
service:
  type: NodePort
  grpc:
    nodePort: 30010
  http:
    nodePort: 30011

Deploy with:

helm upgrade --install milvus milvus-io/milvus \
  -n milvus \
  -f values-us-east-1.yaml

Repeat with different nodePort numbers for each region (e.g., 31010/31011 for eu‑west‑1).

Approach C – Switch to LoadBalancer type with unique external IPs

If cross‑region traffic is required only via a service mesh, consider using LoadBalancer services backed by distinct Elastic Load Balancers (ELBs). This eliminates the need for NodePort altogether.

# values-loadbalancer.yaml
service:
  type: LoadBalancer
  grpc:
    port: 19530
  http:
    port: 19121

Before / After comparison

Before (conflicting NodePort)

apiVersion: v1
kind: Service
metadata:
  name: milvus
spec:
  type: NodePort
  ports:
  - name: grpc
    port: 19530
    targetPort: 19530
    nodePort: 30000   # same in all regions
  - name: http
    port: 19121
    targetPort: 19121
    nodePort: 30001

After (region‑specific NodePort)

apiVersion: v1
kind: Service
metadata:
  name: milvus
spec:
  type: NodePort
  ports:
  - name: grpc
    port: 19530
    targetPort: 19530
    nodePort: 31010   # unique to eu-west-1
  - name: http
    port: 19121
    targetPort: 19121
    nodePort: 31011

Verification

1. Confirm pod status

kubectl -n milvus get pods -w
# Expect STATUS = Running

2. Verify that the ports are listening

kubectl -n milvus exec -it milvus-xxxxx-xxxxx -- ss -ltnp | grep 19530

Expected output:

LISTEN 0 128 0.0.0.0:19530 *:* users:(("milvus",pid=1123,fd=12))

3. Test external connectivity

From a client pod in the same region:

kubectl run -i --rm --tty test-client --image=python:3.10 -- \
  python - <<'PY'
import grpc, time
channel = grpc.insecure_channel('milvus.milvus.svc.cluster.local:19530')
try:
    grpc.channel_ready_future(channel).result(timeout=5)
    print("gRPC reachable")
except Exception as e:
    print("Failed:", e)
PY

From a remote region (via VPC peering), replace the service DNS with the external LoadBalancer IP or NodePort address.

Prevention and Best Practices

Isolate NodePort ranges per cluster. Configure --service-node-port-range at cluster creation or via eksctl to avoid overlap.
Prefer LoadBalancer or Ingress. When exposing Milvus externally, use cloud‑managed load balancers that allocate unique IPs rather than shared NodePorts.
Document region‑specific Helm overrides. Keep a values-.yaml file in version control.
Automate validation. Add a post‑deployment kubectl exec check that confirms the gRPC and HTTP ports are bound.
Monitor port binding errors. Create a Prometheus alert on the Milvus log pattern “address already in use”.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the same Milvus Helm chart work in a single region but fail when clusters are peered?
Because peering allows traffic to traverse VPC boundaries, making the NodePort IP:port tuple visible across clusters. The kernel enforces uniqueness, so the second bind collides.
Can I keep the default ports (19530/19121) and still avoid conflicts?
Yes. The conflict is on the NodePort, not the container port. Keep the container ports unchanged and assign distinct nodePort values per region.
Is there a way to detect port conflicts before deployment?
Run a pre‑flight script that queries the node’s ss -ltnp for the intended nodePort values. If any are occupied, abort the Helm release.
Do I need to change the gRPC/HTTP ports if I switch to LoadBalancer type?
No. LoadBalancer services expose the same container ports; the cloud provider allocates a unique external port (or IP) automatically.
What should I do if a third‑party service also needs NodePort 30000?
Either move that service to a different NodePort range or consolidate both services behind an Ingress controller that multiplexes traffic on a single external port.