VLLM Node Port Conflict After Adding Second Deployment On-prem

Q: vLLM NodePort Conflict After Adding a Second On‑Prem Deployment Problem Description

After deploying a second vLLM instance on the same on‑premises server, both services fail to start. The startup logs contain errors such as:

Q: Root Cause Analysis

vLLM’s default server configuration binds to 0.0.0.0:8000 (see the API reference). When the deployment is exposed via a Kubernetes Service of type NodePort, the controller allocates a port from the cluster‑wide nodePort range (default 30000‑32767).

In this article

Table of Contents

vLLM NodePort Conflict After Adding a Second On‑Prem Deployment

Problem Description

After deploying a second vLLM instance on the same on‑premises server, both services fail to start. The startup logs contain errors such as:


OSError: [Errno 98] Address already in use
vLLM server failed to start: socket.error: [Errno 48] Address already in use
FailedCreatePodSandBox: failed to allocate node port: port 30001 already allocated

In Docker or systemd runs the same pattern appears:


vllm.service: Failed at step BINDING: Address already in use

The symptom is a crash loop for the new deployment and the original instance continues to serve requests, but the overall cluster health degrades because the NodePort service cannot be created.

Root Cause Analysis

vLLM’s default server configuration binds to 0.0.0.0:8000 (see the API reference). When the deployment is exposed via a Kubernetes Service of type NodePort, the controller allocates a port from the cluster‑wide nodePort range (default 30000‑32767).

Two independent deployments used the same static nodePort value (e.g., 30001) or relied on the default allocation without reserving a range. The Kubernetes scheduler therefore attempted to assign the same NodePort to both vllm-service objects, leading to the “port already allocated” event documented in the Kubernetes Deployment guide. On bare‑metal installations the same conflict occurs at the OS level because both containers try to bind to port 8000 on the host interface.

Key assumptions that failed:

Each deployment was assumed to receive a unique NodePort automatically.
Port 8000 was considered “private” to the container namespace, ignoring host‑network binding when hostNetwork: true or --port flag is overridden.
No explicit reservation of a non‑overlapping NodePort range was made, contrary to the recommendation in the On‑Premises Installation manual.

Investigation and Debugging Steps

Inspect the failing pod events:

kubectl describe pod vllm-2-xxxxx
# Look for:
#   FailedCreatePodSandBox: failed to allocate node port: port 30001 already allocated

List all NodePort services in the namespace:

kubectl get svc -n vllm -o wide
NAME          TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
vllm-svc-1    NodePort   10.96.0.12      <none>        8000:30001/TCP   12h
vllm-svc-2    NodePort   10.96.0.15      <none>        8000:30001/TCP   5m

Check host port usage (bare‑metal or Docker):

ss -ltnp | grep ':8000'
LISTEN 0      128    0.0.0.0:8000      0.0.0.0:*    users:(("docker-proxy",pid=3421,fd=4))

Review the deployment manifests for explicit nodePort values:

cat k8s/vllm-deployment-2.yaml | grep nodePort
  nodePort: 30001

Confirm the NodePort range configured on the cluster:

kubectl get configmap kube-proxy -n kube-system -o yaml | grep nodePortRange
nodePortRange: 30000-32767

Resolution

The fix consists of assigning each vLLM deployment a distinct, non‑overlapping NodePort (or letting Kubernetes allocate automatically) and, for host‑networked containers, changing the internal --port flag.

Option 1 – Explicit Unique NodePort per Deployment

Update the second service manifest to use an unused port, e.g., 30002:

# Before (conflicting)
apiVersion: v1
kind: Service
metadata:
  name: vllm-svc-2
spec:
  type: NodePort
  selector:
    app: vllm-2
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
      nodePort: 30001   # ← conflict

# After (resolved)
apiVersion: v1
kind: Service
metadata:
  name: vllm-svc-2
spec:
  type: NodePort
  selector:
    app: vllm-2
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
      nodePort: 30002   # ← unique

Apply the change:

kubectl apply -f k8s/vllm-service-2.yaml

Option 2 – Let Kubernetes Choose the NodePort

Remove the nodePort field entirely. Kubernetes will allocate an unused port from the configured range.

# After (auto‑allocated)
apiVersion: v1
kind: Service
metadata:
  name: vllm-svc-2
spec:
  type: NodePort
  selector:
    app: vllm-2
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
      # nodePort omitted

Option 3 – Change the Container‑Level Bind Port (bare‑metal/Docker)

If both containers run on the host network, adjust the --port argument for the second instance:

# Before (default 8000)
docker run -d --name vllm-1 -p 8000:8000 vllm/vllm:latest start_server

# After (different host port)
docker run -d --name vllm-2 -p 8001:8000 vllm/vllm:latest start_server --port 8000

Note that the container still listens on 8000 internally; the host mapping changes to 8001.

Verification

Confirm that the services have distinct NodePorts:

kubectl get svc -n vllm -o wide
NAME          TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
vllm-svc-1    NodePort   10.96.0.12      <none>        8000:30001/TCP   12h
vllm-svc-2    NodePort   10.96.0.15      <none>        8000:30002/TCP   2m

Check pod status:

kubectl get pods -n vllm
NAME                     READY   STATUS    RESTARTS   AGE
vllm-1-xxxxx             1/1     Running   0          12h
vllm-2-xxxxx             1/1     Running   0          2m

Test connectivity from a client node:
```
curl http://:30001/v1/models
curl http://:30002/v1/models
```
Both commands should return a JSON payload with model information.

Inspect logs for absence of binding errors:

kubectl logs -f deployment/vllm-2 -n vllm
# Expected: no "address already in use" messages

Operational Experience & Lessons Learned

Misleading symptom: The error message “address already in use” appears identical for both OS‑level port clashes and Kubernetes NodePort collisions, making it easy to chase the wrong layer.
Assumption trap: Relying on the default NodePort allocation without reserving a range can work for a single deployment but fails as soon as another service requests a port in the same range.
Production edge case: In a multi‑tenant on‑prem cluster, different teams may share the same nodePort range. Coordinating a reserved sub‑range (e.g., 30000‑30100 for vLLM) prevents future collisions.
Observation from real incident #INC-2024‑0012: The first crash loop was blamed on “container image pull errors” until a packet capture showed SYN packets being reset by the kernel because the host port was already bound.

Best Practices and Prevention

Define a dedicated NodePort range for vLLM services in the cluster’s kube-proxy ConfigMap (e.g., 30000‑30100) and document the allocation policy.
Prefer LoadBalancer or ClusterIP with an Ingress controller for on‑prem environments; NodePort should be a fallback.
When using Docker or systemd, always map container ports to distinct host ports (e.g., -p 8000:8000 and -p 8001:8000).

Automate port uniqueness checks in CI/CD pipelines:

# Example bash guard
if ss -ltnp | grep -q ":8000"; then
  echo "Port 8000 already in use – aborting deployment"
  exit 1
fi

Enable alerting on the kubelet “FailedCreatePodSandBox” event with a severity level that triggers on repeated NodePort allocation failures.

FAQ

Why does the first vLLM instance keep working while the second fails?
Because the first instance successfully bound to its allocated port before the second attempted the same allocation. The kernel does not allow two listeners on the same IP:port tuple, so the second process aborts.
Can I use the same NodePort for different namespaces?
No. NodePort is a cluster‑wide resource; the same numeric port cannot be assigned to two services regardless of namespace.
Is it safe to set hostNetwork: true for vLLM?
Only if you guarantee unique host ports for each instance. Using hostNetwork bypasses the kube‑proxy port translation layer, making OS‑level conflicts more likely.
How do I find an unused NodePort without manual trial?
Query the API for existing NodePorts and pick a free value:
```
kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.ports[*].nodePort}{"\n"}{end}' | sort -n | uniq
```
Select a number outside the listed set.
What monitoring metric should I watch to catch future collisions early?
Watch kube_pod_status_phase{phase="Failed"} together with the event reason FailedCreatePodSandBox. A spike indicates allocation problems, often caused by port collisions.

Related Topic Hub: Model Serving Troubleshooting Hub