Google Gemini batch job fails to start due to node port conflict

Problem – Gemini Batch Job Fails to Start with “node port conflict”

In a multi‑node Google Gemini deployment on Kubernetes, batch processing jobs are scheduled via the gemini-batch controller. Operators observed that newly created jobs never transition to Running. The pod logs contain a clear error:

2026-10-18T07:42:13.214Z ERROR node_port_conflict.go:87] Failed to bind node port 30001: address already in use
2026-10-18T07:42:13.215Z INFO  batch_job_controller.go:112] Job "data‑ingest‑20231018‑001" aborted

Because Gemini relies on a static node port for internal RPC traffic (default 30001), the conflict prevents the Gemini daemon from exposing its gRPC endpoint, causing the entire batch job to fail during the startup phase.

Root Cause – Overlapping NodePort Allocation Across Gemini Nodes

Gemini’s Helm chart (or the custom manifest) creates a Service of type NodePort for each Gemini replica:

apiVersion: v1
kind: Service
metadata:
  name: gemini-node-{{ .Values.replicaId }}
spec:
  type: NodePort
  ports:
    - port: 50051
      targetPort: 50051
      nodePort: 30001   # <-- hard‑coded
  selector:
    app: gemini
    replica-id: "{{ .Values.replicaId }}"

When the chart is installed multiple times (e.g., for separate environments or for canary deployments) the same nodePort value is reused. Kubernetes does not enforce uniqueness across different namespaces, so the first Service that claims 30001 wins, and subsequent Services fail to allocate the port, resulting in the “address already in use” error shown in the logs.

Additionally, some clusters have NodePort range narrowed to 30000‑30010, increasing the likelihood of collisions when many Gemini nodes are deployed.

Debug – Step‑by‑Step Investigation

1. Identify the conflicting Service

kubectl get svc -A -o wide | grep 30001

Typical output:

default          gemini-node-0          NodePort    10.96.0.1   10.244.0.5   30001/TCP   5d
analytics        gemini-node-0          NodePort    10.96.0.2   10.244.1.3   30001/TCP   2d

Two different namespaces (default and analytics) are both trying to bind 30001.

2. Verify the pod that emitted the error

kubectl logs -n default $(kubectl get pods -n default -l app=gemini -o name | head -n1)

Look for the “Failed to bind node port” line; note the pod name and the replica ID.

3. Check the NodePort range configuration

kubectl get cm kube-proxy -n kube-system -o yaml | grep nodePortRange

Example output:

nodePortRange: 30000-30010

4. Confirm that the Gemini daemon is indeed listening on the expected port

kubectl exec -n default $(kubectl get pod -n default -l app=gemini -o name | head -n1) -- ss -ltnp | grep 50051

If the daemon never reaches the ss step, the NodePort bind failure is the blocker.

Solution – Allocate Unique NodePorts or Switch to ClusterIP + Ingress

Option A – Dynamic NodePort Allocation (recommended)

Modify the Helm values (or manifest) to let Kubernetes assign a free port automatically by omitting nodePort:

# Before (hard‑coded)
nodePort: 30001

# After (dynamic)
# nodePort omitted – Kubernetes picks an available port

Update the chart:

helm upgrade gemini ./gemini-chart \
  --set replicaCount=3 \
  --set service.type=NodePort \
  --reuse-values

After the upgrade, retrieve the allocated ports:

kubectl get svc -n default -l app=gemini -o jsonpath='{range .items[*]}{.metadata.name}{" -> "}{.spec.ports[0].nodePort}{"\n"}{end}'

Option B – Reserve a NodePort range per environment

If static ports are required (e.g., for firewall rules), allocate non‑overlapping ranges per namespace:

# values-prod.yaml
service:
  type: NodePort
  nodePortBase: 30020   # start of range for prod

# values-analytics.yaml
service:
  type: NodePort
  nodePortBase: 30030   # start of range for analytics

Then compute the actual port per replica:

nodePort: {{ add .Values.service.nodePortBase .Values.replicaId }}

Option C – Replace NodePort with ClusterIP + Ingress (most robust)

Change the service type to ClusterIP and expose Gemini via an internal Ingress controller that terminates TLS and forwards to the pod IPs. This eliminates host‑level port allocation entirely.

# Before
type: NodePort
nodePort: 30001

# After
type: ClusterIP

Then create an Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gemini-ingress
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  rules:
  - host: gemini.internal.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: gemini-service
            port:
              number: 50051

Verify – Confirm the Batch Job Starts Successfully

1. Check that the Gemini service reports a healthy endpoint:

kubectl get svc -n default gemini-node-0 -o yaml | grep nodePort

2. Inspect the pod logs for the absence of the bind error:

kubectl logs -n default $(kubectl get pod -n default -l app=gemini -o name | head -n1) | grep "Failed to bind"

Expected: no output.

3. Verify the batch job transitions to Running and eventually Completed:

kubectl get batchjobs -n default data-ingest-20231018-001 -o wide

4. Optionally, perform an end‑to‑end gRPC health check:

grpcurl -plaintext gemini.internal.example.com:50051 list

Successful response indicates the service is reachable.

Prevent – Operational Guardrails

• Namespace‑scoped NodePort ranges: Configure the kube‑proxy ConfigMap with distinct nodePortRange per namespace using the --node-port-range flag on the kube‑proxy daemonset.
• Helm chart validation: Add a pre‑install hook that checks for existing NodePort usage via kubectl get svc -A -o jsonpath='{.items[*].spec.ports[?(@.nodePort)]}' and aborts if a conflict is detected.
• Monitoring: Export kube_service_info metrics (Prometheus) and set an alert on kube_service_spec_type{type="NodePort"} with duplicate node_port labels.
• CI/CD policy: Enforce that all new services either use ClusterIP or request a unique NodePort via a central allocation service.
• Documentation: Record the allocated ranges per environment in a version‑controlled config file to avoid manual duplication.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why does the conflict appear only after a Helm upgrade?
   Helm upgrades recreate the Service object. If the new chart still hard‑codes the same nodePort, Kubernetes attempts to re‑allocate it. If another Service has claimed the port in the meantime, the bind fails.
2. Can I use a LoadBalancer service instead of NodePort?
   Yes. A LoadBalancer type abstracts the host port and avoids NodePort collisions, but it requires a cloud provider that supports external load balancers.
3. Is it safe to delete the conflicting Service and let Kubernetes recreate it?
   Deleting the Service will free the port, but any pods that depend on that Service will lose connectivity temporarily. Prefer a rolling update that changes the port allocation instead of a hard delete.
4. How do I discover which node is actually binding the port?
   Run kubectl get pods -o wide -n <namespace> to map the Service’s externalIPs to node names, then SSH into the node and use sudo ss -ltnp | grep 30001 to see the owning process.
5. Will switching to ClusterIP affect existing firewall rules?
   Yes. ClusterIP traffic stays within the cluster network, so external firewall rules targeting the node’s IP/port become irrelevant. Update your network policies accordingly.