Weaviate Scheduler Pods Pending Due To Node Selector Mismatch In GKE

Q: Root Cause

The Weaviate Helm chart defines a nodeSelector for the scheduler component that targets a label cloud.google.com/gke-nodepool=weaviate-pool. In the production cluster the node pool was renamed to weaviate-np during a recent scaling operation, but the label on the nodes was not updated. Consequently, the scheduler pods cannot find any node that satisfies the selector, leading the Kubernetes scheduler to repeatedly attempt placement and eventually mark the pods as Pending.

Q: Debug 1. Observe the pending state

$ kubectl get pods -n weaviate -l app=weaviate-scheduler NAME READY STATUS RESTARTS AGE weaviate-scheduler-5d9c7b9f8c-abcde 0/1 Pending 0 12m 2. Inspect the scheduling events $ kubectl describe pod weaviate-scheduler-5d9c7b9f8c-abcde -n weaviate ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 12m default-scheduler Successfully assigned weaviate/weaviate-scheduler-5d9c7b9f8c-abcde to gke-cluster-1-pool-1-abc123 Warning FailedScheduling 12m default-scheduler 0/3 nodes are available: 3 node(s) didn't match node selector. 3. Verify node labels $ kubectl get nodes --show-labels | grep weaviate gke-cluster-1-pool-1-abc123 Ready ... cloud.google.com/gke-nodepool=default-pool gke-cluster-1-pool-2-def456 Ready ... cloud.google.com/gke-nodepool=weaviate-np 4. Review the Helm values that set the selector # values.yaml (excerpt) scheduler: nodeSelector: cloud.google.com/gke-nodepool: weaviate-pool 5. Correlate the mismatch The selector expects weaviate-pool but the actual node label is weaviate-np. No node matches, causing the failure.

Q: Option B – Adjust the Helm chart to match the existing node label

Modify values.yaml to use the correct selector and redeploy.

Q: Verify 1. Confirm pod scheduling

$ kubectl get pods -n weaviate -l app=weaviate-scheduler NAME READY STATUS RESTARTS AGE weaviate-scheduler-5d9c7b9f8c-abcde 1/1 Running 0 2m 2. Check node selector satisfaction $ kubectl get pod weaviate-scheduler-5d9c7b9f8c-abcde -n weaviate -o jsonpath='{.spec.nodeName}' gke-cluster-1-pool-2-def456 3. Review scheduler logs for errors $ kubectl logs weaviate-scheduler-5d9c7b9f8c-abcde -n weaviate 2024-08-30T12:34:56Z INFO scheduler started 2024-08-30T12:34:57Z INFO node selector matched: cloud.google.com/gke-nodepool=weaviate-np ... Prevent Validate Helm values against actual node labels during CI/CD using a lint step that queries the cluster for expected labels. Pin node selector values to a configurable variable that is sourced from a ConfigMap populated by a cluster‑bootstrap script. Enable a Kubernetes Admission Controller (e.g., PodNodeSelector) to reject pods whose selectors cannot be satisfied, surfacing the problem earlier. Document node pool naming conventions and enforce them through Terraform or GKE API scripts to avoid drift. Related Topic Hub: Vector Databases Troubleshooting Hub

In this article

Table of Contents

Problem

In a Google Kubernetes Engine (GKE) cluster running a Weaviate deployment, the weaviate-scheduler pods remain in the Pending state indefinitely. The kubectl describe pod output shows a scheduling failure related to node selector constraints, despite apparent node capacity.

Root Cause

The Weaviate Helm chart defines a nodeSelector for the scheduler component that targets a label cloud.google.com/gke-nodepool=weaviate-pool. In the production cluster the node pool was renamed to weaviate-np during a recent scaling operation, but the label on the nodes was not updated. Consequently, the scheduler pods cannot find any node that satisfies the selector, leading the Kubernetes scheduler to repeatedly attempt placement and eventually mark the pods as Pending.

Debug

1. Observe the pending state


$ kubectl get pods -n weaviate -l app=weaviate-scheduler
NAME                                 READY   STATUS    RESTARTS   AGE
weaviate-scheduler-5d9c7b9f8c-abcde   0/1     Pending   0          12m

2. Inspect the scheduling events


$ kubectl describe pod weaviate-scheduler-5d9c7b9f8c-abcde -n weaviate
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Normal   Scheduled         12m   default-scheduler  Successfully assigned weaviate/weaviate-scheduler-5d9c7b9f8c-abcde to gke-cluster-1-pool-1-abc123
  Warning  FailedScheduling  12m   default-scheduler  0/3 nodes are available: 3 node(s) didn't match node selector.

3. Verify node labels


$ kubectl get nodes --show-labels | grep weaviate
gke-cluster-1-pool-1-abc123   Ready    ...   cloud.google.com/gke-nodepool=default-pool
gke-cluster-1-pool-2-def456   Ready    ...   cloud.google.com/gke-nodepool=weaviate-np

4. Review the Helm values that set the selector


# values.yaml (excerpt)
scheduler:
  nodeSelector:
    cloud.google.com/gke-nodepool: weaviate-pool

5. Correlate the mismatch

The selector expects weaviate-pool but the actual node label is weaviate-np. No node matches, causing the failure.

Solution

Option A – Align node labels with the selector

Update the node pool label to the value expected by the Helm chart.


# Add the missing label to all nodes in the pool
$ for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=weaviate-np -o name); do
    kubectl label $node cloud.google.com/gke-nodepool=weaviate-pool --overwrite
  done

After relabeling, the scheduler pods can be scheduled without modifying Helm values.

Option B – Adjust the Helm chart to match the existing node label

Modify values.yaml to use the correct selector and redeploy.


# Before (values.yaml)
scheduler:
  nodeSelector:
    cloud.google.com/gke-nodepool: weaviate-pool

# After (values.yaml)
scheduler:
  nodeSelector:
    cloud.google.com/gke-nodepool: weaviate-np

Apply the change:


$ helm upgrade weaviate weaviate/weaviate -n weaviate -f values.yaml

Recommended approach

Option B is safer in managed GKE because node pool labels are automatically managed by GKE. Changing the Helm chart avoids a race condition where GKE may overwrite manual label changes during node pool autoscaling.

Verify

1. Confirm pod scheduling


$ kubectl get pods -n weaviate -l app=weaviate-scheduler
NAME                                 READY   STATUS    RESTARTS   AGE
weaviate-scheduler-5d9c7b9f8c-abcde   1/1     Running   0          2m

2. Check node selector satisfaction


$ kubectl get pod weaviate-scheduler-5d9c7b9f8c-abcde -n weaviate -o jsonpath='{.spec.nodeName}'
gke-cluster-1-pool-2-def456

3. Review scheduler logs for errors


$ kubectl logs weaviate-scheduler-5d9c7b9f8c-abcde -n weaviate
2024-08-30T12:34:56Z INFO scheduler started
2024-08-30T12:34:57Z INFO node selector matched: cloud.google.com/gke-nodepool=weaviate-np
...

Prevent

Validate Helm values against actual node labels during CI/CD using a lint step that queries the cluster for expected labels.
Pin node selector values to a configurable variable that is sourced from a ConfigMap populated by a cluster‑bootstrap script.
Enable a Kubernetes Admission Controller (e.g., PodNodeSelector) to reject pods whose selectors cannot be satisfied, surfacing the problem earlier.
Document node pool naming conventions and enforce them through Terraform or GKE API scripts to avoid drift.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why do other Weaviate pods schedule successfully while the scheduler pod does not?
The other components use the default (no) nodeSelector, so they can run on any node. Only the scheduler pod inherits the explicit selector defined in the Helm chart.
Can I use nodeAffinity instead of nodeSelector to avoid this issue?
nodeAffinity provides more expressive matching (e.g., preferredDuringSchedulingIgnoredDuringExecution) and can include a fallback rule. However, the underlying label still must exist; mismatched labels will still prevent placement unless a soft rule is used.
What if the node pool is auto‑scaled and new nodes miss the label?
GKE automatically propagates the cloud.google.com/gke-nodepool label to new nodes. If you manually override the label, ensure your automation re‑applies the correct value after scaling events.

Is there a way to see all pods that are pending due to node selector mismatches?

Yes. Use the following command:


kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.status.phase=="Pending") | select(.status.conditions[]?.reason=="Unschedulable") | "\(.metadata.namespace) \(.metadata.name) \(.status.conditions[]?.message)"'

Should I keep the scheduler pod on a dedicated node pool?
Separating resource‑intensive components can improve isolation, but it introduces the need for strict label management. If isolation is required, ensure the Helm chart and the infrastructure provisioning scripts use the same label source.