Pod Placement Failure For PyTorch Job On Kind After Node Selector Change

Q: Root Cause

The failure originates from a mismatch between the nodeSelector defined in the PyTorchJob manifest and the labels present on the sole kind node. After the node selector was introduced to isolate GPU‑enabled nodes, the selector points to a label that does not exist on the single‑node cluster, causing the default scheduler to consider the node ineligible for the pod. Because there is only one node, the scheduler has no alternative placement, resulting in a Insufficient cpu or node(s) didn't match node selector scheduling error.

Q: Option 3 – Scale down resource requests to match node capacity

Adjust the cpu and memory fields to values that the kind node can allocate. The following table shows a typical mapping for a default kind node (2 CPU, 4 GiB RAM):

In this article

Table of Contents

Problem: Pod placement failure for a PyTorch job on a kind cluster after changing the node selector

When launching a distributed PyTorch training job on a local kind (Kubernetes IN Docker) cluster, the scheduler repeatedly reports that it cannot place the training pods. The job remains in Pending state even though the single node appears healthy.

Root Cause

The failure originates from a mismatch between the nodeSelector defined in the PyTorchJob manifest and the labels present on the sole kind node. After the node selector was introduced to isolate GPU‑enabled nodes, the selector points to a label that does not exist on the single‑node cluster, causing the default scheduler to consider the node ineligible for the pod. Because there is only one node, the scheduler has no alternative placement, resulting in a Insufficient cpu or node(s) didn't match node selector scheduling error.

In addition, the default resource requests for the PyTorch containers (e.g., cpu: 4, memory: 8Gi) exceed the capacity of the kind node (typically 2 CPUs, 4Gi memory), which compounds the placement issue.

Debug: Investigation Steps

Inspect the pod status and events.

kubectl get pods -n torch -l torch-job-name=my-job -o wide

Sample output:

NAME                         READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
my-job-worker-0              0/1     Pending   0          2m    <none>       kind-control-plane   <none>          <none>

kubectl describe pod my-job-worker-0 -n torch

Key event excerpt:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Normal   Scheduled         2m    default-scheduler  Successfully assigned torch/my-job-worker-0 to kind-control-plane
  Warning  FailedScheduling  2m    default-scheduler  0/1 nodes are available: 1 node(s) didn't match node selector, 1 Insufficient cpu.

Verify node labels.

kubectl get nodes --show-labels

Typical output on a fresh kind cluster:

NAME                 STATUS   ROLES    AGE   VERSION   LABELS
kind-control-plane   Ready    master   10m   v1.28.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=kind-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,type=control-plane

Notice the absence of any custom label such as accelerator=nvidia that the job’s nodeSelector expects.

Check resource requests vs node capacity.

kubectl describe node kind-control-plane | grep -A5 "Allocated resources"

Sample excerpt:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  cpu:                2
  memory:             4Gi
  nvidia.com/gpu:     0

The pod requests cpu: 4, which exceeds the node’s allocatable CPU.

Review the PyTorchJob manifest.

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: my-job
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          nodeSelector:
            accelerator: nvidia
          containers:
          - name: pytorch
            image: pytorch/pytorch:2.0.0
            resources:
              requests:
                cpu: "4"
                memory: "8Gi"
                nvidia.com/gpu: "1"
    Worker:
      replicas: 2
      template:
        spec:
          nodeSelector:
            accelerator: nvidia
          containers:
          - name: pytorch
            image: pytorch/pytorch:2.0.0
            resources:
              requests:
                cpu: "4"
                memory: "8Gi"
                nvidia.com/gpu: "1"

Solution: Align node selector and resource requests with the kind environment

Option 1 – Remove or relax the node selector for local development

For a single‑node kind cluster without GPUs, the simplest fix is to omit the nodeSelector and adjust resource requests to fit the node’s capacity.

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: my-job
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          # nodeSelector removed
          containers:
          - name: pytorch
            image: pytorch/pytorch:2.0.0
            resources:
              requests:
                cpu: "1"
                memory: "2Gi"
    Worker:
      replicas: 2
      template:
        spec:
          # nodeSelector removed
          containers:
          - name: pytorch
            image: pytorch/pytorch:2.0.0
            resources:
              requests:
                cpu: "1"
                memory: "2Gi"

Option 2 – Add the expected label to the kind node

If you want to keep the selector (e.g., to keep the manifest identical to production), label the node accordingly:

kubectl label node kind-control-plane accelerator=nvidia

Then ensure the node reports a fake GPU resource so the scheduler does not reject the request. Install the nvidia.com/gpu device plugin in a dummy mode or use extendedResources via a ResourceQuota that pretends the resource exists. For most local experiments, removing the GPU request is sufficient.

Option 3 – Scale down resource requests to match node capacity

Adjust the cpu and memory fields to values that the kind node can allocate. The following table shows a typical mapping for a default kind node (2 CPU, 4 GiB RAM):

Component	Original Request	Adjusted Request
CPU	4	1
Memory	8Gi	2Gi
GPU	1	0 (removed)

Verify: Confirm that the job schedules and runs

Apply the corrected manifest:

kubectl apply -f pytorchjob-fixed.yaml

Watch pod status:

kubectl get pods -n torch -w

Expected output after a few seconds:

NAME                         READY   STATUS    RESTARTS   AGE
my-job-master-0              1/1     Running   0          30s
my-job-worker-0              1/1     Running   0          30s
my-job-worker-1              1/1     Running   0          30s

Inspect logs to ensure the training script started:

kubectl logs my-job-master-0 -n torch

Sample log excerpt:

[2026-10-30 12:01:02] INFO: Initializing process group with backend NCCL
[2026-10-30 12:01:05] INFO: Rank 0/3 - Training started

Prevent: Practices to avoid similar placement failures

Separate manifests for local vs. production. Use Kustomize overlays or Helm values to toggle nodeSelector and resource requests based on the target cluster.
Validate node labels before applying workloads. A simple pre‑deployment script can run kubectl get nodes --show-labels and abort if required labels are missing.
Define resource limits that respect the smallest node in your fleet. When a workload must run on both GPU‑enabled and CPU‑only nodes, set requests to the lowest common denominator and use limits for higher‑performance nodes.
Enable scheduler simulation. Tools like kubectl dry-run=client combined with kubectl explain pod.spec.schedulerName can surface selector mismatches early.
Automate node labeling. Include a post‑cluster‑creation step that adds development‑specific labels (e.g., accelerator=nvidia) to kind nodes when you intend to emulate production selectors.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the pod stay in Pending even though the node is Ready?
Because the scheduler filters nodes based on the nodeSelector and resource requests. If no node satisfies both criteria, the pod cannot be bound, and it remains pending.
Can I keep the GPU request in a kind cluster that has no GPUs?
Not without a device plugin that advertises a fake nvidia.com/gpu resource. The usual approach is to remove the GPU request for local testing.
Is there a way to make the scheduler ignore the node selector temporarily?
You can patch the pod spec after creation with kubectl patch pod <name> -p '{"spec":{"nodeSelector":null}}', but it’s better to adjust the manifest before deployment.
How do I know the exact resource capacity of my kind node?
Run kubectl describe node <node-name> | grep -A5 "Allocated resources" or inspect .status.allocatable via kubectl get node -o jsonpath='{.items[0].status.allocatable}'.
My production manifest uses nodeSelector: accelerator=nvidia. How can I keep the same file for local development?
Use Kustomize overlays: a base manifest with the selector, and a development overlay that removes or overrides the selector and scales down resources.