TorchServe Pods Pending After Custom Scheduler Can't Match GPU Resource Requests

Q: Debug – Step‑by‑Step Investigation 1. Observe the pod status and events

kubectl get pods -n inference -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES torchserve-abc123 0/1 Pending 0 2m <none> <none> <none> Events:

Q: 2. Verify node resources and taints

kubectl describe node gpu-node-1 | grep -A3 "Allocated resources" Allocated resources: cpu: 28 memory: 112Gi nvidia.com/gpu: 4 kubectl get nodes -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.spec.taints}{'\n'}{end}" gpu-node-1 [{"key":"nvidia.com/gpu","value":"present","effect":"NoSchedule"}] 3. Inspect the TorchServe pod spec apiVersion: serving.k8s.io/v1alpha1 kind: InferenceService metadata: name: torchserve spec: predictor: custom: container: image: myrepo/torchserve:latest resources: limits: nvidia.com/gpu: 1 4. Review the custom scheduler configuration apiVersion: kubescheduler.config.k8s.io/v1beta3 kind: KubeSchedulerConfiguration profiles: - schedulerName: inference-scheduler plugins: queueSort: enabled: - name: PrioritySort preFilter: enabled: - name: CustomGPUFilter filter: disabled: - name: NodeResourcesFit #

In this article

Table of Contents

Problem – TorchServe Pods Remain Pending When Using a Custom Scheduler

In a Kubernetes cluster that hosts GPU‑enabled node pools, a real‑time inference service is deployed with TorchServe. The deployment succeeds, but the resulting pods stay in Pending state indefinitely. The custom scheduler (configured via a scheduler profile or extender) reports that it cannot find any node that satisfies the GPU resource request and the node taints.

Root Cause – Mismatch Between GPU Resource Requests, Node Taints, and Scheduler Filtering Logic

The failure originates from three intertwined issues:

GPU resource name mismatch: The pod requests nvidia.com/gpu while the node’s Allocatable list reports the resource as nvidia.com/gpu only after the NVIDIA device plugin registers. If the plugin is not yet running on a node, the resource appears as 0, causing the scheduler to deem the node insufficient.
Node taints without matching tolerations: GPU nodes are tainted with nvidia.com/gpu=present:NoSchedule to prevent non‑GPU workloads from landing on them. The TorchServe pod spec lacks a corresponding tolerations entry, so the scheduler filters out all GPU nodes.
Custom scheduler filter plugin does not consider GPU resources: The scheduler profile used for inference services overrides the default NodeResourcesFit plugin with a custom extender that only evaluates CPU and memory. Consequently, even if a node reports a free GPU, the extender discards it as “unsuitable”.

Debug – Step‑by‑Step Investigation

1. Observe the pod status and events

kubectl get pods -n inference -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE   NOMINATED NODE   READINESS GATES
torchserve-abc123        0/1     Pending   0          2m   <none>      <none>               <none>

Events:

Warning  FailedScheduling  30s (x4)  custom-scheduler  0/5 nodes are available: 5 Insufficient nvidia.com/gpu, 5 node(s) had taint {key: nvidia.com/gpu, effect: NoSchedule}.

2. Verify node resources and taints

kubectl describe node gpu-node-1 | grep -A3 "Allocated resources"
Allocated resources:
  cpu:                28
  memory:             112Gi
  nvidia.com/gpu:     4

kubectl get nodes -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.spec.taints}{'\n'}{end}"
gpu-node-1   [{"key":"nvidia.com/gpu","value":"present","effect":"NoSchedule"}]

3. Inspect the TorchServe pod spec

apiVersion: serving.k8s.io/v1alpha1
kind: InferenceService
metadata:
  name: torchserve
spec:
  predictor:
    custom:
      container:
        image: myrepo/torchserve:latest
        resources:
          limits:
            nvidia.com/gpu: 1

4. Review the custom scheduler configuration

apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: inference-scheduler
  plugins:
    queueSort:
      enabled:
      - name: PrioritySort
    preFilter:
      enabled:
      - name: CustomGPUFilter
    filter:
      disabled:
      - name: NodeResourcesFit   # <-- default plugin disabled

5. Check the custom filter plugin logs

kubectl logs -n kube-system -l component=custom-scheduler -c scheduler
...
[custom-gpu-filter] node gpu-node-1 rejected: missing GPU resource in request
...

Solution – Align Resource Requests, Tolerations, and Scheduler Logic

1. Add a toleration for the GPU taint

apiVersion: serving.k8s.io/v1alpha1
kind: InferenceService
metadata:
  name: torchserve
spec:
  predictor:
    custom:
      container:
        image: myrepo/torchserve:latest
        resources:
          limits:
            nvidia.com/gpu: 1
        tolerations:
        - key: "nvidia.com/gpu"
          operator: "Equal"
          value: "present"
          effect: "NoSchedule"

2. Re‑enable the standard `NodeResourcesFit` plugin or extend the custom filter to consider GPUs

Option A – Enable the default plugin alongside the custom one:

apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: inference-scheduler
  plugins:
    filter:
      enabled:
      - name: NodeResourcesFit
      - name: CustomGPUFilter

Option B – Modify CustomGPUFilter to read nvidia.com/gpu from node.Status.Allocatable and compare it against pod.Spec.Containers[].Resources.Limits.

3. Ensure the NVIDIA device plugin is running on all GPU nodes before scheduling

kubectl get daemonset -n kube-system nvidia-device-plugin-daemonset
NAME                              DESIRED   CURRENT   READY   UP‑TO‑DATE   AVAILABLE
nvidia-device-plugin-daemonset    5         5         5       5            5

If the daemonset is not ready, the nvidia.com/gpu resource will be reported as zero, causing the scheduler to reject the node.

4. Apply the corrected InferenceService manifest

# Before (failing)
apiVersion: serving.k8s.io/v1alpha1
kind: InferenceService
metadata:
  name: torchserve
spec:
  predictor:
    custom:
      container:
        image: myrepo/torchserve:latest
        resources:
          limits:
            nvidia.com/gpu: 1

# After (fixed)
apiVersion: serving.k8s.io/v1alpha1
kind: InferenceService
metadata:
  name: torchserve
spec:
  predictor:
    custom:
      container:
        image: myrepo/torchserve:latest
        resources:
          limits:
            nvidia.com/gpu: 1
        tolerations:
        - key: "nvidia.com/gpu"
          operator: "Equal"
          value: "present"
          effect: "NoSchedule"

Verify – Confirm That Pods Schedule Successfully and Serve Traffic

Watch the pod transition:

kubectl get pods -n inference -w
NAME          READY   STATUS    RESTARTS   AGE
torchserve-abc123   1/1   Running   0          30s

Inspect the node assignment and GPU allocation:

kubectl describe pod torchserve-abc123 -n inference | grep -A2 "Allocated resources"
Allocated resources:
  nvidia.com/gpu:  1
Node:               gpu-node-3/10.128.0.12

Run a simple inference request to ensure TorchServe is functional:

curl -X POST http://torchserve-abc123.inference.svc.cluster.local/predictions/mnist -T sample.png
{"prediction": 7}

Prevent – Operational Guardrails to Avoid Recurrence

Validate GPU resource availability before creating InferenceService objects: use a webhook that checks node.status.allocatable["nvidia.com/gpu"] against the requested limit.
Standardize node taints and pod tolerations: document that all GPU workloads must include the nvidia.com/gpu=present:NoSchedule toleration.
Keep the NVIDIA device plugin healthy: monitor daemonset nvidia-device-plugin-daemonset readiness; set an alert on ReadyReplicas != DesiredReplicas.
Scheduler configuration review: ensure any custom scheduler profile either extends or re‑enables NodeResourcesFit for GPU resources.
Automated tests in CI/CD: spin up a minimal GPU node pool and run a smoke test that deploys a TorchServe pod with a GPU request, asserting it reaches Running.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the pod schedule when I remove the custom scheduler but fail with it enabled?
The custom scheduler profile disables the default NodeResourcesFit plugin, which is responsible for matching nvidia.com/gpu requests against node allocatable resources. Without it, the scheduler cannot see that a node has a free GPU.
Can I use nodeSelector instead of tolerations for GPU nodes?
Yes, adding nodeSelector: {"nvidia.com/gpu.present": "true"} works if the nodes are labeled accordingly, but you still need a toleration for the NoSchedule taint; otherwise the scheduler will reject the node.
My cluster uses AMD GPUs (resource name amd.com/gpu). How should I adapt the fix?
Replace all occurrences of nvidia.com/gpu with amd.com/gpu in the pod resources.limits, node taints, tolerations, and ensure the corresponding device plugin registers the resource.
Is it safe to remove the GPU taint altogether?
Removing the taint allows non‑GPU workloads to be scheduled on GPU nodes, which can lead to resource contention. The recommended approach is to keep the taint and use tolerations for GPU‑specific workloads.
How do I debug a custom scheduler extender that silently drops nodes?
Enable verbose logging for the extender (e.g., --v=4 on the scheduler container) and inspect the JSON payloads in the extender’s logs. Look for the filterResult field that lists rejected nodes and the reason strings.