MLflow scheduler pod pending due to insufficient compute resources

Problem

In a managed Kubernetes environment, MLflow’s mlflow‑scheduler pod repeatedly remains in Pending state when a training run is submitted. The pod never transitions to Running, causing the entire run to fail with an error similar to:


Error: Unable to schedule pod for run 12345: pod "mlflow-scheduler-abcde" is pending: 
  insufficient cpu, memory, or node selector constraints.

Symptoms observed across multiple runs:

  • MLflow UI shows “Queued” runs that never start.
  • kubectl get pods -n mlflow lists scheduler pods with Pending status.
  • Cluster autoscaler does not provision new nodes.
  • Node affinity/selector logs indicate mismatched labels.

Root Cause

The scheduler pod definition includes explicit resource requests and a nodeSelector that targets a node pool dedicated to GPU‑enabled workloads. In the managed service, those nodes have a hard limit on cpu and memory allocations per pod, and the autoscaler is configured to ignore pods that request resources exceeding the node’s allocatable capacity. The combination of:

  1. Over‑provisioned resource requests (e.g., cpu: "4", memory: "16Gi") that exceed the maximum per‑pod quota on the GPU node pool.
  2. Node selector mismatch – the selector cloud.google.com/gke-nodepool: gpu-pool is applied to every scheduler pod, but the GPU pool only contains nodes with accelerator: nvidia‑t4. When a non‑GPU run is queued, the scheduler still tries to land on that pool, which lacks sufficient generic compute capacity.

Consequently, the Kubernetes scheduler cannot find a node that satisfies both the resource request and the selector, leaving the pod pending indefinitely.

Debug

Step‑by‑step investigation performed in the affected namespace (mlflow).

1. Inspect pod status and events


$ kubectl describe pod mlflow-scheduler-abcde -n mlflow
Name:         mlflow-scheduler-abcde
Namespace:    mlflow
Status:       Pending
...
Events:
  Type    Reason            Age   From               Message
  ----    ------            ----  ----               -------
  Normal  Scheduled         2m    default-scheduler  Successfully assigned mlflow/mlflow-scheduler-abcde to node-1
  Warning FailedScheduling  2m    default-scheduler  0/5 nodes are available: 5 Insufficient cpu, 5 Insufficient memory.
  Warning FailedScheduling  2m    default-scheduler  0/5 nodes are available: 5 node(s) didn't match node selector.

2. Verify resource quotas and limits on the target node pool


$ kubectl get nodes -l cloud.google.com/gke-nodepool=gpu-pool -o jsonpath='{.items[*].status.allocatable}'
cpu: "28"
memory: "112Gi"
pods: "110"

Cluster‑level LimitRange for the namespace:


$ kubectl get limitrange -n mlflow -o yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: mlflow-limitrange
spec:
  limits:
  - default:
      cpu: "2"
      memory: "4Gi"
    defaultRequest:
      cpu: "1"
      memory: "2Gi"
    max:
      cpu: "4"
      memory: "16Gi"
    type: Container

Requests in the scheduler deployment exceed the max values for non‑GPU nodes.

3. Examine the scheduler deployment manifest


apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlflow-scheduler
  namespace: mlflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mlflow-scheduler
  template:
    metadata:
      labels:
        app: mlflow-scheduler
    spec:
      nodeSelector:
        cloud.google.com/gke-nodepool: gpu-pool
      containers:
      - name: scheduler
        image: ghcr.io/mlflow/mlflow:2.9.0
        resources:
          requests:
            cpu: "4"
            memory: "16Gi"
          limits:
            cpu: "4"
            memory: "16Gi"
        env:
        - name: BACKEND_STORE_URI
          value: postgresql://mlflow:pwd@postgres:5432/mlflow
        - name: DEFAULT_ARTIFACT_ROOT
          value: s3://mlflow-artifacts

4. Check autoscaler behavior


$ kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml | grep -i pending
# No pending pods are reported because the scheduler pod is filtered out due to nodeSelector mismatch.

Solution

Two complementary changes resolve the pending state:

1. Adjust resource requests to fit the generic node pool

Reduce the scheduler’s CPU and memory to values within the max limits of the LimitRange and the per‑pod quota of the non‑GPU node pool.

# Before
resources:
  requests:
    cpu: "4"
    memory: "16Gi"
  limits:
    cpu: "4"
    memory: "16Gi"

# After
resources:
  requests:
    cpu: "2"
    memory: "8Gi"
  limits:
    cpu: "2"
    memory: "8Gi"

2. Refine node selector logic

Separate scheduler pods that need GPU resources from those that do not. The scheduler itself does not require a GPU, so remove the selector or apply it conditionally based on an environment variable.

# Before (hard selector)
nodeSelector:
  cloud.google.com/gke-nodepool: gpu-pool

# After (no selector, allowing any node)
# nodeSelector removed
# Optional: add tolerations if needed for taints on GPU nodes
tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

3. Apply the updated manifest


$ kubectl apply -f mlflow-scheduler-deployment.yaml
deployment.apps/mlflow-scheduler configured

After the rollout, the new pod is scheduled on a standard compute node.

Verify

Confirm that the scheduler pod reaches Running and that new training runs start successfully.

Pod status


$ kubectl get pod -l app=mlflow-scheduler -n mlflow
NAME                     READY   STATUS    RESTARTS   AGE
mlflow-scheduler-xyz12   1/1     Running   0          30s

Run execution


$ mlflow runs list -f "status='RUNNING'" --experiment-id 0
run_id                      status   start_time
-------------------------------------------------
12345                       RUNNING  2026-11-09T14:02:13Z

Cluster resources


$ kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-1        12           30%    28Gi            25%
node-2        8            20%    16Gi            15%

Prevent

  • Define separate Deployments for GPU‑bound components. Use distinct nodeSelector or affinity rules only where a GPU is required.
  • Align resource requests with namespace LimitRange and node pool quotas. Automate validation with a CI lint step that parses manifests and fails on over‑provisioned values.
  • Enable the autoscaler to consider pending pods with node selectors. In GKE, set --expander=least-waste and ensure --balance-similar-node-groups is true.
  • Monitor pod pending reasons. Create a Prometheus alert on kube_pod_status_phase{phase="Pending"} with a label filter for reason=Unschedulable.
  • Document default resource profiles. Keep a version‑controlled resources.yaml that lists “small”, “medium”, “large” profiles for MLflow components.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the scheduler pod require a GPU node selector? The original manifest was copied from the training worker deployment, which needs GPUs. The scheduler itself only orchestrates jobs and does not perform GPU‑intensive work, so the selector is unnecessary and blocks scheduling on generic nodes.
  2. How can I see which resource request caused the pending state? Use kubectl describe pod <pod> and look for “Insufficient cpu” or “Insufficient memory” events. The message includes the exact shortfall compared to node allocatable resources.
  3. Will lowering the scheduler’s resources affect run throughput? No. The scheduler is lightweight; 2 CPU / 8 GiB is sufficient for queuing and dispatching jobs. The heavy lifting occurs in the worker pods.
  4. Can I keep the node selector for GPU runs while allowing the scheduler to run elsewhere? Yes. Deploy two separate scheduler Deployments: one with nodeSelector: gpu-pool for GPU‑specific jobs, and a default one without a selector for CPU‑only runs. Use MLflow’s MLFLOW_SCHEDULER_BACKEND environment variable to point to the appropriate service.
  5. What if the autoscaler still does not provision new nodes after fixing the selector? Verify that the cluster’s maxNodeCount and minNodeCount limits allow scaling, and that the node pool’s machine type satisfies the requested resources. Adjust the autoscaler configuration or add a larger machine type to the pool.