Problem
In a managed Kubernetes environment, MLflow’s mlflow‑scheduler pod repeatedly remains in Pending state when a training run is submitted. The pod never transitions to Running, causing the entire run to fail with an error similar to:
Error: Unable to schedule pod for run 12345: pod "mlflow-scheduler-abcde" is pending:
insufficient cpu, memory, or node selector constraints.
Symptoms observed across multiple runs:
- MLflow UI shows “Queued” runs that never start.
kubectl get pods -n mlflowlists scheduler pods withPendingstatus.- Cluster autoscaler does not provision new nodes.
- Node affinity/selector logs indicate mismatched labels.
Root Cause
The scheduler pod definition includes explicit resource requests and a nodeSelector that targets a node pool dedicated to GPU‑enabled workloads. In the managed service, those nodes have a hard limit on cpu and memory allocations per pod, and the autoscaler is configured to ignore pods that request resources exceeding the node’s allocatable capacity. The combination of:
- Over‑provisioned resource requests (e.g.,
cpu: "4",memory: "16Gi") that exceed the maximum per‑pod quota on the GPU node pool. - Node selector mismatch – the selector
cloud.google.com/gke-nodepool: gpu-poolis applied to every scheduler pod, but the GPU pool only contains nodes withaccelerator: nvidia‑t4. When a non‑GPU run is queued, the scheduler still tries to land on that pool, which lacks sufficient generic compute capacity.
Consequently, the Kubernetes scheduler cannot find a node that satisfies both the resource request and the selector, leaving the pod pending indefinitely.
Debug
Step‑by‑step investigation performed in the affected namespace (mlflow).
1. Inspect pod status and events
$ kubectl describe pod mlflow-scheduler-abcde -n mlflow
Name: mlflow-scheduler-abcde
Namespace: mlflow
Status: Pending
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned mlflow/mlflow-scheduler-abcde to node-1
Warning FailedScheduling 2m default-scheduler 0/5 nodes are available: 5 Insufficient cpu, 5 Insufficient memory.
Warning FailedScheduling 2m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
2. Verify resource quotas and limits on the target node pool
$ kubectl get nodes -l cloud.google.com/gke-nodepool=gpu-pool -o jsonpath='{.items[*].status.allocatable}'
cpu: "28"
memory: "112Gi"
pods: "110"
Cluster‑level LimitRange for the namespace:
$ kubectl get limitrange -n mlflow -o yaml
apiVersion: v1
kind: LimitRange
metadata:
name: mlflow-limitrange
spec:
limits:
- default:
cpu: "2"
memory: "4Gi"
defaultRequest:
cpu: "1"
memory: "2Gi"
max:
cpu: "4"
memory: "16Gi"
type: Container
Requests in the scheduler deployment exceed the max values for non‑GPU nodes.
3. Examine the scheduler deployment manifest
apiVersion: apps/v1
kind: Deployment
metadata:
name: mlflow-scheduler
namespace: mlflow
spec:
replicas: 1
selector:
matchLabels:
app: mlflow-scheduler
template:
metadata:
labels:
app: mlflow-scheduler
spec:
nodeSelector:
cloud.google.com/gke-nodepool: gpu-pool
containers:
- name: scheduler
image: ghcr.io/mlflow/mlflow:2.9.0
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "4"
memory: "16Gi"
env:
- name: BACKEND_STORE_URI
value: postgresql://mlflow:pwd@postgres:5432/mlflow
- name: DEFAULT_ARTIFACT_ROOT
value: s3://mlflow-artifacts
4. Check autoscaler behavior
$ kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml | grep -i pending
# No pending pods are reported because the scheduler pod is filtered out due to nodeSelector mismatch.
Solution
Two complementary changes resolve the pending state:
1. Adjust resource requests to fit the generic node pool
Reduce the scheduler’s CPU and memory to values within the max limits of the LimitRange and the per‑pod quota of the non‑GPU node pool.
# Before
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "4"
memory: "16Gi"
# After
resources:
requests:
cpu: "2"
memory: "8Gi"
limits:
cpu: "2"
memory: "8Gi"
2. Refine node selector logic
Separate scheduler pods that need GPU resources from those that do not. The scheduler itself does not require a GPU, so remove the selector or apply it conditionally based on an environment variable.
# Before (hard selector)
nodeSelector:
cloud.google.com/gke-nodepool: gpu-pool
# After (no selector, allowing any node)
# nodeSelector removed
# Optional: add tolerations if needed for taints on GPU nodes
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
3. Apply the updated manifest
$ kubectl apply -f mlflow-scheduler-deployment.yaml
deployment.apps/mlflow-scheduler configured
After the rollout, the new pod is scheduled on a standard compute node.
Verify
Confirm that the scheduler pod reaches Running and that new training runs start successfully.
Pod status
$ kubectl get pod -l app=mlflow-scheduler -n mlflow
NAME READY STATUS RESTARTS AGE
mlflow-scheduler-xyz12 1/1 Running 0 30s
Run execution
$ mlflow runs list -f "status='RUNNING'" --experiment-id 0
run_id status start_time
-------------------------------------------------
12345 RUNNING 2026-11-09T14:02:13Z
Cluster resources
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
node-1 12 30% 28Gi 25%
node-2 8 20% 16Gi 15%
Prevent
- Define separate Deployments for GPU‑bound components. Use distinct
nodeSelectororaffinityrules only where a GPU is required. - Align resource requests with namespace
LimitRangeand node pool quotas. Automate validation with a CI lint step that parses manifests and fails on over‑provisioned values. - Enable the autoscaler to consider pending pods with node selectors. In GKE, set
--expander=least-wasteand ensure--balance-similar-node-groupsis true. - Monitor pod pending reasons. Create a Prometheus alert on
kube_pod_status_phase{phase="Pending"}with a label filter forreason=Unschedulable. - Document default resource profiles. Keep a version‑controlled
resources.yamlthat lists “small”, “medium”, “large” profiles for MLflow components.
Related Topic Hub: Model Serving Troubleshooting Hub
FAQ
- Why does the scheduler pod require a GPU node selector? The original manifest was copied from the training worker deployment, which needs GPUs. The scheduler itself only orchestrates jobs and does not perform GPU‑intensive work, so the selector is unnecessary and blocks scheduling on generic nodes.
- How can I see which resource request caused the pending state? Use
kubectl describe pod <pod>and look for “Insufficient cpu” or “Insufficient memory” events. The message includes the exact shortfall compared to node allocatable resources. - Will lowering the scheduler’s resources affect run throughput? No. The scheduler is lightweight; 2 CPU / 8 GiB is sufficient for queuing and dispatching jobs. The heavy lifting occurs in the worker pods.
- Can I keep the node selector for GPU runs while allowing the scheduler to run elsewhere? Yes. Deploy two separate scheduler Deployments: one with
nodeSelector: gpu-poolfor GPU‑specific jobs, and a default one without a selector for CPU‑only runs. Use MLflow’sMLFLOW_SCHEDULER_BACKENDenvironment variable to point to the appropriate service. - What if the autoscaler still does not provision new nodes after fixing the selector? Verify that the cluster’s
maxNodeCountandminNodeCountlimits allow scaling, and that the node pool’s machine type satisfies the requested resources. Adjust the autoscaler configuration or add a larger machine type to the pool.