Problem – GPU‑intensive CronJob blocks AI training pods after node autoscaling
In a production Kubernetes cluster that runs NVIDIA GPU‑enabled AI training workloads, a nightly CronJob that performs data preprocessing on a GPU started to cause noticeable delays for scheduled training pods. The delays manifested only after the cluster performed an automatic scale‑out of GPU nodes. Operators observed that training pods remained in Pending for several minutes, while the CronJob pods were already running on the newly added nodes.
Root Cause – Uncontrolled GPU resource contention and scheduler priority mismatch
The root cause can be broken down into three interacting factors:
- Identical resource requests: Both the CronJob and the training pods requested
nvidia.com/gpu: 1without any distinction. The scheduler treated them as interchangeable workloads. - Default pod priority: Neither workload defined a
PriorityClass. Consequently, all pods received the system default priority (value0), making them equally eligible for preemption. - Node autoscaling race condition: When the cluster autoscaler added a new GPU node, the scheduler immediately placed the pending CronJob pod on that node because it was the first pod in the scheduling queue. The training pod, which arrived milliseconds later, was forced to wait for the CronJob to finish or for the node to be reclaimed, leading to the observed delay.
Because the CronJob runs daily and consumes a full GPU for up to 30 minutes, it effectively monopolizes newly provisioned GPU capacity, preempting the higher‑value training workloads.
Debug – Step‑by‑step investigation
1. Observe pod states and events
kubectl get pods -n ai -o wide
NAME READY STATUS RESTARTS AGE NODE
training-20231112-abcde 0/1 Pending 0 2m <none>
data-preprocess-20231112-xyz 1/1 Running 0 1m ip-10-0-5-12
Inspect the pending training pod events:
kubectl describe pod training-20231112-abcde -n ai | grep -A3 Events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned ai/ip-10-0-5-12 to ip-10-0-5-12
Warning FailedScheduling 2m default-scheduler 0/1 nodes are available: 1 Insufficient nvidia.com/gpu.
2. Check scheduler logs for pre‑scale‑out ordering
kubectl logs -n kube-system -l component=kube-scheduler -c kube-scheduler | grep -i "gpu"
2026-11-12T03:14:02.123Z scheduler.go:215] "Attempting to schedule pod" pod="ai/data-preprocess-20231112-xyz"
2026-11-12T03:14:02.127Z scheduler.go:215] "Attempting to schedule pod" pod="ai/training-20231112-abcde"
The log shows the CronJob pod was processed first.
3. Verify node autoscaling trigger
kubectl get nodes -l cloud.google.com/gke-nodepool=gpu-pool -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE
ip-10-0-5-12 Ready 12d v1.28.3 10.0.5.12 35.224.1.45 Ubuntu 22.04 LTS
ip-10-0-7-34 Ready 5m v1.28.3 10.0.7.34 35.224.2.67 Ubuntu 22.04 LTS # newly created
4. Examine CronJob spec
apiVersion: batch/v1
kind: CronJob
metadata:
name: data-preprocess
namespace: ai
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: preprocess
image: myrepo/preprocess:latest
resources:
limits:
nvidia.com/gpu: 1
restartPolicy: OnFailure
5. Examine training pod spec
apiVersion: batch/v1
kind: Job
metadata:
name: training-20231112
namespace: ai
spec:
template:
spec:
containers:
- name: trainer
image: myrepo/trainer:2023.11
resources:
limits:
nvidia.com/gpu: 1
restartPolicy: Never
Solution – Isolate CronJob from critical GPU workloads
1. Introduce a dedicated GPU node pool for batch‑type jobs
Create a second node pool with the same GPU type but labeled for low‑priority workloads.
# gcloud command (example for GKE)
gcloud container node-pools create gpu-batch-pool \
--cluster=my-cluster \
--zone=us-central1-a \
--machine-type=n1-standard-8 \
--accelerator type=nvidia-tesla-t4,count=1 \
--node-labels=workload=batch,gpu=true \
--node-taints=workload=batch:NoSchedule
2. Add node affinity to the CronJob so it runs only on the batch pool
apiVersion: batch/v1
kind: CronJob
metadata:
name: data-preprocess
namespace: ai
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: workload
operator: In
values:
- batch
tolerations:
- key: workload
operator: Equal
value: batch
effect: NoSchedule
containers:
- name: preprocess
image: myrepo/preprocess:latest
resources:
limits:
nvidia.com/gpu: 1
restartPolicy: OnFailure
3. Elevate training pods with a higher PriorityClass
Create a priority class for AI training workloads:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: ai-training-high
value: 1000000
globalDefault: false
description: "High priority for production AI training jobs"
Reference it in the training job spec:
apiVersion: batch/v1
kind: Job
metadata:
name: training-20231112
namespace: ai
spec:
priorityClassName: ai-training-high
template:
spec:
containers:
- name: trainer
image: myrepo/trainer:2023.11
resources:
limits:
nvidia.com/gpu: 1
restartPolicy: Never
4. Optional – Enforce GPU quota per node pool
Apply a ResourceQuota that caps the total GPU usage in the batch pool, preventing it from exhausting all GPUs:
apiVersion: v1
kind: ResourceQuota
metadata:
name: gpu-batch-quota
namespace: ai
spec:
hard:
limits.nvidia.com/gpu: "4" # assuming the pool has 4 GPUs total
5. Deploy changes
# Apply new node pool, priority class, and updated manifests
kubectl apply -f priorityclass.yaml
kubectl apply -f cronjob-updated.yaml
kubectl apply -f training-job.yaml
kubectl apply -f gpu-quota.yaml
Verify – Confirm that training pods are no longer delayed
1. Observe scheduling order after a scale‑out event
# Trigger a scale‑out by submitting a training job
kubectl create -f training-job.yaml
# Watch pod events
kubectl get pods -w -n ai
Expected output (simplified):
NAME READY STATUS RESTARTS AGE NODE
training-20231112-abcde 0/1 Running 0 10s ip-10-0-7-34 # scheduled on new GPU node
data-preprocess-20231112-xyz 1/1 Running 0 5m ip-10-0-5-12 # runs on batch pool
2. Validate that the CronJob never lands on the primary training node pool
kubectl get pods -l job-name=data-preprocess -n ai -o jsonpath="{.items[*].spec.nodeName}"
# Output should contain only nodes with label workload=batch
3. Check that preemptible GPU capacity remains available for training
kubectl top nodes -l workload=batch
kubectl top nodes -l workload=training
Metrics should show near‑zero GPU utilization on the training pool during the CronJob window.
Prevent – Operational guardrails to avoid recurrence
- Separate node pools for production AI training and batch‑type GPU jobs; label and taint them accordingly.
- Pod priority classes for all critical workloads; assign lower priority to non‑critical batch jobs.
- GPU ResourceQuota per namespace or node pool to cap aggregate GPU consumption.
- Admission controller policy (e.g., OPA Gatekeeper) that rejects GPU‑requesting pods without an explicit
priorityClassNameor node affinity. - Monitoring alerts on
Pendingpods with GPU requests and onnode.autoscaler.scale_upevents to catch scheduling bottlenecks early. - Automated testing of CronJob manifests in a staging cluster that mirrors production node pool labels and taints.
FAQ – Common follow‑up questions
- Why does the CronJob still consume a GPU if it only does data preprocessing?
The container image includes a TensorFlow library that automatically loads the CUDA driver, causing the Kubernetes runtime to allocate a GPU based on thelimitsfield. Removing the GPU limit or using a CPU‑only image eliminates the allocation. - Can I use
preemptionPolicy: Neveron the CronJob instead of priority classes?
SettingpreemptionPolicy: Neverprevents the CronJob from being preempted, but it does not stop it from preempting higher‑priority pods. Priority classes together withpreemptionPolicy: PreemptLowerPriority(default) give deterministic ordering. - What if I need more than one GPU for the training job?
Define the exact GPU count in the training pod spec (e.g.,limits: nvidia.com/gpu: 4) and ensure the node pool has sufficient capacity. The scheduler will then consider multi‑GPU topology (NUMA, MIG) when placing the pod. - How do I verify which node a pod was scheduled on after autoscaling?
Usekubectl get pod <name> -o wideorkubectl describe pod <name>. TheNodefield shows the exact node name, which can be cross‑referenced with node labels/taints. - Is it safe to run the CronJob on the same GPU pool but with a lower
PriorityClass?
It works only if the training pods have higher priority and the cluster has enough free GPUs. In practice, separating the workloads into distinct node pools removes the race condition entirely and simplifies capacity planning.
Related Topic Hub: GPU Infrastructure Troubleshooting Hub