GPU cronjob causing training pod delays after node autoscaling

Problem – GPU‑intensive CronJob blocks AI training pods after node autoscaling

In a production Kubernetes cluster that runs NVIDIA GPU‑enabled AI training workloads, a nightly CronJob that performs data preprocessing on a GPU started to cause noticeable delays for scheduled training pods. The delays manifested only after the cluster performed an automatic scale‑out of GPU nodes. Operators observed that training pods remained in Pending for several minutes, while the CronJob pods were already running on the newly added nodes.

Root Cause – Uncontrolled GPU resource contention and scheduler priority mismatch

The root cause can be broken down into three interacting factors:

  1. Identical resource requests: Both the CronJob and the training pods requested nvidia.com/gpu: 1 without any distinction. The scheduler treated them as interchangeable workloads.
  2. Default pod priority: Neither workload defined a PriorityClass. Consequently, all pods received the system default priority (value 0), making them equally eligible for preemption.
  3. Node autoscaling race condition: When the cluster autoscaler added a new GPU node, the scheduler immediately placed the pending CronJob pod on that node because it was the first pod in the scheduling queue. The training pod, which arrived milliseconds later, was forced to wait for the CronJob to finish or for the node to be reclaimed, leading to the observed delay.

Because the CronJob runs daily and consumes a full GPU for up to 30 minutes, it effectively monopolizes newly provisioned GPU capacity, preempting the higher‑value training workloads.

Debug – Step‑by‑step investigation

1. Observe pod states and events

kubectl get pods -n ai -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   NODE
training-20231112-abcde               0/1     Pending   0          2m    <none>
data-preprocess-20231112-xyz          1/1     Running   0          1m    ip-10-0-5-12

Inspect the pending training pod events:

kubectl describe pod training-20231112-abcde -n ai | grep -A3 Events
Events:
  Type    Reason            Age   From               Message
  ----    ------            ----  ----               -------
  Normal  Scheduled         2m    default-scheduler  Successfully assigned ai/ip-10-0-5-12 to ip-10-0-5-12
  Warning FailedScheduling   2m    default-scheduler  0/1 nodes are available: 1 Insufficient nvidia.com/gpu.

2. Check scheduler logs for pre‑scale‑out ordering

kubectl logs -n kube-system -l component=kube-scheduler -c kube-scheduler | grep -i "gpu"
2026-11-12T03:14:02.123Z scheduler.go:215] "Attempting to schedule pod" pod="ai/data-preprocess-20231112-xyz"
2026-11-12T03:14:02.127Z scheduler.go:215] "Attempting to schedule pod" pod="ai/training-20231112-abcde"

The log shows the CronJob pod was processed first.

3. Verify node autoscaling trigger

kubectl get nodes -l cloud.google.com/gke-nodepool=gpu-pool -o wide
NAME                                 STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE
ip-10-0-5-12                         Ready       12d   v1.28.3   10.0.5.12     35.224.1.45   Ubuntu 22.04 LTS
ip-10-0-7-34                         Ready       5m    v1.28.3   10.0.7.34     35.224.2.67   Ubuntu 22.04 LTS   # newly created

4. Examine CronJob spec

apiVersion: batch/v1
kind: CronJob
metadata:
  name: data-preprocess
  namespace: ai
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: preprocess
            image: myrepo/preprocess:latest
            resources:
              limits:
                nvidia.com/gpu: 1
          restartPolicy: OnFailure

5. Examine training pod spec

apiVersion: batch/v1
kind: Job
metadata:
  name: training-20231112
  namespace: ai
spec:
  template:
    spec:
      containers:
      - name: trainer
        image: myrepo/trainer:2023.11
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never

Solution – Isolate CronJob from critical GPU workloads

1. Introduce a dedicated GPU node pool for batch‑type jobs

Create a second node pool with the same GPU type but labeled for low‑priority workloads.

# gcloud command (example for GKE)
gcloud container node-pools create gpu-batch-pool \
  --cluster=my-cluster \
  --zone=us-central1-a \
  --machine-type=n1-standard-8 \
  --accelerator type=nvidia-tesla-t4,count=1 \
  --node-labels=workload=batch,gpu=true \
  --node-taints=workload=batch:NoSchedule

2. Add node affinity to the CronJob so it runs only on the batch pool

apiVersion: batch/v1
kind: CronJob
metadata:
  name: data-preprocess
  namespace: ai
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: workload
                    operator: In
                    values:
                    - batch
          tolerations:
          - key: workload
            operator: Equal
            value: batch
            effect: NoSchedule
          containers:
          - name: preprocess
            image: myrepo/preprocess:latest
            resources:
              limits:
                nvidia.com/gpu: 1
          restartPolicy: OnFailure

3. Elevate training pods with a higher PriorityClass

Create a priority class for AI training workloads:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: ai-training-high
value: 1000000
globalDefault: false
description: "High priority for production AI training jobs"

Reference it in the training job spec:

apiVersion: batch/v1
kind: Job
metadata:
  name: training-20231112
  namespace: ai
spec:
  priorityClassName: ai-training-high
  template:
    spec:
      containers:
      - name: trainer
        image: myrepo/trainer:2023.11
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never

4. Optional – Enforce GPU quota per node pool

Apply a ResourceQuota that caps the total GPU usage in the batch pool, preventing it from exhausting all GPUs:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-batch-quota
  namespace: ai
spec:
  hard:
    limits.nvidia.com/gpu: "4"   # assuming the pool has 4 GPUs total

5. Deploy changes

# Apply new node pool, priority class, and updated manifests
kubectl apply -f priorityclass.yaml
kubectl apply -f cronjob-updated.yaml
kubectl apply -f training-job.yaml
kubectl apply -f gpu-quota.yaml

Verify – Confirm that training pods are no longer delayed

1. Observe scheduling order after a scale‑out event

# Trigger a scale‑out by submitting a training job
kubectl create -f training-job.yaml

# Watch pod events
kubectl get pods -w -n ai

Expected output (simplified):

NAME                                 READY   STATUS    RESTARTS   AGE   NODE
training-20231112-abcde               0/1     Running   0          10s   ip-10-0-7-34   # scheduled on new GPU node
data-preprocess-20231112-xyz          1/1     Running   0          5m    ip-10-0-5-12   # runs on batch pool

2. Validate that the CronJob never lands on the primary training node pool

kubectl get pods -l job-name=data-preprocess -n ai -o jsonpath="{.items[*].spec.nodeName}"
# Output should contain only nodes with label workload=batch

3. Check that preemptible GPU capacity remains available for training

kubectl top nodes -l workload=batch
kubectl top nodes -l workload=training

Metrics should show near‑zero GPU utilization on the training pool during the CronJob window.

Prevent – Operational guardrails to avoid recurrence

  • Separate node pools for production AI training and batch‑type GPU jobs; label and taint them accordingly.
  • Pod priority classes for all critical workloads; assign lower priority to non‑critical batch jobs.
  • GPU ResourceQuota per namespace or node pool to cap aggregate GPU consumption.
  • Admission controller policy (e.g., OPA Gatekeeper) that rejects GPU‑requesting pods without an explicit priorityClassName or node affinity.
  • Monitoring alerts on Pending pods with GPU requests and on node.autoscaler.scale_up events to catch scheduling bottlenecks early.
  • Automated testing of CronJob manifests in a staging cluster that mirrors production node pool labels and taints.

FAQ – Common follow‑up questions

  1. Why does the CronJob still consume a GPU if it only does data preprocessing?
    The container image includes a TensorFlow library that automatically loads the CUDA driver, causing the Kubernetes runtime to allocate a GPU based on the limits field. Removing the GPU limit or using a CPU‑only image eliminates the allocation.
  2. Can I use preemptionPolicy: Never on the CronJob instead of priority classes?
    Setting preemptionPolicy: Never prevents the CronJob from being preempted, but it does not stop it from preempting higher‑priority pods. Priority classes together with preemptionPolicy: PreemptLowerPriority (default) give deterministic ordering.
  3. What if I need more than one GPU for the training job?
    Define the exact GPU count in the training pod spec (e.g., limits: nvidia.com/gpu: 4) and ensure the node pool has sufficient capacity. The scheduler will then consider multi‑GPU topology (NUMA, MIG) when placing the pod.
  4. How do I verify which node a pod was scheduled on after autoscaling?
    Use kubectl get pod <name> -o wide or kubectl describe pod <name>. The Node field shows the exact node name, which can be cross‑referenced with node labels/taints.
  5. Is it safe to run the CronJob on the same GPU pool but with a lower PriorityClass?
    It works only if the training pods have higher priority and the cluster has enough free GPUs. In practice, separating the workloads into distinct node pools removes the race condition entirely and simplifies capacity planning.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub