ReplicaSet Scaling Failure After Blue‑green Rollout Of Hugging Face Model

In this article

Table of Contents

ReplicaSet Scaling Failure After Blue‑Green Rollout of a Hugging Face Model

Problem

During a blue‑green deployment of a Hugging Face Transformers API served by TorchServe, the new version’s ReplicaSet does not scale out. Increasing spec.replicas in the Deployment has no effect: no new Pods are created, or the Pods remain in Pending state indefinitely.

Typical symptoms observed in the cluster:

kubectl reports the desired replica count but the availableReplicas stays at the old value.
New Pods appear in kubectl get pods with Status: Pending and never transition to Running.
Cluster events show “Insufficient nvidia.com/gpu” or “node(s) didn’t match node selector”.
Traffic continues to be served by the old version, but the new version never receives any requests.

Root Cause

The blue‑green rollout introduced a change in the pod specification that altered the scheduler constraints:

Additional GPU resource request – the new container image includes a torch.cuda import path, causing the Dockerfile to add nvidia.com/gpu: 1 to resources.requests. The existing node pool does not have GPU‑enabled nodes, so the scheduler cannot place the Pods.
Node selector / tolerations mismatch – a nodeSelector of accelerator: nvidia was added to the new Deployment to target GPU nodes. No nodes carry that label, resulting in “node(s) didn’t match node selector”.
Readiness probe failure – the new image uses a different HTTP port (8081) for TorchServe, but the readiness probe still checks 8080. Pods become Running but immediately become NotReady, causing the ReplicaSet controller to think the desired replica count is not satisfied and to keep creating Pods that are immediately evicted.

In practice, the most common blocker is the unsatisfiable GPU request; the readiness‑probe mismatch often surfaces only after the scheduler places the Pods on a correctly labeled node.

Debug & Investigation

Step‑by‑step diagnostics that revealed the above conditions:

# 1. Verify the ReplicaSet status
kubectl get rs -n ml-serving -l app=huggingface-model -o wide

# Sample output
NAME                                 DESIRED   CURRENT   READY   AGE
huggingface-model-7c9d5b9c9f-abcde   3         3         3       12h
huggingface-model-7c9d5b9c9f-fghij   0         0         0       12h

# 2. Inspect the new ReplicaSet’s Pods
kubectl get pods -n ml-serving -l version=green

# Sample output (all Pending)
NAME                                          READY   STATUS    RESTARTS   AGE
huggingface-model-7c9d5b9c9f-fghij-abcde-0    0/1     Pending   0          2m
huggingface-model-7c9d5b9c9f-fghij-abcde-1    0/1     Pending   0          2m

# 3. Describe one of the pending Pods
kubectl describe pod huggingface-model-7c9d5b9c9f-fghij-abcde-0 -n ml-serving

# Relevant events section
Events:
  Type     Reason                Age   From               Message
  ----     ------                ----  ----               -------
  Warning  FailedScheduling      2m    default-scheduler  0/5 nodes are available: 5 Insufficient nvidia.com/gpu.

# 4. Show the Deployment manifest diff (old vs. new)
kubectl get deployment huggingface-model -n ml-serving -o yaml > old.yaml
# After applying the green version
kubectl get deployment huggingface-model -n ml-serving -o yaml > new.yaml
diff -u old.yaml new.yaml | less

Key differences highlighted by diff:

Field	Old (Blue)	New (Green)
resources.requests.cpu	500m	500m
resources.requests.memory	1Gi	1Gi
resources.requests.nvidia.com/gpu	(none)	1
nodeSelector	(none)	accelerator: nvidia
readinessProbe.httpGet.port	8080	8081

Solution

Three corrective actions are required, depending on the intended deployment environment.

Option A – Deploy to GPU‑enabled nodes

If the model truly requires a GPU, provision a node pool with the appropriate accelerator and label it accordingly.

# 1. Create a GPU node pool (example for GKE)
gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --accelerator type=nvidia-tesla-t4,count=1 \
  --machine-type=n1-standard-4 \
  --node-labels=accelerator=nvidia \
  --num-nodes=3

# 2. Verify node labels
kubectl get nodes -L accelerator

After the nodes are ready, the pending Pods schedule successfully.

Option B – Remove GPU requirement for CPU‑only serving

If the model can run on CPU (e.g., using torchserve --model-store with --device cpu), strip the GPU request and selector from the manifest.

# Before (failing)
resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
    nvidia.com/gpu: "1"
nodeSelector:
  accelerator: nvidia

# After (working)
resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
# nodeSelector removed

Apply the corrected manifest:

kubectl apply -f deployment-green.yaml

Option C – Align readiness probe with the new service port

Update the readiness probe to match the actual TorchServe port (default 8080, overridden to 8081 in the new image).

# Before (probe points to wrong port)
readinessProbe:
  httpGet:
    path: /ping
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

# After (probe corrected)
readinessProbe:
  httpGet:
    path: /ping
    port: 8081
  initialDelaySeconds: 5
  periodSeconds: 10

Verification

Confirm that the new ReplicaSet reaches the desired replica count and that the green pods serve traffic.

# 1. Check ReplicaSet and Pod status
kubectl get rs -n ml-serving -l version=green
kubectl get pods -n ml-serving -l version=green

# Expected output
NAME                                 DESIRED   CURRENT   READY   AGE
huggingface-model-7c9d5b9c9f-fghij   3         3         3       3m

# 2. Verify that the service routes to the green pods
curl -s http://ml-serving.svc.cluster.local/v1/models/my-model:predict -d '{"inputs":"Hello"}' -H "Host: green.my-company.com"

# Expected JSON response from the new model version
{
  "outputs": ["Hi there!"]
}

# 3. Observe metrics (optional)
kubectl top pods -n ml-serving -l version=green

Prevention & Best Practices

Validate resource specifications in CI. Use a linting step that fails if nvidia.com/gpu is requested on clusters without GPU nodes.
Separate GPU and CPU Deployments. Maintain distinct Deployment manifests (or Helm values) for GPU‑enabled and CPU‑only variants.
Automated readiness‑probe sanity check. Include a test that starts a pod from the built image and confirms the probe endpoint returns 200.
Blue‑green rollout sanity test. Before switching traffic, run kubectl rollout status and verify availableReplicas equals desiredReplicas for the new ReplicaSet.
Node labeling policy. Enforce that any manifest containing a nodeSelector must reference a label present on at least one node; otherwise, CI should reject the change.

FAQ

Why do Pods stay in Pending even though the cluster has free CPU and memory?
Because the scheduler also evaluates custom resources (e.g., nvidia.com/gpu) and node selectors. If a pod requests a GPU that no node provides, it remains pending regardless of other resources.
Can I use a toleration instead of a node selector to schedule GPU pods?
Yes, but you still need a node with the matching nvidia.com/gpu resource. Tolerations only bypass taints; they do not create the required resource.
My readiness probe points to the correct port, yet the pod never becomes Ready.
Check the container’s startup logs for TorchServe binding errors. A common cause is the --model-store path being missing, causing the server to exit after the probe succeeds.
How do I confirm which GPU resource a pod is actually requesting?
Run kubectl get pod <name> -o jsonpath='{.spec.containers[*].resources.requests}'. The output will list nvidia.com/gpu if present.
Is there a way to force the ReplicaSet controller to ignore unschedulable pods?
No. The controller respects the scheduler’s outcome. The proper fix is to make the pod schedulable (remove impossible requests or provide appropriate nodes).

Related Topic Hub: Model Serving Troubleshooting Hub