ONNX Runtime ReplicaSet scaling stuck in Pending after increasing replicas in Minikube

ONNX Runtime ReplicaSet scaling stuck in Pending after increasing replicas in Minikube

Problem

During local development with a Minikube cluster, the ONNX Runtime Helm chart is deployed as a standard Deployment. When the replica count is increased (e.g., from 1 to 3) the new pods never transition to Running. They remain in the Pending phase and no additional inference containers start.

Typical symptoms

  • kubectl get pods shows two pods in Running and the new ones in Pending.
  • Events for the pending pods contain messages such as “Failed to mount volume …: volume is already in use” or “0/1 nodes are available: 1 Insufficient cpu”.
  • CPU/Memory metrics on the Minikube node remain well below the limits, yet the scheduler cannot place the pods.

Root Cause

The Helm chart mounts the ONNX model file through a PersistentVolumeClaim defined with accessModes: [ReadWriteOnce]. In a Deployment the same PVC is referenced by every replica. Kubernetes allows a ReadWriteOnce volume to be mounted by only a single pod at a time on a node. When the replica count is increased, the scheduler attempts to start additional pods that need the same PVC, but the volume is already bound to the first pod. The pod therefore stays in Pending with an event similar to:


Warning  FailedMount  12s (x3 over 30s)  kubelet,
default
    MountVolume.SetUp failed for volume "onnx-model-pvc" : 
    volume is already in use by pod "onnx-runtime-6d9c7b5c5f-abcde"

Because the PVC cannot be shared, the ReplicaSet cannot scale beyond a single pod, even though the underlying node has sufficient resources.

Debug

  1. Inspect pod status and events
    kubectl get pods -n onnx-runtime -w

    Typical output:

    NAME                                 READY   STATUS    RESTARTS   AGE
    onnx-runtime-6d9c7b5c5f-abcde        1/1     Running   0          5m
    onnx-runtime-6d9c7b5c5f-fghij        0/1     Pending   0          10s
    onnx-runtime-6d9c7b5c5f-klmno        0/1     Pending   0          10s
  2. Describe a pending pod
    kubectl describe pod onnx-runtime-6d9c7b5c5f-fghij -n onnx-runtime

    Look for the Events section:

    Events:
      Type     Reason       Age   From               Message
      ----     ------       ----  ----               -------
      Warning  FailedMount  30s   kubelet, minikube  MountVolume.SetUp failed for volume "onnx-model-pvc" : volume is already in use by pod "onnx-runtime-6d9c7b5c5f-abcde"
  3. Check the PVC definition
    kubectl get pvc onnx-model-pvc -n onnx-runtime -o yaml

    Key fields:

    accessModes:
    - ReadWriteOnce
    volumeMode: Filesystem
    ...
  4. Verify node resources (optional)
    kubectl top node

    If CPU/Memory are not exhausted, the issue is not a resource shortage.

Solution

Two viable approaches:

1. Change the PVC to a shared access mode

If the underlying storage class supports ReadWriteMany, modify the PVC to use it. This allows all replicas to mount the same volume concurrently.

Before (partial Helm values snippet)

persistence:
  enabled: true
  storageClass: standard
  accessMode: ReadWriteOnce
  size: 1Gi

After

persistence:
  enabled: true
  storageClass: standard
  accessMode: ReadWriteMany   # <-- changed
  size: 1Gi

Apply the change:

helm upgrade onnx-runtime ./onnx-runtime-chart -n onnx-runtime \
  --set persistence.accessMode=ReadWriteMany

2. Switch to a StatefulSet with per‑replica PVCs

If the storage class only offers ReadWriteOnce, use a StatefulSet that creates a unique PVC for each replica via volumeClaimTemplates. This isolates the model volume per pod while preserving the same image.

Before (Deployment)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: onnx-runtime
spec:
  replicas: 1
  selector:
    matchLabels:
      app: onnx-runtime
  template:
    metadata:
      labels:
        app: onnx-runtime
    spec:
      containers:
      - name: onnx
        image: onnxruntime/server:latest
        volumeMounts:
        - name: model
          mountPath: /models
      volumes:
      - name: model
        persistentVolumeClaim:
          claimName: onnx-model-pvc

After (StatefulSet)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: onnx-runtime
spec:
  serviceName: onnx-runtime
  replicas: 3
  selector:
    matchLabels:
      app: onnx-runtime
  template:
    metadata:
      labels:
        app: onnx-runtime
    spec:
      containers:
      - name: onnx
        image: onnxruntime/server:latest
        volumeMounts:
        - name: model
          mountPath: /models
  volumeClaimTemplates:
  - metadata:
      name: model
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: standard
      resources:
        requests:
          storage: 1Gi

Deploy the StatefulSet:

kubectl apply -f onnx-runtime-statefulset.yaml -n onnx-runtime

Verify

  1. Confirm that the new pods reach Running:
    kubectl get pods -n onnx-runtime

    Expected output:

    NAME                     READY   STATUS    RESTARTS   AGE
    onnx-runtime-0           1/1     Running   0          2m
    onnx-runtime-1           1/1     Running   0          1m
    onnx-runtime-2           1/1     Running   0          1m
  2. Check that the volume is bound for each replica (StatefulSet case):
    kubectl get pvc -n onnx-runtime

    Output:

    NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    onnx-runtime-model-0   Bound    pvc-1234abcd-5678-efgh-ijkl-9012mnop   1Gi        RWO            standard       2m
    onnx-runtime-model-1   Bound    pvc-2345bcde-6789-fghi-jklm-0123nopq   1Gi        RWO            standard       1m
    onnx-runtime-model-2   Bound    pvc-3456cdef-7890-ghij-klmn-1234opqr   1Gi        RWO            standard       1m
  3. Run a quick inference request to ensure the server is functional:
    curl -X POST http://localhost:8000/v1/models/model:predict -d '{"inputs": [...]}'
    

    A successful JSON response confirms the runtime is serving.

Prevent

  • Use appropriate volume access modes: Choose ReadWriteMany for shared model files, or isolate per‑replica storage with a StatefulSet.
  • Validate PVC compatibility early: Run kubectl get storageclass -o yaml and verify that the class supports the desired accessModes before deploying.
  • Include a pre‑flight check in CI/CD that renders the Helm chart with the target replica count and runs helm template + kubectl apply --dry-run=client to catch volume‑binding conflicts.
  • Monitor pod scheduling events: Configure an alert on Pending pods with the reason FailedMount or Insufficient to surface the issue before it blocks a release.
  • Document storage expectations in the chart README: clearly state which access mode is required for scaling and which storage classes are compatible.

FAQ

  1. Why does scaling work on a cloud‑hosted cluster but not in Minikube?
    Cloud providers often supply a default storage class that supports ReadWriteMany (e.g., NFS or Azure Files). Minikube’s default standard class is backed by hostPath and only offers ReadWriteOnce, causing the mount conflict.
  2. Can I keep using a Deployment and still share the model file?
    Yes, if you replace the PVC with a ConfigMap (for small models) or a hostPath volume that is readable by all pods. Both approaches avoid the ReadWriteOnce limitation.
  3. What if my storage class truly cannot provide ReadWriteMany?
    Switch to a StatefulSet with per‑replica PVCs, or use a side‑car init container that copies the model from a shared ConfigMap into an emptyDir at pod start.
  4. How do I know which access mode a storage class supports?
    Run kubectl get sc <name> -o jsonpath='{.allowVolumeExpansion}' and inspect the parameters section; most providers document supported modes in the class description.
  5. Is there a way to automatically convert a Deployment to a StatefulSet via Helm?
    Yes, expose a boolean value (e.g., statefulSet.enabled) in the chart and use Helm conditionals to render either a Deployment or a StatefulSet with the appropriate volume template.

Related Topic Hub: Model Serving Troubleshooting Hub