ONNX Runtime ReplicaSet scaling stuck in Pending after increasing replicas in Minikube
Problem
During local development with a Minikube cluster, the ONNX Runtime Helm chart is deployed as a standard Deployment. When the replica count is increased (e.g., from 1 to 3) the new pods never transition to Running. They remain in the Pending phase and no additional inference containers start.
Typical symptoms
kubectl get podsshows two pods inRunningand the new ones inPending.- Events for the pending pods contain messages such as “Failed to mount volume …: volume is already in use” or “0/1 nodes are available: 1 Insufficient cpu”.
- CPU/Memory metrics on the Minikube node remain well below the limits, yet the scheduler cannot place the pods.
Root Cause
The Helm chart mounts the ONNX model file through a PersistentVolumeClaim defined with accessModes: [ReadWriteOnce]. In a Deployment the same PVC is referenced by every replica. Kubernetes allows a ReadWriteOnce volume to be mounted by only a single pod at a time on a node. When the replica count is increased, the scheduler attempts to start additional pods that need the same PVC, but the volume is already bound to the first pod. The pod therefore stays in Pending with an event similar to:
Warning FailedMount 12s (x3 over 30s) kubelet,
default
MountVolume.SetUp failed for volume "onnx-model-pvc" :
volume is already in use by pod "onnx-runtime-6d9c7b5c5f-abcde"
Because the PVC cannot be shared, the ReplicaSet cannot scale beyond a single pod, even though the underlying node has sufficient resources.
Debug
- Inspect pod status and events
kubectl get pods -n onnx-runtime -wTypical output:
NAME READY STATUS RESTARTS AGE onnx-runtime-6d9c7b5c5f-abcde 1/1 Running 0 5m onnx-runtime-6d9c7b5c5f-fghij 0/1 Pending 0 10s onnx-runtime-6d9c7b5c5f-klmno 0/1 Pending 0 10s - Describe a pending pod
kubectl describe pod onnx-runtime-6d9c7b5c5f-fghij -n onnx-runtimeLook for the
Eventssection:Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 30s kubelet, minikube MountVolume.SetUp failed for volume "onnx-model-pvc" : volume is already in use by pod "onnx-runtime-6d9c7b5c5f-abcde" - Check the PVC definition
kubectl get pvc onnx-model-pvc -n onnx-runtime -o yamlKey fields:
accessModes: - ReadWriteOnce volumeMode: Filesystem ... - Verify node resources (optional)
kubectl top nodeIf CPU/Memory are not exhausted, the issue is not a resource shortage.
Solution
Two viable approaches:
1. Change the PVC to a shared access mode
If the underlying storage class supports ReadWriteMany, modify the PVC to use it. This allows all replicas to mount the same volume concurrently.
Before (partial Helm values snippet)
persistence:
enabled: true
storageClass: standard
accessMode: ReadWriteOnce
size: 1Gi
After
persistence:
enabled: true
storageClass: standard
accessMode: ReadWriteMany # <-- changed
size: 1Gi
Apply the change:
helm upgrade onnx-runtime ./onnx-runtime-chart -n onnx-runtime \
--set persistence.accessMode=ReadWriteMany
2. Switch to a StatefulSet with per‑replica PVCs
If the storage class only offers ReadWriteOnce, use a StatefulSet that creates a unique PVC for each replica via volumeClaimTemplates. This isolates the model volume per pod while preserving the same image.
Before (Deployment)
apiVersion: apps/v1
kind: Deployment
metadata:
name: onnx-runtime
spec:
replicas: 1
selector:
matchLabels:
app: onnx-runtime
template:
metadata:
labels:
app: onnx-runtime
spec:
containers:
- name: onnx
image: onnxruntime/server:latest
volumeMounts:
- name: model
mountPath: /models
volumes:
- name: model
persistentVolumeClaim:
claimName: onnx-model-pvc
After (StatefulSet)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: onnx-runtime
spec:
serviceName: onnx-runtime
replicas: 3
selector:
matchLabels:
app: onnx-runtime
template:
metadata:
labels:
app: onnx-runtime
spec:
containers:
- name: onnx
image: onnxruntime/server:latest
volumeMounts:
- name: model
mountPath: /models
volumeClaimTemplates:
- metadata:
name: model
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: standard
resources:
requests:
storage: 1Gi
Deploy the StatefulSet:
kubectl apply -f onnx-runtime-statefulset.yaml -n onnx-runtime
Verify
- Confirm that the new pods reach
Running:kubectl get pods -n onnx-runtimeExpected output:
NAME READY STATUS RESTARTS AGE onnx-runtime-0 1/1 Running 0 2m onnx-runtime-1 1/1 Running 0 1m onnx-runtime-2 1/1 Running 0 1m - Check that the volume is bound for each replica (StatefulSet case):
kubectl get pvc -n onnx-runtimeOutput:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE onnx-runtime-model-0 Bound pvc-1234abcd-5678-efgh-ijkl-9012mnop 1Gi RWO standard 2m onnx-runtime-model-1 Bound pvc-2345bcde-6789-fghi-jklm-0123nopq 1Gi RWO standard 1m onnx-runtime-model-2 Bound pvc-3456cdef-7890-ghij-klmn-1234opqr 1Gi RWO standard 1m - Run a quick inference request to ensure the server is functional:
curl -X POST http://localhost:8000/v1/models/model:predict -d '{"inputs": [...]}'A successful JSON response confirms the runtime is serving.
Prevent
- Use appropriate volume access modes: Choose
ReadWriteManyfor shared model files, or isolate per‑replica storage with aStatefulSet. - Validate PVC compatibility early: Run
kubectl get storageclass -o yamland verify that the class supports the desiredaccessModesbefore deploying. - Include a pre‑flight check in CI/CD that renders the Helm chart with the target replica count and runs
helm template+kubectl apply --dry-run=clientto catch volume‑binding conflicts. - Monitor pod scheduling events: Configure an alert on
Pendingpods with the reasonFailedMountorInsufficientto surface the issue before it blocks a release. - Document storage expectations in the chart README: clearly state which access mode is required for scaling and which storage classes are compatible.
FAQ
- Why does scaling work on a cloud‑hosted cluster but not in Minikube?
Cloud providers often supply a default storage class that supportsReadWriteMany(e.g., NFS or Azure Files). Minikube’s defaultstandardclass is backed by hostPath and only offersReadWriteOnce, causing the mount conflict. - Can I keep using a Deployment and still share the model file?
Yes, if you replace the PVC with aConfigMap(for small models) or ahostPathvolume that is readable by all pods. Both approaches avoid theReadWriteOncelimitation. - What if my storage class truly cannot provide ReadWriteMany?
Switch to aStatefulSetwith per‑replica PVCs, or use a side‑car init container that copies the model from a sharedConfigMapinto anemptyDirat pod start. - How do I know which access mode a storage class supports?
Runkubectl get sc <name> -o jsonpath='{.allowVolumeExpansion}'and inspect theparameterssection; most providers document supported modes in the class description. - Is there a way to automatically convert a Deployment to a StatefulSet via Helm?
Yes, expose a boolean value (e.g.,statefulSet.enabled) in the chart and use Helm conditionals to render either aDeploymentor aStatefulSetwith the appropriate volume template.
Related Topic Hub: Model Serving Troubleshooting Hub