A detailed red and blue robot with glowing eyes

Kubernetes storage class not found error deploying Hugging Face model

by

Problem: “StorageClass not found” when deploying a Hugging Face Transformers model

During the CI/CD pipeline a Deployment that mounts a PersistentVolumeClaim for the model artifacts fails with the following error:

error: failed to create pod sandbox: rpc error: code = Unknown desc = failed to provision volume with StorageClass "fast-ssd": storageclass.storage.k8s.io "fast-ssd" not found

The pod never reaches Running state, and the deployment is marked as ReplicaFailure. This issue blocks the rollout of the model‑serving microservice in every environment (dev, staging, prod).

Root Cause

The manifest references a StorageClass that either does not exist in the target cluster or is named incorrectly. Common underlying reasons:

  • Cluster‑level mismatch: The development cluster has a fast-ssd StorageClass, but the production cluster only provides standard and premium.
  • Namespace‑scoped resources: The StorageClass is defined in a different namespace (it is a cluster‑wide resource, but the definition was applied with --namespace, causing it to be ignored).
  • Operator not installed: The CSI driver that implements the requested class (e.g., ebs.csi.aws.com) is not deployed, so the class is never created.
  • Typo or case‑sensitivity: The manifest uses Fast-SSD while the actual class is fast-ssd.

Because the PVC controller cannot locate the class, it aborts provisioning, leading to the observed error.

Debug: Investigation Steps

  1. Confirm the PVC definition 
    cat deploy/model-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-artifacts-pvc
spec:
  storageClassName: fast-ssd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  2. List available StorageClasses in the cluster 
    kubectl get storageclass
NAME            PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard        kubernetes.io/aws-ebs      Delete          Immediate           true                    45d
premium         kubernetes.io/aws-ebs      Delete          Immediate           true                    45d

    If fast-ssd is missing, the PVC cannot be satisfied.

  3. Inspect the CSI driver deployment (if using a custom driver) 
    kubectl get pods -n kube-system -l app=ebs-csi-driver
NAME                                 READY   STATUS    RESTARTS   AGE
ebs-csi-controller-7d9f8c5c5b-9kz8b   2/2     Running   0          45d
ebs-csi-node-8k9l2                    2/2     Running   0          45d

    Missing driver pods indicate the driver was not installed.

  4. Check for typos in the manifest 
    grep -i storageclassName -R .
  5. Validate that the PVC is bound after fixing the class 
    kubectl get pvc model-artifacts-pvc -o yaml
status:
  phase: Bound

Solution: Align the PVC with an existing StorageClass

Two typical remediation paths:

Option 1 – Use an existing class (quick fix)

Replace fast-ssd with premium (or any class that matches the required performance).

# Before (model-pvc.yaml)
storageClassName: fast-ssd

# After
storageClassName: premium

Option 2 – Create the missing StorageClass (policy‑driven fix)

If the environment requires a dedicated SSD class, create it with the appropriate provisioner.

# fast-ssd-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "10"
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Apply the definition cluster‑wide:

kubectl apply -f fast-ssd-storageclass.yaml

After the class exists, the original PVC definition can stay unchanged.

Verify: Confirm that the model serving pod starts successfully

  1. Re‑apply the PVC (if it was stuck in Pending)
    2. kubectl delete pvc model-artifacts-pvc
kubectl apply -f deploy/model-pvc.yaml
  2. Watch the PVC binding
    3. kubectl get pvc model-artifacts-pvc -w
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
model-artifacts-pvc  Bound    pvc-3c9f2d1a-4b5e-4f9a-9a2d-7e1c6d9f8b2a   10Gi       RWO            premium        5s
  3. Verify the pod reaches Running
    4. kubectl get pods -l app=model-serving -w
NAME                                 READY   STATUS    RESTARTS   AGE
model-serving-7c9b5d8f5c-9kz8b        1/1     Running   0          12s
  4. Check that the model files are present inside the container
    5. kubectl exec -it model-serving-7c9b5d8f5c-9kz8b -- ls /model
config.json  pytorch_model.bin  tokenizer.json

Prevent: Guardrails to avoid future “StorageClass not found” failures

  • Cluster‑wide inventory: Maintain a ConfigMap (e.g., storage-classes) that lists allowed classes per environment. CI pipelines should validate manifests against this inventory.
  • Admission controller policy: Deploy gatekeeper or OPA policies that reject PVCs referencing unknown classes.
  • Automated tests: Include a Helm test or Kuttl test that creates a dummy PVC with the same storageClassName and asserts it becomes Bound before the full deployment runs.
  • Documentation sync: Keep the storage class naming convention documented alongside the IaC repository; enforce naming via linting tools (e.g., kube-linter).
  • Version pinning: When using a CSI driver, pin the driver version in the Helm chart and verify its StorageClass resources after upgrades.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  • Why does the error appear only after a cluster upgrade?

    Upgrades can remove or rename default StorageClasses. Verify the storageclass.kubernetes.io/is-default-class annotation after each upgrade.

  • Can I use a PVC without specifying a StorageClass?

    If a default StorageClass exists, omitting storageClassName will bind to it. Explicitly setting the field is safer for reproducibility.

  • What if I need multiple PVCs with different performance tiers?

    Create separate StorageClasses (e.g., fast-ssd, standard-hdd) and reference the appropriate one per PVC.

  • How do I debug a PVC that stays in Pending even though the StorageClass exists?

    Inspect the controller logs: kubectl logs -n kube-system -l app=csi-provisioner -c provisioner. Look for quota errors, volume limits, or driver misconfiguration.

  • Is it safe to delete a StorageClass that is currently in use?

    No. Deleting a class does not delete existing volumes, but new PVCs cannot be provisioned and existing ones may fail to expand. Remove the class only after migrating dependent PVCs.