A book page with printed text and blank lined paper

TGI checkpoint storage class not found after A/B deployment

by

TGI checkpoint storage class not found after A/B deployment

Problem

During an A/B testing rollout of Text Generation Inference (TGI) instances, the newly created replica fails to start and logs an error similar to:


Failed to load checkpoint: storage class "fast-ssd" not found

Other observed symptoms include:

  • Pod tgi-a-12345 stays in Pending state.
  • kubectl describe pod tgi-a-12345 shows an event:

Events:
  Type     Reason       Age   From               Message
  ----     ------       ----  ----               -------
  Warning  FailedMount  2m    kubelet, node-1    mounting failed: storage class not found

In parallel, the “B” version continues to serve traffic, masking the failure until a health‑check alerts the SRE team.

Root Cause Analysis

The error originates from the TGI deployment guide – Configuring checkpoint storage. TGI expects a PersistentVolumeClaim that references a valid StorageClass. In an A/B testing scenario the following patterns commonly break this contract:

  1. Namespace‑scoped StorageClass omission – A custom class such as fast-ssd was created only in the production namespace. The new Helm release for version “A” was installed in testing, where the class does not exist.
  2. Helm value overwrite – The values.yaml of the second release unintentionally overwrote the shared storageClass definition (see the Helm chart README). The second release then points to a class that was deleted during the upgrade.
  3. Cluster upgrade default change – After a Kubernetes upgrade the default class renamed from standard to standard-rwo. Existing TGI manifests still referenced standard, causing the “storage class not found” error.
  4. Race condition on PVC creation – When two TGI releases request PVCs simultaneously, the controller may attempt to bind the second PVC before the StorageClass CRD is fully reconciled, leading to rejection (see real incident “Concurrent PVC creation for two TGI versions triggered a race condition”).

All these scenarios violate the assumption documented in the official guide that the storage class referenced by spec.storageClassName must be resolvable cluster‑wide at pod creation time.

Investigation and Debugging

Follow these steps to pinpoint the missing class:

  1. Inspect the pod events and PVC status:

kubectl describe pod tgi-a-12345
kubectl get pvc -n testing tgi-model-pvc -o yaml

Typical output indicating the problem:


status:
  phase: Pending
  conditions:
  - type: "Failed"
    reason: "StorageClassNotFound"
    message: "storage class \"fast-ssd\" does not exist"
  1. List available StorageClasses and verify their scope:

kubectl get storageclass


NAME          PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
fast-ssd      kubernetes.io/aws-ebs  Delete          WaitForFirstConsumer   true                45d
standard      kubernetes.io/gce      Delete          Immediate               false               120d

If the expected class is absent, it confirms the root cause.

  1. Check Helm release values for both A and B releases:

helm get values tgi-a -n testing
helm get values tgi-b -n testing

Look for the storageClass key. A common misconfiguration is:


storageClass: fast-ssd   # missing quotes, overridden by later release
  1. Validate that the StorageClass CRD is ready:

kubectl get storageclass fast-ssd -o yaml

If the object exists but the controller has not yet set the provisioner field, you may see a status: {} block, indicating a race condition.

Resolution

Apply the fix that matches the identified cause. Below are three representative solutions.

1. Create or re‑expose the missing StorageClass in the target namespace

Before (missing class):


# No fast-ssd StorageClass in the cluster

After:


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "10"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Apply with:


kubectl apply -f fast-ssd-sc.yaml

2. Align Helm values with the existing class and prevent overwrites

Before (Helm values causing overwrite):


# values.yaml of release B
storageClass: fast-ssd
# later in the same file, a default block resets it
defaultStorageClass: standard

After (explicit, immutable definition):


# values.yaml of both releases
storageClass: fast-ssd
# do not define defaultStorageClass in the same chart

Upgrade the releases:


helm upgrade tgi-a ./charts/tgi -n testing -f values-a.yaml
helm upgrade tgi-b ./charts/tgi -n testing -f values-b.yaml

3. Update manifests to reference the new default class after a cluster upgrade

Before (out‑of‑date reference):


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tgi-model-pvc
spec:
  storageClassName: standard   # no longer exists
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

After (point to the renamed class):


spec:
  storageClassName: standard-rwo

Apply the corrected PVC:


kubectl apply -f pvc-updated.yaml

Verification

Confirm that the pod transitions to Running and that TGI can load the checkpoint:


kubectl get pod tgi-a-12345 -n testing
kubectl logs tgi-a-12345 -n testing | grep "Loading checkpoint"

Expected log snippet:


[INFO] Loading checkpoint from /model-checkpoint/...
[INFO] Checkpoint loaded successfully, model ready to serve.

Additionally, verify PVC binding:


kubectl get pvc tgi-model-pvc -n testing

Output should show Bound status.

Prevention and Best Practices

  • Namespace‑agnostic StorageClass: Create custom classes at the cluster level, not per‑namespace, to avoid accidental omission.
  • Helm value pinning: Store the storage class name in a dedicated ConfigMap or use --set storageClass=fast-ssd on each release to prevent accidental overrides.
  • Post‑upgrade reconciliation: After a Kubernetes version bump, run a script that validates all PersistentVolumeClaim objects against existing StorageClass names.
  • Automated health checks: Add a readiness probe that checks the existence of the checkpoint directory; alert on failures before traffic is routed.
  • Race‑condition mitigation: Serialize PVC creation for parallel TGI releases using a Helm hook that waits for the StorageClass to reach Ready status.

FAQ

  1. Why does the error mention a storage class that I never defined?
    The default storageClassName in the TGI chart is standard. If the cluster’s default class was renamed (e.g., to standard-rwo) the chart still emits the old name, leading to the “not found” error.
  2. Can I use a different StorageClass for each A/B variant?
    Yes, but each variant must reference a class that exists in the target namespace. Declare distinct class names in each Helm values file and ensure the corresponding StorageClass objects are created before the PVCs.
  3. How do I know whether a PVC is stuck because of a missing StorageClass or because of insufficient quota?
    Inspect the PVC’s status.conditions. A missing class yields reason: StorageClassNotFound, while quota issues show reason: InsufficientQuota. The kubectl describe pvc output differentiates the two.
  4. Is it safe to delete a StorageClass that is still referenced by a running TGI pod?
    No. Deleting the class does not affect already bound PVs, but new PVCs (including those created by rolling updates) will fail, causing pod restarts. Remove references first or migrate PVCs to a new class.
  5. What monitoring metric should I watch to catch this early?
    Track kube_pod_status_phase{phase="Pending"} combined with a label filter for pods containing tgi-. Spike in pending pods correlated with “FailedMount” events indicates storage class issues.

Related Topic Hub: Model Serving Troubleshooting Hub