photo of computer cables

GPT-3.5 checkpoint backup fails with storage class not found during DR

by

Problem

During a disaster‑recovery (DR) run‑book, the automated backup job that persists GPT‑3.5 checkpoint files to the configured backup bucket fails with the following error:


2026-11-05T14:32:07Z ERROR backup_job: storage class not found: "fast-ssd-backup"
2026-11-05T14:32:07Z INFO  backup_job: aborting checkpoint upload for model "gpt-3.5"

The failure occurs only in the DR region; the same backup pipeline works normally in the primary region.

Root Cause

The backup job runs inside a Kubernetes Job that references a StorageClass by name. In the primary region the StorageClass fast-ssd-backup exists and points to a provisioner that creates gs:// buckets with the required performance tier. In the DR region the StorageClass object was never created because the infrastructure‑as‑code (IaC) manifest that defines it was scoped to the primary cluster only. When the backup job starts, the CSI driver queries the API server for the class, receives a NotFound response, and aborts the upload.

Additional contributing factors:

  • The backup job uses the dynamicProvisioning=true flag, which forces the driver to create a new PersistentVolumeClaim (PVC) at runtime. Without a matching StorageClass, the PVC remains in Pending state.
  • Alerting rules only fire on PVC creation failures, not on the higher‑level backup job error, so the issue went unnoticed until the DR drill.
  • The DR cluster runs a different version of the CSI driver (v1.4 vs v1.2) that logs the error as “storage class not found” rather than “PVC pending”. This change masked the symptom.

Debug

Step‑by‑step investigation that reproduced the failure:

  1. Inspect the backup job pod logs (already shown above).
  2. Check the PVC created by the job:

$ kubectl -n backup get pvc -l job-name=gpt35-backup-20231105 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gpt35-checkpoint-pvc
  namespace: backup
  labels:
    job-name: gpt35-backup-20231105
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Gi
  storageClassName: fast-ssd-backup
status:
  phase: Pending

The PVC remains Pending because the storage class cannot be resolved.

3. Verify the existence of the storage class in the DR cluster:


$ kubectl get sc fast-ssd-backup
Error from server (NotFound): storageclass.storage.k8s.io "fast-ssd-backup" not found

4. Review the IaC repository for the storage class definition:


# storage-classes/fast-ssd-backup.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd-backup
provisioner: com.google.cloud.storage.csi
parameters:
  type: pd-ssd
  replication-type: regional-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate

The manifest exists but is only applied to the primary cluster via a kustomize overlay.

Solution

Make the storage class definition part of the DR cluster’s provisioning pipeline and add a fallback to a known‑good class.

Step 1 – Add the storage class to the DR overlay


# kustomization.yaml (DR overlay)
resources:
- ../../base
- ../storage-classes/fast-ssd-backup.yaml   # ← new line

Step 2 – Deploy the updated manifest


$ kubectl apply -k overlays/dr
storageclass.storage.k8s.io/fast-ssd-backup configured

Step 3 – Update the backup job to use a fallback class

Modify the job spec to reference a variable that resolves to the primary class if present, otherwise to standard-backup (a class guaranteed to exist in all regions).


# backup-job.yaml (excerpt)
apiVersion: batch/v1
kind: Job
metadata:
  name: gpt35-backup
spec:
  template:
    spec:
      containers:
      - name: backup
        image: myorg/gpt-backup:1.2.0
        env:
        - name: STORAGE_CLASS
          valueFrom:
            configMapKeyRef:
              name: backup-config
              key: storageClass   # now populated by a ConfigMap
        volumeMounts:
        - name: checkpoint
          mountPath: /data/checkpoint
      volumes:
      - name: checkpoint
        persistentVolumeClaim:
          claimName: gpt35-checkpoint-pvc
          storageClassName: $(STORAGE_CLASS)

Update the ConfigMap to include the fallback logic (rendered by Helm or Kustomize):


# backup-config.yaml (template)
apiVersion: v1
kind: ConfigMap
metadata:
  name: backup-config
data:
  storageClass: {{ if .Values.storageClass }}{{ .Values.storageClass }}{{ else }}standard-backup{{ end }}

Step 4 – Redeploy the backup job


$ helm upgrade --install gpt-backup ./chart \
  --set storageClass=fast-ssd-backup \
  --namespace backup

Verify

Confirm that the PVC binds and the backup completes:


$ kubectl -n backup get pvc gpt35-checkpoint-pvc -o jsonpath='{.status.phase}'
Bound

Check the job logs for successful upload:


2026-11-05T14:38:12Z INFO  backup_job: checkpoint uploaded to gs://dr-backup-bucket/gpt-3.5/20231105.ckpt
2026-11-05T14:38:12Z INFO  backup_job: backup job completed successfully

Validate the object exists in the DR bucket:


$ gsutil ls gs://dr-backup-bucket/gpt-3.5/20231105.ckpt
gs://dr-backup-bucket/gpt-3.5/20231105.ckpt

Prevent

  • IaC consistency checks: Add a CI pipeline step that runs kubectl get sc against all target clusters and fails if any expected class is missing.
  • Cross‑region ConfigMap defaults: Store the default storage class name in a ConfigMap that is replicated via kubectl apply --server-side to all clusters.
  • Alert on PVC Pending: Extend Prometheus rules to fire when any PVC in the backup namespace stays in Pending for more than 5 minutes.
  • Version‑agnostic logging: Normalize backup job error messages to a common schema (e.g., {error_code:"STORAGE_CLASS_NOT_FOUND"}) so that alerting does not depend on driver version.
  • Disaster‑recovery test checklist: Include a step that verifies the presence of all storage classes before initiating DR backups.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  • Why does the backup succeed in the primary region but fail in DR? The primary cluster has the fast-ssd-backup StorageClass defined; the DR cluster never received that manifest, so the PVC cannot be provisioned.
  • Can I use a static bucket instead of dynamic provisioning? Yes. Replace the PVC with a direct gsutil cp call and remove the storageClassName reference, but you lose the benefits of CSI‑managed lifecycle and automatic cleanup.
  • What if the CSI driver version changes again? Keep error handling generic (e.g., check for PVC Pending status) and rely on alerting rather than parsing driver‑specific log strings.
  • Is there a way to automatically copy missing StorageClasses to new clusters? Use a GitOps operator (Argo CD, Flux) that watches a central storage-classes directory and syncs it to every cluster in the fleet.
  • How do I test the backup job without affecting production data? Deploy the job in a dedicated namespace with a mock bucket (e.g., gs://test-backup-bucket) and set dryRun=true in the container’s environment variables.