LlamaIndex Storage Class Not Found Error During Hybrid Kubernetes Deployment

Q: Root Cause Analysis

The error originates from a mismatch between the StorageClass resource that LlamaIndex references and the actual storage class available to the Kubernetes control plane in the target cluster.

Q: Investigation and Debugging Steps 1. Verify the StorageClass exists and is cluster‑scoped

kubectl get storageclass -o wide | grep llama-index-sc Expected output:

Q: 2. Inspect the PVC created by LlamaIndex

kubectl describe pvc llama-index-pvc -n llama-index Typical error snippet:

Q: 3. Check CSI driver pods for provisioning errors

# AWS EBS CSI driver kubectl logs -l app=ebs-csi-controller -n kube-system | grep llama-index-sc # Azure Blob CSI driver kubectl logs -l app=blob-csi-controller -n kube-system | grep PermissionDenied Look for messages such as “no provisioner found for the class” or “PermissionDenied”.

In this article

Table of Contents

Problem: “Storage class not found” error during LlamaIndex index initialization in a hybrid Kubernetes deployment

LlamaIndex attempts to create a PersistentVolumeClaim (PVC) for the vector‑store backend (e.g., S3, Azure Blob) using the storage_class_name supplied in its configuration. In a mixed on‑prem / cloud environment the pod fails to start and the following error appears in the LlamaIndex logs:


2024-07-12T14:23:07Z ERROR VectorStore initialization failed: unable to locate persistent storage backend defined by storage_class_name "llama-index-sc"
kubectl get pvc llama-index-pvc -n llama-index
Error from server (NotFound): persistentvolumeclaims "llama-index-pvc" not found

Consequences include:

Index creation aborts, causing data‑persistence failures.
Subsequent queries fall back to an in‑memory store, losing durability.
Deployment pipelines stall waiting for the PVC to become bound.

Root Cause Analysis

The error originates from a mismatch between the StorageClass resource that LlamaIndex references and the actual storage class available to the Kubernetes control plane in the target cluster.

Key factors identified from real incidents and community reports:

Namespace scoping: StorageClass is a cluster‑scoped object, but some administrators mistakenly create it as a namespaced custom resource (e.g., via a Helm chart that scopes it to llama-index). The PVC therefore cannot resolve the name, yielding “storageclass not found”.
Inconsistent names across clusters: Hybrid deployments often reuse the same Helm values file. An automated upgrade renamed the class from llama-index-sc to llama-index-sc-v2 without updating the LlamaIndex config, reproducing the error on both AWS and Azure clusters (see “Automated Helm chart upgrade” incident).
Missing CSI driver provisioner: On‑prem clusters using Ceph CSI lacked the provisioner field that matches the driver name (e.g., ceph.com/rbd). Kubernetes logs reported “no provisioner found for the class”. The same symptom appears in the AWS EBS issue #1275.
IAM / role misconfiguration for cloud CSI drivers: The S3 CSI driver on the on‑prem cluster could not authenticate, causing the driver to reject the StorageClass definition and Kubernetes to treat it as undefined (see “Incorrect IAM role binding” incident).
Version incompatibility: Azure Blob CSI driver version v0.5.0 on Kubernetes v1.24 ignored unknown fields, effectively dropping the StorageClass definition (see “Version mismatch” incident).

In all cases the underlying reason is that the StorageClass object that LlamaIndex expects either does not exist, is not recognized by a provisioner, or is hidden by RBAC/namespace constraints.

Investigation and Debugging Steps

1. Verify the StorageClass exists and is cluster‑scoped

kubectl get storageclass -o wide | grep llama-index-sc

Expected output:

NAME               PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
llama-index-sc     ebs.csi.aws.com            Delete          Immediate           true                    12d

If the command returns nothing, the class is missing or misnamed.

2. Inspect the PVC created by LlamaIndex

kubectl describe pvc llama-index-pvc -n llama-index

Typical error snippet:

Events:
  Type    Reason                Age   From               Message
  ----    ------                ----  ----               -------
  Normal  ProvisioningFailed    2m    persistentvolume-controller  storageclass.storage.k8s.io "llama-index-sc" not found

3. Check CSI driver pods for provisioning errors

# AWS EBS CSI driver
kubectl logs -l app=ebs-csi-controller -n kube-system | grep llama-index-sc

# Azure Blob CSI driver
kubectl logs -l app=blob-csi-controller -n kube-system | grep PermissionDenied

Look for messages such as “no provisioner found for the class” or “PermissionDenied”.

4. Validate IAM / RBAC bindings (cloud drivers)

# Example: verify ServiceAccount used by the driver has the required IAM role
aws iam get-role --role-name eks-s3-csi-driver-role
# Confirm policy includes s3:PutObject, s3:GetObject on the target bucket

5. Confirm Helm values and version compatibility

helm get values llama-index -n llama-index
# Look for storageClassName field
# Verify chart version matches CSI driver version requirements (see AWS EKS docs)

Resolution

Step‑by‑step fix for the most common scenario (renamed StorageClass)

Re‑apply the expected StorageClass (or rename the existing one). Example for AWS EBS:

# before (missing or renamed)
# kubectl get storageclass llama-index-sc   # returns NotFound

# after: create the class
cat > llama-index-sc.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: llama-index-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
EOF

kubectl apply -f llama-index-sc.yaml

Update LlamaIndex configuration if the class name differs (e.g., llama-index-sc-v2).

# llama_index_config.yaml
vector_store:
  type: s3
  storage_class_name: llama-index-sc   # ensure matches the SC name
  bucket: my-llama-index-bucket
  region: us-east-1

Restart the LlamaIndex pod to pick up the new PVC binding:

kubectl rollout restart deployment llama-index -n llama-index

Verify the PVC becomes bound:

kubectl get pvc -n llama-index
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
llama-index-pvc    Bound    pvc-1234abcd-5678-efgh-9012-ijklmnopqrstu   10Gi       RWO            llama-index-sc   30s

Alternative fixes for other root causes

Root Cause	Fix
CSI driver not installed or mismatched provisioner name	Install the correct driver (e.g., `helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver`) and ensure `provisioner` field matches driver name.
IAM role missing S3 permissions	Attach `AmazonS3FullAccess` (or least‑privilege policy) to the driver’s ServiceAccount role.
Namespace‑scoped StorageClass (custom resource)	Re‑create the StorageClass as a cluster‑scoped object (remove `metadata.namespace`).
Version incompatibility (Azure Blob CSI)	Upgrade Azure Blob CSI driver to a version that supports the Kubernetes API version (e.g., v0.9.0 for k8s 1.24).

Validation

After applying the fix, perform the following checks:

Confirm the PVC status:

kubectl get pvc -n llama-index -o jsonpath='{.items[0].status.phase}'
Bound

Inspect LlamaIndex pod logs for successful vector‑store initialization:

kubectl logs deployment/llama-index -n llama-index | grep "VectorStore initialized"
2024-07-12T14:24:12Z INFO VectorStore initialized using storage class "llama-index-sc"

Run a simple index write/read test (Python example):

from llama_index import VectorStore, SimpleDirectoryReader

store = VectorStore.from_config({
    "type": "s3",
    "bucket": "my-llama-index-bucket",
    "region": "us-east-1",
    "storage_class_name": "llama-index-sc"
})
doc = SimpleDirectoryReader("data").load_data()[0]
store.add_documents([doc])
print("Document persisted, count:", store.count())

Expected output shows a non‑zero count and no exceptions.

Operational Experience and Lessons Learned

Misleading symptom: The LlamaIndex error mentions “storage class not found”, but the underlying issue can be an IAM permission error that prevents the CSI driver from provisioning the volume. Always check driver logs.
Cross‑cluster naming consistency: Using a single Helm values file across on‑prem and cloud clusters is convenient, but any drift (e.g., a chart upgrade that renames the class) propagates instantly. Pin the storage_class_name per‑environment or use a templating tool that updates both sides.
Namespace confusion: Although StorageClass is cluster‑scoped, some Helm charts mistakenly create it inside a namespace. This silently creates a custom resource that never matches the PVC request, leading to the “not found” error.
Provisioner visibility: The Kubernetes controller manager logs “no provisioner found for the class” before the pod logs appear. Checking kubectl describe storageclass for the provisioner field can quickly rule out driver mismatches.

Best Practices and Prevention

Maintain a storage-classes.yaml manifest in version control and apply it before any Helm chart that references it.
Use distinct names per driver (e.g., aws-s3-sc, azure-blob-sc) and map them in the LlamaIndex config via environment‑specific overrides.
Enable storageclass.kubernetes.io/is-default-class only for a single class per cluster to avoid accidental PVC binding to the wrong driver.

Automate validation in CI/CD pipelines:

# Verify StorageClass exists
kubectl get storageclass ${SC_NAME} --no-headers || exit 1
# Verify driver pods are healthy
kubectl get pods -l app=${DRIVER_NAME} -n kube-system -o jsonpath='{.items[*].status.containerStatuses[0].ready}'
true

Configure alerts on PVC pending state for longer than 2 minutes:

alert: PVCStuckPending
expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} > 120
for: 2m
labels:
  severity: warning
annotations:
  summary: "PVC {{ $labels.persistentvolumeclaim }} in namespace {{ $labels.namespace }} is pending"

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the error appear only on the on‑prem cluster?
Because the on‑prem cluster uses a different CSI driver (Ceph) and the StorageClass was defined for the AWS EBS driver only. The PVC request cannot be satisfied, resulting in “storageclass not found”. Deploy a matching Ceph StorageClass or use a cloud‑agnostic provisioner.
Can I use a single StorageClass name for both S3 and Azure Blob?
Yes, if the underlying CSI drivers share the same provisioner identifier, but it is safer to keep separate names (e.g., s3-sc, azure-blob-sc) to avoid cross‑driver confusion and to simplify troubleshooting.
How do I know which provisioner a StorageClass uses?
Inspect the provisioner field:
```
kubectl get storageclass llama-index-sc -o yaml | grep provisioner
provisioner: ebs.csi.aws.com
```
Match this value with the CSI driver documentation (AWS EKS “Using Amazon S3 with CSI driver”, Azure “Blob CSI driver”).
What if the CSI driver logs “no provisioner found for the class”?
The driver either is not installed, is failing to start, or the provisioner field in the StorageClass does not match the driver’s name. Re‑install the driver and ensure the StorageClass definition aligns with the driver’s provisioner string.
Is it safe to set allowVolumeExpansion: true for vector‑store PVCs?
Yes. Vector stores can grow as new embeddings are added. Ensure the underlying storage (e.g., S3 bucket) has no hard quota limits, and update the PVC size when needed:
```
kubectl patch pvc llama-index-pvc -n llama-index -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
```