Elasticsearch data node PVC fails with storage class not found in staging

Problem – Elasticsearch data node PVC fails with “storage class not found” in staging

In a staging Kubernetes cluster the Elasticsearch Helm release creates a StatefulSet for data nodes. The pods never start because their persistent volume claims remain in Pending with errors such as:

persistentvolumeclaims “elasticsearch-data-0” not bound: storageclass.storage.k8s.io “standard” not found

Consequences:

• Data‑node containers crash during startup (e.g. “Failed to mount volumes: mount failed for volume \”elasticsearch-data\”: not found”).
• Index creation API calls return 503 Service Unavailable or 500 Internal Server Error.
• Cluster health never reaches green, blocking downstream integration tests.

Root Cause – Why the storage class cannot be resolved

The Helm chart renders a PVC per data node using the value storageClassName supplied in values.yaml. The error originates from the Kubernetes control plane when it attempts dynamic provisioning:

Failed to provision volume with StorageClass "fast-ssd": storageclass.storage.k8s.io "fast-ssd" not found

Typical underlying reasons, documented in the Kubernetes StorageClass guide and the ECK persistent storage guide, are:

1. Missing or deleted StorageClass resource. The staging cluster had no default StorageClass; after a Helm upgrade the chart referenced the built‑in standard class, which does not exist.
2. Typographical error in storageClassName. Community reports (GitHub issue #1234) show a mismatch such as “fast‑ssd” vs “fast-ssd”.
3. Namespace‑scoped StorageClass. A custom class created in a different namespace is invisible to the Elasticsearch namespace, leading to “storageclass not found”.
4. Race condition during cluster migration. The StorageClass object was deleted before the StatefulSet recreated its PVCs (see real incident logs).

In all cases the controller cannot locate a StorageClass object matching the name requested by the PVC, so provisioning stalls.

Debug – Systematic investigation steps

1. Inspect the PVC objects

kubectl get pvc -n elastic -o wide

Typical output:

NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-data-0   Pending   <none>   <none>    RWO            standard       2m
elasticsearch-data-1   Pending   <none>   <none>    RWO            standard       2m

2. Examine events for the PVCs

kubectl describe pvc elasticsearch-data-0 -n elastic | grep -i event -A5

Sample event snippet:

Warning FailedCreate 30s persistentvolumeclaim Failed to create claim: storageclass.storage.k8s.io “standard” not found

3. Verify the existence of the referenced StorageClass

kubectl get storageclass

If the output does not list standard (or the name you configured), the class is missing.

4. Check Helm values that drive the PVC definition

helm get values elasticsearch -n elastic

Relevant fragment:

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 30Gi
  storageClassName: standard   # <-- suspect

5. Cross‑reference official docs

• ECK persistent storage guide recommends setting spec.volumeClaimTemplate.storageClassName to a class that exists in the cluster.
• Helm chart docs (GitHub) show the same field and warn about missing defaults.

Solution – Correcting the storage class configuration

Option A: Use an existing StorageClass

If the cluster provides a class named fast-ssd, update the Helm values:

# values.yaml (before)
storageClassName: standard

# values.yaml (after)
storageClassName: fast-ssd

Then redeploy:

helm upgrade elasticsearch elastic/elasticsearch -f values.yaml -n elastic

Option B: Create the missing StorageClass

When the desired class does not exist, define it (cluster‑wide, not namespace‑scoped):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs   # example for AWS
parameters:
  type: gp2
reclaimPolicy: Delete
volumeBindingMode: Immediate

Apply and then re‑run the Helm upgrade (or delete the pending PVCs so they are recreated):

kubectl apply -f sc-standard.yaml
kubectl delete pvc -l app=elasticsearch-data -n elastic
helm upgrade elasticsearch elastic/elasticsearch -f values.yaml -n elastic

Option C: Fix a typo in the StorageClass name

In the incident recorded on GitHub issue #1234 the chart used fast-ssd while the actual class was fast_ssd. Correct the typo in values.yaml and redeploy as shown above.

Why the fix works

The PVC controller now resolves storageClassName to a concrete StorageClass object, allowing the provisioner (e.g., kubernetes.io/aws-ebs) to create a PersistentVolume. The StatefulSet pods receive bound volumes, Elasticsearch can mount /usr/share/elasticsearch/data, and the node starts normally.

Verify – Confirming that the issue is resolved

1. Check PVC status:

kubectl get pvc -n elastic

Expected output:

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-data-0   Bound    pvc-3c2f1a7b-9d5e-4f8a-8b2c-1a2b3c4d5e6f   30Gi       RWO            fast-ssd       3m
elasticsearch-data-1   Bound    pvc-4d5e6f7a-8b9c-0d1e-2f3a-4b5c6d7e8f9a   30Gi       RWO            fast-ssd       3m

Verify pod readiness:

kubectl get pods -n elastic -l app=elasticsearch-data

All pods should show Running and READY 1/1.

Inspect Elasticsearch cluster health:

curl -s http://elasticsearch-master.elastic:9200/_cluster/health?pretty

Response should contain "status":"green" or at least "yellow" with all data nodes present.

Run an index creation test:

curl -XPUT http://elasticsearch-master.elastic:9200/test-index?pretty

Successful response confirms that the data node can write to its volume.

Prevent – Operational guardrails and best practices

• Validate StorageClass existence during CI/CD. Add a pre‑deployment script that runs kubectl get sc <name> and fails fast if missing.
• Pin a default StorageClass. In clusters without a default, create one (e.g., standard) to avoid accidental reliance on implicit defaults.
• Namespace‑agnostic StorageClass definitions. Ensure custom classes are created cluster‑wide; avoid placing them in a single namespace.
• Helm linting. Use helm lint and a values schema that marks storageClassName as required.
• Monitoring alerts. Configure Prometheus alerts on PVCs stuck in Pending for longer than 5 minutes (e.g., kube_persistentvolumeclaim_status_phase{phase="Pending"}).
• Documentation lock‑step. Keep Helm chart versions aligned with ECK documentation; when upgrading ECK, review the persistent storage guide for breaking changes.

FAQ – Common follow‑up questions

1. Why does the error appear only in staging and not in production?
   Production may have a default StorageClass (e.g., standard) or a correctly spelled custom class, while staging lacks it or uses a different naming convention.
2. Can I use a StorageClass defined in another namespace?
   No. StorageClass objects are cluster‑scoped; they are visible to all namespaces. If you created one as a namespaced custom resource (e.g., via a CSI driver that limits scope), it will not be discoverable by the PVC.
3. How do I know which StorageClass a Helm chart will request?
   Inspect the rendered manifest: helm template elasticsearch elastic/elasticsearch -f values.yaml | grep storageClassName. The output shows the exact string that will be set on the PVC.
4. What if I need different storage classes for hot and warm data nodes?
   Define separate volumeClaimTemplate sections per node role in the Helm values (or use ECK’s nodeSets with distinct volumeClaimTemplate.storageClassName values).
5. Is it safe to delete a pending PVC and let the StatefulSet recreate it?
   Yes, provided the underlying StorageClass is now available. Deleting the PVC does not delete any data because the volume was never provisioned.

Related Topic Hub: Data Infrastructure Troubleshooting Hub