Weaviate Persistent Volume Mount Failure In Hybrid Kubernetes Cluster

Q: Root Cause Analysis

The failure originates from a mismatch between the StorageClass provisioned for the PVC and the node where the pod is scheduled:

Q: Resolution

Apply the following changes to the Helm values file (values.yaml) and redeploy.

In this article

Table of Contents

Weaviate Persistent Volume Mount Failure in Hybrid Kubernetes Cluster

Problem Description

When deploying Weaviate via the official Helm chart in a hybrid Kubernetes cluster (on‑premises nodes mixed with AWS or Azure cloud nodes), the Weaviate pods get stuck in Init:CreateContainerConfigError or ContainerCreating state. The kubelet logs contain errors such as:


MountVolume.SetUp failed for volume "weaviate-data" : access denied by filesystem
MountVolume.SetUp failed for volume "weaviate-data" : no such file or directory

Symptoms observed:

Pod never reaches Running phase.
Helm install completes without error, but kubectl get pods shows the above mount failures.
Only a subset of nodes (usually the cloud nodes) exhibit the problem; on‑prem nodes succeed.

Root Cause Analysis

The failure originates from a mismatch between the StorageClass provisioned for the PVC and the node where the pod is scheduled:

Mixed storage backends – On‑prem nodes use an nfs‑based StorageClass, while cloud nodes rely on the CSI driver for AWS EBS or Azure Disk. The Helm chart defaults to a single PVC name (weaviate-data) without node affinity constraints, so the scheduler may place a Weaviate pod on a node that cannot satisfy the requested volume type.
CSI driver absence or version mismatch – Some cloud VMs lack the required CSI driver plugin, leading to the generic “access denied by filesystem” error documented in the Kubernetes CSI driver guide (Kubernetes CSI driver documentation).
SecurityContext fsGroup misconfiguration – The Helm chart’s default securityContext.fsGroup does not match the group ownership of the underlying NFS export or EBS volume, resulting in permission denials on both environments (Weaviate Helm Chart – Persistent storage).
Node affinity not set – Without a nodeSelector or affinity rule, the scheduler ignores the fact that certain nodes lack the correct StorageClass, reproducing the scenario described in GitHub issue weaviate/weaviate#2152 and helm/charts#1245.

Investigation and Debugging Steps

Follow these concrete steps to isolate the cause:

Inspect PVC and PV details:
```
kubectl describe pvc weaviate-data
kubectl get pv -o wide | grep weaviate-data
```
Expected output shows the storageClassName and nodeAffinity fields. If nodeAffinity is empty, the volume can be bound to any node.
Check pod scheduling:
```
kubectl get pod weaviate-0 -o jsonpath='{.spec.nodeName}'
```
If the node name corresponds to a cloud VM while the PVC uses an NFS StorageClass, you have a mismatch.
Review kubelet logs on the failing node:
```
journalctl -u kubelet | grep weaviate-data
```
Look for messages like “access denied by filesystem” or “no such file or directory”.

Validate CSI driver installation on the node:

kubectl get csidrivers
kubectl describe csidriver ebs.csi.aws.com   # for AWS
kubectl describe csidriver disk.csi.azure.com # for Azure

If the driver is missing, the node cannot provision the volume.

Examine security context of the pod:
```
kubectl get pod weaviate-0 -o yaml | grep -A5 securityContext
```
Confirm that fsGroup matches the group ownership of the underlying volume (e.g., 2000 for NFS export).

Resolution

Apply the following changes to the Helm values file (values.yaml) and redeploy.

1. Separate StorageClasses per environment

# values.yaml (before)
persistence:
  enabled: true
  storageClass: weaviate-sc   # single class used for all nodes
  size: 100Gi

# values.yaml (after)
persistence:
  enabled: true
  storageClass: ""
  size: 100Gi
  volumeClaimTemplates:
    - metadata:
        name: weaviate-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi
        storageClassName: {{ .Values.nodeSelector.kubernetes.io/cloud == "true" ? "aws-ebs-sc" : "nfs-sc" }}

2. Add node affinity to force correct placement

# values.yaml (after – add pod affinity)
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: "kubernetes.io/hostname"
              operator: In
              values:
                - {{ .Values.nodeSelector.kubernetes.io/cloud == "true" ? "cloud-node-*" : "onprem-node-*" }}

3. Set appropriate `fsGroup`

# values.yaml (after – securityContext)
securityContext:
  runAsUser: 1000
  fsGroup: 2000   # matches NFS export or EBS volume group

4. Ensure CSI drivers are installed on all cloud nodes

On each cloud VM, run:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/deploy/kubernetes/manifest.yaml
# or for Azure
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/kubernetes/driver.yaml

5. Redeploy Helm chart

helm upgrade --install weaviate weaviate/weaviate -f values.yaml

Verification

After applying the changes, confirm successful mounting:

Check pod status:
```
kubectl get pods -l app=weaviate -w
```
All pods should reach Running without init errors.
Inspect volume mount inside the container:
```
kubectl exec weaviate-0 -- df -h /var/lib/weaviate
```
Output should list the provisioned volume (e.g., /dev/nvme0n1 for EBS or the NFS mount point).
Review Weaviate logs for normal start‑up messages:
```
kubectl logs weaviate-0 -c weaviate
```
Look for lines such as "Weaviate started successfully" and no mount‑related errors.

Run a quick health check:

curl -s http://$(kubectl get svc weaviate -o jsonpath='{.spec.clusterIP}'):8080/v1/.well-known/ready

Response should be {"status":"ready"}.

Operational Experience and Lessons Learned

Misleading symptom: The error “access denied by filesystem” suggested a permission problem, but the real blocker was node‑level storage incompatibility.
Assumption that a single PVC works across heterogeneous nodes: In hybrid clusters, StorageClasses are often tied to specific cloud providers; using a unified PVC without affinity leads to the “no such file or directory” error when the driver cannot locate the requested volume.
SecurityContext alignment: The default fsGroup in the Helm chart (1000) does not match NFS export permissions (2000), causing denial on both on‑prem and cloud nodes.
Deployment guardrails: Adding explicit nodeSelector or affinity rules prevents the scheduler from placing pods on incompatible nodes, a pattern highlighted in the real incident “Hybrid cluster where on‑prem nodes use NFS‑based storage class while cloud nodes use AWS EBS”.

Best Practices and Prevention

Define distinct StorageClass objects per environment and bind PVCs via storageClassName conditional logic.
Use nodeAffinity or nodeSelector to enforce placement of stateful workloads on nodes that have the required CSI driver installed.
Align securityContext.fsGroup with the underlying volume’s group ownership; make it configurable per environment.
Enable monitoring of kubelet mount errors (e.g., Prometheus metric kubelet_volume_mount_failed_total) and set alerts for spikes.
Validate CSI driver health during cluster bootstrap and enforce it via a DaemonSet readiness probe.

FAQ

Why does the mount succeed on on‑prem nodes but fail on cloud nodes? The PVC references an NFS‑based StorageClass. Cloud nodes lack an NFS client or appropriate CSI driver, so the kubelet cannot attach the volume, resulting in “access denied by filesystem”.
Can I use a single StorageClass for both NFS and EBS? No. StorageClasses encapsulate provider‑specific provisioners and parameters. Mixing them requires separate PVCs or dynamic selection logic as shown in the values.yaml example.
How do I know which CSI driver is missing on a node? Run kubectl get csidrivers and compare the list against the drivers installed on each node (check /var/lib/kubelet/plugins/ or the DaemonSet logs for the driver).
What role does fsGroup play in these errors? The pod’s processes run with the group ID defined by fsGroup. If the underlying volume’s directory is owned by a different group, the kernel denies write access, surfacing as “access denied by filesystem”.
After fixing the affinity, pods still show “no such file or directory”. Verify that the underlying disk actually exists on the node (e.g., lsblk for EBS) and that the storageClassName matches the node’s provisioner. A stale PV bound to a different node can cause this error.

Related Topic Hub: Vector Databases Troubleshooting Hub