Redis Pod Stuck Pending PVC On Edge Node After Node Reboot

Q: Problem – Redis pod stuck pending PVC on an edge node after node reboot

A Redis StatefulSet deployed to an edge‑computing node fails to start. The pod remains in Pending because its PersistentVolumeClaim (redis-data) never becomes Bound. The symptom blocks data persistence and makes the Redis service unavailable.

Q: Root Cause – How the edge node reboot broke the PVC binding

Edge clusters often use a local‑path provisioner (or any local storage class) to achieve low‑latency persistence. The provisioner creates a PersistentVolume (PV) that is bound to a specific node via nodeAffinity:

Q: Debug – Systematic investigation steps 1. Verify node and storage class status

# Node health kubectl get nodes -o wide # Example output NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME edge-node-01 Ready 45d v1.27.3 10.0.1.5 <none> Ubuntu 22.04 LTS 5.15.0-78-generic docker://24.0.5 # StorageClass definition kubectl get sc local-path -o yaml Check that the local-path storage class exists and that volumeBindingMode is set to WaitForFirstConsumer. Using Immediate on edge nodes can cause premature PV creation before the node is Ready (official docs).

Q: 2. Inspect the PVC and its bound PV (if any)

kubectl get pvc redis-data -o yaml kubectl get pv -l claimName=redis-data -o yaml Key fields to look for:

Q: 3. Review controller manager logs for binding errors

kubectl -n kube-system logs -l component=controller-manager -c kube-controller-manager | grep -i "volumebindingcontroller" Typical log snippet:

Q: 4. Check for taints and tolerations

# List node taints kubectl describe node edge-node-01 | grep -i taint # Show StatefulSet tolerations kubectl get sts redis -o yaml | grep -i tolerations -A3 5. Validate underlying filesystem # On the edge node ssh root@10.0.1.5 ls -l /var/lib/kubelet/pods/*/volumes/kubernetes.io~local-path/redis-data If the directory is missing, the provisioner could not recreate the PV after reboot.

Q: Option C – Add tolerations for node taints

If a maintenance taint exists, update the Redis StatefulSet:

In this article

Table of Contents

Problem – Redis pod stuck pending PVC on an edge node after node reboot

A Redis StatefulSet deployed to an edge‑computing node fails to start. The pod remains in Pending because its PersistentVolumeClaim (redis-data) never becomes Bound. The symptom blocks data persistence and makes the Redis service unavailable.

Typical pod description:

kubectl describe pod redis-0
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 node(s) had volume node affinity conflict.
  Warning  FailedMount       1m    kubelet            MountVolume.SetUp failed for volume "redis-data": mount failed: no such device

Typical PVC description:

kubectl describe pvc redis-data
...
Status:  Pending
Events:
  Type    Reason                     Age   From                     Message
  ----    ------                     ----  ----                     -------
  Warning VolumeBindingController  3m    volumebinding-controller  no suitable node found for volume

Root Cause – How the edge node reboot broke the PVC binding

Edge clusters often use a local‑path provisioner (or any local storage class) to achieve low‑latency persistence. The provisioner creates a PersistentVolume (PV) that is bound to a specific node via nodeAffinity:

PV attribute	Value
spec.nodeAffinity	requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: – matchExpressions: – key: kubernetes.io/hostname, operator: In, values: [edge-node-01]
spec.local.path	/var/lib/kubelet/pods/…/volumes/kubernetes.io~local-path/redis-data

When the node reboots, two things can happen (as reported in the evidence package):

Local PV deletion: The local-path provisioner watches NodeReady events. On a reboot it may delete the PV because the underlying directory disappears temporarily (GitHub issue #101123).
Device name change: A kernel upgrade or hardware re‑enumeration can rename block devices (e.g., /dev/sdb → /dev/sdc), leaving the PV pointing to a non‑existent device (incident).

Both cases break the nodeAffinity match or the mount operation, causing the PVC to stay Pending with errors such as:

"FailedMount" event with message "mount failed: no such device"
"volumebindingcontroller: no suitable node found for volume"

Additionally, if a taint (e.g., edge-node=maintenance:NoSchedule) was added during the reboot and the Redis StatefulSet lacks a matching toleration, the scheduler will never place the pod, and the PVC binding controller will not find a suitable node (Kubernetes docs).

Debug – Systematic investigation steps

1. Verify node and storage class status

# Node health
kubectl get nodes -o wide
# Example output
NAME          STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
edge-node-01  Ready       45d   v1.27.3   10.0.1.5      <none>        Ubuntu 22.04 LTS    5.15.0-78-generic docker://24.0.5

# StorageClass definition
kubectl get sc local-path -o yaml

Check that the local-path storage class exists and that volumeBindingMode is set to WaitForFirstConsumer. Using Immediate on edge nodes can cause premature PV creation before the node is Ready (official docs).

2. Inspect the PVC and its bound PV (if any)

kubectl get pvc redis-data -o yaml
kubectl get pv -l claimName=redis-data -o yaml

Key fields to look for:

spec.storageClassName – must match the provisioner.
spec.volumeName – if empty, no PV was created.
PV spec.nodeAffinity – ensure the hostname matches the edge node.

3. Review controller manager logs for binding errors

kubectl -n kube-system logs -l component=controller-manager -c kube-controller-manager | grep -i "volumebindingcontroller"

Typical log snippet:

time="2024-08-22T14:12:03Z" level=error msg="no suitable node found for volume redis-data (local-path) – node affinity mismatch"

4. Check for taints and tolerations

# List node taints
kubectl describe node edge-node-01 | grep -i taint

# Show StatefulSet tolerations
kubectl get sts redis -o yaml | grep -i tolerations -A3

5. Validate underlying filesystem

# On the edge node
ssh root@10.0.1.5
ls -l /var/lib/kubelet/pods/*/volumes/kubernetes.io~local-path/redis-data

If the directory is missing, the provisioner could not recreate the PV after reboot.

Solution – Restoring PVC binding and getting Redis running

Option A – Re‑create the missing PV manually (quick fix)

If the PV was deleted, create a new one that matches the PVC’s storage class and node affinity.

# Before (missing PV)
# No PV object exists for claim redis-data

# After – create PV
cat > pv-redis-data.yaml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-redis-data
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  local:
    path: /mnt/edge-storage/redis-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - edge-node-01
EOF

kubectl apply -f pv-redis-data.yaml

Then bind the PVC:

kubectl patch pvc redis-data -p '{"spec":{"volumeName":"pv-redis-data"}}'

Option B – Adjust the StorageClass to use `WaitForFirstConsumer`

Modify the storage class so PV creation is deferred until a pod actually requests it, preventing orphaned PVs after node reboot.

# Before – Immediate binding
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
volumeBindingMode: Immediate

# After – WaitForFirstConsumer
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

Apply the updated class and delete the stale PVC (it will be recreated by the StatefulSet).

kubectl delete pvc redis-data
kubectl apply -f redis-statefulset.yaml   # triggers new PVC creation

Option C – Add tolerations for node taints

If a maintenance taint exists, update the Redis StatefulSet:

# Before – no tolerations
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  ...

# After – toleration added
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  ...
  template:
    spec:
      tolerations:
      - key: "edge-node"
        operator: "Exists"
        effect: "NoSchedule"

Option D – Ensure the local-path provisioner runs as a DaemonSet

Verify the provisioner is scheduled on the edge node after reboot:

kubectl get ds -n local-path-provisioner
kubectl describe ds local-path-provisioner -n local-path-provisioner | grep -i node

If the DaemonSet is not running, redeploy it:

kubectl rollout restart ds local-path-provisioner -n local-path-provisioner

Verify – Confirm that Redis is healthy and data persists

# Verify PVC is bound
kubectl get pvc redis-data
NAME         STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-data   Bound    pv-redis-data 5Gi        RWO            local-path     2m

# Verify pod is Running
kubectl get pod -l app=redis
NAME       READY   STATUS    RESTARTS   AGE
redis-0    1/1     Running   0          1m

# Check Redis persistence
kubectl exec redis-0 -- redis-cli INFO persistence
# Expected fields
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1724567890
rdb_last_bgsave_status:ok

Optionally, write a key, restart the pod, and ensure it survives:

kubectl exec redis-0 -- redis-cli SET foo bar
kubectl delete pod redis-0   # forces restart
kubectl exec redis-0 -- redis-cli GET foo   # should return "bar"

Prevent – Operational guardrails for edge deployments

Use WaitForFirstConsumer binding mode for any local storage class to avoid premature PV creation.
Label edge nodes consistently (e.g., edge=true) and reference those labels in both nodeAffinity and nodeSelector of the StatefulSet.
Persist local-path provisioner configuration across node reboots (store /etc/kubernetes/manifests or use a DaemonSet with hostPath that survives reboot).
Monitor node taints and PVC events with alerts on PersistentVolumeClaim status Pending and on controller manager logs containing volumebindingcontroller.
Automate PV recreation via a small controller that watches for PersistentVolume deletions with a specific storageClassName and recreates them using the original nodeAffinity.
Validate device paths after kernel or firmware upgrades – run a post‑upgrade script that updates any local PV definitions if block device names change.

FAQ – Common follow‑up questions

Why does the PVC stay Pending only after a node reboot?
Because the local‑path provisioner deletes the PV when the node becomes NotReady, and the PVC cannot find a matching node with the required nodeAffinity until a new PV is created.
Can I use a remote storage class (e.g., NFS) on edge nodes to avoid this issue?
Yes, remote CSI drivers do not rely on node‑specific nodeAffinity, but they add latency. If low latency is required, fixing the local‑path workflow is preferable.
How do I know which storage class a PVC is using?
Inspect the PVC YAML: kubectl get pvc <name> -o jsonpath='{.spec.storageClassName}'. Ensure it matches the provisioner you intend to use.
What alert should I set for this failure mode?
Create a Prometheus alert on kube_persistentvolumeclaim_status_phase{phase="Pending"} combined with a label filter for the Redis namespace, and a separate alert on controller manager logs containing volumebindingcontroller: no suitable node found.
Is it safe to delete a PVC that is stuck Pending?
Yes, deleting the PVC removes the binding request. When the StatefulSet recreates the PVC, a new PV will be provisioned if the storage class and node conditions are correct.

Related Topic Hub: Data Infrastructure Troubleshooting Hub