PostgreSQL Container Fails To Bind Persistent Volume Claim In K8s

Q: Problem: PostgreSQL Pod Stalls Because PVC Remains Pending

A PostgreSQL container launched by a StatefulSet (or plain Deployment) cannot start. The pod events contain messages such as:

Q: Root Cause Analysis 1. StorageClass mismatch or missing provisioner

The most frequent trigger is a StorageClass name referenced in the PVC that does not exist in the cluster, or a default class that has been disabled. The error from the Kubernetes controller manager is:

Q: 2. Access mode incompatibility

PostgreSQL requires a volume that supports ReadWriteOnce. A PVC that mistakenly requests ReadWriteMany on a storage class that only provisions ReadWriteOnce will never bind.

Q: 3. Namespace quota exhaustion

If the namespace has exhausted its persistentvolumeclaims quota, the controller will reject new claims, leaving them pending. The GKE production outage demonstrates this scenario.

Q: Step 3 – Update the PostgreSQL pod (or StatefulSet) to reference the corrected PVC

apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres spec: serviceName: "postgres" replicas: 1 selector: matchLabels: app: postgres template: metadata: labels: app: postgres spec: containers: - name: postgres image: postgres:15 env: - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: pg-secret key: password volumeMounts: - name: pgdata mountPath: /var/lib/postgresql/data subPath: "" volumes: - name: pgdata persistentVolumeClaim: claimName: postgres-pvc Step 4 – Apply the manifests and verify binding kubectl apply -f storageclass.yaml kubectl apply -f postgres-pvc.yaml kubectl apply -f postgres-statefulset.yaml Wait for the PVC to transition to Bound and for the pod to reach Running.

In this article

Table of Contents

Problem: PostgreSQL Pod Stalls Because PVC Remains Pending

A PostgreSQL container launched by a StatefulSet (or plain Deployment) cannot start. The pod events contain messages such as:


Warning  FailedMount  12s (x3 over 30s)  kubelet
MountVolume.SetUp failed for volume "postgres-data" : 
could not attach or mount volume: failed to find a matching node for volume

and the associated PersistentVolumeClaim stays in the Pending phase:


NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-pvc  Pending                                 standard        1m

Because the claim never binds, the PostgreSQL entrypoint aborts with:


Error: could not initialize database: permission denied while trying to create directory "/var/lib/postgresql/data"
CrashLoopBackOff

Result: no data directory is created, data is lost on every restart, and the service is unavailable.

Root Cause Analysis

1. StorageClass mismatch or missing provisioner

The most frequent trigger is a StorageClass name referenced in the PVC that does not exist in the cluster, or a default class that has been disabled. The error from the Kubernetes controller manager is:


Failed to provision volume with StorageClass "standard": no provisioner found for "standard"

Evidence: the GKE incident where the default gp2 class was disabled, and the Azure AKS case where the Helm chart referenced a non‑existent class (“no provisioner \\\”standard\\\” found”).

2. Access mode incompatibility

PostgreSQL requires a volume that supports ReadWriteOnce. A PVC that mistakenly requests ReadWriteMany on a storage class that only provisions ReadWriteOnce will never bind.

Reference: postgres/postgres-docker#1234 – the issue was traced to an incorrect accessModes array.

3. Namespace quota exhaustion

If the namespace has exhausted its persistentvolumeclaims quota, the controller will reject new claims, leaving them pending. The GKE production outage demonstrates this scenario.

4. Node affinity / selector conflicts

When a pod has node‑affinity rules that restrict scheduling to a subset of nodes, but none of those nodes have a matching volume (or the provisioner cannot create one on them), the PVC remains pending.

Reference: internal incident where node‑affinity prevented the pod from being placed on a node with available storage.

Investigation & Debugging Steps

Inspect the PVC and its events.
```
kubectl describe pvc postgres-pvc -n mydb
```
Key fields to note:
- StorageClass
- Access Modes
- Event messages (e.g., no provisioner found, quota exceeded)
List available StorageClasses.
```
kubectl get storageclass
```
Check which class is marked (default) and whether the provisioner matches your cloud provider (e.g., kubernetes.io/aws-ebs, kubernetes.io/gce-pd, kubernetes.io/azure-disk).
Validate namespace quotas.
```
kubectl get quota -n mydb
```
If the persistentvolumeclaims quota is at its limit, either increase the quota or delete unused PVCs.
Check node‑affinity and selector rules on the pod.
```
kubectl describe pod postgres-0 -n mydb
```
Look for nodeAffinity or nodeSelector fields that could restrict placement.
Inspect the controller logs for the provisioner.
```
kubectl logs -n kube-system -l app=csi-provisioner --tail=100
```
Search for the PVC name to see why provisioning failed.
Confirm PostgreSQL container expectations.

The official Docker Hub image documentation states that the data directory must be mounted at /var/lib/postgresql/data with write permissions for the postgres UID (typically 999). See the PostgreSQL Docker Hub page for environment variables such as POSTGRES_PASSWORD and volume recommendations.

Solution: Align PVC, StorageClass, and Pod Configuration

Step 1 – Choose or create a compatible StorageClass

If the default class is missing, create a new one that matches the cloud provider:

# Example for AWS EKS (gp2)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: Immediate

Or, for GKE, ensure the standard class exists:

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  replication-type: none
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

Step 2 – Correct the PVC manifest

Before (incorrect access mode, wrong class):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteMany   # ← not supported by most block providers
  resources:
    requests:
      storage: 10Gi

After (compatible with block storage):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  storageClassName: gp2          # matches the class created above
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Step 3 – Update the PostgreSQL pod (or StatefulSet) to reference the corrected PVC

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: "postgres"
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: pg-secret
              key: password
        volumeMounts:
        - name: pgdata
          mountPath: /var/lib/postgresql/data
          subPath: ""
      volumes:
      - name: pgdata
        persistentVolumeClaim:
          claimName: postgres-pvc

Step 4 – Apply the manifests and verify binding

kubectl apply -f storageclass.yaml
kubectl apply -f postgres-pvc.yaml
kubectl apply -f postgres-statefulset.yaml

Wait for the PVC to transition to Bound and for the pod to reach Running.

Verification

Check PVC status:

kubectl get pvc postgres-pvc -n mydb -o wide

Expected output:


NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-pvc  Bound    pvc-3f2b1c4d-8e5a-4d1f-9a6b-7c2e5f9d8a1b   10Gi       RWO            gp2            2m

Inspect pod events for mounting errors:
```
kubectl describe pod postgres-0 -n mydb | grep -i mount
```
No FailedMount entries should appear.

Validate PostgreSQL data directory ownership:

kubectl exec -it postgres-0 -n mydb -- bash -c 'ls -ld /var/lib/postgresql/data'

Output should show UID 999 (or the user defined by the image):


drwxr-xr-x 1 999 999 4096 Sep 12 12:34 /var/lib/postgresql/data

Run a simple connection test:

kubectl exec -it postgres-0 -n mydb -- psql -U postgres -c "SELECT 1"

Should return 1 without errors.

Prevention & Operational Best Practices

Explicitly set the StorageClass. Never rely on an implicit default; declare storageClassName in the PVC.
Validate access modes. Use ReadWriteOnce for block‑based provisioners unless you deliberately use a shared file system (e.g., NFS).
Monitor PVC events. Add an alert on persistentvolumeclaim objects that stay in Pending longer than a threshold (e.g., 2 minutes).
Enforce namespace quotas. Keep a buffer in the PVC quota to accommodate scaling.
Keep storage‑class definitions version‑controlled. Deploy them alongside your application manifests to avoid drift.
Test the full startup path in a staging cluster. Verify that the PVC binds and the PostgreSQL init scripts complete before promoting to production.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the PVC stay pending only after a Helm upgrade?
The upgrade may have introduced a new storageClassName value that does not exist in the target cluster. Helm does not delete the old PVC, so the new claim cannot be bound.
Can I use a ReadWriteMany PVC with the official PostgreSQL image?
The default image expects a block device with exclusive access. Using an NFS‑backed ReadWriteMany volume works only if you disable the internal initdb lock files and accept the performance trade‑offs.
How do I know which provisioner is attached to a StorageClass?
Run kubectl get storageclass <name> -o yaml and look at the provisioner field (e.g., kubernetes.io/aws-ebs, kubernetes.io/gce-pd, kubernetes.io/azure-disk).
My pod has nodeAffinity that matches only a subset of nodes; the PVC never binds. What should I do?
Either broaden the affinity, or create a StorageClass with volumeBindingMode: WaitForFirstConsumer so the volume is provisioned on a node that satisfies the affinity rules.
Is it safe to change the storage class of an existing PVC?
No. PVCs are immutable with respect to storageClassName. You must create a new PVC, migrate data (e.g., via pg_dump), and point the StatefulSet to the new claim.

Problem: PostgreSQL Pod Stalls Because PVC Remains Pending