Nginx Ingress Controller PVC Pending After Adding Logging Volume

Q: 2. VolumeBindingMode: WaitForFirstConsumer

When the StorageClass is configured with volumeBindingMode: WaitForFirstConsumer, provisioning is deferred until a pod that consumes the PVC is scheduled. If the ingress controller pod has a nodeSelector or affinity that excludes all nodes where the storage backend can provision a volume (e.g., AZ‑specific EBS volumes), the controller never becomes the “first consumer”. This produces the event “volumebinding pending: waiting for first consumer to be scheduled”, as documented in the Kubernetes StorageClass guide.

Q: 4. Namespace‑level ResourceQuota

A ResourceQuota limiting requests.storage can silently reject the PVC request, leaving it pending with an event similar to “quota exceeded for storage” (see GitHub issue kubernetes/kubernetes#110456).

Q: Scenario B – WaitForFirstConsumer Blocking Provisioning

Either change the binding mode to Immediate or ensure the pod can be scheduled on a node that satisfies the storage topology.

In this article

Table of Contents

Problem: NGINX Ingress Controller PVC Stays Pending After Adding a Logging Volume

The NGINX Ingress Controller is deployed via the official Helm chart. After extending the chart to mount a dedicated PersistentVolumeClaim (PVC) for request logging and cache files, the PVC nginx-logging never reaches the Bound phase. The controller pod remains in Pending and the logs that should be written to /var/log/nginx are missing.

Typical symptoms observed in the cluster:

kubectl get pods -n ingress-nginx shows the controller pod in Pending or CrashLoopBackOff after the extra volume is added.
kubectl get pvc -n ingress-nginx reports STATUS CLAIM STORAGECLASS CAPACITY ACCESS MODES AGE with the new PVC in Pending.
Events from kubectl describe pvc nginx-logging -n ingress-nginx contain messages such as:

Events:
  Type    Reason                     Age   From                         Message
  ----    ------                     ----  ----                         -------
  Normal  ProvisioningFailed         2m    persistentvolume-controller  Failed to provision volume with StorageClass "custom-logging": no provisioner found
  Warning VolumeBindingPending     1m    persistentvolume-controller  volumebinding pending: waiting for first consumer to be scheduled

These symptoms prevent the ingress controller from serving traffic, breaking the AI model‑serving microservices that rely on it for routing and TLS termination.

Root Cause Analysis

1. StorageClass Mismatch or Missing Provisioner

The Helm chart adds the PVC definition:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-logging
  namespace: ingress-nginx
spec:
  storageClassName: custom-logging
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

If custom-logging does not exist, or the underlying CSI driver does not register a provisioner for it, the controller logs the error shown above (“no provisioner found”). This matches the Kubernetes PV/PVC documentation which states that a PVC remains pending when no suitable PV or dynamic provisioner can be found.

2. VolumeBindingMode: `WaitForFirstConsumer`

When the StorageClass is configured with volumeBindingMode: WaitForFirstConsumer, provisioning is deferred until a pod that consumes the PVC is scheduled. If the ingress controller pod has a nodeSelector or affinity that excludes all nodes where the storage backend can provision a volume (e.g., AZ‑specific EBS volumes), the controller never becomes the “first consumer”. This produces the event “volumebinding pending: waiting for first consumer to be scheduled”, as documented in the Kubernetes StorageClass guide.

3. Access Mode Incompatibility

In multi‑node clusters (EKS, GKE, on‑prem), the default provisioner may only support ReadWriteOnce. If the ingress controller is deployed with a DaemonSet that creates a pod on every node, each pod would need its own volume, but a single ReadWriteOnce claim cannot be attached to multiple nodes simultaneously. This scenario is discussed in GitHub issue kubernetes/ingress-nginx#10234.

4. Namespace‑level ResourceQuota

A ResourceQuota limiting requests.storage can silently reject the PVC request, leaving it pending with an event similar to “quota exceeded for storage” (see GitHub issue kubernetes/kubernetes#110456).

Investigation and Debugging Steps

Inspect the PVC and StorageClass
```
kubectl get pvc nginx-logging -n ingress-nginx -o yaml
kubectl get sc custom-logging -o yaml
```
Check that storageClassName matches an existing class and that the class has a provisioner field.
Review PVC events
```
kubectl describe pvc nginx-logging -n ingress-nginx
```
Look for messages like “no provisioner found” or “volumebinding pending”.
Validate node selector / affinity of the controller pod
```
kubectl get daemonset nginx-ingress-controller -n ingress-nginx -o yaml | grep -A5 nodeSelector
```
If the selector restricts pods to a subset of nodes, verify that those nodes are in a zone supported by the storage backend.
Check ResourceQuota
```
kubectl get quota -n ingress-nginx -o yaml
```
Confirm that requests.storage is not exceeded.
Examine CSI driver logs (if applicable)
```
kubectl logs -n kube-system -l app=csi-driver -c driver
```
Search for “no volume plugin found” or “access mode not supported”.

Simulate a manual PVC bind (use a test namespace):

cat > test-pvc.yaml <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
  namespace: default
spec:
  storageClassName: custom-logging
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

kubectl apply -f test-pvc.yaml
kubectl describe pvc test-pvc

If this PVC also stays pending, the problem is at the storage class level rather than the ingress controller.

Resolution

Scenario A – Missing or Misnamed StorageClass

Create the expected StorageClass or correct the name in the Helm values.

# Before (values.yaml snippet)
controller:
  extraVolumes:
    - name: logging
      persistentVolumeClaim:
        claimName: nginx-logging
  extraVolumeMounts:
    - name: logging
      mountPath: /var/log/nginx

# After – add correct storageClassName
controller:
  extraVolumes:
    - name: logging
      persistentVolumeClaim:
        claimName: nginx-logging
        storageClass: standard   # or the actual class name
  extraVolumeMounts:
    - name: logging
      mountPath: /var/log/nginx

Or create the missing class:

cat > custom-logging-sc.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: custom-logging
provisioner: kubernetes.io/gce-pd   # example for GKE
parameters:
  type: pd-standard
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

kubectl apply -f custom-logging-sc.yaml

Scenario B – WaitForFirstConsumer Blocking Provisioning

Either change the binding mode to Immediate or ensure the pod can be scheduled on a node that satisfies the storage topology.

# Modify existing class
kubectl patch storageclass custom-logging -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl patch storageclass custom-logging -p '{"volumeBindingMode":"Immediate"}' --type=merge

Alternatively, remove the restrictive nodeSelector from the DaemonSet:

kubectl edit daemonset nginx-ingress-controller -n ingress-nginx
# Delete or adjust the nodeSelector block

Scenario C – Access Mode Mismatch

If the controller runs as a DaemonSet on multiple nodes, switch to a ReadWriteMany capable backend (e.g., NFS, Azure Files) or change the deployment to a single‑replica Deployment.

# Example using NFS
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-logging-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  nfs:
    path: /exports/nginx-logs
    server: nfs.example.com
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-logging
  namespace: ingress-nginx
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

Scenario D – ResourceQuota Exceeded

Increase the quota or delete unused PVCs to free space.

kubectl edit quota ingress-nginx -n ingress-nginx
# Adjust hard.requests.storage to a higher value

Verification

Confirm the PVC is Bound:

kubectl get pvc nginx-logging -n ingress-nginx
# Expected output:
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-logging  Bound    pvc-1234abcd-5678-efgh-9012-ijklmnopqrstu  5Gi        RWO            standard       2m

Check that the controller pod starts and mounts the volume:

kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx
kubectl describe pod nginx-ingress-controller-xxxx -n ingress-nginx | grep -A3 Volumes

Validate logging works:

kubectl exec -n ingress-nginx -c controller nginx-ingress-controller-xxxx -- ls /var/log/nginx
# Should list access.log error.log

Run a functional request through the ingress and verify log entry:

curl -I http://my-service.example.com/healthz
kubectl exec -n ingress-nginx -c controller nginx-ingress-controller-xxxx -- grep my-service /var/log/nginx/access.log

Operational Experience & Lessons Learned

Misleading symptom: The controller pod remained in Pending while other pods in the same namespace ran fine, leading engineers to initially suspect network policies.
Common incorrect assumption: “The default StorageClass is always present.” In GKE clusters where standard was disabled, the PVC referenced a non‑existent class, reproducing the “no provisioner found” error.
Production edge case: In an EKS multi‑AZ setup, the WaitForFirstConsumer mode delayed provisioning until the pod was scheduled, but the ingress controller DaemonSet used a nodeSelector that excluded the AZs where the EBS volume could be created.
Lesson: Always audit StorageClass parameters, volumeBindingMode, and pod affinity rules when adding extra volumes to system‑critical components.

Best Practices and Prevention

Declare the storage class explicitly in the Helm values file; avoid relying on the cluster default.
Prefer Immediate binding for system components that must start before any pod is scheduled, unless topology constraints are required.
When using a DaemonSet, choose a storage backend that supports ReadWriteMany or switch to a single‑replica Deployment for log aggregation.

Set up alerts on PVC events:

kubectl get events --field-selector reason=ProvisioningFailed -A

Include a health check that verifies the existence of the log directory inside the controller container.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does the PVC stay pending only after adding extraVolumes?
Because the Helm chart adds a new PVC that references a StorageClass which either does not exist, has an incompatible volumeBindingMode, or uses an access mode unsupported by the underlying CSI driver.
How can I see which StorageClass a PVC is trying to use?
```
kubectl get pvc nginx-logging -n ingress-nginx -o jsonpath='{.spec.storageClassName}'
```
If the output is empty, the PVC falls back to the cluster’s default StorageClass.
Can I use a ReadWriteMany volume for the ingress controller logs?
Yes, but only if the underlying storage provider supports RWX (e.g., NFS, Azure Files, GCP Filestore). Otherwise the PVC will remain pending with an error like “access mode not supported”.
What does the event “volumebinding pending: waiting for first consumer to be scheduled” mean?
It indicates the StorageClass is configured with WaitForFirstConsumer. Provisioning is deferred until a pod that consumes the PVC is scheduled on a node that satisfies the volume’s topology constraints.
How do I troubleshoot a quota‑related PVC pending state?
Check the namespace’s ResourceQuota:
```
kubectl get quota -n ingress-nginx -o yaml
```
If hard.requests.storage is exceeded, either increase the quota or delete unused PVCs.