Redis Pod Creation Denied By Admission Controller During API Gateway Scaling

Q: Root Cause

The cluster enforces a Pod Security Admission (PSA) policy configured in restricted mode. The policy requires every pod to carry a specific label (app.kubernetes.io/name=redis) and to request cpu and memory limits. The Helm chart used for Redis does not propagate these labels to newly created pods during a scale‑out operation, causing the admission webhook to reject the pod creation.

In this article

Table of Contents

Problem

When the API gateway scales out, the Helm release that deploys the Redis cache attempts to create additional Redis pods. The pod creation fails with an admission controller error, preventing the cache layer from meeting the traffic demand.

Typical console output:


$ kubectl scale deployment redis-cache --replicas=5
deployment.apps/redis-cache scaled
$ kubectl get pods -w
NAME               READY   STATUS    RESTARTS   AGE
redis-cache-0      1/1     Running   0          12m
redis-cache-1      1/1     Running   0          12m
redis-cache-2      0/1     Pending   0          0s

Pending pod logs:


$ kubectl describe pod redis-cache-2
Name:         redis-cache-2
Namespace:    production
...
Events:
  Type     Reason                     Age   From               Message
  ----     ------                     ----  ----               -------
  Normal   Scheduled                  5s    default-scheduler  Successfully assigned production/redis-cache-2 to node-12
  Warning  FailedCreatePodSandbox     4s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = admission webhook "pod-security.k8s.io" denied the request: pod must have the label "app.kubernetes.io/name=redis"

Root Cause

The cluster enforces a Pod Security Admission (PSA) policy configured in restricted mode. The policy requires every pod to carry a specific label (app.kubernetes.io/name=redis) and to request cpu and memory limits. The Helm chart used for Redis does not propagate these labels to newly created pods during a scale‑out operation, causing the admission webhook to reject the pod creation.

Underlying mechanisms:

Admission webhook – intercepts CREATE pod requests and validates them against the PSA constraints.
Label selector enforcement – the webhook checks the presence of mandatory labels defined in the PodSecurityPolicy (or PSA configuration) before allowing the pod to be admitted.
Helm templating – the replicaCount scaling path re‑uses the podTemplate but omits the mandatory label due to a conditional block that only runs on initial install.

Debug

Step‑by‑step investigation performed in the affected namespace:

Inspect the admission webhook configuration:


$ kubectl get validatingwebhookconfiguration pod-security -o yaml | grep -A5 "rules:"
rules:
- apiGroups:
  - ""
  apiVersions:
  - v1
  resources:
  - pods
  operations:
  - CREATE

Review the PSA configuration applied to the namespace:


$ kubectl get namespace production -o yaml | grep -i pod-security.kubernetes.io
metadata:
  annotations:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Show the pod template from the Helm chart (pre‑upgrade):


apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "redis.fullname" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "redis.name" . }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "redis.name" . }}
        # NOTE: missing mandatory label for PSA
    spec:
      containers:
      - name: redis
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        resources:
          limits:
            cpu: {{ .Values.resources.limits.cpu }}
            memory: {{ .Values.resources.limits.memory }}

The missing label (app.kubernetes.io/name=redis) is the direct cause of the denial.

Solution

Two complementary changes were applied:

1. Update the Helm chart to always emit the required label

Before:


metadata:
  labels:
    app.kubernetes.io/name: {{ include "redis.name" . }}
    # missing mandatory label

After:


metadata:
  labels:
    app.kubernetes.io/name: {{ include "redis.name" . }}
    app.kubernetes.io/component: redis
    app.kubernetes.io/instance: {{ .Release.Name }}
    # mandatory label enforced by PSA
    app.kubernetes.io/name: redis

2. Adjust the PSA exception to accept the existing label set (optional)

If modifying the chart is not feasible, a namespace‑level exemption can be added:


apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: redis-psp-exempt
spec:
  allowedLabels:
  - "app.kubernetes.io/name"
  - "app.kubernetes.io/component"
  - "app.kubernetes.io/instance"

However, the preferred approach is to make the chart compliant, keeping the PSA enforcement strict.

Verify

After applying the chart changes and redeploying:


$ helm upgrade --install redis ./redis-chart -n production
$ kubectl rollout status deployment/redis-cache -n production
deployment "redis-cache" successfully rolled out
$ kubectl get pods -l app.kubernetes.io/name=redis -n production
NAME               READY   STATUS    RESTARTS   AGE
redis-cache-0      1/1     Running   0          3m
redis-cache-1      1/1     Running   0          3m
redis-cache-2      1/1     Running   0          2m
redis-cache-3      1/1     Running   0          2m
redis-cache-4      1/1     Running   0          2m

Metrics from Prometheus confirm the cache hit ratio remains stable while the API gateway traffic spikes, indicating successful scaling.

Prevent

Chart linting: Add a Helm test that validates the rendered pod template contains all mandatory PSA labels.
CI gate: Use kubeval or conftest with OPA policies to reject PRs that omit required labels.
Namespace policy documentation: Keep a version‑controlled list of PSA requirements per namespace and expose it to developers.
Automated health checks: Deploy a CronJob that periodically attempts to create a dummy pod with the required labels; alert on failures.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the pod get created successfully on the first install but fails on subsequent scaling?
The initial install uses the helm install path, which applies a different set of template values that include the mandatory label. The scaling path re‑uses the podTemplate without re‑evaluating the conditional block that adds the label, leading to the omission.
Can I disable the PSA webhook for a single deployment?
Yes, by adding the annotation pod-security.kubernetes.io/enforce=privileged to the deployment metadata, but this weakens the security posture and is not recommended.
How do I find which label(s) are required by the PSA?
Inspect the PodSecurityPolicy or the PSA configuration in the namespace annotations. The allowedLabels field (or custom OPA rules) lists the required keys.
Is there a way to see the exact decision made by the admission webhook?
Enable audit logging for the admission.k8s.io group. The audit log entry contains the objectRef and the responseStatus.message that shows the exact validation failure.
Will upgrading the Kubernetes version change the behavior of this admission controller?
Kubernetes 1.25 introduced PSA as a built‑in feature, replacing the older PSP. If the cluster is upgraded, ensure the PSA mode and any custom webhook configurations are reviewed for compatibility.