a laptop computer sitting on top of a wooden table

Admission controller rejection of GPT-4 pod missing securityContext in canary

by

Problem – Admission Controller Rejects GPT‑4 Canary Pods

During a staged rollout of a new GPT‑4 inference service, the canary pods never reach the Running state. The Kubernetes API server returns a 403 error from the pod-security-admission webhook and a custom model‑policy webhook. Typical log excerpts look like:


Error from server (Forbidden): admission webhook "pod-security-admission" denied the request:
securityContext.runAsUser is required
metadata.labels: Invalid value: "{app: gpt4-canary}": must match label selector 'app.kubernetes.io/component=gpt4'
admission webhook "custom-model-policy" denied the request: securityContext.allowPrivilegeEscalation must be false

The failure blocks the canary deployment, causing a full rollout pause and risking SLA violations for the AI SaaS platform.

Root Cause – Missing Required securityContext and Label Constraints

Two independent admission controls enforce the Pod Security Standards (PSS) “restricted” profile and a custom model policy:

  • PodSecurityAdmission (PSA) webhook – Requires runAsNonRoot: true, runAsUser, runAsGroup, and readOnlyRootFilesystem: true in securityContext. It also enforces the label selector app.kubernetes.io/component=gpt4 for all GPT‑4 workloads.
  • Custom model‑policy webhook – Checks that allowPrivilegeEscalation: false is explicitly set and that the pod carries the mandatory component label.

In the canary manifest the securityContext fields were omitted to keep the Helm chart minimal, and the required label was missing because the chart used a generic app: gpt4 label. This violates both the official PSS documentation and the custom policy introduced after a recent security hardening sprint (see the production incident on GKE where runAsUser and runAsGroup were omitted).

Debug – Investigation Steps

1. Capture the Admission Rejection


$ kubectl describe pod gpt4-canary-5f9d8c7b9c-abcde
Events:
  Type     Reason  Age   From                     Message
  ----     ------  ----  ----                     -------
  Warning  Failed  2s    pod-security-admission   securityContext.runAsUser is required
  Warning  Failed  2s    custom-model-policy      securityContext.allowPrivilegeEscalation must be false

2. Verify PSA Policy Mode


$ kubectl get ns gpt4-canary -o jsonpath='{.metadata.labels.pod-security.kubernetes.io/enforce}'
restricted

3. Inspect the Helm values used for the canary


$ helm get values gpt4-inference --revision 12 --output yaml
...
securityContext: {}
labels:
  app: gpt4
  tier: canary

4. Review the custom webhook configuration


apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: custom-model-policy
webhooks:
- name: model-policy.example.com
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["pods"]
  clientConfig:
    service:
      name: model-policy-webhook
      namespace: kube-system
    caBundle: 
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

5. Reproduce the failure with a minimal pod


apiVersion: v1
kind: Pod
metadata:
  name: test-gpt4
  labels:
    app: gpt4
spec:
  containers:
  - name: inference
    image: openai/gpt4-inference:latest

Running kubectl apply -f test.yaml yields the same admission errors, confirming the root cause.

Solution – Add Required Security Context and Labels

The fix consists of two parts: (1) enrich the pod securityContext to satisfy the “restricted” PSS, and (2) ensure the mandatory component label is present. Below are the before/after manifests and the Helm value overrides.

Before – Faulty Canary Manifest (excerpt)


apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpt4-canary
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpt4
  template:
    metadata:
      labels:
        app: gpt4
        tier: canary
    spec:
      containers:
      - name: inference
        image: openai/gpt4-inference:{{ .Values.imageTag }}
        resources:
          limits:
            cpu: "4"
            memory: "16Gi"
      # securityContext omitted

After – Corrected Canary Manifest (excerpt)


apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpt4-canary
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: gpt4
      app.kubernetes.io/component: gpt4
      tier: canary
  template:
    metadata:
      labels:
        app.kubernetes.io/name: gpt4
        app.kubernetes.io/component: gpt4
        tier: canary
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 3000
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
      containers:
      - name: inference
        image: openai/gpt4-inference:{{ .Values.imageTag }}
        resources:
          limits:
            cpu: "4"
            memory: "16Gi"
        securityContext:
          capabilities:
            drop: ["ALL"]

Helm Value Override (values.yaml)


securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 3000
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false

labels:
  app.kubernetes.io/name: gpt4
  app.kubernetes.io/component: gpt4
  tier: canary

Applying the updated chart (helm upgrade --install gpt4-inference ./chart -f values.yaml) creates pods that pass both admission controllers.

Verify – Confirming Successful Canary Rollout

  • Pod status – All canary pods reach Running without admission errors.
  • Admission logs – No entries from pod-security-admission or custom-model-policy for the new pods.
  • Metrics – Inference latency and request count appear in Prometheus under gpt4_canary_* labels.

$ kubectl get pods -l app.kubernetes.io/component=gpt4 -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE
gpt4-canary-5f9d8c7b9c-abcde   1/1     Running   0          2m    10.12.3.45   gke-node-1
gpt4-canary-5f9d8c7b9c-fghij   1/1     Running   0          2m    10.12.3.46   gke-node-2

Prevent – Operational Guardrails

Preventive Action Implementation
Enforce Helm linting for securityContext Add a helm lint rule or a pre-commit hook that checks for runAsUser, runAsGroup, and readOnlyRootFilesystem fields.
Label validation CI job Run kubectl apply --dry-run=client -f manifests/ and grep for missing app.kubernetes.io/component=gpt4 label.
PodSecurityAdmission audit mode Set audit mode for the restricted policy in non‑production clusters to surface violations before they block deployments.
Custom webhook unit tests Include integration tests that submit a minimal pod manifest and assert a 200 response from the webhook.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the pod pass locally but fail in the canary namespace?
    The canary namespace is labeled with pod-security.kubernetes.io/enforce=restricted, which activates stricter PSA checks than the default namespace.
  2. Can I disable the custom model‑policy webhook for canary testing?
    You can set the webhook’s failurePolicy to Ignore in a test cluster, but in production it is recommended to fix the manifest rather than bypass security checks.
  3. Do I need to set securityContext.runAsNonRoot if I already set runAsUser?
    Yes. The “restricted” profile explicitly requires runAsNonRoot: true in addition to a non‑zero runAsUser.
  4. What label selector should I use for future model versions?
    Follow the convention app.kubernetes.io/name=gpt4 and app.kubernetes.io/component=gpt4. If you introduce a new model family, change the component value accordingly (e.g., gpt4‑v2).
  5. Is readOnlyRootFilesystem mandatory for all containers?
    Under the “restricted” PSS it is required for every container. If a container needs write access, use an emptyDir volume and mount it at the required path.