a purple and black background with a purple and black logo

Admission controller rejecting PyTorch pod creation on kind cluster

by

Problem

When attempting to launch a PyTorch training job on a local kind cluster, the pod creation fails with an admission controller error. The user sees the following output from kubectl apply and kubectl describe pod:

$ kubectl apply -f pytorch-job.yaml
pod/pytorch-trainer created
error: error when creating "pytorch-job.yaml": admission webhook "podsecuritypolicy.k8s.io" denied the request: 
spec.securityContext.runAsUser: Invalid value: 0: running as root is forbidden by policy

$ kubectl describe pod pytorch-trainer
Name:         pytorch-trainer
Namespace:    default
...
Events:
  Type     Reason                     Age   From                     Message
  ----     ------                     ----  ----                     -------
  Warning  FailedCreatePodSandbox     5s    kubelet, kind-control-plane  Failed to create pod sandbox: rpc error: code = Unknown desc = 
    admission webhook "podsecuritypolicy.k8s.io" denied the request: spec.securityContext.runAsUser: Invalid value: 0: running as root is forbidden by policy

The pod never reaches the Running state, and the training job never starts.

Root Cause

The failure originates from the PodSecurityPolicy (PSP) admission controller that is enabled by default in the kind cluster configuration used for this project. The PSP in effect (restricted-psp) disallows:

  • Running containers as UID 0 (root).
  • Using privileged mode.
  • Mounting hostPath volumes.

The official PyTorch Docker images (pytorch/pytorch:latest) are built to run as root by default. When the pod spec does not explicitly set a non‑root runAsUser, the container inherits UID 0, triggering the PSP denial.

Debug

Step‑by‑step investigation:

  1. Inspect the admission controller chain to confirm PSP is active.
$ kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration -A | grep podsecuritypolicy
validatingwebhookconfiguration.admissionregistration.k8s.io/podsecuritypolicy-webhook  1
  1. Review the PSP object that is applied to the default namespace.
$ kubectl get psp restricted-psp -o yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted-psp
spec:
  runAsUser:
    rule: MustRunAsNonRoot
  privileged: false
  allowPrivilegeEscalation: false
  volumes:
    - 'configMap'
    - 'secret'
    - 'emptyDir'
  ...
  1. Check the pod spec generated by the PyTorch job manifest.
$ kubectl get pod pytorch-trainer -o yaml
...
spec:
  containers:
  - name: pytorch
    image: pytorch/pytorch:latest
    securityContext:
      # No runAsUser defined → defaults to root (0)

Because runAsUser is missing, the container runs as root, violating the PSP.

Solution

Two practical approaches:

Option 1 – Override the container user

Modify the pod spec to run as a non‑root UID (e.g., 1000) and drop unnecessary capabilities.

# pytorch-job.yaml (before)
apiVersion: batch/v1
kind: Job
metadata:
  name: pytorch-trainer
spec:
  template:
    spec:
      containers:
      - name: pytorch
        image: pytorch/pytorch:latest
        command: ["python", "train.py"]
      restartPolicy: Never

# pytorch-job.yaml (after)
apiVersion: batch/v1
kind: Job
metadata:
  name: pytorch-trainer
spec:
  template:
    spec:
      securityContext:
        runAsUser: 1000          # Enforce non‑root
        runAsGroup: 1000
        fsGroup: 1000
      containers:
      - name: pytorch
        image: pytorch/pytorch:latest
        command: ["python", "train.py"]
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
      restartPolicy: Never

Option 2 – Adjust the PSP for the development namespace

If running as root is unavoidable (e.g., legacy images), create a permissive PSP and bind it to a dedicated service account used only for local development.

# dev-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: dev-allow-root
spec:
  runAsUser:
    rule: RunAsAny
  privileged: false
  allowPrivilegeEscalation: true
  volumes:
    - '*'
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny

# dev-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: dev-psp-role
  namespace: default
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['dev-allow-root']

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-psp-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: dev-sa
  namespace: default
roleRef:
  kind: Role
  name: dev-psp-role
  apiGroup: rbac.authorization.k8s.io

Then reference the service account in the job:

# pytorch-job.yaml (using dev-sa)
...
spec:
  template:
    spec:
      serviceAccountName: dev-sa
      containers:
      - name: pytorch
        image: pytorch/pytorch:latest
        command: ["python", "train.py"]
      restartPolicy: Never

Why the fix works

  • Option 1 satisfies the existing restricted-psp by providing a non‑root UID, aligning with the MustRunAsNonRoot rule.
  • Option 2 creates a scoped exception that only the designated service account can use, preserving security for other workloads while allowing the PyTorch image to run unchanged.

Verify

After applying the chosen fix, confirm that the pod is admitted and the training job starts:

$ kubectl apply -f pytorch-job.yaml
job.batch/pytorch-trainer created

$ kubectl get pods -w
NAME                     READY   STATUS    RESTARTS   AGE
pytorch-trainer-xxxxxx   1/1     Running   0          10s

Check the container logs to ensure the training script is executing:

$ kubectl logs -f pytorch-trainer-xxxxxx
[INFO] Starting training...
Epoch 1/10 - loss: 0.5432
...

Optionally, verify the effective UID inside the container:

$ kubectl exec -it pytorch-trainer-xxxxxx -- id
uid=1000(gpuser) gid=1000(gpuser) groups=1000(gpuser)

Prevent

  • Include a securityContext.runAsUser in all pod templates that use images built to run as root.
  • Maintain a separate development namespace with a permissive PSP bound to a dedicated service account; keep production namespaces locked down with restricted-psp.
  • Automate PSP compliance checks in CI pipelines (e.g., using kube-score or conftest).
  • Document image provenance and required security contexts in the repository README to avoid accidental root runs.
  • Enable the PodSecurity admission controller (v1.25+) with the restricted profile as the default for production clusters, and use the baseline profile for local clusters where flexibility is needed.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the same PyTorch image work on Minikube but not on Kind?

    Minikube often runs with the alwaysAllow PSP or without PSP enabled, whereas Kind’s default configuration includes a restrictive PSP. The difference in admission controller policies explains the discrepancy.

  2. Can I disable the PSP admission controller entirely for local development?

    Yes, you can start Kind with --disable-admission-plugins=PodSecurityPolicy, but this removes a valuable security guard. Prefer scoped PSPs or explicit security contexts instead.

  3. My custom PyTorch Dockerfile already sets USER 1000. Why am I still seeing the denial?

    Check that the pod spec does not override the image’s USER with a securityContext.runAsUser of 0. Also verify that the image’s USER is not reset by an entrypoint script.

  4. Is there a way to see which PSP is evaluated for a given pod?

    Use kubectl auth can-i create pod --as=system:serviceaccount:default:default --namespace=default --dry-run=client -o yaml or inspect the admission webhook logs (e.g., journalctl -u kube-apiserver) to see the PSP name that caused the denial.

  5. How do I test PSP changes without affecting existing workloads?

    Create a temporary namespace, bind the new PSP to a dedicated service account, and run a test pod there. This isolates the policy change from production namespaces.