Kubernetes admission controller rejects Docker image missing security label in production

Problem

During a rolling update of the AI inference service in the production Kubernetes cluster, the admission controller rejected the new pods with the following error:


Error from server (Forbidden): admission webhook "security-labels.k8s.io" denied the request: 
container "inference" in pod "inference-7f9c5d9c9b-ktz9l" is missing required security label 
com.myorg.security.level=high

The deployment never progressed past the Pending state, and the kubectl describe pod output showed the same admission failure repeatedly.

Root Cause

The cluster enforces a custom admission webhook that validates Docker image metadata for compliance with internal security policies. The webhook checks for a specific LABEL in the image manifest:

  • com.myorg.security.level=high
  • Optional: com.myorg.security.scan=passed

In the production CI/CD pipeline, the Dockerfile for the AI inference container omitted the required label due to a recent refactor that introduced a multi‑stage build. The final stage copied the binary from a builder stage but did not inherit the label definitions, resulting in an image without the security label. The webhook therefore rejected the pod.

Debug

Step‑by‑step investigation performed in the incident:

  1. Inspect the failing pod
kubectl describe pod inference-7f9c5d9c9b-ktz9l

Key excerpt:

Events:
  Type     Reason                     Age   From                     Message
  ----     ------                     ----  ----                     -------
  Warning  FailedCreatePodAdmission    2m    security-labels.k8s.io   container "inference" is missing required security label com.myorg.security.level=high
  1. Check the image manifest for labels
docker pull registry.myorg.com/ai/inference:2023-11-15
docker inspect registry.myorg.com/ai/inference:2023-11-15 --format='{{json .Config.Labels}}' | jq .

Result:

{
  "org.opencontainers.image.created": "2023-11-15T08:12:34Z",
  "org.opencontainers.image.version": "v2.4.1"
}

Notice the absence of com.myorg.security.level.

  1. Review the Dockerfile used in CI
# Dockerfile (stage 1)
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY src/ .

# Dockerfile (stage 2) – final image
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /app /app
# NOTE: No LABEL statements here
CMD ["python", "-m", "inference"]

The LABEL directives were present in the original single‑stage Dockerfile but were inadvertently removed when the multi‑stage build was introduced.

  1. Validate the webhook configuration
kubectl get validatingwebhookconfiguration security-labels -o yaml

Relevant snippet:

rules:
- apiGroups: [""]
  apiVersions: ["v1"]
  operations: ["CREATE"]
  resources: ["pods"]
  scope: "Namespaced"
clientConfig:
  service:
    name: security-labels-webhook
    namespace: kube-system
  caBundle: ...

The webhook logic (simplified) checks the image labels via the containerd image metadata API.

Solution

Restore the required security label in the Docker image and ensure future builds cannot omit it.

Dockerfile fix

Add the label in the final stage (or in a dedicated LABEL block that is copied across stages).

Before:

# Dockerfile (stage 2) – final image
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /app /app
CMD ["python", "-m", "inference"]

After:

# Dockerfile (stage 2) – final image with security label
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /app /app
LABEL com.myorg.security.level="high" \
      com.myorg.security.scan="passed"
CMD ["python", "-m", "inference"]

Alternatively, define the label in the builder stage and propagate it using --label during the docker build command:

docker build \
  --label com.myorg.security.level=high \
  --label com.myorg.security.scan=passed \
  -t registry.myorg.com/ai/inference:2023-11-15 .

CI pipeline safeguard

Introduce a lint step that verifies required labels before pushing the image:

# .gitlab-ci.yml snippet
image_label_check:
  stage: test
  script:
    - |
      IMAGE="registry.myorg.com/ai/inference:${CI_COMMIT_TAG}"
      docker pull $IMAGE
      MISSING=$(docker inspect $IMAGE --format='{{json .Config.Labels}}' \
        | jq -r 'keys[] as $k | select($k | test("^com\\.myorg\\.security\\.")) | empty')
      if [ -z "$MISSING" ]; then
        echo "Required security labels present."
      else
        echo "Missing security labels!" && exit 1
      fi

Verify

After rebuilding and pushing the corrected image, re‑apply the deployment and confirm the pod reaches Running state.

# Re‑deploy
kubectl set image deployment/inference inference=registry.myorg.com/ai/inference:2023-11-15 --record

# Watch pod status
kubectl get pods -l app=inference -w

Expected output:

NAME                     READY   STATUS    RESTARTS   AGE
inference-7f9c5d9c9b-ktz9l   1/1     Running   0          12s

Re‑run the label inspection to confirm presence:

docker inspect registry.myorg.com/ai/inference:2023-11-15 \
  --format='{{json .Config.Labels}}' | jq .

Result:

{
  "com.myorg.security.level": "high",
  "com.myorg.security.scan": "passed",
  "org.opencontainers.image.created": "2023-11-15T08:12:34Z",
  "org.opencontainers.image.version": "v2.4.1"
}

Prevent

  • Label enforcement in CI: Use the lint step shown above or integrate hadolint rules that flag missing LABEL directives.
  • Immutable webhook policy: Keep the webhook configuration version‑controlled and audit changes via GitOps.
  • Automated image signing: Pair the label check with Notary or Cosign signatures; the webhook can reject unsigned images as an additional guard.
  • Documentation: Add a “Security Labels” checklist to the Dockerfile template used by all AI services.
  • Monitoring: Emit a custom Prometheus metric (admission_controller_label_missing_total) from the webhook and alert on spikes.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does the admission controller reject the image even though the Dockerfile contains a LABEL?
    If a multi‑stage build is used, labels defined in an earlier stage are not automatically carried over to the final stage. The final image must have the label explicitly set, either via a LABEL instruction in the last stage or by passing --label to docker build.
  2. Can I bypass the webhook for testing?
    Temporarily you can add the annotation admission.k8s.io/ignore=true to the pod spec, but this defeats the security policy and should never be used in production. The proper fix is to add the required labels.
  3. How do I know which labels the webhook expects?
    Inspect the webhook’s ConfigMap or source code. In this environment the required keys are documented in security-labels.yaml and include com.myorg.security.level (mandatory) and com.myorg.security.scan (optional).
  4. Is there a way to make the label check part of the image build process?
    Yes. Use a Dockerfile linter (e.g., hadolint) with a custom rule, or add a post‑build verification step that runs docker inspect and fails the CI job if the label is missing.
  5. What if the image is built by an external vendor?
    Require the vendor to sign the image and provide a signed attestation that includes the security label. The webhook can be extended to verify the attestation before allowing the pod to start.