Problem
During a rolling update of the AI inference service in the production Kubernetes cluster, the admission controller rejected the new pods with the following error:
Error from server (Forbidden): admission webhook "security-labels.k8s.io" denied the request:
container "inference" in pod "inference-7f9c5d9c9b-ktz9l" is missing required security label
com.myorg.security.level=high
The deployment never progressed past the Pending state, and the kubectl describe pod output showed the same admission failure repeatedly.
Root Cause
The cluster enforces a custom admission webhook that validates Docker image metadata for compliance with internal security policies. The webhook checks for a specific LABEL in the image manifest:
com.myorg.security.level=high- Optional:
com.myorg.security.scan=passed
In the production CI/CD pipeline, the Dockerfile for the AI inference container omitted the required label due to a recent refactor that introduced a multi‑stage build. The final stage copied the binary from a builder stage but did not inherit the label definitions, resulting in an image without the security label. The webhook therefore rejected the pod.
Debug
Step‑by‑step investigation performed in the incident:
- Inspect the failing pod
kubectl describe pod inference-7f9c5d9c9b-ktz9l
Key excerpt:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodAdmission 2m security-labels.k8s.io container "inference" is missing required security label com.myorg.security.level=high
- Check the image manifest for labels
docker pull registry.myorg.com/ai/inference:2023-11-15
docker inspect registry.myorg.com/ai/inference:2023-11-15 --format='{{json .Config.Labels}}' | jq .
Result:
{
"org.opencontainers.image.created": "2023-11-15T08:12:34Z",
"org.opencontainers.image.version": "v2.4.1"
}
Notice the absence of com.myorg.security.level.
- Review the Dockerfile used in CI
# Dockerfile (stage 1)
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY src/ .
# Dockerfile (stage 2) – final image
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /app /app
# NOTE: No LABEL statements here
CMD ["python", "-m", "inference"]
The LABEL directives were present in the original single‑stage Dockerfile but were inadvertently removed when the multi‑stage build was introduced.
- Validate the webhook configuration
kubectl get validatingwebhookconfiguration security-labels -o yaml
Relevant snippet:
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
scope: "Namespaced"
clientConfig:
service:
name: security-labels-webhook
namespace: kube-system
caBundle: ...
The webhook logic (simplified) checks the image labels via the containerd image metadata API.
Solution
Restore the required security label in the Docker image and ensure future builds cannot omit it.
Dockerfile fix
Add the label in the final stage (or in a dedicated LABEL block that is copied across stages).
Before:
# Dockerfile (stage 2) – final image
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /app /app
CMD ["python", "-m", "inference"]
After:
# Dockerfile (stage 2) – final image with security label
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /app /app
LABEL com.myorg.security.level="high" \
com.myorg.security.scan="passed"
CMD ["python", "-m", "inference"]
Alternatively, define the label in the builder stage and propagate it using --label during the docker build command:
docker build \
--label com.myorg.security.level=high \
--label com.myorg.security.scan=passed \
-t registry.myorg.com/ai/inference:2023-11-15 .
CI pipeline safeguard
Introduce a lint step that verifies required labels before pushing the image:
# .gitlab-ci.yml snippet
image_label_check:
stage: test
script:
- |
IMAGE="registry.myorg.com/ai/inference:${CI_COMMIT_TAG}"
docker pull $IMAGE
MISSING=$(docker inspect $IMAGE --format='{{json .Config.Labels}}' \
| jq -r 'keys[] as $k | select($k | test("^com\\.myorg\\.security\\.")) | empty')
if [ -z "$MISSING" ]; then
echo "Required security labels present."
else
echo "Missing security labels!" && exit 1
fi
Verify
After rebuilding and pushing the corrected image, re‑apply the deployment and confirm the pod reaches Running state.
# Re‑deploy
kubectl set image deployment/inference inference=registry.myorg.com/ai/inference:2023-11-15 --record
# Watch pod status
kubectl get pods -l app=inference -w
Expected output:
NAME READY STATUS RESTARTS AGE
inference-7f9c5d9c9b-ktz9l 1/1 Running 0 12s
Re‑run the label inspection to confirm presence:
docker inspect registry.myorg.com/ai/inference:2023-11-15 \
--format='{{json .Config.Labels}}' | jq .
Result:
{
"com.myorg.security.level": "high",
"com.myorg.security.scan": "passed",
"org.opencontainers.image.created": "2023-11-15T08:12:34Z",
"org.opencontainers.image.version": "v2.4.1"
}
Prevent
- Label enforcement in CI: Use the lint step shown above or integrate
hadolintrules that flag missingLABELdirectives. - Immutable webhook policy: Keep the webhook configuration version‑controlled and audit changes via GitOps.
- Automated image signing: Pair the label check with Notary or Cosign signatures; the webhook can reject unsigned images as an additional guard.
- Documentation: Add a “Security Labels” checklist to the Dockerfile template used by all AI services.
- Monitoring: Emit a custom Prometheus metric (
admission_controller_label_missing_total) from the webhook and alert on spikes.
Related Topic Hub: Distributed Systems Troubleshooting Hub
FAQ
- Why does the admission controller reject the image even though the Dockerfile contains a LABEL?
If a multi‑stage build is used, labels defined in an earlier stage are not automatically carried over to the final stage. The final image must have the label explicitly set, either via aLABELinstruction in the last stage or by passing--labeltodocker build. - Can I bypass the webhook for testing?
Temporarily you can add the annotationadmission.k8s.io/ignore=trueto the pod spec, but this defeats the security policy and should never be used in production. The proper fix is to add the required labels. - How do I know which labels the webhook expects?
Inspect the webhook’s ConfigMap or source code. In this environment the required keys are documented insecurity-labels.yamland includecom.myorg.security.level(mandatory) andcom.myorg.security.scan(optional). - Is there a way to make the label check part of the image build process?
Yes. Use aDockerfilelinter (e.g.,hadolint) with a custom rule, or add a post‑build verification step that runsdocker inspectand fails the CI job if the label is missing. - What if the image is built by an external vendor?
Require the vendor to sign the image and provide a signed attestation that includes the security label. The webhook can be extended to verify the attestation before allowing the pod to start.