Problem
Deploying an AI inference service on a GPU‑enabled Kubernetes cluster fails with an admission controller error:
Error from server (Forbidden): admission webhook "gpu-policy.example.com" denied the request:
missing required label "model-version" on pod spec;
requested GPU count 2 exceeds the per‑pod limit of 1
The MutatingAdmissionWebhook enforces two policies for every pod that schedules on GPU nodes:
- Each pod must carry a
model-versionlabel that matches a known model release. - The number of GPUs requested (
nvidia.com/gpu) cannot exceed1per pod (to guarantee fair sharing).
Because the pod specification omitted the required label and requested two GPUs, the webhook rejected the creation request.
Root Cause
The failure originates from the webhook’s validation logic, which runs after any mutating step but before the pod is persisted. The webhook examines the incoming Pod object and applies the following checks (simplified Go code excerpt):
func (v *Validator) ValidatePod(pod *corev1.Pod) error {
// 1. Verify required label
if _, ok := pod.Labels["model-version"]; !ok {
return fmt.Errorf("missing required label \"model-version\" on pod spec")
}
// 2. Enforce GPU quota per pod
gpuQty := pod.Spec.Containers[0].Resources.Requests[corev1.ResourceName("nvidia.com/gpu")]
if gpuQty.Cmp(resource.MustParse("1")) > 0 {
return fmt.Errorf("requested GPU count %s exceeds the per‑pod limit of 1", gpuQty.String())
}
return nil
}
In the deployment manifest the model-version label was omitted, and the container requested 2 GPUs to satisfy a high‑throughput inference workload. Both checks failed, causing the webhook to return a 403 Forbidden response.
Debug
Step‑by‑step investigation:
- Inspect the API server response using
kubectl:
$ kubectl apply -f model-serving.yaml
Error from server (Forbidden): admission webhook "gpu-policy.example.com" denied the request: missing required label "model-version" on pod spec; requested GPU count 2 exceeds the per-pod limit of 1
- Examine the webhook logs to confirm which rule triggered:
$ kubectl logs -n kube-system -l app=gpu-policy-webhook -c webhook
2024-10-12T08:15:23Z INFO validating pod model-serving-5c9d8f9c9-abcde
2024-10-12T08:15:23Z WARN missing required label "model-version"
2024-10-12T08:15:23Z WARN GPU request 2 exceeds limit 1
2024-10-12T08:15:23Z ERROR admission denied
- Review the pod spec that triggered the webhook:
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-serving
spec:
replicas: 2
selector:
matchLabels:
app: model-serving
template:
metadata:
labels:
app: model-serving
spec:
containers:
- name: inference
image: registry.example.com/ai-model:latest
resources:
requests:
nvidia.com/gpu: 2
limits:
nvidia.com/gpu: 2
Both the missing label and the GPU request are evident.
Solution
Adjust the deployment manifest to satisfy the webhook’s policies:
Before (non‑compliant)
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-serving
spec:
replicas: 2
selector:
matchLabels:
app: model-serving
template:
metadata:
labels:
app: model-serving
spec:
containers:
- name: inference
image: registry.example.com/ai-model:latest
resources:
requests:
nvidia.com/gpu: 2
limits:
nvidia.com/gpu: 2
After (compliant)
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-serving
spec:
replicas: 2
selector:
matchLabels:
app: model-serving
template:
metadata:
labels:
app: model-serving
model-version: "v1.3.2" # <-- required label added
spec:
containers:
- name: inference
image: registry.example.com/ai-model:latest
resources:
requests:
nvidia.com/gpu: 1 # <-- respect per‑pod limit
limits:
nvidia.com/gpu: 1
Key changes:
- Added
model-version: "v1.3.2"to the pod template labels. - Reduced GPU request/limit from
2to1per pod. - If higher throughput is required, increase the
replicascount instead of per‑pod GPU count.
Verify
Re‑apply the corrected manifest and confirm successful creation:
$ kubectl apply -f model-serving.yaml
deployment.apps/model-serving configured
$ kubectl get pods -l app=model-serving -o wide
NAME READY STATUS RESTARTS AGE IP NODE
model-serving-5c9d8f9c9-abcde 1/1 Running 0 12s 10.244.1.12 gpu-node-1
model-serving-5c9d8f9c9-fghij 1/1 Running 0 12s 10.244.1.13 gpu-node-2
Additional validation steps:
- Check that the label is present:
$ kubectl get pod model-serving-5c9d8f9c9-abcde -o jsonpath='{.metadata.labels}'
{"app":"model-serving","model-version":"v1.3.2"}
- Confirm GPU allocation via
nvidia-smiinside the container:
$ kubectl exec -it model-serving-5c9d8f9c9-abcde -- nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM4... Off | 00000000:3B:00.0 Off | 0 N/A |
| N/A 33C P0 30W / 400W | 0MiB / 40960MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
Prevent
Implement safeguards to avoid recurrence:
- Admission policy documentation: Keep a version‑controlled
READMEdescribing required labels and GPU limits for all AI workloads. - CI linting: Add a
kube-linterrule that checks for the presence ofmodel-versionand validates GPU request values before merging manifests. - Policy as code: Store the webhook's validation rules in a GitOps repository; any change to limits triggers a review.
- Automated tests: Include an integration test that attempts to create a pod missing the label and expects a
403response, ensuring the webhook remains active. - Monitoring & alerting: Export webhook rejection metrics (e.g.,
admission_webhook_denied_total{reason="missing_label"}) to Prometheus and create alerts for spikes.
Related Topic Hub: Distributed Systems Troubleshooting Hub
FAQ
- Why does the webhook reject pods that already have a
model-versionlabel?
If the label value does not match an entry in the webhook’s whitelist (e.g., a stale version), the validator treats it as missing. Update the whitelist or use an approved version. - Can I request more than one GPU per pod if the workload truly needs it?
The current policy caps GPUs at 1 per pod to improve node sharing. To exceed the limit, request a policy change via the platform team, or split the workload across multiple pods. - Is there a way to automatically add the required label?
Deploy a MutatingAdmissionWebhook that injectsmodel-versionbased on image tags. However, the existing validating webhook will still reject pods that exceed GPU limits, so both policies must be satisfied. - How do I debug a rejection when the webhook logs are not accessible?
Enable--v=6on the API server to capture admission request/response payloads, or usekubectl auth can-i create pod --as=system:serviceaccount:...to isolate RBAC vs. webhook failures. - What if I need to run a batch job that temporarily needs 2 GPUs?
Create a separateJobresource with a distinctapiVersion: batch/v1and request a customResourceQuotathat the webhook recognizes as an exception, or use a dedicated namespace with a relaxed webhook configuration.