Haystack pod admission controller rejection after adding GPU resource request

Problem: Haystack pod rejected by admission controller after adding GPU request

When deploying a Haystack inference service to a GPU‑enabled Kubernetes cluster (A100/H100 nodes), the pod creation fails with an admission controller error. The pod never reaches the Pending state; instead the API server returns a 400 Bad Request response indicating an invalid or missing GPU resource specification.

Root Cause

The cluster enforces a GPUQuotaAdmission webhook that validates:

• Presence of the nvidia.com/gpu resource request in the pod spec.
• Alignment of the request with the node’s accelerator label (e.g., accelerator=nvidia-a100).
• Compliance with a GPUResourcePolicy ConfigMap that defines allowed limits per namespace.

In the Haystack deployment manifest the GPU request was added only to the containers section, while the initContainers and the pod‑level resources field remained empty. The webhook treats the pod as “mixed” – some containers request GPUs, others do not – and rejects it because the policy requires a **uniform** GPU request across all containers.

Additionally, the request used the legacy resource name gpu instead of the current nvidia.com/gpu, causing the validator to treat the field as missing.

Debug: Investigation Steps

1. Capture the API server response

kubectl apply -f haystack-gpu.yaml
Error from server (BadRequest): admission webhook "gpuquota.k8s.io" denied the request: 
[GPUQuotaAdmission] Pod "haystack-inference-xyz" in namespace "ml" 
does not satisfy GPU resource policy: missing or mismatched nvidia.com/gpu request

2. Examine the pod manifest that triggered the rejection

apiVersion: apps/v1
kind: Deployment
metadata:
  name: haystack-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: haystack
  template:
    metadata:
      labels:
        app: haystack
    spec:
      containers:
      - name: haystack
        image: haystack:latest
        resources:
          requests:
            gpu: 1               # ← legacy name
          limits:
            gpu: 1
      initContainers:
      - name: model-downloader
        image: downloader:latest
        # No GPU request here

3. Verify the webhook configuration

kubectl get validatingwebhookconfigurations gpuquota -o yaml | grep -A5 "rules:"

Key excerpt:

rules:
- apiGroups: [""]
  apiVersions: ["v1"]
  operations: ["CREATE"]
  resources: ["pods"]
  scope: "Namespaced"
  failurePolicy: Fail
  matchPolicy: Exact
  objectSelector:
    matchExpressions:
    - key: gpu-enabled
      operator: In
      values: ["true"]

4. Check the policy ConfigMap

kubectl -n kube-system get configmap gpu-resource-policy -o yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-resource-policy
  namespace: kube-system
data:
  max-gpus-per-namespace: "4"
  allowed-gpu-types: "nvidia-a100,nvidia-h100"

5. Confirm node labels

kubectl get nodes -L accelerator

NAME          STATUS   ROLES    AGE   VERSION   ACCELERATOR
gpu-node-01   Ready       45d   v1.27.3   nvidia-a100
gpu-node-02   Ready       45d   v1.27.3   nvidia-h100

Solution: Align pod spec with GPU admission policy

1. Use the correct resource name

Replace gpu with nvidia.com/gpu in both requests and limits.

2. Declare GPU request at the pod level (or uniformly for all containers)

When an initContainer does not need a GPU, the pod‑level resources field must still contain the GPU request so the webhook sees a consistent declaration.

3. Ensure the namespace is labeled for GPU workloads

kubectl label namespace ml gpu-enabled=true --overwrite

4. Updated manifest (before/after)

Aspect	Before	After
Resource name	gpu	nvidia.com/gpu
Placement of request	Only in container spec	Pod‑level resources + container spec (uniform)
Namespace label	Missing	gpu-enabled=true

# haystack-gpu.yaml (fixed)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: haystack-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: haystack
  template:
    metadata:
      labels:
        app: haystack
    spec:
      # Uniform GPU request at pod level
      resources:
        requests:
          nvidia.com/gpu: 1
        limits:
          nvidia.com/gpu: 1
      containers:
      - name: haystack
        image: haystack:latest
        resources:
          # Optional: repeat for clarity; must match pod level
          requests:
            nvidia.com/gpu: 1
          limits:
            nvidia.com/gpu: 1
        ports:
        - containerPort: 8080
      initContainers:
      - name: model-downloader
        image: downloader:latest
        # No GPU request needed; inherits pod‑level request

Verify: Confirm successful admission and scheduling

1. Apply the corrected manifest

kubectl apply -f haystack-gpu.yaml

Expected output:

deployment.apps/haystack-inference created

2. Observe pod status

kubectl get pods -n ml -w

NAME                                 READY   STATUS    RESTARTS   AGE
haystack-inference-5c9d8b7f9c-9kz2l   1/1     Running   0          12s

3. Verify GPU allocation on the node

kubectl describe pod haystack-inference-5c9d8b7f9c-9kz2l -n ml | grep -i gpu

    Limits:
      nvidia.com/gpu:  1
    Requests:
      nvidia.com/gpu:  1

4. Confirm node has the expected GPU device

kubectl get node gpu-node-01 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'

8

Prevent: Guardrails to avoid recurrence

• Policy‑driven CI linting: Add a kubeval or conftest rule that checks for the presence of nvidia.com/gpu in resources when the namespace carries the gpu-enabled=true label.
• Admission webhook testing: Include a negative test case in the cluster’s e2e suite that deliberately submits a pod with mismatched GPU specs and asserts a 400 response.
• Namespace labeling automation: Use a NamespaceLifecycle controller that automatically adds gpu-enabled=true to any namespace annotated with team=gpu, reducing manual errors.
• Documentation sync: Keep the internal runbook updated with the exact resource name (nvidia.com/gpu) and the requirement for pod‑level uniformity.
• Monitoring alert: Create a Prometheus alert on admission_controller_rejection_total{reason="GPUQuotaAdmission"} to catch future regressions early.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

1. Why does the webhook reject a pod that only some containers request GPUs?
   The GPUQuotaAdmission policy enforces a uniform GPU request across the entire pod to guarantee that the scheduler can treat the pod as a single GPU‑aware unit. Mixed‑container requests break this assumption, so the webhook denies the pod.
2. Can I request multiple GPU types (e.g., A100 and H100) in the same pod?
   No. The policy’s allowed-gpu-types list is evaluated per pod, and the scheduler can only bind a pod to a node with a single accelerator label. Mixing types requires separate deployments or a multi‑node job.
3. Is gpu ever a valid resource name?
   Only in very old clusters that used the legacy NVIDIA device plugin. Modern clusters (v1.22+) require the fully qualified name nvidia.com/gpu. Using the legacy name will always fail validation.
4. How do I debug a similar admission rejection for other custom resources?
   Inspect the webhook’s logs (usually in the kube-system namespace) with kubectl logs -n kube-system -l app=gpu-admission-webhook, and reproduce the request with kubectl apply -f ... --dry-run=client -o yaml to see the exact object sent to the API server.
5. Can I bypass the GPU admission controller for a one‑off test?
   Temporarily set the webhook’s failurePolicy to Ignore or add an admission.k8s.io/ignore annotation to the pod, but this defeats the policy’s safety guarantees and should never be used in production.