NVIDIA GPU CRD Validation Failure In Kubernetes

Q: Investigation – Debugging the Validation Failure Inspect the CRD definitions installed by the operator: kubectl get crd gpuallocations.nvidia.com -o yaml | grep -A5 "openAPIV3Schema" Expected output includes the spec.count and spec.migProfile fields. Check the API version used in the manifest: cat gpu-allocation.yaml | grep "apiVersion" If it shows nvidia.com/v1alpha1, the manifest is outdated. Review the admission webhook logs (usually in the gpu-operator namespace): kubectl logs -n gpu-operator -l app=gpu-operator-webhook -c webhook Look for lines containing "spec.gpus" or "migProfile" to confirm which schema rule failed. Validate the resource name referenced in the pod spec: kubectl describe pod my-trainer-pod | grep nvidia.com If the output shows nvidia.com/gpu-count, the pod is using the deprecated name. Confirm the Kubernetes version: kubectl version --short K8s 1.28+ requires strict OpenAPI compliance, which explains newly‑enforced optional fields. Solution – Align Manifests with the Current GPU Operator Schema 1. Update API version and kind

Change the CRD reference from v1alpha1 to v1beta1 and ensure the kind matches the installed CRD.

Q: 2. Use the correct resource name in pod specs

# Before (deprecated) resources: limits: nvidia.com/gpu-count: 2 # After (current) resources: limits: nvidia.com/gpu: 2 3. Add required fields for MIG (if applicable) If the cluster has MIG enabled, the migProfile field must be present. Example for an A100 node requesting a 1g.5gb profile:

Q: 4. Re‑apply the corrected manifests

kubectl apply -f gpu-allocation.yaml kubectl apply -f training-pod.yaml 5. (Optional) Pin the GPU Operator version To avoid future schema surprises, lock the Helm chart version and set operator.crd.enabled=false after the initial install, then manage CRDs via a version‑controlled repository.

In this article

Table of Contents

Problem – CRD Validation Failure When Deploying NVIDIA GPU Workloads

After upgrading the NVIDIA GPU Operator (or during a fresh installation) AI training pods are rejected by the Kubernetes API server with errors such as:


admission webhook "validation.gpu-operator.nvidia.com" denied the request: spec.gpus: Invalid value: "": required property "count" missing
error: unable to recognize "gpu-crd.yaml": no matches for kind "GPU" in version "nvidia.com/v1alpha1"
validation failed: spec.resourceName: Invalid value: "nvidia.com/gpu-count": must be one of ["nvidia.com/gpu"]
Webhook "gpu-operator-validation" failed with error: failed to decode request body: json: unknown field "migProfile"

These errors prevent the pod from being created, causing AI training jobs to stall and consuming valuable GPU capacity.

Root Cause – Schema Mismatch and Resource Naming Drift

The NVIDIA GPU Operator defines several Custom Resource Definitions (CRDs) that describe how GPUs are requested, allocated, and optionally partitioned with MIG. The validation failures stem from one or more of the following mismatches:

API version drift: Manifests still reference nvidia.com/v1alpha1 while the operator now serves nvidia.com/v1beta1 (see the GPU Operator documentation).
Resource name change: The device plugin renamed the resource from nvidia.com/gpu-count to nvidia.com/gpu (see the Device Plugin README).
New required fields: Enabling MIG on A100 nodes introduced the migProfile field in the GPUAllocation CRD. Older manifests that omit this field are rejected (GitHub issue #657).
Stricter OpenAPI v3 enforcement: Kubernetes 1.28+ validates optional fields more aggressively, causing manifests that previously omitted empty objects to fail (Kubernetes CRD versioning docs).

In short, the admission webhook validation.gpu-operator.nvidia.com is rejecting the request because the submitted custom resource does not satisfy the current OpenAPI schema.

Investigation – Debugging the Validation Failure

Inspect the CRD definitions installed by the operator:
```
kubectl get crd gpuallocations.nvidia.com -o yaml | grep -A5 "openAPIV3Schema"
```
Expected output includes the spec.count and spec.migProfile fields.
Check the API version used in the manifest:
```
cat gpu-allocation.yaml | grep "apiVersion"
```
If it shows nvidia.com/v1alpha1, the manifest is outdated.
Review the admission webhook logs (usually in the gpu-operator namespace):
```
kubectl logs -n gpu-operator -l app=gpu-operator-webhook -c webhook
```
Look for lines containing "spec.gpus" or "migProfile" to confirm which schema rule failed.
Validate the resource name referenced in the pod spec:
```
kubectl describe pod my-trainer-pod | grep nvidia.com
```
If the output shows nvidia.com/gpu-count, the pod is using the deprecated name.
Confirm the Kubernetes version:
```
kubectl version --short
```
K8s 1.28+ requires strict OpenAPI compliance, which explains newly‑enforced optional fields.

Solution – Align Manifests with the Current GPU Operator Schema

1. Update API version and kind

Change the CRD reference from v1alpha1 to v1beta1 and ensure the kind matches the installed CRD.

# Before (invalid on v1.9+)
apiVersion: nvidia.com/v1alpha1
kind: GPUAllocation
metadata:
  name: training-gpu
spec:
  count: 2

# After (compatible with GPU Operator v1.9)
apiVersion: nvidia.com/v1beta1
kind: GPUAllocation
metadata:
  name: training-gpu
spec:
  count: 2
  # Include migProfile only when MIG is enabled; otherwise omit

2. Use the correct resource name in pod specs

# Before (deprecated)
resources:
  limits:
    nvidia.com/gpu-count: 2

# After (current)
resources:
  limits:
    nvidia.com/gpu: 2

3. Add required fields for MIG (if applicable)

If the cluster has MIG enabled, the migProfile field must be present. Example for an A100 node requesting a 1g.5gb profile:

apiVersion: nvidia.com/v1beta1
kind: GPUAllocation
metadata:
  name: mig-a100
spec:
  count: 1
  migProfile: "1g.5gb"

4. Re‑apply the corrected manifests

kubectl apply -f gpu-allocation.yaml
kubectl apply -f training-pod.yaml

5. (Optional) Pin the GPU Operator version

To avoid future schema surprises, lock the Helm chart version and set operator.crd.enabled=false after the initial install, then manage CRDs via a version‑controlled repository.

Verification – Confirming the Fix Works

Check that the custom resource was created without errors:
```
kubectl get gpuallocation training-gpu -o yaml
```

Verify the pod schedules and runs:

kubectl get pod training-pod -o wide
kubectl logs training-pod -c trainer

Inside the running container, confirm GPU visibility:
```
kubectl exec -it training-pod -- nvidia-smi
```
Output should list the allocated GPU(s) and, if MIG is used, the appropriate MIG instances.
Inspect the webhook metrics (if Prometheus is enabled) for a drop in validation_failure_total counters.

Prevention – Guardrails to Avoid Future CRD Validation Breakages

Version‑controlled CRD manifests: Store the exact CRD YAMLs matching the GPU Operator version in Git and reference them in CI pipelines.
Schema linting: Run kubectl apply --dry-run=client -f <manifest> as part of the PR check; it will surface OpenAPI mismatches early.
Helm values lock: Set operator.crd.version=nvidia.com/v1beta1 in values.yaml and upgrade the chart only after reviewing the release notes.
Monitoring webhook rejections: Alert on validation.gpu-operator.nvidia.com admission webhook 5xx or 4xx responses via Prometheus rule:
```
rate(admission_webhook_response_total{webhook="validation.gpu-operator.nvidia.com",code=~"4.."}[5m]) > 0
```
Document MIG requirements: Include a migProfile field in all GPUAllocation manifests for clusters where MIG is enabled; otherwise explicitly set migEnabled: false if the operator supports it.

FAQ – Common Follow‑Up Questions

Why does the pod work on a dev cluster but fail after upgrading the GPU Operator?
Because the dev cluster still runs an older operator version that serves the v1alpha1 CRD. The upgrade introduced v1beta1 with stricter validation, causing legacy manifests to be rejected.
Can I keep using the old nvidia.com/gpu-count resource name?
No. The device plugin removed that alias in v1.9. Updating the pod spec to nvidia.com/gpu is required; otherwise the admission webhook will reject the request.
My workload uses MIG but I don’t want to specify migProfile in every manifest. Is there a default?
The operator does not apply a default MIG profile. You must either specify migProfile per allocation or disable MIG on the node pool via the operator’s migManager.enabled=false setting.
How can I test a manifest against the current CRD schema without applying it?
Use kubectl apply --dry-run=client -f <file> or the kubeval tool pointed at the CRD’s OpenAPI v3 schema.
After fixing the CRD, pods still show “Insufficient nvidia.com/gpu” errors.
Check that the node’s allocatable field lists the expected GPU count (kubectl describe node <node> | grep nvidia.com/gpu) and that the NVIDIA Device Plugin DaemonSet is healthy. A mismatch often indicates the plugin failed to register the devices after the operator upgrade.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub