background pattern

Kubernetes CRD validation failure during AI job deployment

by

Kubernetes CRD Validation Failure During AI Job Deployment

Problem

When deploying an AI training job to the staging cluster (Kubernetes v1.24) via Helm, the kubectl apply step aborts with a CRD validation error. The custom resource (AITrainingJob) is accepted by the Helm chart, but the API server rejects it, preventing the job from being created. The failure allows malformed configurations to slip through CI, leading to runtime crashes once the job is scheduled.

Typical error output:


error: error validating "ai-training-job.yaml": error validating data: ValidationError(AITrainingJob.spec): missing required field "resources", unknown field "gpuCount" in io.k8s.api.core.v1.PodSpec

Impact includes:

  • Helm release stuck in FAILED state.
  • Prometheus alerts fire for helm_release_failed.
  • Staging environment cannot be used for end‑to‑end testing of new model versions.

Root Cause

The AITrainingJob CRD defines an OpenAPI v3 schema that is out of sync with the Helm chart’s generated manifest. Two specific mismatches caused the validation failure:

  1. Missing required field in the schema: The CRD marks spec.resources as required, but the Helm values file omits it for jobs that rely on default resource limits. The API server therefore rejects any manifest lacking this field.
  2. Stale field name: The chart still references a legacy field gpuCount that was renamed to gpu in the CRD version v1beta2. Because the CRD schema does not allow gpuCount, the API server reports an “unknown field” error.

Both issues stem from a version drift between the CRD definition (stored in the cluster) and the Helm chart source code. Since the staging cluster was upgraded to Kubernetes 1.24, the CRD version was also bumped from v1beta1 to v1, but the chart was never updated accordingly.

Debug

Step‑by‑step investigation that reproduced the failure:

  1. Render the Helm chart without applying it:

helm template ai-pipeline ./charts/ai-pipeline \
  --values values/staging.yaml \
  > rendered.yaml
  1. Inspect the rendered custom resource:

cat rendered.yaml | grep -A5 "kind: AITrainingJob"

Result:


apiVersion: pipelines.example.com/v1beta2
kind: AITrainingJob
metadata:
  name: model-x-train
spec:
  model: "model-x"
  dataset: "s3://datasets/x"
  gpuCount: 4
  1. Check the live CRD schema:

kubectl get crd aitrainingjobs.pipelines.example.com -o yaml \
  | yq '.spec.versions[] | select(.name=="v1") | .schema.openAPIV3Schema'

Key excerpt of the schema:


spec:
  type: object
  required:
    - model
    - dataset
    - resources
  properties:
    model:
      type: string
    dataset:
      type: string
    resources:
      type: object
      required:
        - cpu
        - memory
    gpu:
      type: integer
  1. Validate the manifest against the schema using kubectl apply --dry-run=client:

kubectl apply -f rendered.yaml --dry-run=client
error: error validating "rendered.yaml": error validating data: ValidationError(AITrainingJob.spec): missing required field "resources", unknown field "gpuCount"

These steps confirm that the rendered manifest does not satisfy the CRD schema.

Solution

Two corrective actions were required:

1. Align Helm chart values with the CRD schema

Update values/staging.yaml to provide the required resources block and replace gpuCount with the new gpu field.

Before (values/staging.yaml)


aiTrainingJob:
  model: "model-x"
  dataset: "s3://datasets/x"
  gpuCount: 4

After


aiTrainingJob:
  model: "model-x"
  dataset: "s3://datasets/x"
  resources:
    cpu: "8"
    memory: "32Gi"
  gpu: 4

2. Bump the chart’s apiVersion to match the cluster CRD

Modify the template that renders the custom resource to use apiVersion: pipelines.example.com/v1 instead of v1beta2.

Before (template snippet)


apiVersion: pipelines.example.com/v1beta2
kind: AITrainingJob
metadata:
  name: {{ .Release.Name }}-{{ .Values.aiTrainingJob.model }}
spec:
{{ toYaml .Values.aiTrainingJob | indent 2 }}

After


apiVersion: pipelines.example.com/v1
kind: AITrainingJob
metadata:
  name: {{ .Release.Name }}-{{ .Values.aiTrainingJob.model }}
spec:
{{ toYaml .Values.aiTrainingJob | indent 2 }}

After committing these changes, re‑run the Helm upgrade.


helm upgrade --install ai-pipeline ./charts/ai-pipeline \
  --values values/staging.yaml \
  --namespace ai-pipeline

The release now succeeds and the AITrainingJob object appears in the cluster.

Verify

Verification steps to ensure the fix works in staging:

  1. Run a client‑side dry‑run:

kubectl apply -f rendered.yaml --dry-run=client
# No output means validation passed
  1. Check the actual object:

kubectl get aitrainingjobs pipelines.example.com -n ai-pipeline -o yaml

Expected fields:


spec:
  model: "model-x"
  dataset: "s3://datasets/x"
  resources:
    cpu: "8"
    memory: "32Gi"
  gpu: 4
  1. Confirm the job pod starts:

kubectl get pods -n ai-pipeline -l app=ai-training-job

All pods should transition to Running and Prometheus should show the ai_training_job_success_total metric increment.

Prevent

Preventative measures to avoid recurrence:

  • Schema‑driven CI lint: Add a step that runs kubectl apply --dry-run=client against the rendered Helm manifest in the CI pipeline.
  • Helm chart tests: Use helm test with a dedicated test chart that includes a minimal AITrainingJob manifest.
  • OPA Gatekeeper policy: Enforce that all AITrainingJob resources contain spec.resources and do not contain deprecated fields.
  • Version bump checklist: Whenever the cluster’s CRD version is upgraded, update the chart’s apiVersion and run helm template diff checks.
  • Documentation sync: Keep the CRD reference documentation (OpenAPI spec) in a version‑controlled file and generate Helm values schema from it (e.g., using helm-schema-gen).

FAQ

  1. Why does the validation error appear only in staging and not in dev?
    The dev cluster still runs the legacy CRD version v1beta2, which does not require the resources field. Staging was upgraded to v1, introducing stricter schema enforcement.
  2. Can I disable CRD validation to bypass the error?
    You can set --validate=false on kubectl apply, but this defeats the purpose of schema enforcement and can lead to runtime failures. The proper fix is to align manifests with the CRD.
  3. How do I discover which fields are required by a CRD?
    Use kubectl explain aitrainingjob.spec or inspect the CRD’s OpenAPI schema via kubectl get crd <name> -o yaml and look for the required array under spec.properties.
  4. What if I need to keep the old field name for backward compatibility?
    Add an x-kubernetes-preserve-unknown-fields: true entry for the legacy field in the CRD schema, or implement a conversion webhook that maps gpuCount to gpu during admission.
  5. Is there a way to automatically update Helm values when the CRD schema changes?
    Yes. Generate a JSON schema from the CRD (e.g., kubectl get crd <name> -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema}' > schema.json) and feed it to helm schema-gen to produce an up‑to‑date values.schema.json file that Helm can validate against.

Related Topic Hub: Distributed Systems Troubleshooting Hub

Related Articles