Kubernetes CRD validation errors after schema update

Problem – Validation Errors After Updating a CRD Schema

After a recent schema change to a shared Custom Resource Definition (CRD) used by multiple AI/ML services, kubectl apply and CI/CD pipelines began failing with errors such as:

error: admission webhook "v1.crdvalidation.k8s.io" denied the request: spec: Required value

Other observed messages include:

• validation failed: spec.replicas: Invalid value: "two": must be an integer
• error converting YAML to JSON: yaml: line 12: did not find expected ':'
• no matches for kind "MLJob" in version "ml.example.com/v2beta1"

These errors manifest during creation or update of custom resources across several namespaces, causing model deployment pipelines to stall and production inference services to become unavailable.

Root Cause – How Schema Changes Break Existing Resources

The CRD schema is defined using an OpenAPI v3 validation block. When a new field is marked required: true or an existing field’s type is altered, the built‑in CRD validation webhook (v1.crdvalidation.k8s.io) re‑evaluates every incoming object against the updated schema.

Key mechanisms involved (see Kubernetes API conventions and the CRD versioning guide):

• Structural schema enforcement – Starting with Kubernetes 1.22, preserveUnknownFields: false is the default, causing any field not explicitly described in the schema to be rejected.
• Version bump side effects – Adding a new apiVersion (e.g., v1beta2) without removing the old version can lead to “no matches for kind” errors if clients still reference the removed version.
• Conversion webhook expectations – When spec.versions[].served is true for multiple versions, a conversion webhook must translate objects between versions. Missing conversion logic results in schema mismatches for existing resources.

In the real incident on GKE (see evidence package), the team added a required field metadata.annotations.aiPlatform to the MLJob CRD without providing a migration path. Existing MLJob objects lacked this field, so the validation webhook rejected them with “spec: Required value”.

Debug – Investigation Steps

1. Inspect the CRD definition

kubectl get crd mljobs.ml.example.com -o yaml > crd.yaml

Key sections to review:

Section	What to Look For
spec.validation.openAPIV3Schema	Required fields, type definitions, preserveUnknownFields
spec.versions	Served vs storage versions, conversion webhook config
metadata.annotations	Any version‑specific notes about migration

2. Reproduce the error with a minimal manifest

cat > broken.yaml <<EOF
apiVersion: ml.example.com/v1beta1
kind: MLJob
metadata:
  name: sample-job
spec:
  model: my-model
  # missing the newly required field `metadata.annotations.aiPlatform`
EOF

kubectl apply -f broken.yaml

Expected output (matches the logs observed in the field):

error: admission webhook "v1.crdvalidation.k8s.io" denied the request: spec: Required value

3. Verify the cluster version and CRD compatibility

kubectl version --short

Managed clusters (EKS, GKE, AKS) enforce stricter validation starting with Kubernetes 1.22 (see the EKS version matrix). If the cluster was upgraded recently (e.g., from 1.22 to 1.24), the new validation rules may be the trigger.

4. Check conversion webhook logs (if applicable)

kubectl logs -n ml-platform deployment/mljob-conversion-webhook -c webhook

Look for messages like “conversion failed: missing required field”. Missing conversion logic is a common cause noted in GitHub issue #102317.

Solution – Fixing the Validation Errors

Option A: Add a Migration Job to Populate Required Fields

When a field becomes required, back‑fill existing resources before the schema is enforced.

# migration-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: mljob-migration
spec:
  template:
    spec:
      serviceAccountName: ml-platform-admin
      restartPolicy: OnFailure
      containers:
      - name: migrate
        image: bitnami/kubectl:latest
        command:
        - /bin/sh
        - -c
        - |
          kubectl get mljobs.ml.example.com -A -o json | \
          jq -c '.items[] | select(.metadata.annotations.aiPlatform == null)' | \
          while read -r obj; do
            name=$(echo "$obj" | jq -r .metadata.name)
            ns=$(echo "$obj" | jq -r .metadata.namespace)
            kubectl patch mljob $name -n $ns \
              --type merge -p '{"metadata":{"annotations":{"aiPlatform":"default"}}}'
          done

Run the job, verify that all resources now contain the annotation, then re‑apply the updated CRD.

Option B: Relax the Schema Temporarily

If an immediate migration is not feasible, modify the CRD to make the new field optional and set preserveUnknownFields: true for backward compatibility.

# before (failing schema)
spec:
  validation:
    openAPIV3Schema:
      type: object
      required:
      - spec
      properties:
        spec:
          type: object
          required:
          - model
          - replicas
          properties:
            model:
              type: string
            replicas:
              type: integer
            # New required field added
            metadata:
              type: object
              required:
              - annotations
              properties:
                annotations:
                  type: object
                  required:
                  - aiPlatform
                  properties:
                    aiPlatform:
                      type: string

# after (temporary relaxation)
spec:
  preserveUnknownFields: true   # <--- added
  validation:
    openAPIV3Schema:
      type: object
      required:
      - spec
      properties:
        spec:
          type: object
          required:
          - model
          - replicas
          properties:
            model:
              type: string
            replicas:
              type: integer
            metadata:
              type: object
              properties:               # removed required list
                annotations:
                  type: object
                  properties:
                    aiPlatform:
                      type: string

Deploy the relaxed CRD, run the migration job, then revert to the strict schema once all resources are updated.

Option C: Use Versioned Conversion

Introduce a new storage version (e.g., v1beta2) that includes the required field, and implement a conversion webhook that injects a default value for existing objects.

# crd.yaml snippet with conversion
spec:
  versions:
  - name: v1beta1
    served: true
    storage: false
  - name: v1beta2
    served: true
    storage: true
    schema:
      openAPIV3Schema: ...   # includes required field
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: mljob-converter
          namespace: ml-platform
          path: /convert
      conversionReviewVersions: ["v1"]

The webhook adds metadata.annotations.aiPlatform: "default" when converting from v1beta1 to v1beta2, eliminating the validation failure without a separate migration job.

Verify – Confirming the Fix

1. Re‑apply the updated CRD (or the relaxed version) and ensure kubectl get crd shows the new schema.
2. Re‑run the minimal manifest used in the debug step; it should now succeed:

kubectl apply -f broken.yaml
customresource.ml.example.com/sample-job created

3. Check that all existing resources contain the required field:

kubectl get mljobs.ml.example.com -A -o jsonpath='{range .items[*]}{.metadata.name}{" -> "}{.metadata.annotations.aiPlatform}{"\n"}{end}'

All entries should output a non‑empty value (e.g., default).

4. Monitor the admission webhook logs for a period of 15‑30 minutes to ensure no further validation rejections appear.

Prevent – Best Practices for Evolving CRDs

• Never add a required field without a migration path. Prefer making fields optional first, then back‑fill, then mark required in a subsequent version.
• Use versioned CRDs with storage version separation. Keep the old version served for a deprecation window while a conversion webhook handles defaults.
• Pin preserveUnknownFields: false explicitly. Document the impact in the CRD changelog to avoid surprises when clusters upgrade.
• Automate schema compatibility checks. Run kubectl diff -f in CI pipelines and validate that existing resources pass against the new schema using kubectl apply --dry-run=client.
• Leverage the Kubernetes API conventions. Follow the guidelines in the API conventions for field naming, requiredness, and versioning.

FAQ – Common Follow‑Up Questions

1. Why does the error appear only after a cluster upgrade?
   Managed clusters (EKS/GKE/AKS) enable stricter OpenAPI validation starting with Kubernetes 1.22. Fields that were previously ignored become enforced, causing “spec: Required value” failures.
2. Can I disable the built‑in CRD validation webhook?
   No. The validation webhook is integral to the API server. The correct approach is to adjust the CRD schema or provide a conversion webhook.
3. How do I test that a new required field won’t break existing resources?
   Export all existing resources, remove the new field, and run kubectl apply --dry-run=client -f <exported.yaml>. If the dry run fails, you need a migration strategy.
4. What is the difference between preserveUnknownFields: true and structural schemas?
   When true, the API server accepts fields not described in the schema, effectively bypassing validation for those fields. Setting it to false (the default in recent versions) forces strict structural validation.
5. Do I need to bump the apiVersion when adding required fields?
   Best practice is to introduce a new version (e.g., v1beta2) and mark the previous version as deprecated. This allows a conversion webhook to handle defaulting without breaking existing manifests.

Related Topic Hub: Distributed Systems Troubleshooting Hub