Azure VM AI Microservice Deployment Fails With Custom Resource Definition Error

Q: Debug – Step‑by‑Step Investigation 1. Verify Kubernetes version on the control plane

kubectl version --short Typical output from a mis‑aligned cluster:

Q: 2. Inspect the CRD definition that caused the failure

kubectl get crd mycrd.mygroup.com -o yaml Key fields to look for:

Q: 3. Check CRD registration status

kubectl get crd mycrd.mygroup.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}' If the result is False or empty, the CRD has not been fully established yet.

Q: 4. Review Flux logs for ordering information

kubectl logs -n flux-system deployment/flux-controller-manager | grep mycrd Typical log snippet indicating a race:

Q: 5. Confirm Helm chart CRD version

grep -A3 "apiVersion:" charts/myai-service/crds/mycrd.yaml Example output from a chart built for v1.24:

Q: Solution – Align Versions, Control Installation Order, and Guard Against Overwrites 1. Regenerate CRDs for the target Kubernetes version

Use controller-gen or kubebuilder with --output-version=v1 to produce a 1.22‑compatible schema (omit openAPIV3Schema if not supported).

Q: 2. Pin Helm’s CRD handling to “skip” on upgrade

Set crd.keep=true in helm upgrade to avoid overwriting an already‑installed CRD with an incompatible version.

In this article

Table of Contents

Problem – CRD Validation Failure When Deploying AI Microservices on Azure VMs

In a hybrid‑cloud environment the control plane runs Kubernetes (e.g., v1.22) on Azure Virtual Machines, while AI microservices are delivered via Helm charts through a Flux CD GitOps pipeline. Deployments consistently abort with errors such as:

error: unable to recognize "myservice.yaml": no matches for kind "MyAIService" in version "mygroup/v1beta1"
CustomResourceDefinition "mycrd.mygroup.com" is invalid: spec.versions[0].schema.openAPIV3Schema: Invalid value: ...

Other observed symptoms include intermittent “admission webhook denied the request” messages and “metadata.annotations: Too long” failures when large model descriptors are stored directly in the CRD.

Root Cause – Mismatched CRD Schema Versions and Race Conditions

The failures stem from two tightly coupled issues that are common on self‑managed Azure VM clusters:

API‑server version skew. Helm charts were generated against Kubernetes 1.24 CRD schemas (which require spec.versions[0].schema.openAPIV3Schema fields) while the actual API server runs 1.22. The older API server rejects the newer schema, producing the “Invalid value” validation error. This is documented in the Kubernetes CRD versioning guide.
GitOps race condition. Flux CD applies CRDs before the API server has finished registering them. Subsequent resources that reference the new kind are processed while the CRD is still “Pending”, leading to “no matches for kind” errors. The Flux discussion #2191 describes this exact scenario.

Both issues violate the expectation that the control plane and CRD definitions are in lock‑step, a requirement emphasized in the Azure VM‑based Kubernetes provisioning documentation.

Debug – Step‑by‑Step Investigation

1. Verify Kubernetes version on the control plane

kubectl version --short

Typical output from a mis‑aligned cluster:

Server Version: v1.22.6
Client Version: v1.25.0

2. Inspect the CRD definition that caused the failure

kubectl get crd mycrd.mygroup.com -o yaml

Key fields to look for:

spec:
  versions:
  - name: v1beta1
    served: true
    storage: true
    schema:
      openAPIV3Schema:   # <-- present only in 1.24+ charts
        type: object
        properties: ...

3. Check CRD registration status

kubectl get crd mycrd.mygroup.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'

If the result is False or empty, the CRD has not been fully established yet.

4. Review Flux logs for ordering information

kubectl logs -n flux-system deployment/flux-controller-manager | grep mycrd

Typical log snippet indicating a race:

time="2024-03-15T12:04:12Z" level=info msg="Applying resource" kind=MyAIService name=myservice-01
time="2024-03-15T12:04:12Z" level=error msg="no matches for kind \"MyAIService\" in version \"mygroup/v1beta1\""

5. Confirm Helm chart CRD version

grep -A3 "apiVersion:" charts/myai-service/crds/mycrd.yaml

Example output from a chart built for v1.24:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycrd.mygroup.com
spec:
  versions:
  - name: v1beta1
    schema:   # <-- present only in newer charts

Solution – Align Versions, Control Installation Order, and Guard Against Overwrites

1. Regenerate CRDs for the target Kubernetes version

Use controller-gen or kubebuilder with --output-version=v1 to produce a 1.22‑compatible schema (omit openAPIV3Schema if not supported).

# Before (generated for 1.24)
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycrd.mygroup.com
spec:
  versions:
  - name: v1beta1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        ...

# After (compatible with 1.22)
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycrd.mygroup.com
spec:
  versions:
  - name: v1beta1
    served: true
    storage: true
    # schema omitted because 1.22 does not enforce it

2. Pin Helm’s CRD handling to “skip” on upgrade

Set crd.keep=true in helm upgrade to avoid overwriting an already‑installed CRD with an incompatible version.

helm upgrade myai-service ./charts/myai-service \
  --install \
  --set crd.keep=true \
  --namespace ai-services

3. Enforce CRD‑first installation in Flux

Split the HelmRelease into two separate releases: one that only installs CRDs, and a second that deploys the microservice resources. Use dependsOn to guarantee ordering.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: myai-crds
spec:
  chart:
    spec:
      chart: myai-service
      version: "1.3.0"
      sourceRef:
        kind: HelmRepository
        name: myrepo
  values:
    crds:
      install: true
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: myai-service
spec:
  dependsOn:
  - name: myai-crds
  chart:
    spec:
      chart: myai-service
      version: "1.3.0"
      sourceRef:
        kind: HelmRepository
        name: myrepo
  values:
    crds:
      install: false

4. Align client and server versions

Upgrade the Azure VM node pool to match the kubectl client (or downgrade the client). Azure VM scale‑set upgrade logs should show the node version; ensure it is ≥ 1.22 when the API server is 1.22.

az vmss update \
  --resource-group rg-ai \
  --name vmss-k8s-nodes \
  --set upgradePolicy.mode=Automatic \
  --set virtualMachineProfile.extensionProfile.extensions[0].typeHandlerVersion=1.22

Verification – Confirm the Fix Works End‑to‑End

1. Re‑apply the CRD and watch the establishment condition

kubectl apply -f charts/myai-service/crds/mycrd.yaml
kubectl get crd mycrd.mygroup.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'

Expected output: True

2. Deploy a sample AI microservice resource

cat > sample.yaml <<EOF
apiVersion: mygroup/v1beta1
kind: MyAIService
metadata:
  name: demo-service
spec:
  modelPath: /models/resnet50.onnx
  replicas: 2
EOF

kubectl apply -f sample.yaml
kubectl get myaiservice demo-service -o yaml

Successful kubectl get output confirms that the API server accepted the resource.

3. Validate Flux reconciliation

kubectl get helmrelease -n ai-services -o yaml | grep -A2 "status:"

All HelmReleases should show Ready: true and no pending CRD errors.

Prevention – Guardrails for Future Deployments

Version‑aware CI pipeline. Generate CRDs with the same --kube-version flag used in production clusters.
Explicit CRD install step. In any GitOps workflow, separate CRD manifests from application manifests and enforce a dependency graph.
Admission webhook health checks. Monitor apiserver_admission_webhook_failure_total metrics; alert on spikes that could indicate schema mismatches.
Annotation size limit awareness. Keep model metadata out of annotations; store large blobs in ConfigMaps or Secrets (max 1 MiB) to avoid “metadata.annotations: Too long” errors.
Node‑to‑API version policy. Use Azure VMSS rolling upgrade policies that prevent accidental downgrade of nodes below the control‑plane version.

FAQ – Common Follow‑Up Questions

Why does the CRD work locally with kubectl apply but fail in the Flux pipeline?
Flux may apply the dependent resources before the CRD registration completes. Splitting the HelmRelease and using dependsOn ensures proper ordering.
Can I keep the newer CRD schema and still run on an older Kubernetes version?
No. The API server validates the schema against its own version. Either upgrade the control plane or downgrade the CRD generation to match the server.
What is the purpose of the crd.keep flag in Helm?
It tells Helm to skip CRD upgrades on subsequent releases, preventing accidental schema overwrites that could break existing resources.
How do I detect a version skew before it causes a deployment failure?
Automate a pre‑flight check that compares kubectl version --short output against the apiVersion fields in your chart’s CRDs. Fail the CI job if they differ.
Is there a way to increase the annotation size limit for model configuration?
The limit (262 144 bytes) is hard‑coded in the API server. Store large configuration in a Secret or ConfigMap and reference it from the CRD instead.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub