Google Gemini CRD Validation Failure On Kind After Operator Install

Q: Problem – Gemini CRD Validation Failure on Kind After Operator Install

When applying a GeminiModel or GeminiEndpoint custom resource to a local kind cluster, kubectl apply aborts with validation errors such as:

Q: Root Cause Analysis Schema version mismatch

The Gemini operator ships CRDs that target Kubernetes v1.25+ (official install docs). The spec.version field was introduced in v1.25 and is required by the OpenAPI v3 schema. A kind cluster running Kubernetes 1.24 (as in the real incident) does not recognize this field, resulting in:

Q: Deprecated CRD API version

Older kind images (e.g., kindest/node:v1.22.9) expose the v1beta1 CRD API, which has been removed in v1.26. The operator’s manifest still declares apiVersion: apiextensions.k8s.io/v1beta1, leading to “no matches for kind” errors during resource creation.

Q: Feature‑gate and required fields

In some Minikube setups the CustomResourceValidation feature gate is disabled. The API server therefore skips schema validation on CRD creation, but the operator’s admission webhook still enforces required fields (model, endpointUrl, etc.). This mismatch surfaces as webhook rejections after the resource is accepted:

Q: Investigation and Debugging 1. Verify Kubernetes version and CRD API groups

kubectl version --short kubectl get crd | grep gemini Expected output on a failing kind cluster:

Q: 2. Inspect the installed CRD schema

kubectl get crd geminimodels.gemini.google.com -o yaml | grep -A5 "openAPIV3Schema" Look for the presence of spec.version and the required list. Missing entries indicate a mismatched operator version.

Q: 3. Capture the exact error from kubectl apply

kubectl apply -f gemini-model.yaml Typical output on a mismatched cluster:

Q: 4. Check the operator pod logs for webhook errors

kubectl logs -l app=gemini-operator -c webhook Sample log excerpt:

Q: Resolution – Align CRD Versions and Provide Required Fields Option A – Upgrade the local cluster to Kubernetes ≥ 1.25

For kind, pull a newer node image and recreate the cluster:

In this article

Table of Contents

Problem – Gemini CRD Validation Failure on Kind After Operator Install

When applying a GeminiModel or GeminiEndpoint custom resource to a local kind cluster, kubectl apply aborts with validation errors such as:

error: error validating “gemini-model.yaml”: error validating data: ValidationError(GeminiModel.spec): missing required field “model” in io.k8s.api.core.v1.ConfigMap

error: unable to recognize “gemini-deployment.yaml”: no matches for kind “GeminiModel” in version “gemini.google.com/v1”

These failures prevent the Gemini operator from reconciling the resource, breaking local development workflows.

Root Cause Analysis

Schema version mismatch

The Gemini operator ships CRDs that target Kubernetes v1.25+ (official install docs). The spec.version field was introduced in v1.25 and is required by the OpenAPI v3 schema. A kind cluster running Kubernetes 1.24 (as in the real incident) does not recognize this field, resulting in:

error: error validating “gemini-model.yaml”: error validating data: ValidationError(GeminiModel.spec): unknown field “version”

Deprecated CRD API version

Older kind images (e.g., kindest/node:v1.22.9) expose the v1beta1 CRD API, which has been removed in v1.26. The operator’s manifest still declares apiVersion: apiextensions.k8s.io/v1beta1, leading to “no matches for kind” errors during resource creation.

Feature‑gate and required fields

In some Minikube setups the CustomResourceValidation feature gate is disabled. The API server therefore skips schema validation on CRD creation, but the operator’s admission webhook still enforces required fields (model, endpointUrl, etc.). This mismatch surfaces as webhook rejections after the resource is accepted:

Admission webhook “gemini-operator-validation” denied the request: spec.version: Invalid value: “v1beta1”: version not supported

Investigation and Debugging

1. Verify Kubernetes version and CRD API groups

kubectl version --short
kubectl get crd | grep gemini

Expected output on a failing kind cluster:

Server Version: v1.24.0
gemini.google.com   v1   true   GeminiModel
gemini.google.com   v1   true   GeminiEndpoint

2. Inspect the installed CRD schema

kubectl get crd geminimodels.gemini.google.com -o yaml | grep -A5 "openAPIV3Schema"

Look for the presence of spec.version and the required list. Missing entries indicate a mismatched operator version.

3. Capture the exact error from `kubectl apply`

kubectl apply -f gemini-model.yaml

Typical output on a mismatched cluster:

error: error validating "gemini-model.yaml": error validating data: ValidationError(GeminiModel.spec): unknown field "version"

4. Check the operator pod logs for webhook errors

kubectl logs -l app=gemini-operator -c webhook

Sample log excerpt:

time="2024-09-07T12:34:56Z" level=error msg="admission webhook denied request" kind=GeminiModel version=v1 error="spec.version: Invalid value: \"v1beta1\": version not supported"

5. Verify feature‑gate status (Minikube only)

kubectl -n kube-system get configmap kube-apiserver -o yaml | grep CustomResourceValidation

If the flag is false, the API server will not enforce schema validation.

Resolution – Align CRD Versions and Provide Required Fields

Option A – Upgrade the local cluster to Kubernetes ≥ 1.25

For kind, pull a newer node image and recreate the cluster:

# Delete existing cluster
kind delete cluster --name gemini-dev

# Create a new cluster with v1.26 node image
cat > kind-config.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.26.3
EOF

kind create cluster --name gemini-dev --config kind-config.yaml

After the upgrade, re‑install the operator and apply the resource. The spec.version field is now recognized.

Option B – Pin the operator to a pre‑v1.25 compatible release

If upgrading the cluster is not possible, use the v0.9.2 release of the Gemini operator, which still ships v1beta1 CRDs without the spec.version field.

# Install older manifest
kubectl apply -f https://github.com/google/gemini-operator/releases/download/v0.9.2/gemini-operator.yaml

Then adjust your custom resource to omit spec.version and ensure required fields match the older schema:

Before (v1.25+ manifest)

apiVersion: gemini.google.com/v1
kind: GeminiModel
metadata:
  name: my-model
spec:
  version: v1
  model: "text-bison@001"
  parameters:
    temperature: 0.7

After (v0.9.2 compatible)

apiVersion: gemini.google.com/v1
kind: GeminiModel
metadata:
  name: my-model
spec:
  model: "text-bison@001"
  parameters:
    temperature: 0.7

Option C – Enable the `CustomResourceValidation` feature gate (Minikube)

Restart the API server with the gate enabled:

minikube ssh
sudo sed -i 's/^KUBE_APISERVER_ARGS=.*/KUBE_APISERVER_ARGS="--feature-gates=CustomResourceValidation=true"/' /etc/kubernetes/manifests/kube-apiserver.yaml
sudo systemctl restart kubelet

After the API server restarts, the schema validation will align with the operator’s expectations.

Verification – Confirm the Fix

1. Re‑apply the custom resource

kubectl apply -f gemini-model.yaml

Successful output:

geminiModel.gemini.google.com/my-model created

2. Check the resource status

kubectl get geminimodel my-model -o yaml

Look for a status.conditions entry indicating Ready:

status:
  conditions:
  - type: Ready
    status: "True"
    reason: Reconciled
    message: Model deployed successfully

3. Verify operator logs for reconciliation

kubectl logs -l app=gemini-operator -c controller | grep my-model

Expected log line:

time="2024-09-07T12:45:02Z" level=info msg="GeminiModel reconciled" name=my-model namespace=default

Prevention – Guardrails for Future Development

Cluster version policy: Enforce a minimum Kubernetes version (≥ 1.25) for any repository that includes the Gemini operator. Encode this in CI via kindest/node:v1.26 images.
CRD version pinning: Keep the operator version in sync with the cluster’s apiextensions.k8s.io version. Use helm upgrade --set crd.version=v1 or similar flags.
Schema validation CI step: Run kubectl apply --dry-run=client -f <manifest> against a fresh cluster to catch mismatches early.
Feature‑gate audit: Document required API server flags (e.g., CustomResourceValidation) in the onboarding guide for Minikube/Docker Desktop users.
Admission webhook testing: Include a negative test case that deliberately omits a required field and asserts the webhook error message. This protects against silent schema drift.

FAQ – Common Follow‑Up Questions

Why does the same manifest work on GKE but fail on kind?
GKE clusters run Kubernetes ≥ 1.25 with the v1 CRD API, while many local kind images still use 1.22‑1.24 and expose only v1beta1. The operator’s CRDs rely on fields introduced in v1, causing validation failures on older clusters.
Can I keep using the latest operator on a 1.24 cluster?
Only by downgrading the operator to a release that still publishes v1beta1 CRDs (e.g., v0.9.2). Otherwise you must upgrade the cluster.
What exact fields are required in a GeminiModel manifest?
According to the official CRD reference, spec.model is mandatory. In v1.25+ schemas, spec.version is also required and must be set to v1. Optional fields include parameters and metadata.annotations.
How do I know if the CustomResourceValidation feature gate is disabled?
Inspect the API server manifest or run:
```
kubectl -n kube-system get configmap kube-apiserver -o yaml | grep CustomResourceValidation
```
If the value is false, enable it and restart the API server.
Is there a way to automatically adjust manifests for the cluster version?
Yes. Use a templating tool (e.g., kustomize or helm) with a conditional that strips spec.version when .Capabilities.KubeVersion.Major is less than 1.25.

Related Topic Hub: LLM Systems Troubleshooting Hub