curl -s http://langchain-service.ai.svc.cluster.local/healthz # Expected: {"status":"ok"} Prevent – Best Practices for Safe CRD Evolution Versioned CRDs: Keep separate CRD versions (e.g., v1alpha1, v1beta1) and use conversion webhooks to migrate objects. Never add required fields without a migration path. Introduce new fields as optional, then gradually enforce them. Helm chart linting: Use helm lint with --set helm.sh/strict=true to catch schema mismatches before release. Canary validation pipeline: Deploy the new CRD to a dedicated test namespace, run a kubectl get <cr> -o yaml diff against production resources. Automated integration tests that apply the full Helm chart to a fresh cluster and verify that existing resources are accepted. Monitoring: Alert on apiserver events of type Warning with reason FailedCreate for CRDs. FAQ – Common Follow‑up Questions Why does the upgrade succeed when I use --skip-crds? Skipping CRD updates bypasses the validation step, allowing the Helm release to complete, but the operator will still encounter schema mismatches at runtime. Can I use preserveUnknownFields: true to avoid this issue? Setting preserveUnknownFields: true disables strict validation, which may hide schema problems and lead to runtime errors. It is not recommended for production operators. How do I roll back if the migration job corrupts existing resources? Create a backup of the original resources (e.g., kubectl get langchain -n ai -o yaml > backup.yaml) before running the job. Restoring is as simple as kubectl apply -f backup.yaml. Is there a way to make Helm automatically handle CRD migrations? Helm 3 does not manage CRD lifecycle beyond creation. Use a pre‑upgrade hook (helm.sh/hook: pre-upgrade) to run migration jobs before applying the new chart. What versioning strategy should I adopt for future breaking changes? Adopt v1 for stable, immutable schemas and introduce v2 as a new CRD name (e.g., langchains.v2.langchain.com). Provide a conversion webhook or a separate migration tool to transition objects. Related Topic Hub: RAG Systems Troubleshooting Hub

LangChain Operator CRD Validation Failure During Canary Helm Upgrade

Q: Problem – CRD Validation Failure During Canary Helm Upgrade

During a canary rollout of the LangChain operator via Helm, the upgrade aborts with a CRD validation error. The Helm release never reaches the deployed state, and the operator pod is not created.

Q: Root Cause – Schema Mismatch and Missing Required Fields

The canary chart bundles a newer version of the LangChainOperator CRD that adds a new required field modelConfig to the spec of LangChain resources. The existing stable CRD (installed in the cluster) does not declare this field as required, and the canary CustomResource manifest still omits it. When Helm attempts to apply the updated CRD, the Kubernetes API server validates the new schema against the existing custom resources. Because the existing resources lack modelConfig, the validation fails, and the upgrade is rejected.

Q: Debug – Investigation Process 1. Verify the CRD version in the cluster

kubectl get crd langchains.langchain.com -o yaml | grep "version:" -A2 Expected output shows the currently installed version (e.g., v2.2.0).

Q: 2. Compare the CRD schemas between stable and canary charts

# Export stable CRD kubectl get crd langchains.langchain.com -o yaml > stable-crd.yaml # Extract canary CRD from chart helm template ./langchain-chart --set image.tag=v2.3.1-canary \ --show-only templates/crd.yaml > canary-crd.yaml # Diff the schemas diff -u stable-crd.yaml canary-crd.yaml | less The diff highlights the added required: [modelConfig] entry.

Q: 3. Inspect existing LangChain resources

kubectl get langchain -n ai -o yaml Typical output (stable manifest) lacks modelConfig:

Q: 4. Reproduce the validation error directly

kubectl apply -f canary-crd.yaml # => error: ... required property "modelConfig" not found in schema 5. Check Helm's --skip-crds behavior Running the upgrade with --skip-crds bypasses the CRD update, allowing the canary deployment to proceed, but leaves the schema mismatch unaddressed.

Q: Solution – Align CRD Schema and Migrate Existing Resources Approach

1. Update the canary CRD to make modelConfig optional for backward compatibility. 2. Provide a migration Job that patches existing LangChain resources with a default modelConfig value. 3. Apply the updated CRD before the Helm upgrade.

Q: Step‑by‑step implementation Before – Problematic CRD snippet (canary) spec: validation: openAPIV3Schema: type: object properties: spec: type: object required: - modelName - modelVersion - modelConfig #

kubectl get crd langchains.langchain.com -o yaml | grep "required:" -A3 Output should no longer list modelConfig under required.

Q: 2. Existing resources contain the new field

kubectl get langchain my-chain -n ai -o yaml | grep modelConfig -A2 Expected snippet:

Q: 3. Operator pod status

kubectl get pods -n ai -l app=langchain-operator -o wide Both langchain-operator-stable-... and langchain-operator-canary-... should be Running.

In this article

Table of Contents

Problem – CRD Validation Failure During Canary Helm Upgrade

During a canary rollout of the LangChain operator via Helm, the upgrade aborts with a CRD validation error. The Helm release never reaches the deployed state, and the operator pod is not created.

Observed logs (Helm):

helm upgrade --install langchain-canary ./langchain-chart \
  --namespace ai \
  --set image.tag=v2.3.1-canary \
  --wait

Error: UPGRADE FAILED: cannot install CRD "langchains.langchain.com": 
spec.validation.openAPIV3Schema.properties.spec.required: 
required property "modelConfig" not found in schema

kubectl describe of the failing CRD:

kubectl describe crd langchains.langchain.com
Name:         langchains.langchain.com
Namespace:    
Labels:       
Annotations:  
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
...
Validation:  OpenAPI v3
  schema:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          required:
          - modelName
          - modelVersion
          properties:
            modelName:
              type: string
            modelVersion:
              type: string
            # NOTE: modelConfig missing from required list

Result: Helm aborts before any resources are applied, leaving the cluster in a partially upgraded state.

Root Cause – Schema Mismatch and Missing Required Fields

The canary chart bundles a newer version of the LangChainOperator CRD that adds a new required field modelConfig to the spec of LangChain resources. The existing stable CRD (installed in the cluster) does not declare this field as required, and the canary CustomResource manifest still omits it. When Helm attempts to apply the updated CRD, the Kubernetes API server validates the new schema against the existing custom resources. Because the existing resources lack modelConfig, the validation fails, and the upgrade is rejected.

Key points:

The canary CRD version (v2.3.1) introduces a breaking schema change.
The Helm chart does not include a migration step to add the missing field to existing resources.
Kubernetes treats CRD schema changes as immutable for existing objects unless preserveUnknownFields is set to true, causing the validation error.

Debug – Investigation Process

1. Verify the CRD version in the cluster

kubectl get crd langchains.langchain.com -o yaml | grep "version:" -A2

Expected output shows the currently installed version (e.g., v2.2.0).

2. Compare the CRD schemas between stable and canary charts

# Export stable CRD
kubectl get crd langchains.langchain.com -o yaml > stable-crd.yaml

# Extract canary CRD from chart
helm template ./langchain-chart --set image.tag=v2.3.1-canary \
  --show-only templates/crd.yaml > canary-crd.yaml

# Diff the schemas
diff -u stable-crd.yaml canary-crd.yaml | less

The diff highlights the added required: [modelConfig] entry.

3. Inspect existing LangChain resources

kubectl get langchain -n ai -o yaml

Typical output (stable manifest) lacks modelConfig:

apiVersion: langchain.langchain.com/v1
kind: LangChain
metadata:
  name: my-chain
spec:
  modelName: gpt-4
  modelVersion: "1.0"

4. Reproduce the validation error directly

kubectl apply -f canary-crd.yaml
# => error: ... required property "modelConfig" not found in schema

5. Check Helm’s `--skip-crds` behavior

Running the upgrade with --skip-crds bypasses the CRD update, allowing the canary deployment to proceed, but leaves the schema mismatch unaddressed.

Solution – Align CRD Schema and Migrate Existing Resources

Approach

1. Update the canary CRD to make modelConfig optional for backward compatibility.
2. Provide a migration Job that patches existing LangChain resources with a default modelConfig value.
3. Apply the updated CRD before the Helm upgrade.

Step‑by‑step implementation

Before – Problematic CRD snippet (canary)

spec:
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          required:
          - modelName
          - modelVersion
          - modelConfig   # <-- newly required
          properties:
            modelName:
              type: string
            modelVersion:
              type: string
            modelConfig:
              type: object
              properties:
                temperature:
                  type: number

After – Updated CRD with optional field

spec:
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          required:
          - modelName
          - modelVersion
          # modelConfig removed from required list
          properties:
            modelName:
              type: string
            modelVersion:
              type: string
            modelConfig:
              type: object
              nullable: true
              properties:
                temperature:
                  type: number

Patch Job to add default `modelConfig`

apiVersion: batch/v1
kind: Job
metadata:
  name: langchain-migrate-modelconfig
  namespace: ai
spec:
  template:
    spec:
      serviceAccountName: langchain-operator
      restartPolicy: OnFailure
      containers:
      - name: patcher
        image: bitnami/kubectl:latest
        command:
        - /bin/sh
        - -c
        - |
          for name in $(kubectl get langchain -n ai -o name); do
            kubectl patch $name -n ai --type merge -p '{"spec":{"modelConfig":{"temperature":0.7}}}'
          done

Apply the fixed CRD and migration Job before the Helm upgrade

# Apply updated CRD
kubectl apply -f fixed-crd.yaml

# Run migration
kubectl apply -f migrate-job.yaml
# Wait for Job completion
kubectl wait --for=condition=complete job/langchain-migrate-modelconfig -n ai --timeout=120s

Run the canary Helm upgrade

helm upgrade --install langchain-canary ./langchain-chart \
  --namespace ai \
  --set image.tag=v2.3.1-canary \
  --wait

The upgrade now succeeds, and the operator pod rolls out alongside the stable version.

Verify – Confirming Successful Deployment

1. CRD validation

kubectl get crd langchains.langchain.com -o yaml | grep "required:" -A3

Output should no longer list modelConfig under required.

2. Existing resources contain the new field

kubectl get langchain my-chain -n ai -o yaml | grep modelConfig -A2

Expected snippet:

spec:
  modelName: gpt-4
  modelVersion: "1.0"
  modelConfig:
    temperature: 0.7

3. Operator pod status

kubectl get pods -n ai -l app=langchain-operator -o wide

Both langchain-operator-stable-... and langchain-operator-canary-... should be Running.

4. Functional test

curl -s http://langchain-service.ai.svc.cluster.local/healthz
# Expected: {"status":"ok"}

Prevent – Best Practices for Safe CRD Evolution

Versioned CRDs: Keep separate CRD versions (e.g., v1alpha1, v1beta1) and use conversion webhooks to migrate objects.
Never add required fields without a migration path. Introduce new fields as optional, then gradually enforce them.
Helm chart linting: Use helm lint with --set helm.sh/strict=true to catch schema mismatches before release.
Canary validation pipeline: Deploy the new CRD to a dedicated test namespace, run a kubectl get <cr> -o yaml diff against production resources.
Automated integration tests that apply the full Helm chart to a fresh cluster and verify that existing resources are accepted.
Monitoring: Alert on apiserver events of type Warning with reason FailedCreate for CRDs.

FAQ – Common Follow‑up Questions

Why does the upgrade succeed when I use --skip-crds?
Skipping CRD updates bypasses the validation step, allowing the Helm release to complete, but the operator will still encounter schema mismatches at runtime.
Can I use preserveUnknownFields: true to avoid this issue?
Setting preserveUnknownFields: true disables strict validation, which may hide schema problems and lead to runtime errors. It is not recommended for production operators.
How do I roll back if the migration job corrupts existing resources?
Create a backup of the original resources (e.g., kubectl get langchain -n ai -o yaml > backup.yaml) before running the job. Restoring is as simple as kubectl apply -f backup.yaml.
Is there a way to make Helm automatically handle CRD migrations?
Helm 3 does not manage CRD lifecycle beyond creation. Use a pre‑upgrade hook (helm.sh/hook: pre-upgrade) to run migration jobs before applying the new chart.
What versioning strategy should I adopt for future breaking changes?
Adopt v1 for stable, immutable schemas and introduce v2 as a new CRD name (e.g., langchains.v2.langchain.com). Provide a conversion webhook or a separate migration tool to transition objects.

Related Topic Hub: RAG Systems Troubleshooting Hub