ModelCollection CRD Creation Returns 422 After Argo CD Sync

Q: 2. Insufficient RBAC for the Argo CD service account

The Argo CD server runs under the service account system:serviceaccount:argocd:argocd-server. The RBAC rules granted to this account often include get, list, and watch for core resources but omit create on the custom resource modelcollections.serving.kubeflow.org. When the webhook attempts to read the admission request, the API server returns PermissionDenied, which the webhook propagates as a 422 error.

Q: 3. Namespace selector misconfiguration in the MutatingWebhookConfiguration

In some clusters the MutatingWebhookConfiguration for KFServing is scoped to a specific namespace selector (e.g., kserve). Argo CD creates resources in the argocd namespace, causing the webhook to be invoked for a namespace it cannot access. The webhook pod then fails to read the request body, emitting “failed to read request body” and returning 422.

Q: Investigation and Debugging Inspect Argo CD sync logs $ kubectl -n argocd logs deployment/argocd-server -c argocd-server | grep "422" 2024-08-15T10:12:34Z error: failed to create resource: Unprocessable Entity (HTTP 422): admission webhook "modelcollection.kserve.io" denied the request: spec.predictor: Required value Validate the CRD schema $ kubectl get crd modelcollections.serving.kubeflow.org -o yaml | grep -A5 "spec:" ... required: - predictor ... Check the manifest that Argo CD is applying apiVersion: serving.kubeflow.org/v1beta1 kind: ModelCollection metadata: name: fraud-detection namespace: argocd spec: # predictor field missing Verify RBAC for the Argo CD service account $ kubectl auth can-i create modelcollections.serving.kubeflow.org --as=system:serviceaccount:argocd:argocd-server -n argocd no Inspect the MutatingWebhookConfiguration $ kubectl get mutatingwebhookconfiguration kserve-webhook -o yaml | grep -A4 "namespaceSelector" namespaceSelector: matchExpressions: - key: kserve.io/inject operator: In values: - "true" Capture a request/response trace with kubectl proxy and tcpdump # On the node where the webhook runs sudo tcpdump -i any -nn port 443 and host kserve-webhook.kserve.svc Observed a TCP reset after the webhook pod attempted to read the request body, confirming the namespace‑selector mismatch. Resolution 1. Add the missing required field to the manifest

Update the GitOps source so that every ModelCollection includes a valid spec.predictor block.

Q: 2. Grant Argo CD the required RBAC

Create a ClusterRole and a ClusterRoleBinding (or a namespace‑scoped Role if you prefer) that allows create on the modelcollections.serving.kubeflow.org resource.

Q: 4. Re‑run the Argo CD sync

$ argocd app sync model-serving SYNC STATUS: Synced Verification CR creation $ kubectl get modelcollection fraud-detection -n argocd -o yaml apiVersion: serving.kubeflow.org/v1beta1 kind: ModelCollection metadata: name: fraud-detection namespace: argocd spec: predictor: name: fraud-predictor ... Admission webhook logs $ kubectl -n kserve logs deployment/kserve-webhook -c webhook | grep fraud-detection 2024-08-15T10:15:02Z INFO admission webhook allowed creation of ModelCollection fraud-detection Argo CD health check $ argocd app get model-serving Health: Healthy Status: Synced RBAC test $ kubectl auth can-i create modelcollections.serving.kubeflow.org --as=system:serviceaccount:argocd:argocd-server -n argocd yes Prevention and Best Practices AreaRecommendation CRD schema evolution Pin the KFServing version in GitOps manifests and run a CI validation step that checks generated CRs against the OpenAPI schema (e.g., kubectl apply --dry-run=client -f). RBAC for GitOps operators Maintain a dedicated ClusterRole for Argo CD that includes all custom resources used by your pipelines. Periodically audit with kubectl auth can-i. Webhook namespace selectors Avoid overly restrictive selectors unless you have a clear multi‑tenant policy. Document required namespace labels in the onboarding checklist. Observability Enable audit logging for admission webhook denials and forward them to a central log store. Alert on admission webhook ... denied the request with severity “critical”. Testing after K8s upgrades Run a regression suite that creates a minimal set of custom resources after each minor upgrade. Capture failures early before they reach production. Related Topic Hub: Distributed Systems Troubleshooting Hub

In this article

Table of Contents

Problem: ModelCollection CRD creation returns 422 after Argo CD sync

In an on‑premise Kubernetes 1.27 cluster running the KFServing (now KServe) model serving stack, an Argo CD application that contains a ModelCollection manifest fails during the sync phase. The controller logs show a 422 Unprocessable Entity response from the modelcollection.kserve.io admission webhook, with errors such as:

admission webhook “modelcollection.kserve.io” denied the request: spec.predictor: Required value

rpc error: code = PermissionDenied desc = user “system:serviceaccount:argocd:argocd-server” cannot create resource “modelcollections.serving.kubeflow.org” in API group “serving.kubeflow.org”

The sync aborts, the ModelCollection never appears in the cluster, and the deployment pipeline stalls.

Root Cause Analysis

1. Admission webhook validation changes in Kubernetes 1.27

Kubernetes 1.27 introduced stricter OpenAPI schema validation for custom resources. The KFServing ModelCollection CRD added a new required field spec.predictor (see the KFServing ModelCollection specification). Manifests generated before the upgrade lack this field, causing the webhook to reject the request with spec.predictor: Required value.

2. Insufficient RBAC for the Argo CD service account

The Argo CD server runs under the service account system:serviceaccount:argocd:argocd-server. The RBAC rules granted to this account often include get, list, and watch for core resources but omit create on the custom resource modelcollections.serving.kubeflow.org. When the webhook attempts to read the admission request, the API server returns PermissionDenied, which the webhook propagates as a 422 error.

3. Namespace selector misconfiguration in the MutatingWebhookConfiguration

In some clusters the MutatingWebhookConfiguration for KFServing is scoped to a specific namespace selector (e.g., kserve). Argo CD creates resources in the argocd namespace, causing the webhook to be invoked for a namespace it cannot access. The webhook pod then fails to read the request body, emitting “failed to read request body” and returning 422.

Investigation and Debugging

Inspect Argo CD sync logs


$ kubectl -n argocd logs deployment/argocd-server -c argocd-server | grep "422"
2024-08-15T10:12:34Z error: failed to create resource: Unprocessable Entity (HTTP 422): admission webhook "modelcollection.kserve.io" denied the request: spec.predictor: Required value

Validate the CRD schema


$ kubectl get crd modelcollections.serving.kubeflow.org -o yaml | grep -A5 "spec:" 
...
required:
- predictor
...

Check the manifest that Argo CD is applying


apiVersion: serving.kubeflow.org/v1beta1
kind: ModelCollection
metadata:
  name: fraud-detection
  namespace: argocd
spec:
  # predictor field missing

Verify RBAC for the Argo CD service account


$ kubectl auth can-i create modelcollections.serving.kubeflow.org --as=system:serviceaccount:argocd:argocd-server -n argocd
no

Inspect the MutatingWebhookConfiguration


$ kubectl get mutatingwebhookconfiguration kserve-webhook -o yaml | grep -A4 "namespaceSelector"
namespaceSelector:
  matchExpressions:
  - key: kserve.io/inject
    operator: In
    values:
    - "true"

Capture a request/response trace with kubectl proxy and tcpdump
```
# On the node where the webhook runs
sudo tcpdump -i any -nn port 443 and host kserve-webhook.kserve.svc
```
Observed a TCP reset after the webhook pod attempted to read the request body, confirming the namespace‑selector mismatch.

Resolution

1. Add the missing required field to the manifest

Update the GitOps source so that every ModelCollection includes a valid spec.predictor block.

Before


apiVersion: serving.kubeflow.org/v1beta1
kind: ModelCollection
metadata:
  name: fraud-detection
  namespace: argocd
spec: {}

After


apiVersion: serving.kubeflow.org/v1beta1
kind: ModelCollection
metadata:
  name: fraud-detection
  namespace: argocd
spec:
  predictor:
    name: fraud-predictor
    containers:
    - image: myregistry/fraud-model:latest
      name: fraud-model
      ports:
      - containerPort: 8080

2. Grant Argo CD the required RBAC

Create a ClusterRole and a ClusterRoleBinding (or a namespace‑scoped Role if you prefer) that allows create on the modelcollections.serving.kubeflow.org resource.

RBAC manifest


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd-modelcollection-writer
rules:
- apiGroups: ["serving.kubeflow.org"]
  resources: ["modelcollections"]
  verbs: ["create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd-modelcollection-writer-binding
subjects:
- kind: ServiceAccount
  name: argocd-server
  namespace: argocd
roleRef:
  kind: ClusterRole
  name: argocd-modelcollection-writer
  apiGroup: rbac.authorization.k8s.io

3. Adjust the MutatingWebhookConfiguration namespace selector

If you want the KFServing webhook to process resources in the argocd namespace, either remove the selector or add a matching label.

Option A – Remove selector (simpler for small clusters):


kubectl patch mutatingwebhookconfiguration kserve-webhook \
  --type=json -p='[{"op":"remove","path":"/webhooks/0/namespaceSelector"}]'

Option B – Label the argocd namespace:


kubectl label namespace argocd kserve.io/inject=true

4. Re‑run the Argo CD sync


$ argocd app sync model-serving
SYNC STATUS: Synced

Verification

CR creation


$ kubectl get modelcollection fraud-detection -n argocd -o yaml
apiVersion: serving.kubeflow.org/v1beta1
kind: ModelCollection
metadata:
  name: fraud-detection
  namespace: argocd
spec:
  predictor:
    name: fraud-predictor
    ...

Admission webhook logs


$ kubectl -n kserve logs deployment/kserve-webhook -c webhook | grep fraud-detection
2024-08-15T10:15:02Z INFO admission webhook allowed creation of ModelCollection fraud-detection

Argo CD health check


$ argocd app get model-serving
Health: Healthy
Status: Synced

RBAC test


$ kubectl auth can-i create modelcollections.serving.kubeflow.org --as=system:serviceaccount:argocd:argocd-server -n argocd
yes

Prevention and Best Practices

Area	Recommendation
CRD schema evolution	Pin the KFServing version in GitOps manifests and run a CI validation step that checks generated CRs against the OpenAPI schema (e.g., `kubectl apply --dry-run=client -f`).
RBAC for GitOps operators	Maintain a dedicated `ClusterRole` for Argo CD that includes all custom resources used by your pipelines. Periodically audit with `kubectl auth can-i`.
Webhook namespace selectors	Avoid overly restrictive selectors unless you have a clear multi‑tenant policy. Document required namespace labels in the onboarding checklist.
Observability	Enable audit logging for admission webhook denials and forward them to a central log store. Alert on `admission webhook ... denied the request` with severity “critical”.
Testing after K8s upgrades	Run a regression suite that creates a minimal set of custom resources after each minor upgrade. Capture failures early before they reach production.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does the error say “spec.predictor: Required value” only after upgrading to Kubernetes 1.27?
Kubernetes 1.27 enforces the OpenAPI schema defined by the CRD more strictly. The KFServing ModelCollection CRD added spec.predictor as a required field in that version, so manifests that omitted it are now rejected by the admission webhook.
Can I keep the old manifests and avoid changing them?
Yes, by downgrading the CRD to a previous version that does not require spec.predictor, but this is discouraged. The preferred approach is to update the manifests and version‑pin the CRD to avoid future breaking changes.
How do I know which service account Argo CD uses for resource creation?
Argo CD server uses the argocd-server service account in the argocd namespace. You can confirm with:
```
$ kubectl get deployment argocd-server -n argocd -o jsonpath='{.spec.template.spec.serviceAccountName}'
argocd-server
```
Is the webhook timeout (“context deadline exceeded”) related to the 422 error?
When the webhook cannot read the request (e.g., due to RBAC or namespace‑selector issues), it returns an error that the API server translates into a 422. The timeout is a symptom of the same underlying permission problem.
Should I use a MutatingWebhookConfiguration or a ValidatingWebhookConfiguration for ModelCollection?
KFServing ships both. The validation webhook is responsible for schema checks (the source of the 422). The mutating webhook injects sidecars and can be scoped with namespace selectors. Ensure both are correctly configured for the namespaces where Argo CD operates.

Problem: ModelCollection CRD creation returns 422 after Argo CD sync