MLflow Training Job Rejected By Kubernetes Admission Controller

Q: Root Cause Analysis

MLflow uses the Kubernetes plugin to translate a mlflow run into a pod spec. The generated pod inherits resource requests, limits, and image references directly from the MLflow project definition (MLflow Projects – Kubernetes Backend).

Q: 1. Capture the pod spec generated by MLflow

mlflow run . -P backend=kubernetes \ --no-conda \ -P image=private-registry.com/mlflow-train:latest \ -P cpu=8 -P memory=16Gi \ --dry-run > pod.yaml Inspect pod.yaml for resource requests, limits, securityContext, and image fields.

Q: 2. Attempt manual pod creation

kubectl apply -f pod.yaml The error returned will be identical to the one observed from MLflow and includes the admission controller name.

Q: 3. Query active admission controllers

kubectl api-versions | grep admissionregistration.k8s.io kubectl get validatingwebhookconfigurations kubectl get mutatingwebhookconfigurations 4. Inspect ResourceQuota and OPA constraints in the namespace kubectl get resourcequota -n mlflow-prod -o yaml kubectl get constraint -n mlflow-prod -o yaml 5. Review PodSecurityPolicy or PodSecurity admission status kubectl get psp kubectl get podsecuritypolicy -o yaml | grep -i runAsUser 6. Examine controller logs for detailed denial reasons kubectl logs -n kube-system -l app=gatekeeper -c manager kubectl logs -n kube-system -l app=image-policy-webhook Typical log excerpt (Gatekeeper):

Q: Scenario B – OPA Gatekeeper Image Restriction

Before – Image from a private registry not listed in the allowed-registries constraint.

Q: Scenario C – PodSecurityPolicy Requires Non‑Root

Before – No securityContext, defaults to root.

In this article

Table of Contents

Problem Description

When launching an MLflow training run that uses the Kubernetes backend, the pod creation fails with an admission‑controller error. The MLflow client reports a kubectl error similar to the following:


Error from server (Forbidden): pods "mlflow-train-7f9c8d5b9-xyz" is forbidden: 
exceeded quota: compute-resources, request: cpu=8, memory=16Gi

Other observed messages include:

admission webhook "validation.gatekeeper.sh" denied the request: container image "myregistry.com/xyz" is not allowed by policy
Failed to create pod: PodSecurityPolicy validation failed: container must not run as root
Admission webhook "imagepolicywebhook.k8s.io" denied the request: image "untrusted.registry.com/abc" is not in the allowed list

These rejections prevent the training job from starting, causing downstream pipeline failures and SLA breaches.

Root Cause Analysis

MLflow uses the Kubernetes plugin to translate a mlflow run into a pod spec. The generated pod inherits resource requests, limits, and image references directly from the MLflow project definition (MLflow Projects – Kubernetes Backend).

The cluster enforces several admission controllers:

ResourceQuota: Caps total CPU/memory per namespace. A pod requesting more than the remaining quota triggers the exceeded quota error (see Kubernetes Resource Quotas documentation).
OPA Gatekeeper (or other policy engines): Enforces custom constraints such as allowed registries or maximum CPU per pod. Violations produce messages like container image ... is not allowed by policy (see GitHub issue #5123).
PodSecurityPolicy (PSP) / PodSecurity Standards: Requires non‑root containers, specific Linux capabilities, etc. MLflow’s default container runs as root, leading to PSP rejections (Kubeflow Forum thread).
ImagePolicyWebhook: Restricts images to a whitelist; private registry images are blocked unless explicitly allowed (GitHub issue #4567).

Thus, the admission controller rejections are not random failures but policy violations caused by a mismatch between the pod spec emitted by MLflow and the cluster’s security/compliance configuration.

Investigation and Debugging

Follow these steps to isolate the exact policy causing the denial.

1. Capture the pod spec generated by MLflow


mlflow run . -P backend=kubernetes \
    --no-conda \
    -P image=private-registry.com/mlflow-train:latest \
    -P cpu=8 -P memory=16Gi \
    --dry-run > pod.yaml

Inspect pod.yaml for resource requests, limits, securityContext, and image fields.

2. Attempt manual pod creation


kubectl apply -f pod.yaml

The error returned will be identical to the one observed from MLflow and includes the admission controller name.

3. Query active admission controllers


kubectl api-versions | grep admissionregistration.k8s.io
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

4. Inspect ResourceQuota and OPA constraints in the namespace


kubectl get resourcequota -n mlflow-prod -o yaml
kubectl get constraint -n mlflow-prod -o yaml

5. Review PodSecurityPolicy or PodSecurity admission status


kubectl get psp
kubectl get podsecuritypolicy -o yaml | grep -i runAsUser

6. Examine controller logs for detailed denial reasons


kubectl logs -n kube-system -l app=gatekeeper -c manager
kubectl logs -n kube-system -l app=image-policy-webhook

Typical log excerpt (Gatekeeper):


[2024-05-12T14:23:01Z] {"msg":"denied","kind":"AdmissionReview","resource":"pods","namespace":"mlflow-prod","name":"mlflow-train-7f9c8d5b9-xyz","reason":"container image \"myregistry.com/xyz\" is not allowed by policy"}

Resolution

Fixes depend on the offending policy. Below are three common scenarios with before/after examples.

Scenario A – ResourceQuota Exceeded

Before (pod.yaml)


spec:
  containers:
  - name: mlflow-train
    image: private-registry.com/mlflow-train:latest
    resources:
      requests:
        cpu: "8"
        memory: "16Gi"
      limits:
        cpu: "8"
        memory: "16Gi"

The namespace mlflow-prod has a quota of 100 CPU cores; existing workloads already consume 95 cores.

After – Reduce requests or increase quota


# Option 1: Lower request
spec:
  containers:
  - name: mlflow-train
    resources:
      requests:
        cpu: "4"
        memory: "8Gi"
      limits:
        cpu: "4"
        memory: "8Gi"


# Option 2: Raise quota (admin action)
kubectl patch resourcequota compute-resources -n mlflow-prod \
  -p '{"spec":{"hard":{"requests.cpu":"150","requests.memory":"300Gi"}}}'

Scenario B – OPA Gatekeeper Image Restriction

Before – Image from a private registry not listed in the allowed-registries constraint.


spec:
  containers:
  - name: mlflow-train
    image: private-registry.com/mlflow-train:latest

After – Add registry to constraint or use an allowed image


# Patch the constraint (requires cluster‑admin)
kubectl patch constraint allowed-registries -n mlflow-prod \
  --type merge -p '{"spec":{"parameters":{"allowedRegistries":["private-registry.com","docker.io"]}}}'

Or change the MLflow project to reference an allowed image.

Scenario C – PodSecurityPolicy Requires Non‑Root

Before – No securityContext, defaults to root.


spec:
  containers:
  - name: mlflow-train
    image: private-registry.com/mlflow-train:latest

After – Add non‑root security context


spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
  containers:
  - name: mlflow-train
    image: private-registry.com/mlflow-train:latest
    securityContext:
      allowPrivilegeEscalation: false

Alternatively, adjust the PSP to allow root if policy permits (not recommended).

Validation

After applying the fix, rerun the MLflow job and verify pod creation:


mlflow run . -P backend=kubernetes
kubectl get pods -n mlflow-prod -w

Successful creation will show a pod in Running state:


NAME                         READY   STATUS    RESTARTS   AGE
mlflow-train-7f9c8d5b9-xyz   1/1     Running   0          12s

Additional checks:

Confirm that the pod’s resource usage stays within the quota: kubectl top pod mlflow-train-7f9c8d5b9-xyz -n mlflow-prod
Inspect the pod’s security context: kubectl get pod mlflow-train-7f9c8d5b9-xyz -o yaml | grep runAs
Check that the image is from an allowed registry: kubectl describe pod mlflow-train-7f9c8d5b9-xyz | grep Image

Prevention and Best Practices

Policy Area	Recommended Guardrails
Resource Quotas	Automate quota consumption monitoring (e.g., Prometheus alert on `kube_resourcequota` usage > 80%). Keep MLflow default resource requests modest; expose them as configurable parameters.
Image Policies	Maintain a central list of allowed registries in the Gatekeeper constraint. Use MLflow `--image` flag to reference whitelisted images only.
PodSecurity	Adopt the PodSecurity Standards (restricted profile) and embed a non‑root `securityContext` in the base training image. Document the required fields in the MLflow project template.
Admission Webhooks	Version‑pin admission controllers and keep their logs aggregated. Create a “dry‑run” CI job that attempts to create a pod from a sample MLflow spec to catch regressions early.

Integrate these checks into the CI/CD pipeline that builds and pushes training images, ensuring that any change that would violate a policy fails the pipeline before reaching production.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the same MLflow run succeed in dev but fail in prod?
Production namespaces often have stricter ResourceQuota and PSP settings. Verify that the dev namespace’s quotas and policies match production, or adjust the job’s resource requests accordingly.
Can I disable the admission controller for MLflow pods?
Disabling a controller cluster‑wide is discouraged. Instead, tailor the pod spec (e.g., add required securityContext, use allowed images) or create an exemption constraint if the policy framework supports it.
How do I know which admission controller rejected the pod?
The error message includes the webhook name (e.g., validation.gatekeeper.sh, imagepolicywebhook.k8s.io). Inspect the corresponding ValidatingWebhookConfiguration for the policy definition.
My MLflow project uses dynamic resource values (e.g., ${CPU}). How can I ensure they stay within quota?
Parameterize resources in the mlflow run command and add a pre‑flight script that queries the namespace’s remaining quota via kubectl get resourcequota -o jsonpath='{.items[*].status.hard}' before launching the job.
Is there a way to see the exact OPA constraint that blocked my image?
OPA Gatekeeper logs the constraintName in the denial message. You can also run kubectl get constraint -o yaml and search for the image field in the spec.parameters section.