LlamaIndex training loop rejected by admission controller Kubernetes

Problem Description

The LlamaIndex training loop fails to start in a Kubernetes‑based ML platform. The pod creation request is rejected by the llamaindex-validation admission webhook. Typical error output from kubectl describe pod looks like:

Error from server (BadRequest): admission webhook "llamaindex-validation" denied the request: unauthorized access to secret "llama-index-config"

Other observed rejections include:

• “pod llamaindex-trainer-7f9c9d is forbidden: exceeded quota: cpu”
• “memory limit \”8Gi\” exceeds node allocatable memory \”4Gi\” – admission webhook rejected”
• “serviceaccount \”default\” is not authorized to use the required pod security policy – admission controller denied”
• “failed to create pod: admission webhook \”llamaindex-webhook\” returned error: request body too large (exceeds maxAdmissionReviewSize)”

These messages match the Common errors list from the evidence package and cause the training job to abort before any training code runs.

Root Cause Analysis

The LlamaIndex Helm chart installs a custom admission webhook (llamaindex-webhook) that validates every training pod against three security and resource policies:

1. RBAC & Secret Access – The webhook checks that the pod’s ServiceAccount has secrets/access permission for the secret referenced in LLAMA_INDEX_CONFIG. If the ServiceAccount is default or lacks the rolebinding, the webhook returns “unauthorized access to secret”. (See Training Loop API reference.)
2. Resource Quota & Limits – The webhook compares resources.requests/limits in the pod spec against the namespace’s ResourceQuota. Exceeding CPU or memory triggers “exceeded quota” or “memory limit exceeds node capacity”. (Documented in the Kubernetes Integration guide.)
3. Pod Security Policy / ServiceAccount – Pods must use a ServiceAccount that is bound to the PSP required by LlamaIndex. The default ServiceAccount is not authorized in many hardened clusters (e.g., OpenShift). This results in “serviceaccount is not authorized”.

Additionally, the webhook enforces a maximum AdmissionReview payload size (maxAdmissionReviewSize). When the training manifest includes a large list of volume mounts or a massive envFrom block, the request body can exceed the default limit, causing the “request body too large” error.

Investigation and Debugging

Below is a reproducible debugging workflow that was used in GitHub Issue #842 and #915.

1. Inspect the webhook event:

kubectl get events --namespace llama-index -w | grep llamaindex-validation

Typical output:

2024-05-12T14:22:31Z  Warning  AdmissionWebhook  llamaindex-validation  pod "llamaindex-trainer-xxxx" denied: unauthorized access to secret "llama-index-config"

2. Review the pod manifest generated by LlamaIndex (usually in /tmp/llamaindex-trainer.yaml).

cat /tmp/llamaindex-trainer.yaml

Key sections to verify:

apiVersion: v1
kind: Pod
metadata:
  name: llamaindex-trainer-xxxx
spec:
  serviceAccountName: default
  containers:
  - name: trainer
    image: ghcr.io/run-llama/llama-index:latest
    resources:
      limits:
        cpu: "4"
        memory: "8Gi"
    envFrom:
    - secretRef:
        name: llama-index-config

3. Check RBAC bindings for the ServiceAccount:

kubectl get rolebinding -n llama-index -o yaml | grep -A3 "subjects:"

If no binding grants secrets/access to the ServiceAccount, the webhook will reject.

4. Validate ResourceQuota and node capacity:

kubectl describe quota -n llama-index
kubectl get nodes -o jsonpath="{.items[*].status.allocatable.memory}"

Compare the requested memory and cpu against these values.

5. Inspect webhook configuration (Helm values or ConfigMap):

kubectl get cm llamaindex-webhook-config -n llama-index -o yaml

Look for maxAdmissionReviewSize and resource limit defaults.

Resolution

Apply the following fixes based on the identified cause. Each fix includes a Before and After snippet.

1. Grant Secret Access to a Dedicated ServiceAccount

Before (using default ServiceAccount):

serviceAccountName: default

After (create and bind a dedicated SA):

# Create ServiceAccount
kubectl create serviceaccount llamaindex-trainer -n llama-index

# Bind role that includes secret access
kubectl create rolebinding llamaindex-secret-access \
  --clusterrole=secret-reader \
  --serviceaccount=llama-index:llamaindex-trainer \
  -n llama-index

# Update pod spec (or Helm values)
serviceAccountName: llamaindex-trainer

Why it works: The webhook explicitly checks for the secrets/access permission. Binding the role satisfies the policy, eliminating the “unauthorized access to secret” error (see Issue #842).

2. Align Resource Requests with Namespace Quota and Node Capacity

Before (exceeds quota):

resources:
  limits:
    cpu: "8"
    memory: "12Gi"

After (adjusted to quota limits, e.g., 4 CPU, 6 Gi memory):

resources:
  limits:
    cpu: "4"
    memory: "6Gi"
  requests:
    cpu: "2"
    memory: "4Gi"

Update the Helm chart values (values.yaml) accordingly:

trainer:
  resources:
    limits:
      cpu: "4"
      memory: "6Gi"
    requests:
      cpu: "2"
      memory: "4Gi"

Why it works: The webhook validates the pod against the namespace ResourceQuota and node allocatable resources. Matching the limits prevents “exceeded quota” and “memory limit exceeds node capacity” rejections (see Issue #915).

3. Use an Authorized ServiceAccount with PodSecurityPolicy

Before (default SA, no PSP binding):

serviceAccountName: default

After (bind to PSP):

# Create PSP (if not already present)
kubectl apply -f - <

maxAdmissionReviewSize: 1048576

maxAdmissionReviewSize: 5242880

kubectl edit cm llamaindex-webhook-config -n llama-index
kubectl rollout restart deployment llamaindex-webhook -n llama-index

# Re‑run the training command (LlamaIndex CLI)
llamaindex train --config config.yaml

# Check pod status
kubectl get pods -n llama-index -w

NAME                         READY   STATUS    RESTARTS   AGE
llamaindex-trainer-abc123    1/1     Running   0          12s