kube-controller-manager OOM crash after adding new AI model CRDs

Problem: kube‑controller‑manager OOM crashes after adding AI model CRDs

In a production cluster that hosts dozens of AI model serving workloads, a new set of Custom Resource Definitions (CRDs) was introduced to manage model lifecycle (e.g., Model, ModelVersion, ModelDeployment). Shortly after the rollout, the kube-controller-manager pod began terminating with an OOMKilled status. The crash cascaded into pod‑scheduling failures, because the controller manager could no longer reconcile nodes, endpoints, and the new model resources.

Typical symptoms observed:

  • Repeated OOMKilled restarts of the kube-controller-manager pod.
  • Scheduler logs showing “no nodes available” errors.
  • High memory usage (> 2 GiB) on the control‑plane VM.
  • Increased latency in API server responses during reconciliation spikes.

Root Cause

The newly added model CRDs trigger a high‑frequency reconciliation loop in the custom controllers that are registered with the controller manager. Each reconciliation performs the following steps:

  1. List all Model resources (often > 1 000 entries).
  2. For each model, fetch associated ConfigMap and Secret objects containing model binaries and credentials.
  3. Deserialize large protobuf payloads (up to 200 MiB per model) to compute checksum and version compatibility.
  4. Update status sub‑resources, which generates a write‑back to the API server.

Because the custom controllers were registered as unstructured controllers inside the kube-controller-manager binary (via the --controllers flag), their reconciliation loops share the same process memory space as the built‑in controllers. The combination of:

  • Large in‑memory model descriptors.
  • Frequent full‑list watches (no field selectors).
  • Lack of pagination or caching in the custom controller code.

caused the controller manager’s heap to grow until the cgroup OOM killer terminated the process. The crash was not a simple configuration typo; it was a resource‑exhaustion pathology introduced by the new CRDs and their controller implementation.

Debug: Investigation Steps

1. Verify OOM termination

kubectl -n kube-system get pod kube-controller-manager-$(hostname) -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}'
# Output: OOMKilled

2. Inspect memory usage over time

kubectl top pod -n kube-system kube-controller-manager-$(hostname)
# Example output:
# NAME                                 CPU(cores)   MEMORY(bytes)
# kube-controller-manager-node1        150m         2100Mi

3. Correlate spikes with custom controller activity

kubectl logs -n kube-system kube-controller-manager-$(hostname) --since=1h | grep "reconcile Model"
# Sample log lines:
# 2024-11-14T10:12:03Z INFO reconcile Model: processing 1024 items
# 2024-11-14T10:12:04Z INFO reconcile ModelVersion: fetched 2048 bytes payload

4. Capture a heap profile

# Enable pprof endpoint (if not already)
kubectl -n kube-system port-forward svc/kube-controller-manager 10257:10257 &
curl -s http://localhost:10257/debug/pprof/heap?seconds=30 > /tmp/kcm-heap.prof
# Analyze with go tool pprof
go tool pprof -http=:8080 /tmp/kcm-heap.prof

The profile highlighted a large allocation chain in github.com/yourorg/model-controller/pkg/reconciler where the entire ModelSpec protobuf was kept in a slice for the duration of the reconcile loop.

5. Review controller registration

# kube-controller-manager manifest (static pod)
cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep -- '--controllers'
# Output:
# - --controllers=*,model

Embedding the custom controller directly inside the controller manager process means any memory leak or high‑water usage directly impacts the core control plane.

Solution: Refactor and Resource‑Limit the Model Controllers

1. Externalize the custom controllers

Run the model controllers as separate Deployments rather than as in‑process controllers. This isolates their memory consumption and allows independent scaling.

Before (in‑process registration):


# /etc/kubernetes/manifests/kube-controller-manager.yaml
spec:
  containers:
  - name: kube-controller-manager
    image: k8s.gcr.io/kube-controller-manager:v1.28.2
    command:
    - kube-controller-manager
    - --controllers=*,model
    - --kubeconfig=/etc/kubernetes/controller-manager.conf

After (external Deployment):


# model-controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-controller
  namespace: kube-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-controller
  template:
    metadata:
      labels:
        app: model-controller
    spec:
      serviceAccountName: model-controller-sa
      containers:
      - name: model-controller
        image: yourrepo/model-controller:v2.1.0
        resources:
          limits:
            memory: "1Gi"
          requests:
            memory: "512Mi"
        args:
        - --kubeconfig=/etc/kubernetes/controller-manager.conf
        volumeMounts:
        - name: kubeconfig
          mountPath: /etc/kubernetes
      volumes:
      - name: kubeconfig
        hostPath:
          path: /etc/kubernetes

Remove the model entry from the --controllers flag and restart the static pod.

2. Add pagination and field selectors

Modify the reconciler to watch only the subset of models that changed, using Informer filters.


// Before: full list on every reconcile
func (r *ModelReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var models modelv1.ModelList
    if err := r.List(ctx, &models); err != nil {
        return ctrl.Result{}, err
    }
    // process entire list...
}

// After: filtered watch + pagination
func (r *ModelReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&modelv1.Model{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
        WithOptions(controller.Options{
            MaxConcurrentReconciles: 5,
        }).
        Complete(r)
}

3. Limit in‑memory payload size

Store large binary blobs (model weights) in a dedicated object store (e.g., S3) and keep only a reference (URL) in the ModelSpec. This reduces heap pressure.


// ModelSpec before
type ModelSpec struct {
    Binary []byte `json:"binary"` // up to 200MiB
}

// ModelSpec after
type ModelSpec struct {
    BlobURL string `json:"blobURL"` // S3 pre‑signed URL
}

4. Enforce memory limits on the controller manager

Even after externalizing the controller, set a hard limit on the controller manager to prevent it from being killed by other components.


# /etc/kubernetes/manifests/kube-controller-manager.yaml (add resources)
resources:
  limits:
    memory: "2Gi"
  requests:
    memory: "1Gi"

Verify: Confirm the Issue Is Resolved

  1. Observe the controller manager pod for at least 24 h:
kubectl -n kube-system get pod kube-controller-manager-$(hostname) -w
# Expect no OOMKilled events.
  1. Check memory consumption:
kubectl top pod -n kube-system kube-controller-manager-$(hostname)
# Should stay below 1.5 GiB.
  1. Validate model controller health:
kubectl get deployment model-controller -n kube-system
kubectl logs deployment/model-controller -n kube-system | grep "reconcile Model"
# Logs should show processing of only changed resources, not full list scans.
  1. Confirm pod scheduling works:
kubectl get pods -A | grep Pending
# No pending pods due to scheduler errors.

Prevent: Operational Guardrails

  • Monitoring: Add a Prometheus alert on process_resident_memory_bytes{job="kube-controller-manager"} crossing 80 % of the limit.
  • Alerting: Configure a PagerDuty notification for OOM events on the control‑plane nodes.
  • Testing: Include load‑testing of custom controllers in CI pipelines, simulating > 1 000 model resources.
  • Configuration Review: Enforce a policy that any new CRD controller must run as an external Deployment with explicit resource limits.
  • Documentation: Record the maximum expected size of model descriptors and enforce size validation in the webhook admission controller.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does the controller manager OOM only after adding the model CRDs?
    The new CRDs introduced a high‑frequency, high‑volume reconciliation loop that runs inside the same process as the built‑in controllers. The unbounded list‑and‑process pattern caused the heap to grow beyond the node’s memory capacity.
  2. Can I keep the custom controller inside the controller manager if I add pagination?
    Even with pagination, the controller manager shares a single Go runtime. A bug or unexpected load in any in‑process controller can still affect the whole control plane. Externalizing is the safest approach.
  3. What memory limit should I set for the controller manager?
    Set the limit based on observed peak usage after refactoring. In most clusters, 2 GiB provides headroom for built‑in controllers while still protecting the node from OOM.
  4. How do I detect similar OOM patterns for future custom controllers?
    Enable the process_resident_memory_bytes metric for each controller pod, and create a Grafana dashboard that shows per‑controller memory trends. Alert on rapid growth or sustained usage above 75 % of the request.
  5. Is it safe to store model binaries directly in a ConfigMap?
    No. ConfigMaps have a size limit of 1 MiB and are kept in etcd, which is not designed for large binary payloads. Use an external object store and reference the location in the CRD.