Problem: kube‑controller‑manager OOM crashes after adding AI model CRDs
In a production cluster that hosts dozens of AI model serving workloads, a new set of Custom Resource Definitions (CRDs) was introduced to manage model lifecycle (e.g., Model, ModelVersion, ModelDeployment). Shortly after the rollout, the kube-controller-manager pod began terminating with an OOMKilled status. The crash cascaded into pod‑scheduling failures, because the controller manager could no longer reconcile nodes, endpoints, and the new model resources.
Typical symptoms observed:
- Repeated
OOMKilledrestarts of thekube-controller-managerpod. - Scheduler logs showing “no nodes available” errors.
- High memory usage (> 2 GiB) on the control‑plane VM.
- Increased latency in API server responses during reconciliation spikes.
Root Cause
The newly added model CRDs trigger a high‑frequency reconciliation loop in the custom controllers that are registered with the controller manager. Each reconciliation performs the following steps:
- List all
Modelresources (often > 1 000 entries). - For each model, fetch associated
ConfigMapandSecretobjects containing model binaries and credentials. - Deserialize large protobuf payloads (up to 200 MiB per model) to compute checksum and version compatibility.
- Update status sub‑resources, which generates a write‑back to the API server.
Because the custom controllers were registered as unstructured controllers inside the kube-controller-manager binary (via the --controllers flag), their reconciliation loops share the same process memory space as the built‑in controllers. The combination of:
- Large in‑memory model descriptors.
- Frequent full‑list watches (no field selectors).
- Lack of pagination or caching in the custom controller code.
caused the controller manager’s heap to grow until the cgroup OOM killer terminated the process. The crash was not a simple configuration typo; it was a resource‑exhaustion pathology introduced by the new CRDs and their controller implementation.
Debug: Investigation Steps
1. Verify OOM termination
kubectl -n kube-system get pod kube-controller-manager-$(hostname) -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}'
# Output: OOMKilled
2. Inspect memory usage over time
kubectl top pod -n kube-system kube-controller-manager-$(hostname)
# Example output:
# NAME CPU(cores) MEMORY(bytes)
# kube-controller-manager-node1 150m 2100Mi
3. Correlate spikes with custom controller activity
kubectl logs -n kube-system kube-controller-manager-$(hostname) --since=1h | grep "reconcile Model"
# Sample log lines:
# 2024-11-14T10:12:03Z INFO reconcile Model: processing 1024 items
# 2024-11-14T10:12:04Z INFO reconcile ModelVersion: fetched 2048 bytes payload
4. Capture a heap profile
# Enable pprof endpoint (if not already)
kubectl -n kube-system port-forward svc/kube-controller-manager 10257:10257 &
curl -s http://localhost:10257/debug/pprof/heap?seconds=30 > /tmp/kcm-heap.prof
# Analyze with go tool pprof
go tool pprof -http=:8080 /tmp/kcm-heap.prof
The profile highlighted a large allocation chain in github.com/yourorg/model-controller/pkg/reconciler where the entire ModelSpec protobuf was kept in a slice for the duration of the reconcile loop.
5. Review controller registration
# kube-controller-manager manifest (static pod)
cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep -- '--controllers'
# Output:
# - --controllers=*,model
Embedding the custom controller directly inside the controller manager process means any memory leak or high‑water usage directly impacts the core control plane.
Solution: Refactor and Resource‑Limit the Model Controllers
1. Externalize the custom controllers
Run the model controllers as separate Deployments rather than as in‑process controllers. This isolates their memory consumption and allows independent scaling.
Before (in‑process registration):
# /etc/kubernetes/manifests/kube-controller-manager.yaml
spec:
containers:
- name: kube-controller-manager
image: k8s.gcr.io/kube-controller-manager:v1.28.2
command:
- kube-controller-manager
- --controllers=*,model
- --kubeconfig=/etc/kubernetes/controller-manager.conf
After (external Deployment):
# model-controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-controller
namespace: kube-system
spec:
replicas: 3
selector:
matchLabels:
app: model-controller
template:
metadata:
labels:
app: model-controller
spec:
serviceAccountName: model-controller-sa
containers:
- name: model-controller
image: yourrepo/model-controller:v2.1.0
resources:
limits:
memory: "1Gi"
requests:
memory: "512Mi"
args:
- --kubeconfig=/etc/kubernetes/controller-manager.conf
volumeMounts:
- name: kubeconfig
mountPath: /etc/kubernetes
volumes:
- name: kubeconfig
hostPath:
path: /etc/kubernetes
Remove the model entry from the --controllers flag and restart the static pod.
2. Add pagination and field selectors
Modify the reconciler to watch only the subset of models that changed, using Informer filters.
// Before: full list on every reconcile
func (r *ModelReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var models modelv1.ModelList
if err := r.List(ctx, &models); err != nil {
return ctrl.Result{}, err
}
// process entire list...
}
// After: filtered watch + pagination
func (r *ModelReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&modelv1.Model{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
WithOptions(controller.Options{
MaxConcurrentReconciles: 5,
}).
Complete(r)
}
3. Limit in‑memory payload size
Store large binary blobs (model weights) in a dedicated object store (e.g., S3) and keep only a reference (URL) in the ModelSpec. This reduces heap pressure.
// ModelSpec before
type ModelSpec struct {
Binary []byte `json:"binary"` // up to 200MiB
}
// ModelSpec after
type ModelSpec struct {
BlobURL string `json:"blobURL"` // S3 pre‑signed URL
}
4. Enforce memory limits on the controller manager
Even after externalizing the controller, set a hard limit on the controller manager to prevent it from being killed by other components.
# /etc/kubernetes/manifests/kube-controller-manager.yaml (add resources)
resources:
limits:
memory: "2Gi"
requests:
memory: "1Gi"
Verify: Confirm the Issue Is Resolved
- Observe the controller manager pod for at least 24 h:
kubectl -n kube-system get pod kube-controller-manager-$(hostname) -w
# Expect no OOMKilled events.
- Check memory consumption:
kubectl top pod -n kube-system kube-controller-manager-$(hostname)
# Should stay below 1.5 GiB.
- Validate model controller health:
kubectl get deployment model-controller -n kube-system
kubectl logs deployment/model-controller -n kube-system | grep "reconcile Model"
# Logs should show processing of only changed resources, not full list scans.
- Confirm pod scheduling works:
kubectl get pods -A | grep Pending
# No pending pods due to scheduler errors.
Prevent: Operational Guardrails
- Monitoring: Add a Prometheus alert on
process_resident_memory_bytes{job="kube-controller-manager"}crossing 80 % of the limit. - Alerting: Configure a PagerDuty notification for OOM events on the control‑plane nodes.
- Testing: Include load‑testing of custom controllers in CI pipelines, simulating > 1 000 model resources.
- Configuration Review: Enforce a policy that any new CRD controller must run as an external Deployment with explicit resource limits.
- Documentation: Record the maximum expected size of model descriptors and enforce size validation in the webhook admission controller.
Related Topic Hub: Distributed Systems Troubleshooting Hub
FAQ
- Why does the controller manager OOM only after adding the model CRDs?
The new CRDs introduced a high‑frequency, high‑volume reconciliation loop that runs inside the same process as the built‑in controllers. The unbounded list‑and‑process pattern caused the heap to grow beyond the node’s memory capacity. - Can I keep the custom controller inside the controller manager if I add pagination?
Even with pagination, the controller manager shares a single Go runtime. A bug or unexpected load in any in‑process controller can still affect the whole control plane. Externalizing is the safest approach. - What memory limit should I set for the controller manager?
Set the limit based on observed peak usage after refactoring. In most clusters, 2 GiB provides headroom for built‑in controllers while still protecting the node from OOM. - How do I detect similar OOM patterns for future custom controllers?
Enable theprocess_resident_memory_bytesmetric for each controller pod, and create a Grafana dashboard that shows per‑controller memory trends. Alert on rapid growth or sustained usage above 75 % of the request. - Is it safe to store model binaries directly in a ConfigMap?
No. ConfigMaps have a size limit of 1 MiB and are kept in etcd, which is not designed for large binary payloads. Use an external object store and reference the location in the CRD.