Kubernetes Controller Manager Crash With Elasticsearch In Air-gapped Environment

Q: Problem Description

In a private, air‑gapped Kubernetes cluster the kube-controller-manager pod repeatedly enters CrashLoopBackOff. The crash is triggered when the Elastic Cloud on Kubernetes (ECK) operator attempts to deploy Elasticsearch for log aggregation. Typical log excerpts are:

Q: 1. Mount the custom CA bundle into the controller manager

Create a ConfigMap that contains the root CA and reference it in the static pod manifest (or kubeadm config).

In this article

Table of Contents

Problem Description

In a private, air‑gapped Kubernetes cluster the kube-controller-manager pod repeatedly enters CrashLoopBackOff. The crash is triggered when the Elastic Cloud on Kubernetes (ECK) operator attempts to deploy Elasticsearch for log aggregation. Typical log excerpts are:


2026-06-05T12:34:56Z panic: x509: certificate signed by unknown authority
goroutine 1 [running]:
k8s.io/kubernetes/cmd/kube-controller-manager/app.run(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app.go:345 +0x1c3
...

Additional errors appear in the controller manager’s startup sequence:


failed to verify certificate: x509: certificate signed by unknown authority
error fetching updates from https://artifacts.elastic.co: dial tcp: lookup artifacts.elastic.co: no such host
failed to pull image "docker.elastic.co/elasticsearch/elasticsearch:7.17.0": repository does not exist or is inaccessible

These symptoms match community reports such as GitHub issue kubernetes/kubernetes#108123 and real incidents in financial services and telecom environments.

Root Cause Analysis

The controller manager validates TLS certificates for any HTTPS endpoint it contacts, including the ECK operator’s webhook server and the Elasticsearch image registry. In an air‑gapped environment:

The cluster’s nodes trust only a custom root CA that is not part of the default /etc/kubernetes/pki/ca.crt bundle.
ECK’s default deployment references the public Docker registry (docker.elastic.co) and the public Elastic artifact repository (artifacts.elastic.co) for images and plugins.
The controller manager is started without the --root-ca-file flag pointing to the custom CA bundle, nor is a ConfigMap with the CA mounted into /etc/kubernetes/pki as described in the Kubernetes “Secrets and ConfigMaps for storing custom CA bundles in air‑gapped clusters” documentation.
During the Elasticsearch operator’s admission webhook registration, the controller manager attempts to contact the webhook over HTTPS. Because the webhook presents a certificate signed by the custom CA, the controller manager cannot verify it and panics (certificate signed by unknown authority).
When the operator later tries to pull the Elasticsearch image, DNS resolution for docker.elastic.co fails (no external DNS) and the image pull error propagates back to the controller manager health‑check, causing another panic.

In short, the controller manager crashes because it cannot validate TLS certificates and cannot reach external repositories that the ECK operator assumes are reachable.

Investigation and Debugging Steps

Inspect controller manager logs. Use kubectl logs -n kube-system kube-controller-manager-xxxx and look for the “certificate signed by unknown authority” and DNS lookup failures.

Verify the CA bundle present on the node.


$ ls -l /etc/kubernetes/pki/ca.crt
-rw-r--r-- 1 root root 1234 Jan 10 12:00 /etc/kubernetes/pki/ca.crt
$ openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text | grep Subject:
    Subject: CN=kubernetes

If the custom CA is missing, the verification will fail.

Check the ECK operator deployment.


$ kubectl -n elastic-system get deployment elastic-operator -o yaml | grep image
      image: docker.elastic.co/eck/eck-operator:1.9.0

The image points to a public registry.

Confirm DNS resolution inside a pod.


$ kubectl run -i --rm debug --image=busybox --restart=Never -- nslookup artifacts.elastic.co
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
*** server can't find artifacts.elastic.co: NXDOMAIN

Validate the webhook TLS secret.
```
$ kubectl -n elastic-system get secret elastic-webhook-server-cert -o yaml | grep tls.crt
  tls.crt: 
```
Decode and inspect the certificate chain to ensure the custom root CA is included.

Resolution

The fix consists of three parts: (1) provide the custom CA bundle to the controller manager, (2) configure the ECK operator to use a locally‑mirrored Elasticsearch image and plugin repository, and (3) ensure the webhook server presents a certificate chain trusted by the controller manager.

1. Mount the custom CA bundle into the controller manager

Create a ConfigMap that contains the root CA and reference it in the static pod manifest (or kubeadm config).

Before:


# /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - name: kube-controller-manager
    image: k8s.gcr.io/kube-controller-manager:v1.27.0
    command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    # missing --root-ca-file flag

After:


# Create ConfigMap with custom CA
kubectl create configmap custom-root-ca \
  --from-file=ca.crt=/opt/custom-ca/root-ca.pem \
  -n kube-system

# Updated static pod manifest
apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - name: kube-controller-manager
    image: k8s.gcr.io/kube-controller-manager:v1.27.0
    command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --root-ca-file=/etc/custom-ca/ca.crt
    volumeMounts:
    - name: custom-ca
      mountPath: /etc/custom-ca
  volumes:
  - name: custom-ca
    configMap:
      name: custom-root-ca

After editing the manifest, the kubelet reloads the pod and the controller manager starts with the proper trust store.

2. Use a private image registry for Elasticsearch

Mirror the required Elasticsearch image into an internal registry (e.g., registry.local:5000) and update the Elasticsearch resource spec.

Before:


apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.17.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false

The operator will try to pull from docker.elastic.co.

After:


apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.17.0
  image: registry.local:5000/elasticsearch:7.17.0   # private registry
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        imagePullSecrets:
        - name: private-reg-secret

3. Provide the custom CA to the ECK webhook server

Generate a TLS secret that includes the custom root CA and mount it into the operator.


# Generate cert signed by custom CA
openssl req -new -keyout tls.key -out tls.csr -subj "/CN=elastic-webhook.elastic-system.svc"
openssl x509 -req -in tls.csr -CA /opt/custom-ca/root-ca.pem -CAkey /opt/custom-ca/root-ca-key.pem -CAcreateserial -out tls.crt -days 365

# Create secret
kubectl create secret tls elastic-webhook-server-cert \
  --cert=tls.crt --key=tls.key -n elastic-system

# Patch operator deployment to mount the CA bundle
kubectl -n elastic-system patch deployment elastic-operator --type='json' -p='[
  {"op":"add","path":"/spec/template/spec/volumes/-","value":{"name":"custom-ca","configMap":{"name":"custom-root-ca"}}},
  {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/-","value":{"name":"custom-ca","mountPath":"/etc/custom-ca"}}
]'

Now the webhook presents a certificate chain that the controller manager can validate using the same /etc/custom-ca/ca.crt file.

Validation

Confirm the controller manager is running without panics:


$ kubectl -n kube-system get pod -l component=kube-controller-manager
NAME                                 READY   STATUS    RESTARTS   AGE
kube-controller-manager-xxxx        1/1     Running   0          2m

Check controller manager logs for the absence of TLS errors:


$ kubectl logs -n kube-system kube-controller-manager-xxxx | grep -i "certificate"
(no output)

Verify the ECK operator is healthy:


$ kubectl -n elastic-system get pod -l control-plane=elastic-operator
NAME                     READY   STATUS    RESTARTS   AGE
elastic-operator-abc123  1/1     Running   0          3m

Ensure Elasticsearch pods start correctly:


$ kubectl get pods -l common.k8s.elastic.co/type=elasticsearch
NAME                         READY   STATUS    RESTARTS   AGE
quickstart-es-default-0      1/1     Running   0          2m
quickstart-es-default-1      1/1     Running   0          2m
quickstart-es-default-2      1/1     Running   0          2m

Run a quick health check against the Elasticsearch HTTP endpoint using the custom CA:


$ curl --cacert /opt/custom-ca/root-ca.pem https://quickstart-es-http:9200/_cluster/health?pretty
{
  "cluster_name" : "quickstart",
  "status" : "green",
  ...
}

Operational Best Practices and Prevention

Store custom CAs in a dedicated ConfigMap and mount them into every control‑plane component that performs TLS verification. Follow the pattern described in the Kubernetes “Secrets and ConfigMaps for storing custom CA bundles in air‑gapped clusters” documentation.
Mirror all external container images and artifact repositories. Use a private registry and a local artifact-mirror (e.g., artifacts.local) for Elasticsearch plugins.
Pin the --root-ca-file flag in the kube‑controller‑manager manifest. This prevents accidental fallback to the default system trust store.
Enable a health‑check probe for the ECK webhook. Configure the controller manager’s --secure-port and --tls-cert-file to point to the same CA bundle used by the webhook.
Automate CA bundle updates. When the internal PKI rotates, roll out a rolling update of the ConfigMap and restart the static pods (kubelet will handle it).
Monitor TLS handshake failures. Add a Prometheus alert on apiserver_audit_error_total{reason="certificate_invalid"} or on controller manager logs containing “x509”.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the controller manager panic instead of just logging a TLS error?
The controller manager treats TLS verification failures as fatal during startup because it cannot guarantee secure communication with critical components (e.g., the admission webhook). The panic is intentional to avoid operating in an insecure state.
Can I avoid mounting a custom CA by disabling certificate verification?
Disabling verification (e.g., --insecure-skip-tls-verify) is not supported for the controller manager and would expose the control plane to man‑in‑the‑middle attacks. Providing the proper CA bundle is the recommended approach.
Do I need to update the kubelet’s trust store as well?
Only if the kubelet itself performs TLS verification against external services (e.g., pulling images via HTTPS). For the controller manager, mounting the CA into its pod is sufficient.
What if I cannot mirror the Elasticsearch image?
Use imagePullPolicy: IfNotPresent after manually loading the image onto each node (e.g., ctr images import) and reference the local image name without a registry prefix.
How can I verify which CA bundle the controller manager is actually using?
Exec into the static pod and run:
```
$ cat /proc/$(pidof kube-controller-manager)/environ | tr '\\0' '\\n' | grep ROOT_CA_FILE
ROOT_CA_FILE=/etc/custom-ca/ca.crt
```
This confirms the flag is set and points to the expected file.