Problem Description
In an air‑gapped Kubernetes cluster the Elasticsearch scheduler pod (managed by the Elastic Cloud on Kubernetes (ECK) operator) remains in Pending with events such as:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m default-scheduler 0/5 nodes are available: 5 Insufficient cpu.
Warning FailedScheduling 2m default-scheduler 0/5 nodes are available: 5 Insufficient memory.
Warning FailedScheduling 2m default-scheduler node(s) had taints {edge-node=true:NoSchedule}, that the pod didn't tolerate.
Because the cluster cannot reach external registries, the image pull also fails, producing ImagePullBackOff errors that further block scheduling.
Root Cause Analysis
The failure stems from a combination of three factors that are common in air‑gapped environments:
- Resource requests exceed node capacity. The ECK default
resources.requestsfor Elasticsearch pods are 1 CPU and 2 GiB memory (ECK spec). The edge nodes in the deployment provide only 2 vCPU and 4 GiB total, but a portion is already consumed by other workloads, leaving insufficient headroom. - Node taints without matching tolerations. The cluster applies a custom taint
edge-node=true:NoScheduleto protect critical workloads. The Elasticsearch manifest does not declare the corresponding toleration, so the scheduler rejects placement (Kubernetes scheduler docs). - Image pull failures due to air‑gap. Without access to
docker.elastic.co, the pod cannot download the Elasticsearch container image, resulting inImagePullBackOff. The scheduler interprets this as “no nodes available to satisfy pod’s image pull policy” (Elastic air‑gapped guide).
All three conditions cause the scheduler to emit Insufficient cpu, Insufficient memory, or node(s) had taints … that the pod didn't tolerate messages, preventing pod placement.
Investigation and Debugging
Follow these steps to reproduce the diagnostic information:
- Inspect pod events and node resources.
kubectl get pods -n elastic-system -o wide kubectl describe pod elastic-operator-xxxxx -n elastic-systemTypical output:
Name: elastic-operator-xxxxx
Namespace: elastic-system
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 1m default-scheduler 0/3 nodes are available: 3 Insufficient cpu.
Warning FailedScheduling 1m default-scheduler node(s) had taints {edge-node=true:NoSchedule}, that the pod didn't tolerate
- Check node allocatable resources.
kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.status.allocatable.cpu}{'\t'}{.status.allocatable.memory}{'\n'}{end}"Example output:
node-1 2000m 4096Mi
node-2 2000m 4096Mi
node-3 2000m 4096Mi
If the sum of existing pod requests already consumes most of the allocatable CPU/memory, the new Elasticsearch pod cannot be scheduled.
- Verify taints on the nodes.
kubectl describe node node-1 | grep -i taintTypical output:
Taints: edge-node=true:NoSchedule
- Confirm image pull status.
kubectl get pod elastic-operator-xxxxx -n elastic-system -o jsonpath="{.status.containerStatuses[0].state.waiting.reason}"Output may be
ImagePullBackOff. Check the operator logs:
kubectl logs deployment/elastic-operator -n elastic-system
Look for messages such as:
Failed to pull image "docker.elastic.co/eck/eck-operator:1.9.0": rpc error: code = Unknown desc = failed to resolve reference "docker.elastic.co/eck/eck-operator:1.9.0"
Resolution
Apply the following changes in order of impact.
1. Adjust resource requests to fit node capacity
Reduce the default CPU and memory requests for Elasticsearch master and data nodes. Example Elasticsearch manifest before and after:
Before (default)
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 8.9.0
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
resources:
requests:
cpu: "1"
memory: 2Gi
After (tailored for edge nodes)
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 8.9.0
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
resources:
requests:
cpu: "500m"
memory: 1Gi
limits:
cpu: "1"
memory: 2Gi
Explanation: Reducing requests to 500 mCPU and 1 GiB memory allows three pods to coexist on a 2 vCPU / 4 GiB node while preserving limits to prevent runaway usage.
2. Add tolerations for the custom edge‑node taint
Insert a toleration block into the Elasticsearch spec:
spec:
nodeSets:
- name: default
podTemplate:
spec:
tolerations:
- key: "edge-node"
operator: "Equal"
value: "true"
effect: "NoSchedule"
3. Pre‑load Elasticsearch images and configure imagePullSecrets
In an air‑gapped cluster, mirror the required images to a local registry (e.g., registry.local:5000) and reference them:
spec:
version: 8.9.0
image: registry.local:5000/eck/eck-operator:1.9.0
imagePullSecrets:
- name: local-registry-secret
Create the secret once:
kubectl create secret docker-registry local-registry-secret \
--docker-server=registry.local:5000 \
--docker-username=admin \
--docker-password=******** \
--docker-email=admin@example.com \
-n elastic-system
4. Redeploy the operator and Elasticsearch cluster
kubectl apply -f elastic-operator.yaml
kubectl apply -f elasticsearch.yaml
Verification
Confirm that the pods transition to Running and that the operator reports a healthy cluster.
kubectl get pods -n elastic-system
NAME READY STATUS RESTARTS AGE
elastic-operator-xxxxx 1/1 Running 0 2m
quickstart-es-default-0 1/1 Running 0 1m
quickstart-es-default-1 1/1 Running 0 1m
quickstart-es-default-2 1/1 Running 0 1m
Check the Elasticsearch health API:
curl -k -u elastic:changeme https://quickstart-es-http:9200/_cluster/health?pretty
Expected output includes "status" : "green".
Validate that no pending events remain:
kubectl describe pod quickstart-es-default-0 -n elastic-system | grep -i Events -A5
No FailedScheduling or ImagePullBackOff lines should appear.
Prevention and Best Practices
- Capacity planning. Use
kubectl top nodesandkubectl describe nodeto track allocatable resources before adding heavy stateful workloads. - Explicit resource sizing. Override ECK defaults for all environments, especially edge or air‑gapped clusters, as recommended in the ECK deployment guide.
- Document node taints. Maintain a central list of custom taints and ensure all manifests include matching tolerations.
- Image mirroring strategy. Automate periodic mirroring of Elastic images to an internal registry and keep
imagePullSecretsversion‑controlled. - Monitoring alerts. Configure alerts on
kube_pod_status_phase{phase="Pending"}and on events containing “Insufficient cpu” or “Insufficient memory” to catch resource saturation early.
Related Topic Hub: Data Infrastructure Troubleshooting Hub
FAQ
- Why does the pod stay
Pendingeven though the node shows free CPU?
Because the scheduler evaluates requests, not actual free capacity. If the sum of all requests exceeds allocatable resources, the pod cannot be placed. - Can I keep the default resource requests and just add more nodes?
Yes, adding nodes with sufficientallocatableCPU/memory resolves the scheduling error, but in air‑gapped sites node count is often limited; adjusting requests is usually more practical. - Do I need to add tolerations for every custom taint in the cluster?
Only for pods that must run on tainted nodes. Review the cluster’s taint policy and add tolerations to the ElasticsearchpodTemplate.spec.tolerationssection as needed. - How can I verify which image the operator is trying to pull?
Inspect theimagefield in the operator deployment or runkubectl describe pod <pod> -n elastic-systemand look for theImageline underContainers. - Is it safe to lower Elasticsearch memory requests below 1 GiB?
Elasticsearch requires enough heap for stable operation. The official docs recommend at least 1 GiB heap (i.e., 2 GiB JVM memory). Reducing requests below that may cause out‑of‑memory kills; instead keep limits higher while lowering requests.