Problem
When deploying the Prometheus server on an edge‑computing Kubernetes node, the API server returns an admission‑controller error and the Prometheus custom resource (CR) never becomes active. Metric collection and alerting stop working for the edge site.
Typical error observed:
$ kubectl apply -f prometheus-edge.yaml
error: admission webhook "prometheus-operator-admission.k8s.io" denied the request:
[edge‑policy] deployment of Prometheus is rejected:
resource limits exceed allowed maximum (cpu: 500m, memory: 256Mi)
The failure prevents the prometheus pod from being scheduled, and the Prometheus CR remains in Pending state.
Root Cause
The edge cluster runs a custom admission controller chain that enforces:
- Maximum CPU and memory per pod (500 mCPU, 256 MiB RAM).
- Mandatory
nodeSelectoredge=truefor workloads that must run on edge nodes. - Explicit
tolerationsfor theedge-taintapplied to all edge nodes. - Prohibited use of the
hostNetworkflag.
The default Prometheus manifest generated by kube-prometheus-stack requests 1 CPU and 2 GiB memory, does not include the required nodeSelector, and omits the toleration. Consequently, the admission controller (implemented with OPA Gatekeeper constraints) rejects the creation request.
Debug
Follow these steps to reproduce the failure and gather evidence.
1. Inspect the failing resource
$ kubectl get prometheus -n monitoring edge-prom -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: edge-prom
namespace: monitoring
spec:
resources:
limits:
cpu: "1"
memory: 2Gi
# missing nodeSelector and tolerations
2. Review the admission webhook logs
$ kubectl logs -n gatekeeper-system -l control-plane=controller-manager \
-c manager --tail=100 | grep edge‑policy
2026-11-15T08:12:34Z INFO admission {"uid":"12345","kind":"Prometheus","name":"edge-prom","namespace":"monitoring","msg":"rejected","reason":"resource limits exceed allowed maximum (cpu: 500m, memory: 256Mi)"}
3. Examine the Gatekeeper constraint that blocks the deployment
$ kubectl get constrainttemplates.template.gatekeeper.sh edge‑resource‑limits -o yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: edge-resource-limits
spec:
crd:
spec:
names:
kind: EdgeResourceLimits
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package edge.resource.limits
violation[{"msg": msg}] {
input.review.object.kind == "Prometheus"
limits := input.review.object.spec.resources.limits
cpu := parse_quantity(limits.cpu)
mem := parse_quantity(limits.memory)
cpu > 500000000 # 500m
msg := sprintf("resource limits exceed allowed maximum (cpu: %v, memory: %v)", [limits.cpu, limits.memory])
}
4. Verify node taints and labels
$ kubectl get nodes -l edge=true -o wide
NAME STATUS ROLES AGE VERSION LABELS
edge-node-1 Ready 12d v1.27.3 edge=true
$ kubectl describe node edge-node-1 | grep Taint
Taints: edge-taint=NoSchedule:NoExecute
Solution
Two complementary approaches resolve the issue:
Option A – Adjust the Prometheus manifest to satisfy existing policies
Reduce resource requests/limits, add the required nodeSelector and tolerations, and ensure hostNetwork is false.
Before (failing manifest):
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: edge-prom
namespace: monitoring
spec:
resources:
limits:
cpu: "1"
memory: 2Gi
# missing nodeSelector, tolerations
After (compliant manifest):
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: edge-prom
namespace: monitoring
spec:
resources:
limits:
cpu: "500m"
memory: 256Mi
nodeSelector:
edge: "true"
tolerations:
- key: "edge-taint"
operator: "Exists"
effect: "NoSchedule"
hostNetwork: false
Apply the updated manifest:
$ kubectl apply -f prometheus-edge.yaml
prometheus.monitoring.coreos.com/edge-prom created
Option B – Relax the admission controller policy for Prometheus
If the edge workload truly requires more resources, modify the Gatekeeper constraint to allow a higher limit for the Prometheus kind.
Update the EdgeResourceLimits constraint:
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: EdgeResourceLimits
metadata:
name: edge-prometheus-limits
spec:
match:
kinds:
- apiGroups: ["monitoring.coreos.com"]
kinds: ["Prometheus"]
parameters: {}
# raise limits for Prometheus only
enforcementAction: dryrun # change to "deny" after testing
After validation, change enforcementAction back to deny and re‑apply.
Verify
Confirm that the Prometheus pod is running on the edge node and that metrics are being scraped.
$ kubectl get pods -n monitoring -l app=prometheus
NAME READY STATUS RESTARTS AGE
prometheus-edge-prom-0 2/2 Running 0 2m
$ kubectl describe pod prometheus-edge-prom-0 -n monitoring | grep Node
Node: edge-node-1/10.0.0.5
$ curl -s http://edge-node-1:9090/-/ready
{"status":"ready"}
Check that the admission controller no longer logs rejections for the Prometheus resource.
Prevent
- Policy‑as‑code testing: integrate OPA/Gatekeeper constraint unit tests into CI pipelines (e.g.,
conftest) to catch mismatches before deployment. - Manifest linting: enforce a lint rule that all
PrometheusCRs includenodeSelector,tolerations, and resource limits within the edge profile. - Observability: ship a side‑car that streams admission‑controller decisions to a central log store; set alerts on
deniedevents for critical workloads. - Documentation: maintain a version‑controlled “edge‑policy guide” that lists required fields for each workload type.
- Resource budgeting: use the
VerticalPodAutoscaleron edge nodes to keep actual usage within the policy envelope, reducing the need for manual limit adjustments.
Related Topic Hub: Observability Troubleshooting Hub
FAQ
- Why does the same Prometheus manifest work on central clusters but fail on edge nodes?
Edge clusters have stricter Gatekeeper constraints (CPU/memory caps, mandatory node selectors) that are not present in the central clusters. The manifest exceeds those caps, triggering the denial. - Can I disable the admission controller for a single namespace?
Gatekeeper constraints can be scoped withmatch.namespace. Create a separate constraint that excludes themonitoringnamespace, but this weakens policy enforcement and is not recommended. - What if I need more than 500 mCPU for Prometheus on the edge?
Either raise the limit in the GatekeeperEdgeResourceLimitsconstraint for thePrometheuskind, or split the workload (e.g., run a remote write to a central Prometheus and keep a lightweightprometheus-agenton the edge). - How do I debug a denial that does not show a clear message?
Enable--v=6on the Gatekeeper controller manager to get verbose logs, and inspect theAdmissionReviewobject in the webhook request viakubectl get --raw /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations. - Is there a way to automatically add the required nodeSelector/tolerations to all Prometheus CRs?
Yes. Use aMutatingAdmissionWebhook(or Gatekeeper’smutationfeature) that injects the missing fields based on a policy template.