Prometheus deployment rejected by admission controller on edge node

Problem

When deploying the Prometheus server on an edge‑computing Kubernetes node, the API server returns an admission‑controller error and the Prometheus custom resource (CR) never becomes active. Metric collection and alerting stop working for the edge site.

Typical error observed:


$ kubectl apply -f prometheus-edge.yaml
error: admission webhook "prometheus-operator-admission.k8s.io" denied the request: 
[edge‑policy] deployment of Prometheus is rejected: 
resource limits exceed allowed maximum (cpu: 500m, memory: 256Mi)

The failure prevents the prometheus pod from being scheduled, and the Prometheus CR remains in Pending state.

Root Cause

The edge cluster runs a custom admission controller chain that enforces:

  • Maximum CPU and memory per pod (500 mCPU, 256 MiB RAM).
  • Mandatory nodeSelector edge=true for workloads that must run on edge nodes.
  • Explicit tolerations for the edge-taint applied to all edge nodes.
  • Prohibited use of the hostNetwork flag.

The default Prometheus manifest generated by kube-prometheus-stack requests 1 CPU and 2 GiB memory, does not include the required nodeSelector, and omits the toleration. Consequently, the admission controller (implemented with OPA Gatekeeper constraints) rejects the creation request.

Debug

Follow these steps to reproduce the failure and gather evidence.

1. Inspect the failing resource


$ kubectl get prometheus -n monitoring edge-prom -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: edge-prom
  namespace: monitoring
spec:
  resources:
    limits:
      cpu: "1"
      memory: 2Gi
  # missing nodeSelector and tolerations

2. Review the admission webhook logs


$ kubectl logs -n gatekeeper-system -l control-plane=controller-manager \
    -c manager --tail=100 | grep edge‑policy
2026-11-15T08:12:34Z    INFO    admission    {"uid":"12345","kind":"Prometheus","name":"edge-prom","namespace":"monitoring","msg":"rejected","reason":"resource limits exceed allowed maximum (cpu: 500m, memory: 256Mi)"}

3. Examine the Gatekeeper constraint that blocks the deployment


$ kubectl get constrainttemplates.template.gatekeeper.sh edge‑resource‑limits -o yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: edge-resource-limits
spec:
  crd:
    spec:
      names:
        kind: EdgeResourceLimits
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package edge.resource.limits
      violation[{"msg": msg}] {
        input.review.object.kind == "Prometheus"
        limits := input.review.object.spec.resources.limits
        cpu := parse_quantity(limits.cpu)
        mem := parse_quantity(limits.memory)
        cpu > 500000000   # 500m
        msg := sprintf("resource limits exceed allowed maximum (cpu: %v, memory: %v)", [limits.cpu, limits.memory])
      }

4. Verify node taints and labels


$ kubectl get nodes -l edge=true -o wide
NAME          STATUS   ROLES    AGE   VERSION   LABELS
edge-node-1   Ready       12d   v1.27.3   edge=true

$ kubectl describe node edge-node-1 | grep Taint
Taints:             edge-taint=NoSchedule:NoExecute

Solution

Two complementary approaches resolve the issue:

Option A – Adjust the Prometheus manifest to satisfy existing policies

Reduce resource requests/limits, add the required nodeSelector and tolerations, and ensure hostNetwork is false.

Before (failing manifest):


apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: edge-prom
  namespace: monitoring
spec:
  resources:
    limits:
      cpu: "1"
      memory: 2Gi
  # missing nodeSelector, tolerations

After (compliant manifest):


apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: edge-prom
  namespace: monitoring
spec:
  resources:
    limits:
      cpu: "500m"
      memory: 256Mi
  nodeSelector:
    edge: "true"
  tolerations:
  - key: "edge-taint"
    operator: "Exists"
    effect: "NoSchedule"
  hostNetwork: false

Apply the updated manifest:


$ kubectl apply -f prometheus-edge.yaml
prometheus.monitoring.coreos.com/edge-prom created

Option B – Relax the admission controller policy for Prometheus

If the edge workload truly requires more resources, modify the Gatekeeper constraint to allow a higher limit for the Prometheus kind.

Update the EdgeResourceLimits constraint:


apiVersion: constraints.gatekeeper.sh/v1beta1
kind: EdgeResourceLimits
metadata:
  name: edge-prometheus-limits
spec:
  match:
    kinds:
    - apiGroups: ["monitoring.coreos.com"]
      kinds: ["Prometheus"]
  parameters: {}
  # raise limits for Prometheus only
  enforcementAction: dryrun   # change to "deny" after testing

After validation, change enforcementAction back to deny and re‑apply.

Verify

Confirm that the Prometheus pod is running on the edge node and that metrics are being scraped.


$ kubectl get pods -n monitoring -l app=prometheus
NAME                     READY   STATUS    RESTARTS   AGE
prometheus-edge-prom-0   2/2     Running   0          2m

$ kubectl describe pod prometheus-edge-prom-0 -n monitoring | grep Node
Node:               edge-node-1/10.0.0.5

$ curl -s http://edge-node-1:9090/-/ready
{"status":"ready"}

Check that the admission controller no longer logs rejections for the Prometheus resource.

Prevent

  • Policy‑as‑code testing: integrate OPA/Gatekeeper constraint unit tests into CI pipelines (e.g., conftest) to catch mismatches before deployment.
  • Manifest linting: enforce a lint rule that all Prometheus CRs include nodeSelector, tolerations, and resource limits within the edge profile.
  • Observability: ship a side‑car that streams admission‑controller decisions to a central log store; set alerts on denied events for critical workloads.
  • Documentation: maintain a version‑controlled “edge‑policy guide” that lists required fields for each workload type.
  • Resource budgeting: use the VerticalPodAutoscaler on edge nodes to keep actual usage within the policy envelope, reducing the need for manual limit adjustments.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

  1. Why does the same Prometheus manifest work on central clusters but fail on edge nodes?
    Edge clusters have stricter Gatekeeper constraints (CPU/memory caps, mandatory node selectors) that are not present in the central clusters. The manifest exceeds those caps, triggering the denial.
  2. Can I disable the admission controller for a single namespace?
    Gatekeeper constraints can be scoped with match.namespace. Create a separate constraint that excludes the monitoring namespace, but this weakens policy enforcement and is not recommended.
  3. What if I need more than 500 mCPU for Prometheus on the edge?
    Either raise the limit in the Gatekeeper EdgeResourceLimits constraint for the Prometheus kind, or split the workload (e.g., run a remote write to a central Prometheus and keep a lightweight prometheus-agent on the edge).
  4. How do I debug a denial that does not show a clear message?
    Enable --v=6 on the Gatekeeper controller manager to get verbose logs, and inspect the AdmissionReview object in the webhook request via kubectl get --raw /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations.
  5. Is there a way to automatically add the required nodeSelector/tolerations to all Prometheus CRs?
    Yes. Use a MutatingAdmissionWebhook (or Gatekeeper’s mutation feature) that injects the missing fields based on a policy template.