Grafana UI unreachable after blue‑green promotion due to network policy

Problem – Grafana UI Unreachable After Green Promotion in a Blue‑Green Deployment

In a Kubernetes cluster that hosts an AI inference service using a blue‑green deployment pattern, the green namespace is promoted to receive production traffic. Immediately after promotion, Grafana (running in the monitoring namespace) fails to:

  • Serve its web UI (HTTP 503/404 errors from the Ingress).
  • Scrape metrics from Prometheus endpoints that reside in the newly promoted green namespace.
  • Display dashboards that depend on data sources (Prometheus, Loki) that are now isolated.

Operators see the following symptoms in the logs and monitoring tools.

Observed Logs


$ kubectl logs -n monitoring grafana-7c9d5f6d9b-ktz9x
2024-10-12T08:15:23Z error  failed to get datasource "Prometheus (green)" : Get "http://prometheus-green.monitoring.svc:9090/api/v1/query?query=up": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-10-12T08:15:24Z error  failed to load dashboard "AI Inference Overview": datasource "Prometheus (green)" not found
2024-10-12T08:15:30Z error  failed to start HTTP server: listen tcp 0.0.0.0:3000: bind: permission denied

$ kubectl describe ingress -n monitoring grafana
Name:             grafana
Namespace:        monitoring
Address:          10.0.12.34
Default backend: default-http-backend:80 (10.0.5.2:80)
Rules:
  Host                     Path    Backends
  ----                     ----    --------
  grafana.example.com
                           /   grafana:3000 ()
Annotations:
  kubernetes.io/ingress.class: nginx
  nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/16
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  5m    nginx-ingress-controller  Ingress monitoring/grafana

Root Cause – NetworkPolicy Restricts Traffic to the Green Namespace

The cluster enforces a NetworkPolicy per environment namespace. When the green namespace (ai-green) is created, a default deny policy is applied:


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: ai-green
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This policy blocks all inbound and outbound traffic unless explicitly allowed. The existing monitoring stack (Grafana, Prometheus) resides in the monitoring namespace and has network policies that only permit traffic to the ai-blue namespace. When the green namespace is promoted, the following assumptions fail:

  • Assumption 1: Grafana can reach any Prometheus endpoint in the cluster because the cluster network is “open”. In reality, NetworkPolicy isolates namespaces.
  • Assumption 2: The Ingress whitelist (10.0.0.0/16) automatically includes the green pods. The whitelist only applies to source IPs, not to pod‑to‑pod communication which is governed by NetworkPolicy.
  • Assumption 3: Service IPs are reachable without additional egress rules. The deny-all policy also blocks egress from green pods to the monitoring services, causing Grafana’s own HTTP server to be unable to bind to the node port when the pod runs with hostNetwork: false and the CNI enforces egress restrictions.

The result is a cascade of timeouts and “permission denied” errors, preventing Grafana from both serving its UI and pulling metrics.

Debug – Step‑by‑Step Investigation

  1. Confirm NetworkPolicy enforcement
    
    $ kubectl get networkpolicy -n ai-green
    NAME        POD-SELECTOR   AGE
    deny-all    {}             12m
    
  2. Check connectivity from Grafana to Prometheus (green)
    
    $ kubectl exec -n monitoring -it $(kubectl get pod -n monitoring -l app=grafana -o jsonpath='{.items[0].metadata.name}') -- curl -s -o /dev/null -w "%{http_code}" http://prometheus-green.monitoring.svc:9090/api/v1/query?query=up
    000
    

    Exit code 7 indicates a connection failure.

  3. Inspect egress rules for the green namespace
    
    $ kubectl get networkpolicy -n ai-green -o yaml
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: deny-all
      namespace: ai-green
    spec:
      podSelector: {}
      policyTypes:
      - Ingress
      - Egress
    
  4. Validate Ingress whitelist effect
    
    $ curl -s -o /dev/null -w "%{http_code}" http://grafana.example.com
    503
    

    The 503 originates from the NGINX Ingress controller, not from Grafana itself.

  5. Capture traffic with tcpdump on a node hosting a green pod
    
    # On the node
    $ sudo tcpdump -i eth0 -nn host 10.0.12.34 and port 3000 -c 5
    

    No packets are observed, confirming that the pod never receives inbound traffic.

Solution – Adjust NetworkPolicies to Permit Monitoring Traffic

The fix consists of two complementary policy updates:

1. Allow Grafana (monitoring namespace) to query Prometheus in the green namespace


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring-ingress
  namespace: ai-green
spec:
  podSelector:
    matchLabels:
      app: prometheus
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
      podSelector:
        matchLabels:
          app: grafana
    ports:
    - protocol: TCP
      port: 9090

2. Permit inbound traffic to Grafana’s Service from the external IP range


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: grafana-allow-ingress
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app: grafana
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 10.0.0.0/16
    ports:
    - protocol: TCP
      port: 3000

3. (Optional) Add egress rule for green pods to reach monitoring services


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-green-egress-to-monitoring
  namespace: ai-green
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: monitoring
    ports:
    - protocol: TCP
      port: 3000   # Grafana UI
    - protocol: TCP
      port: 9090   # Prometheus

Before / After Comparison

Aspect Before After
Grafana UI access 503 from Ingress, no pod traffic 200 OK, UI loads
Prometheus scrape (green) Timeout, “context deadline exceeded” Successful HTTP 200, metrics visible
Dashboard rendering Datasource not found errors Dashboards populate with live data

Verify – Confirm That the Issue Is Resolved

  1. Check Ingress health
    
    $ curl -s -o /dev/null -w "%{http_code}" http://grafana.example.com
    200
    
  2. Validate Grafana can query Prometheus (green)
    
    $ kubectl exec -n monitoring -it $(kubectl get pod -n monitoring -l app=grafana -o jsonpath='{.items[0].metadata.name}') -- curl -s http://prometheus-green.monitoring.svc:9090/api/v1/query?query=up | jq '.status'
    "success"
    
  3. Inspect Grafana logs for datasource errors
    
    $ kubectl logs -n monitoring $(kubectl get pod -n monitoring -l app=grafana -o jsonpath='{.items[0].metadata.name}') | grep "failed to get datasource"
    # No output – datasource connections succeed
    
  4. Confirm metrics appear on dashboards

    Open http://grafana.example.com/d/ai-inference-overview and verify that panels show non‑stale data from the green Prometheus instance.

Prevent – Guardrails to Avoid Recurrence

  • Policy Templates per Environment – Store baseline NetworkPolicy YAML files in a version‑controlled policy/ directory and apply them via kustomize during namespace creation.
  • Automated Policy Validation – Use kubectl netpol validate (or OPA Gatekeeper policies) in CI to ensure that any new namespace includes at least one ingress rule for monitoring and one egress rule for prometheus services.
  • Observability Alerts – Create Prometheus alerts for “Grafana datasource connection failures” and “Ingress 5xx rates > 1% for Grafana”. Example rule:
    
    - alert: GrafanaDatasourceFailure
      expr: rate(grafana_datasource_failure_total[5m]) > 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Grafana cannot reach one or more datasources"
        description: "Check NetworkPolicy for the target namespace."
    
  • Blue‑Green Promotion Script – Extend the promotion automation to patch the target namespace with the allow-monitoring-ingress and allow-green-egress-to-monitoring policies before traffic cut‑over.
  • Documentation Checklist – Add a “NetworkPolicy verification” step to the runbook for any new environment rollout.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

  1. Why does Grafana work with the blue namespace but fail after promoting green?

    The blue namespace already had explicit NetworkPolicy rules allowing monitoring traffic. The green namespace was created with a default deny policy, so after promotion no rules existed to let Grafana reach its data sources.

  2. Can I use a single “allow-all” NetworkPolicy for monitoring instead of per‑namespace rules?

    Yes, but it defeats the purpose of namespace isolation. A scoped policy that only permits traffic from the monitoring namespace to specific ports provides a balance between security and operability.

  3. Do I need to modify the Ingress whitelist when promoting green?

    No. The whitelist controls source IP ranges, not pod‑to‑pod traffic. The missing piece was the intra‑cluster NetworkPolicy, not the Ingress source range.

  4. How can I test NetworkPolicy changes without affecting production?

    Deploy a temporary “probe” pod in the target namespace and use curl or nc to verify connectivity to the monitoring services before applying the policy cluster‑wide.

  5. What if I use Calico’s GlobalNetworkPolicy instead of per‑namespace policies?

    Global policies can be used, but you must still create explicit allow rules for the monitoring namespace. The same principle applies: deny‑by‑default policies require explicit permits for cross‑namespace traffic.