Problem – Grafana UI Unreachable After Green Promotion in a Blue‑Green Deployment
In a Kubernetes cluster that hosts an AI inference service using a blue‑green deployment pattern, the green namespace is promoted to receive production traffic. Immediately after promotion, Grafana (running in the monitoring namespace) fails to:
- Serve its web UI (HTTP 503/404 errors from the Ingress).
- Scrape metrics from Prometheus endpoints that reside in the newly promoted green namespace.
- Display dashboards that depend on data sources (Prometheus, Loki) that are now isolated.
Operators see the following symptoms in the logs and monitoring tools.
Observed Logs
$ kubectl logs -n monitoring grafana-7c9d5f6d9b-ktz9x
2024-10-12T08:15:23Z error failed to get datasource "Prometheus (green)" : Get "http://prometheus-green.monitoring.svc:9090/api/v1/query?query=up": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-10-12T08:15:24Z error failed to load dashboard "AI Inference Overview": datasource "Prometheus (green)" not found
2024-10-12T08:15:30Z error failed to start HTTP server: listen tcp 0.0.0.0:3000: bind: permission denied
$ kubectl describe ingress -n monitoring grafana
Name: grafana
Namespace: monitoring
Address: 10.0.12.34
Default backend: default-http-backend:80 (10.0.5.2:80)
Rules:
Host Path Backends
---- ---- --------
grafana.example.com
/ grafana:3000 ()
Annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/16
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 5m nginx-ingress-controller Ingress monitoring/grafana
Root Cause – NetworkPolicy Restricts Traffic to the Green Namespace
The cluster enforces a NetworkPolicy per environment namespace. When the green namespace (ai-green) is created, a default deny policy is applied:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: ai-green
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
This policy blocks all inbound and outbound traffic unless explicitly allowed. The existing monitoring stack (Grafana, Prometheus) resides in the monitoring namespace and has network policies that only permit traffic to the ai-blue namespace. When the green namespace is promoted, the following assumptions fail:
- Assumption 1: Grafana can reach any Prometheus endpoint in the cluster because the cluster network is “open”. In reality,
NetworkPolicyisolates namespaces. - Assumption 2: The Ingress whitelist (
10.0.0.0/16) automatically includes the green pods. The whitelist only applies to source IPs, not to pod‑to‑pod communication which is governed byNetworkPolicy. - Assumption 3: Service IPs are reachable without additional egress rules. The
deny-allpolicy also blocks egress from green pods to themonitoringservices, causing Grafana’s own HTTP server to be unable to bind to the node port when the pod runs withhostNetwork: falseand the CNI enforces egress restrictions.
The result is a cascade of timeouts and “permission denied” errors, preventing Grafana from both serving its UI and pulling metrics.
Debug – Step‑by‑Step Investigation
- Confirm NetworkPolicy enforcement
$ kubectl get networkpolicy -n ai-green NAME POD-SELECTOR AGE deny-all {} 12m - Check connectivity from Grafana to Prometheus (green)
$ kubectl exec -n monitoring -it $(kubectl get pod -n monitoring -l app=grafana -o jsonpath='{.items[0].metadata.name}') -- curl -s -o /dev/null -w "%{http_code}" http://prometheus-green.monitoring.svc:9090/api/v1/query?query=up 000Exit code 7 indicates a connection failure.
- Inspect egress rules for the green namespace
$ kubectl get networkpolicy -n ai-green -o yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all namespace: ai-green spec: podSelector: {} policyTypes: - Ingress - Egress - Validate Ingress whitelist effect
$ curl -s -o /dev/null -w "%{http_code}" http://grafana.example.com 503The 503 originates from the NGINX Ingress controller, not from Grafana itself.
- Capture traffic with tcpdump on a node hosting a green pod
# On the node $ sudo tcpdump -i eth0 -nn host 10.0.12.34 and port 3000 -c 5No packets are observed, confirming that the pod never receives inbound traffic.
Solution – Adjust NetworkPolicies to Permit Monitoring Traffic
The fix consists of two complementary policy updates:
1. Allow Grafana (monitoring namespace) to query Prometheus in the green namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-monitoring-ingress
namespace: ai-green
spec:
podSelector:
matchLabels:
app: prometheus
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: monitoring
podSelector:
matchLabels:
app: grafana
ports:
- protocol: TCP
port: 9090
2. Permit inbound traffic to Grafana’s Service from the external IP range
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: grafana-allow-ingress
namespace: monitoring
spec:
podSelector:
matchLabels:
app: grafana
policyTypes:
- Ingress
ingress:
- from:
- ipBlock:
cidr: 10.0.0.0/16
ports:
- protocol: TCP
port: 3000
3. (Optional) Add egress rule for green pods to reach monitoring services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-green-egress-to-monitoring
namespace: ai-green
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
name: monitoring
ports:
- protocol: TCP
port: 3000 # Grafana UI
- protocol: TCP
port: 9090 # Prometheus
Before / After Comparison
| Aspect | Before | After |
|---|---|---|
| Grafana UI access | 503 from Ingress, no pod traffic | 200 OK, UI loads |
| Prometheus scrape (green) | Timeout, “context deadline exceeded” | Successful HTTP 200, metrics visible |
| Dashboard rendering | Datasource not found errors | Dashboards populate with live data |
Verify – Confirm That the Issue Is Resolved
- Check Ingress health
$ curl -s -o /dev/null -w "%{http_code}" http://grafana.example.com 200 - Validate Grafana can query Prometheus (green)
$ kubectl exec -n monitoring -it $(kubectl get pod -n monitoring -l app=grafana -o jsonpath='{.items[0].metadata.name}') -- curl -s http://prometheus-green.monitoring.svc:9090/api/v1/query?query=up | jq '.status' "success" - Inspect Grafana logs for datasource errors
$ kubectl logs -n monitoring $(kubectl get pod -n monitoring -l app=grafana -o jsonpath='{.items[0].metadata.name}') | grep "failed to get datasource" # No output – datasource connections succeed - Confirm metrics appear on dashboards
Open
http://grafana.example.com/d/ai-inference-overviewand verify that panels show non‑stale data from the green Prometheus instance.
Prevent – Guardrails to Avoid Recurrence
- Policy Templates per Environment – Store baseline
NetworkPolicyYAML files in a version‑controlledpolicy/directory and apply them viakustomizeduring namespace creation. - Automated Policy Validation – Use
kubectl netpol validate(or OPA Gatekeeper policies) in CI to ensure that any new namespace includes at least one ingress rule formonitoringand one egress rule forprometheusservices. - Observability Alerts – Create Prometheus alerts for “Grafana datasource connection failures” and “Ingress 5xx rates > 1% for Grafana”. Example rule:
- alert: GrafanaDatasourceFailure expr: rate(grafana_datasource_failure_total[5m]) > 0 for: 2m labels: severity: critical annotations: summary: "Grafana cannot reach one or more datasources" description: "Check NetworkPolicy for the target namespace." - Blue‑Green Promotion Script – Extend the promotion automation to patch the target namespace with the
allow-monitoring-ingressandallow-green-egress-to-monitoringpolicies before traffic cut‑over. - Documentation Checklist – Add a “NetworkPolicy verification” step to the runbook for any new environment rollout.
Related Topic Hub: Observability Troubleshooting Hub
FAQ
- Why does Grafana work with the blue namespace but fail after promoting green?
The blue namespace already had explicit NetworkPolicy rules allowing monitoring traffic. The green namespace was created with a default deny policy, so after promotion no rules existed to let Grafana reach its data sources.
- Can I use a single “allow-all” NetworkPolicy for monitoring instead of per‑namespace rules?
Yes, but it defeats the purpose of namespace isolation. A scoped policy that only permits traffic from the
monitoringnamespace to specific ports provides a balance between security and operability. - Do I need to modify the Ingress whitelist when promoting green?
No. The whitelist controls source IP ranges, not pod‑to‑pod traffic. The missing piece was the intra‑cluster NetworkPolicy, not the Ingress source range.
- How can I test NetworkPolicy changes without affecting production?
Deploy a temporary “probe” pod in the target namespace and use
curlorncto verify connectivity to the monitoring services before applying the policy cluster‑wide. - What if I use Calico’s GlobalNetworkPolicy instead of per‑namespace policies?
Global policies can be used, but you must still create explicit allow rules for the monitoring namespace. The same principle applies: deny‑by‑default policies require explicit permits for cross‑namespace traffic.