Kubernetes NetworkPolicy blocking API gateway inbound traffic after policy update

Problem

The API gateway service (gateway-prod) suddenly stopped receiving inbound HTTP requests after a routine NetworkPolicy update. The symptom manifested as 5xx errors from downstream services and connection timeouts observed by clients.

Observed logs

2026-10-05T14:32:10.123Z kubelet[1234]: E0905 14:32:10.123456    1234 network_plugin.go:215] Failed to program iptables rules: iptables-restore v1.8.4: line 23: Invalid argument
2026-10-05T14:32:10.124Z kube-proxy[5678]: I0905 14:32:10.124567    5678 proxy.go:123] Service "gateway-prod" has no endpoints
2026-10-05T14:32:12.001Z calico-node[2345]: 2026-10-05 14:32:12.001 [ERR] Policy "allow-gateway-ingress" denied traffic from pod "frontend-7c9d9f5b5c-9k2lm" to service "gateway-prod"

Impact:

• Client requests time out (average latency ↑ from 30 ms to >30 s).
• Downstream micro‑services report “connection refused”.
• Auto‑scaling metrics for the gateway show zero traffic, causing scale‑down events.

Root Cause

The updated NetworkPolicy allow-gateway-ingress unintentionally restricted both inbound traffic to the gateway pods and outbound traffic from the gateway to other services. The policy was intended to allow traffic only from the frontend namespace, but the selector logic used podSelector: {} in the ingress rule, which matches *all* pods in the namespace, and the egress rule omitted the required to clause, resulting in a default deny for all egress.

Key misconfiguration details:

• Ingress rule used namespaceSelector: {matchLabels: {name: frontend}} with an empty podSelector, causing the policy to apply to *every* pod in the frontend namespace, including the gateway itself, which then blocked external traffic that does not originate from a pod (e.g., LoadBalancer IP).
• Egress rule was defined without a to selector, so the policy fell back to “deny all egress” for the gateway pods, preventing them from reaching downstream services (databases, auth services, etc.).
• The policy was applied with policyTypes: [Ingress, Egress] but the intention was to control only ingress.

Debug

Step‑by‑step investigation that led to the root cause:

1. Identify affected pods

kubectl get pods -n prod -l app=gateway-prod -o wide

All gateway pods reported Ready status but had IPTABLES drops in the CNI logs.

2. Inspect the NetworkPolicy

kubectl get networkpolicy allow-gateway-ingress -n prod -o yaml

Output (truncated):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-ingress
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: gateway-prod
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
      podSelector: {}
  egress: []

The empty podSelector under from matches all pods in frontend, and the empty egress list translates to “deny all egress”.

3. Check CNI logs for drops

kubectl logs -n kube-system -l k8s-app=calico-node -c calico-node --tail=100 | grep deny

Sample log entry:

2026-10-05 14:32:12.001 [ERR] Policy "allow-gateway-ingress" denied traffic from pod "frontend-7c9d9f5b5c-9k2lm" (10.244.2.5) to pod "gateway-prod-6d9f7c8d9b-2xv9k" (10.244.3.12) on port 80

4. Validate iptables rules generated by the policy

kubectl exec -n prod -c calico-node $(kubectl get pods -n kube-system -l k8s-app=calico-node -o name | head -n1) -- iptables -S | grep gateway-prod

Result showed a -A INPUT -s 10.244.2.0/24 -d 10.244.3.12 -j DROP rule.

Solution

Rewrite the NetworkPolicy to explicitly allow inbound traffic from the frontend namespace on the required ports and to permit egress to all services needed by the gateway.

Before (faulty policy)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-ingress
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: gateway-prod
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
      podSelector: {}
  egress: []

After (corrected policy)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-ingress
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: gateway-prod
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
      podSelector:
        matchLabels:
          role: client
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 5432   # example DB port
    - protocol: TCP
      port: 6379   # example Redis port

Key changes:

• Added explicit podSelector under from to match only client pods (e.g., role: client).
• Specified the required ports for ingress.
• Introduced an egress rule that permits traffic to any namespace (namespaceSelector: {}) on the ports the gateway needs.
• Retained policyTypes: [Ingress, Egress] to keep egress control explicit.

Apply the corrected policy:

kubectl apply -f corrected-gateway-np.yaml

Verify

After applying the new policy, perform the following checks:

1. Confirm NetworkPolicy is active

kubectl get networkpolicy allow-gateway-ingress -n prod -o wide

Test inbound connectivity

curl -s -o /dev/null -w "%{http_code}" http://gateway-prod.prod.svc.cluster.local/healthz
# Expected output: 200

Test egress to downstream services

kubectl exec -n prod -c gateway-prod $(kubectl get pod -n prod -l app=gateway-prod -o jsonpath="{.items[0].metadata.name}") -- curl -s http://auth-service.prod.svc.cluster.local/ping
# Expected output: pong

Inspect iptables for drops

kubectl exec -n kube-system -l k8s-app=calico-node -c calico-node -- iptables -S | grep DROP | grep gateway-prod
# Should return no lines related to the gateway

Monitor metrics

Verify that request latency returns to baseline (<30 ms) and that the gateway’s request count rises in Prometheus.

Prevent

Adopt the following guardrails to avoid similar regressions:

• Policy Review Checklist
  

  Item	Verification
  Ingress from selectors are specific	Confirm podSelector is not empty unless intentional
  Egress rules defined when policyTypes includes Egress	Ensure at least one to clause or explicitly allow all
  Ports listed match service requirements	Cross‑check with Service and Deployment specs

• Automated CI linting – Use kube‑val or conftest policies to reject NetworkPolicies with empty selectors.
• Canary rollout – Apply new NetworkPolicy to a staging namespace first; run smoke tests before promoting to production.
• Observability – Enable Calico’s policyLog level and create alerts on “policy denied” log spikes.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why did the gateway lose inbound traffic even though the policy only targeted the frontend namespace?
   The empty podSelector under from matched *all* pods in the frontend namespace, including the LoadBalancer IP that terminates outside the cluster. Since the policy applied to the gateway pods, any traffic not matching the selector (e.g., external IP) was dropped.
2. Can I use namespaceSelector: {} to allow all namespaces for egress?
   Yes. An empty namespaceSelector matches every namespace. Pair it with a podSelector or specific ports to avoid a blanket “allow all” that defeats security.
3. How do I debug which NetworkPolicy is denying traffic when multiple policies exist?
   Enable the CNI’s policy logging (e.g., calicoctl node run --policy-log-level=debug) and grep for “denied”. The log includes the policy name, source pod, destination pod, and port, allowing you to pinpoint the offending rule.
4. Is it safe to omit the egress section if I only need to control ingress?
   If policyTypes includes Egress, an empty egress array defaults to “deny all egress”. Either remove Egress from policyTypes or explicitly allow required egress destinations.
5. What monitoring alerts should I set for NetworkPolicy failures?
   Create a Prometheus alert on the calico_policy_denied_total metric (or equivalent for your CNI) with a threshold of >0 over a 5‑minute window. Pair it with a log‑based alert on “policy denied” messages.