a group of blue boxes

Kubernetes NetworkPolicy blocking model-serving pods from Redis cache

by

Problem – Model‑Serving Pods Cannot Reach Redis Cache

In a production Kubernetes cluster the AI inference workloads run in the model-serving namespace. The Redis cache that stores model artifacts lives in the data-services namespace and is exposed via the Service redis-data-services (port 6379).

After a recent deployment of a default‑deny NetworkPolicy in model-serving, inference requests began timing out after ~30 seconds. The pod logs contain errors such as:


dial tcp redis-data-services.data-services.svc.cluster.local:6379: i/o timeout
dial tcp 10.96.12.34:6379: connect: connection refused

CoreDNS also reports DNS resolution failures for the Redis service:


lookup redis-data-services.data-services.svc.cluster.local failed: context deadline exceeded

These symptoms indicate that network traffic from the model‑serving pods to the Redis endpoint is being blocked.

Root Cause – Egress Rules Missing for Cross‑Namespace Redis Traffic

Kubernetes NetworkPolicy defaults to “allow all” unless a policy selects a pod. Once a policy exists, any traffic not explicitly allowed is denied (default deny). The newly applied policy in model-serving only permitted HTTP egress (ports 80/443) and omitted a rule for Redis (port 6379) and for the target namespace.

According to the NetworkPolicy documentation on namespaces, egress to a different namespace requires a namespaceSelector in the egress rule. The missing selector caused the CNI (Calico/Canal) to drop packets, as seen in Calico logs:


Dropped packet from pod model-serving-abcde to 10.0.0.0/24:6379: policy denied (egress)

Community reports (e.g., GitHub issue kubernetes/kubernetes#115331 and Stack Overflow 73123456) describe the same pattern: a default‑deny policy that allows only HTTP inadvertently blocks Redis traffic.

Debug – Step‑by‑Step Investigation

  1. Confirm DNS resolution from a model‑serving pod: 
    
kubectl exec -n model-serving $(kubectl get pod -n model-serving -l app=inference -o jsonpath='{.items[0].metadata.name}') -- nslookup redis-data-services.data-services.svc.cluster.local

    Expected output shows the service IP; if it times out, add a DNS‑allow rule (see below).

  2. Test TCP connectivity using nc or curl: 
    
kubectl exec -n model-serving $(kubectl get pod -n model-serving -l app=inference -o jsonpath='{.items[0].metadata.name}') -- nc -vz redis-data-services.data-services.svc.cluster.local 6379

    Result: Connection timed out confirms egress block.

  3. Inspect existing NetworkPolicies in the namespace: 
    
kubectl get networkpolicy -n model-serving -o yaml

    Typical offending policy:

    
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-http-egress
  namespace: model-serving
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
  4. Check CNI logs for denied packets: 
    
kubectl logs -n kube-system -l k8s-app=calico-node --tail=100 | grep "policy denied"

    Sample line:

    
Dropped packet from pod model-serving-xyz to 10.96.12.34:6379: policy denied (egress)

Solution – Add an Egress Rule for Redis Across Namespaces

Two approaches are common:

  • Extend the existing policy with an additional egress rule that selects the data-services namespace and port 6379.
  • Create a dedicated policy that explicitly allows Redis traffic, keeping the HTTP policy unchanged.

Before – Existing Policy (simplified)


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-http-egress
  namespace: model-serving
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443

After – Updated Policy Including Redis


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-http-and-redis-egress
  namespace: model-serving
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  # HTTP/HTTPS to anywhere
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
  # Redis in data-services namespace
  - to:
    - namespaceSelector:
        matchLabels:
          name: data-services
    ports:
    - protocol: TCP
      port: 6379

Key points:

  • namespaceSelector matches the target namespace via a label (ensure the namespace has name=data-services label; add it if missing).
  • The rule is additive; it does not affect existing HTTP allowances.

If the cluster uses Calico, you may need to add a selector on the namespace:


kubectl label namespace data-services name=data-services --overwrite

Verification – Confirm Connectivity Restored

  1. Re‑apply the policy: 
    
kubectl apply -f allow-http-and-redis-egress.yaml
  2. Re‑run the TCP test from a model‑serving pod: 
    
kubectl exec -n model-serving $(kubectl get pod -n model-serving -l app=inference -o jsonpath='{.items[0].metadata.name}') -- nc -vz redis-data-services.data-services.svc.cluster.local 6379

    Expected output:

    
Connection to redis-data-services.data-services.svc.cluster.local 6379 port [tcp/6379] succeeded!
  3. Check application logs for the disappearance of timeout errors.
  4. Validate that DNS still resolves (nslookup) and that other egress traffic (HTTP) remains functional.

Prevention – Guardrails and Best Practices

  • Policy Templates with Explicit Cross‑Namespace Rules: Maintain a reusable YAML snippet that includes egress to known service namespaces (e.g., data-services for Redis, monitoring for Prometheus).
  • Namespace Labelling Convention: Enforce a label such as name=<namespace> on every namespace; this simplifies namespaceSelector expressions.
  • Automated Policy Validation: Integrate a CI check (e.g., using kube-score or conftest) that fails if a policy selects pods without an egress rule for required service ports.
  • Observability: Add alerts on networkpolicy_denied_egress_total (Prometheus metric exposed by most CNI plugins) for critical ports (6379) and namespaces.
  • Incremental Rollout: Apply new policies with kubectl apply --dry-run=client and monitor kubectl get events for any NetworkPolicy denial messages before full deployment.

FAQ – Common Follow‑Up Questions

  1. Why does DNS resolution succeed but the TCP connection still time out?
    Because NetworkPolicy only controls IP traffic; DNS queries are allowed by default, but the subsequent egress to port 6379 is denied.
  2. Can I use an ipBlock instead of a namespaceSelector for Redis?
    Yes, but hard‑coding CIDRs couples the policy to the current cluster IP range. Using namespaceSelector is more portable and adapts to IP changes.
  3. My cluster uses Cilium – does the same YAML work?
    Cilium implements the standard NetworkPolicy API, so the rule above works. Ensure Cilium’s ciliumEndpointSlice is up‑to‑date; otherwise, you may need to add a CiliumNetworkPolicy with equivalent egress.
  4. How do I debug a similar issue for another service (e.g., PostgreSQL on port 5432)?
    Replicate the steps: verify DNS, test TCP with nc, inspect existing policies, and add an egress rule that selects the target namespace and port 5432.
  5. Is there a way to see which pods are currently blocked by a NetworkPolicy?
    Tools like kubectl netpol (from the kubectl-netpol plugin) or CNI‑specific utilities (e.g., calicoctl policy show) can list denied flows based on iptables or BPF logs.

Related Topic Hub: Distributed Systems Troubleshooting Hub