photo of computer cables

Canary Elasticsearch pods timing out after Calico network policy change

by

Problem – Canary Elasticsearch Pods Time Out After Calico NetworkPolicy Change

During a canary rollout of a new Elasticsearch version, pods labeled app=es-canary were unable to reach the primary elasticsearch-master service. All HTTP requests to port 9200 and transport requests to port 9300 resulted in connection‑timeout errors, causing indexing and query failures and halting the rollout.

java.net.ConnectException: Connection timed out (connection timeout=30s)
org.elasticsearch.transport.TransportException: [es-canary-0] failed to connect to node [es-master-0]
no alive nodes found in your cluster

The failure manifested as repeated health‑check failures in the canary deployment pipeline and an alert from the SRE team indicating “Elasticsearch canary pods unreachable”.

Root Cause – Calico Policy Over‑Restricts Egress for Canary Pods

Calico NetworkPolicy was updated to enforce a stricter egress rule that only allowed traffic to pods with label role=backend. The policy was written as:


apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: backend-egress
  namespace: production
spec:
  selector: all()
  egress:
  - action: Allow
    destination:
      selector: role == 'backend'

Because the primary Elasticsearch pods carry the label role=master (or no role label at all), the egress rule did not match, and Calico’s default‑deny behavior blocked all outbound traffic from the canary pods. The Calico NetworkPolicy API reference states that an empty selector matches no pods, which explains why the policy effectively isolated the canary pods.

Additionally, the Elasticsearch network settings documentation requires both HTTP (9200) and transport (9300) ports to be reachable from any node that participates in the cluster. The policy change violated this requirement, leading to the observed timeouts.

Debug – Investigation Steps

1. Verify Symptom with Logs and Metrics


kubectl logs -n production es-canary-0
2023-11-12T08:45:12.345Z WARN  [es-canary-0] failed to connect to node [es-master-0] (org.elasticsearch.transport.TransportException)
java.net.ConnectException: Connection timed out (connection timeout=30s)

Metrics from Prometheus showed a spike in es_http_client_requests_total{status="timeout"} for the canary deployment.

2. Confirm NetworkPolicy Effect


kubectl get networkpolicy -n production -o yaml backend-egress

Calico logs (visible via journalctl -u calico-node) contained entries such as:

DENIED egress from pod production/es-canary-0 to 10.96.0.10:9200

3. Test Connectivity from Canary Pod


kubectl exec -n production es-canary-0 -- curl -s -o /dev/null -w "%{http_code}" http://elasticsearch-master:9200

Result: 000 (no response). Using nc -vz elasticsearch-master 9200 also timed out.

4. Inspect Service & Endpoints


kubectl get svc -n production elasticsearch-master -o yaml
kubectl get endpoints -n production elasticsearch-master -o yaml

Endpoints resolved correctly (IP 10.96.0.10), confirming DNS and service configuration were not at fault.

5. Review Policy Scope and Namespace Selectors

GitHub issue projectcalico/calico#4932 describes a similar failure when a namespaceSelector unintentionally excluded the monitoring namespace, leading to DNS resolution failures. In our case, the policy lacked a namespaceSelector but the selector: all() applied cluster‑wide, affecting the canary pods.

Solution – Adjust Calico NetworkPolicy to Permit Elasticsearch Traffic

Before (Blocking Policy)


apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: backend-egress
  namespace: production
spec:
  selector: all()
  egress:
  - action: Allow
    destination:
      selector: role == 'backend'

After (Allow Elasticsearch HTTP & Transport)


apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: es-canary-egress
  namespace: production
spec:
  selector: app == 'es-canary'
  egress:
  # Allow traffic to any pod that is part of the Elasticsearch cluster
  - action: Allow
    destination:
      selector: app == 'elasticsearch'   # matches master, data, ingest pods
      ports:
      - 9200
      - 9300
  # Preserve existing backend allowance
  - action: Allow
    destination:
      selector: role == 'backend'

The updated policy explicitly matches the canary pods (app=es-canary) and permits egress to any pod labeled app=elasticsearch on the required ports. This aligns with the ECK security guide, which recommends allowing both HTTP and transport ports for intra‑cluster communication.

Apply the Fixed Policy


kubectl apply -f es-canary-egress.yaml

After applying, Calico logs no longer show DENIED entries for the canary pod.

Verify – Confirm Connectivity and Functional Health

1. Re‑run Connectivity Checks


kubectl exec -n production es-canary-0 -- curl -s -o /dev/null -w "%{http_code}" http://elasticsearch-master:9200

Expected output: 200


kubectl exec -n production es-canary-0 -- nc -vz elasticsearch-master 9300
Connection to elasticsearch-master 9300 port [tcp/*] succeeded!

2. Observe Elasticsearch Client Logs


2023-11-12T08:52:01.112Z INFO  [es-canary-0] successfully connected to node [es-master-0] (org.elasticsearch.transport.TransportService)

3. Check Health‑Check Metrics

Prometheus query es_cluster_health_status{cluster="canary"} returns green for the canary deployment.

4. Verify Rollout Completion

The CI/CD pipeline proceeds past the canary stage, and the new Elasticsearch version is promoted to production.

Prevent – Best Practices for NetworkPolicy Management in Elasticsearch Deployments

  • Label Consistency: Ensure all Elasticsearch pods share a common label (e.g., app=elasticsearch) that can be referenced by policies.
  • Policy Scope: Use selector that targets only the intended pods (e.g., app == 'es-canary') instead of all() which applies globally.
  • Port Whitelisting: Explicitly list both HTTP (9200) and transport (9300) ports in egress rules; omit them and you’ll encounter silent timeouts.
  • Policy Ordering: When multiple policies exist, remember that Calico evaluates them cumulatively; a default‑deny policy can override a later allow rule if selectors don’t match.
  • Testing Before Promotion: Deploy a temporary pod that mimics the canary label and run connectivity checks against the Elasticsearch service before applying new policies.
  • Observability: Enable Calico flow logs (or audit logs) and monitor for DENIED egress messages; correlate them with application timeouts.

FAQ – Common Follow‑Up Questions

  1. Why did the canary pods still resolve the elasticsearch-master DNS name but fail to connect?
    Because DNS resolution succeeds (service exists) but the egress traffic is blocked by Calico, resulting in a “no alive nodes found” error after the client times out.
  2. Do I need to allow traffic to the Elasticsearch service IP or to individual pod IPs?
    Both are covered by the same rule; Calico evaluates the destination selector against pod labels, which matches the pod IPs behind the service. Allowing the service IP alone would not bypass the pod‑level deny.
  3. Can I use a namespaceSelector to simplify the policy?
    Yes, but ensure the selector includes the namespace where the Elasticsearch pods run. A missing namespace (as seen in the fintech incident) will inadvertently block traffic.
  4. What ports must be opened for an Elasticsearch cluster in Kubernetes?
    At minimum, HTTP (9200) for REST calls and transport (9300) for inter‑node communication. If TLS is enabled, also allow the corresponding TLS ports (default same numbers).
  5. How do I debug a similar issue if the policy uses podSelector: {}?
    An empty selector matches no pods, effectively isolating the source. Replace it with a concrete label selector or remove the rule if you intend “allow all”.

Related Topic Hub: Data Infrastructure Troubleshooting Hub