Nginx serving stale data after Kubernetes pod update

Problem – Nginx Serves Stale Data After a Kubernetes Pod Update

In an on‑premises data‑center, Nginx (or the Nginx Ingress Controller) is used as a reverse proxy for a micro‑services architecture. Service endpoints are discovered through the Kubernetes EndpointSlice API. After a rolling update or a pod crash, Nginx continues to forward traffic to the old pod IPs for several minutes, producing 5xx errors or returning stale responses.

Typical log excerpts:

2023/09/12 14:23:07 [error] 12#12: *245 upstream server temporarily disabled while connecting to upstream, client: 10.1.2.34, server: my-service, request: "GET /api/v1/resource HTTP/1.1", upstream: "http://10.244.1.12:8080/api/v1/resource", host: "api.example.com"
2023/09/12 14:23:12 [error] 12#12: *247 no live upstreams while connecting to upstream, client: 10.1.2.34, server: my-service, request: "GET /api/v1/resource HTTP/1.1", upstream: "http://my-service.namespace.svc.cluster.local:8080/", host: "api.example.com"
2023/09/12 14:23:15 [error] 12#12: *250 resolver timed out while resolving "my-service.namespace.svc.cluster.local"

These symptoms indicate that Nginx’s upstream list is out‑of‑sync with the current set of pods.

Root Cause – Why Nginx Keeps Using Outdated EndpointSlices

The underlying mechanisms involved are:

• Kubernetes EndpointSlice API: Provides a scalable representation of service endpoints. Updates are propagated to watchers based on the controller’s sync interval.
• Nginx dynamic upstream reconfiguration: Nginx Plus can reload upstreams without a full reload (see NGINX Plus documentation – Dynamic reconfiguration of upstream servers). The open‑source Nginx relies on DNS resolution and the resolver directive with a valid cache time.
• Ingress‑NGINX controller sync loop: The controller watches EndpointSlice objects and writes an Nginx configuration file. By default the sync interval is 30 seconds, but the controller caches the last known state and only updates the config when the EndpointSlice version changes (see Ingress‑NGINX – Syncing with EndpointSlices).

In the observed incidents the following assumptions failed:

1. DNS caching: Nginx was configured with a static DNS resolver and a long valid period (e.g., valid=60s). After a pod IP was removed, the DNS cache still returned the stale IP until the cache expired.
2. Controller sync latency: The Ingress‑NGINX controller’s sync interval (or the --watch-namespace filter) was too high, causing a delay of up to 5 minutes before the new EndpointSlice was reflected in the generated upstream block (see GitHub issue #8452).
3. Keep‑alive connections: Existing keep‑alive connections to a pod remained open after the pod termination, so Nginx continued to reuse those sockets until they timed out (financial services incident).
4. Health checks disabled: Without active health checks Nginx never detected that an upstream became unavailable, leading to “no live upstreams” errors after a blue‑green deployment (telecom operator incident).

Debug – Investigation and Diagnostic Steps

1. Verify the current EndpointSlice content

$ kubectl get endpointslice my-service -n namespace -o yaml
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: my-service-abcde
  namespace: namespace
addressType: IPv4
endpoints:
- addresses:
  - 10.244.1.12
  conditions:
    ready: true
- addresses:
  - 10.244.1.15
  conditions:
    ready: true
ports:
- name: http
  port: 8080
  protocol: TCP

Confirm that the terminated pod IP (e.g., 10.244.1.12) is no longer present.

2. Check Nginx’s upstream configuration

$ docker exec -it nginx cat /etc/nginx/conf.d/upstream_my-service.conf
upstream my-service {
    server 10.244.1.12:8080 max_fails=3 fail_timeout=30s;
    server 10.244.1.15:8080 max_fails=3 fail_timeout=30s;
}

If the stale IP is still listed, the controller has not regenerated the config.

3. Inspect the resolver cache (if DNS‑based upstreams are used)

$ curl -s http://127.0.0.1:8080/nginx_status | grep resolver
resolver 10.96.0.10 valid=60s;

Validate the valid duration; a long TTL will keep stale DNS answers.

4. Review controller logs for sync activity

$ kubectl logs -n ingress-nginx deploy/ingress-nginx-controller | grep EndpointSlice
2023/09/12 14:22:58 [info] sync: processing EndpointSlice "my-service-abcde"
2023/09/12 14:23:30 [info] sync: no changes detected for EndpointSlice "my-service-abcde"

If the “no changes detected” message appears after a pod termination, the controller missed the update.

5. Check keep‑alive socket state

$ ss -tnp | grep 10.244.1.12
ESTAB 0 0 10.1.2.34:443 10.244.1.12:8080 users:(("nginx",pid=12,fd=45))

Active sockets to the terminated pod indicate that Nginx is reusing stale connections.

Solution – Fixing Stale EndpointSlice Propagation

1. Reduce DNS cache TTL (if using DNS upstreams)

Update the resolver directive to a shorter valid period (e.g., 5 seconds) and enable resolver_timeout:

# Before
resolver 10.96.0.10 valid=60s;

# After
resolver 10.96.0.10 valid=5s;
resolver_timeout 2s;

This forces Nginx to re‑query the cluster DNS quickly after a pod IP change.

2. Enable Nginx Plus dynamic upstream reconfiguration (if licensed)

Replace static upstream blocks with the dynamic_resolver module:

# Before (static upstream)
upstream my-service {
    server 10.244.1.12:8080;
    server 10.244.1.15:8080;
}

# After (dynamic)
upstream my-service {
    zone my-service 64k;
    server my-service.namespace.svc.cluster.local:8080 resolve;
}

With resolve, Nginx automatically updates the upstream list when DNS answers change, matching the behavior described in the official Nginx Plus docs.

3. Accelerate Ingress‑NGINX controller sync

Adjust the controller’s --sync-period flag (default 30 s) to a lower value, e.g., 5 s, and enable the --watch-endpointslice flag if not already set:

# Deployment spec snippet
containers:
- name: controller
  args:
  - /nginx-ingress-controller
  - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
  - --election-id=ingress-controller-leader
  - --sync-period=5s
  - --watch-endpointslice=true

This reduces the window during which stale EndpointSlice data is cached (see GitHub issue #106927).

4. Enable active health checks

Add the proxy_next_upstream and proxy_connect_timeout directives, or enable Nginx Plus health checks:

# Example for open‑source Nginx
location / {
    proxy_pass http://my-service;
    proxy_next_upstream error timeout http_502 http_503 http_504;
    proxy_connect_timeout 2s;
    proxy_read_timeout 5s;
}

Health checks cause Nginx to drop connections to unresponsive pods, preventing “no live upstreams” logs.

5. Force connection termination on pod shutdown

Configure the pod’s terminationGracePeriodSeconds to a low value (e.g., 5 s) and add a pre‑stop hook that closes listening sockets, ensuring Nginx’s keep‑alive connections are closed promptly.

Verify – Confirming the Fix Works

1. EndpointSlice reflects new pods:

   
   $ kubectl get endpointslice my-service -n namespace -o jsonpath='{.endpoints[*].addresses}'
   10.244.1.15
   

2. Nginx upstream file is updated:

   
   $ cat /etc/nginx/conf.d/upstream_my-service.conf
   upstream my-service {
       server 10.244.1.15:8080;
   }
   

3. No stale DNS entries (if DNS‑based):

   
   $ dig @10.96.0.10 my-service.namespace.svc.cluster.local +short
   10.244.1.15
   

4. Successful request flow:

   
   $ curl -s -o /dev/null -w "%{http_code}" http://api.example.com/api/v1/resource
   200
   

5. Metrics show zero 5xx from upstream (Prometheus query):

   
   sum(rate(nginx_upstream_response_total{status=~"5.."}[1m]))
   0
   

Prevent – Operational Guardrails and Best Practices

• Set DNS cache TTL ≤ 10 s for service names used in upstreams.
• Use Nginx Plus dynamic reconfiguration when possible; it eliminates the need for full reloads.
• Configure Ingress‑NGINX sync period ≤ 5 s in environments with frequent rollouts.
• Enable active health checks (either Nginx Plus health_check or open‑source proxy_next_upstream) to drop dead backends quickly.
• Limit keep‑alive timeout on the upstream side (e.g., keepalive_timeout 30s;) to ensure stale sockets are reclaimed.
• Monitor EndpointSlice sync lag via a custom alert:

  
  max(kube_endpoint_slice_sync_timestamp - kube_endpoint_slice_last_observed_timestamp) > 30
  

• Test rolling updates in a staging cluster with the same resolver valid and sync settings before promotion.

FAQ – Common Follow‑Up Questions

1. Why does the issue appear only after a rolling update and not on initial deployment?
   During the initial deployment the DNS cache is empty, so Nginx resolves the correct IPs. A rolling update replaces pod IPs while Nginx still holds cached DNS answers or static upstream entries, exposing the stale‑data window.
2. Can I rely solely on the resolver directive without changing the controller sync interval?
   Yes, if you use DNS‑based upstreams with a short valid period (≤ 5 s) and enable resolve in the upstream block. However, the controller still needs to update the Nginx configuration for annotations or custom settings, so a reasonable sync period is still recommended.
3. How do I verify which IP Nginx actually used for a request?
   Enable the log_format upstream_debug with the $upstream_addr variable:

   
   log_format upstream_debug '$remote_addr - $remote_user [$time_local] "$request" '
                            'upstream: $upstream_addr status: $status';
   access_log /var/log/nginx/upstream_debug.log upstream_debug;
   

   The log will show the exact IP address Nginx connected to.

4. What impact does disabling keep‑alive have?
   Disabling keep‑alive forces Nginx to open a new TCP connection for each request, which eliminates the risk of reusing a closed pod socket but increases latency and CPU usage. Prefer reducing keepalive_timeout instead of disabling it entirely.
5. Is there a way to force Nginx to reload automatically when an EndpointSlice changes?
   The Ingress‑NGINX controller already triggers a reload when it detects a change. If you need immediate reloads, you can enable the --watch-endpointslice flag and set --sync-period to a low value, or use a sidecar that watches EndpointSlice events and sends SIGHUP to the Nginx master process.

Related Topic Hub: Distributed Systems Troubleshooting Hub