Nginx proxy pass fails after Kubernetes deployment

Problem Description

After a rolling update of a Deployment in a Kubernetes cluster, the NGINX Ingress controller started returning 502 Bad Gateway and 504 Gateway Timeout for the affected host. The HTTP response body was empty, and the NGINX error log contained entries such as:

2024/07/31 12:45:12 [error] 12#12: *12345 connect() failed (111: Connection refused) while connecting to upstream, client: 10.2.1.45, server: example.com, request: "GET /api/v1/resource HTTP/1.1", upstream: "http://my-service.default.svc.cluster.local:8080/api/v1/resource", host: "example.com"
2024/07/31 12:45:12 [error] 12#12: *12346 host not found in upstream "my-service.default.svc.cluster.local", client: 10.2.1.45, server: example.com, request: "GET /api/v1/resource HTTP/1.1", upstream: "http://my-service.default.svc.cluster.local:8080/api/v1/resource", host: "example.com"
2024/07/31 12:45:13 [error] 12#12: *12347 upstream prematurely closed connection while reading response header from upstream, client: 10.2.1.45, server: example.com, request: "GET /api/v1/resource HTTP/1.1", upstream: "http://my-service.default.svc.cluster.local:8080/api/v1/resource", host: "example.com"

Impact:

• External clients receive 502/504 errors.
• Service-level agreement (SLA) violations.
• Increased alert noise on nginx_ingress_controller_upstream_response_time_seconds and nginx_ingress_controller_requests metrics.

Root Cause Analysis

The failure stemmed from a combination of three common Kubernetes‑NGINX integration pitfalls:

1. Service selector mismatch after Deployment change – The new Deployment used a different app label (app: my-service-v2) while the existing Service still selected app: my-service. Consequently the Service resolved to an empty endpoint list, producing the log “no upstreams are available” described in the NGINX upstream troubleshooting guide.
2. Kubernetes DNS outage – A coredns pod crash coincided with the rollout, causing temporary name resolution failures for my-service.default.svc.cluster.local. This manifested as “host not found in upstream” (see the GitHub issue #92758).
3. Trailing slash in proxy_pass URL – A ConfigMap change introduced a trailing slash (proxy_pass http://my-service.default.svc.cluster.local/;). According to the NGINX proxy_pass documentation, this causes NGINX to replace the entire request URI, leading to mismatched paths and “upstream sent unexpected response while reading response header”.

Because NGINX resolves upstream hosts at request time, any of the above conditions results in the observed 502/504 errors.

Investigation and Debugging

Step‑by‑step diagnostics that reproduced the issue:

1. Inspect NGINX error logs for the failing request.
2. Verify Service endpoints:

kubectl get svc my-service -n default -o jsonpath='{.spec.selector}'
kubectl get endpoints my-service -n default

Output showed an empty endpoint list:

NAME         ENDPOINTS   AGE
my-service   <none>      12m

Check Deployment pod labels:

kubectl get deploy my-service -n default -o yaml | grep -A3 selector
kubectl get pods -l app=my-service -n default

The new pods carried app: my-service-v2, confirming the selector mismatch (similar to the incident “Service selector was mismatched, causing NGINX to resolve the Service to an empty endpoint list”).

Validate DNS resolution from the Ingress pod:

kubectl exec -n ingress-nginx $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -n ingress-nginx -o name | head -n1) -- nslookup my-service.default.svc.cluster.local

The response was “server can't find my-service.default.svc.cluster.local: NXDOMAIN”, matching the “host not found in upstream” error.

Inspect the NGINX ConfigMap for proxy_pass syntax:

kubectl -n ingress-nginx get configmap nginx-configuration -o yaml | grep proxy_pass

Found:

proxy_pass: "http://my-service.default.svc.cluster.local/";

The trailing slash was the cause of the path‑rewriting issue.

Resolution

The fix required three coordinated changes:

1. Align Service selector with Deployment labels

Before (service.yaml):

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: default
spec:
  selector:
    app: my-service          # <-- outdated selector
  ports:
  - protocol: TCP
    port: 8080
    targetPort: http

After (updated selector):

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: default
spec:
  selector:
    app: my-service-v2      # <-- matches new pods
  ports:
  - protocol: TCP
    port: 8080
    targetPort: http

2. Restore CoreDNS health

Restart the failing CoreDNS pod and ensure the deployment has the desired replica count:

kubectl -n kube-system rollout restart deployment coredns
kubectl -n kube-system get pods -l k8s-app=kube-dns

After the rollout, nslookup from the Ingress pod succeeded.

3. Remove trailing slash from proxy_pass

Before (nginx‑configmap.yaml):

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  proxy-pass: "http://my-service.default.svc.cluster.local/";

After (no trailing slash, per NGINX Docs – Configuring NGINX as a reverse proxy):

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  proxy-pass: "http://my-service.default.svc.cluster.local";

Apply the updated ConfigMap and reload the Ingress controller:

kubectl apply -f nginx-configmap.yaml
kubectl -n ingress-nginx rollout restart deployment ingress-nginx-controller

Verification

Validate that the upstream is now reachable and the errors disappear:

1. Check Service endpoints:

kubectl get endpoints my-service -n default

Expected output:

NAME         ENDPOINTS                         AGE
my-service   10.244.1.12:8080,10.244.2.8:8080   2m

Confirm DNS resolution from the Ingress pod:

kubectl exec -n ingress-nginx $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -n ingress-nginx -o name | head -n1) -- nslookup my-service.default.svc.cluster.local

Should return the IPs shown above.

Issue a curl request through the Ingress:

curl -I https://example.com/api/v1/resource

Expected HTTP status: 200 OK and no 502/504.

Observe NGINX logs for absence of upstream errors:

kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=20 | grep -i "upstream"

No lines containing “connect() failed”, “host not found”, or “upstream prematurely closed”.

Operational Experience

• Misleading symptom: The initial 502 error suggested the backend was down, but the root cause was a label mismatch – a classic “service selector drift” after a rollout.
• DNS timing: A brief CoreDNS outage coincided with the deployment, amplifying the problem. In production, a rolling update should be coordinated with a health check on the kube-dns deployment.
• ConfigMap pitfalls: Adding a trailing slash in proxy_pass silently altered request URIs, leading to backend 404s that appeared as upstream errors. Always validate ConfigMap syntax with nginx -t inside a temporary pod.
• Readiness probes: Pods that failed their readiness probe were excluded from the Service endpoints, which prevented NGINX from routing to them. Ensure probes reflect actual service readiness.

Best Practices and Prevention

• Use a single source of truth for label values (e.g., Helm chart .Values.labels.app) to avoid selector drift.
• Enable Service topology aware routing (topologyKeys) to reduce cross‑zone latency and limit the impact of a single pod failure.
• Monitor coredns health via kube-system/coredns pod restarts and expose coredns_up metrics to alert on DNS outages.
• Validate NGINX configuration changes with a pre‑deployment test pod:

kubectl run nginx-test --image=nginx:stable-alpine --restart=Never --command -- sh -c "nginx -t && cat /etc/nginx/nginx.conf"

• Configure Ingress health checks using the nginx.ingress.kubernetes.io/proxy-connect-timeout and nginx.ingress.kubernetes.io/proxy-read-timeout annotations to surface upstream timeouts early.
• Leverage kubectl wait --for=condition=available on Services before triggering a rollout that updates the Ingress.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does NGINX return 502 only after a Deployment rollout?
   Because the new pods may carry different labels, causing the Service selector to match zero endpoints. NGINX then has no upstream to route to, producing “no upstreams are available”.
2. How can I tell if a DNS issue is causing “host not found in upstream”?
   Run an nslookup or dig from the Ingress pod for the Service name. If it fails, check CoreDNS pod status and logs.
3. Does a trailing slash in proxy_pass always cause path mismatches?
   Yes. According to the NGINX proxy_pass documentation, a trailing slash causes NGINX to replace the full request URI, which can lead to 404s or “upstream sent unexpected response”. Remove the slash unless intentional path rewriting is required.
4. What alert thresholds should I set for upstream errors?
   Alert on a spike of nginx_ingress_controller_upstream_response_time_seconds > 5s or a rate of nginx_ingress_controller_requests{status=5xx} exceeding 1% of total traffic for 5 minutes.
5. Can I use a headless Service with NGINX Ingress?
   Yes, but you must set serviceName to the headless Service and ensure the Ingress controller’s resolve annotation is enabled (nginx.ingress.kubernetes.io/upstream-vhost) so that DNS SRV records are resolved per request.