Nginx Webhook Timeout During Long GPU Training Jobs

Q: Investigation and Debugging Steps 1. Capture Nginx error logs

kubectl -n monitoring logs -l app=nginx-ingress-controller \ --tail=200 | grep "upstream timed out" Typical output:

Q: 2. Verify current timeout settings

kubectl -n ingress-nginx exec $(kubectl -n ingress-nginx get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[0].metadata.name}") -- \ cat /etc/nginx/nginx.conf | grep -E "proxy_(connect|send|read)_timeout" Typical output (default values):

Q: 3. Inspect the webhook request payload size

kubectl exec -it training-pod-0 -- \ cat /var/log/training/webhook_payload.json | wc -c If the size exceeds the default client_max_body_size (1 MiB), Nginx may buffer to disk, further delaying the response.

Q: 4. Perform a packet capture on the ingress node

sudo tcpdump -i eth0 -nn -s0 -w /tmp/webhook.pcap host observability.example.com and port 443 Analysis with tshark confirms that the TCP stream remains open for >300 s without any ACK from the upstream, matching the timeout window.

Q: 3. Disable request buffering for large webhook payloads

The configuration-snippet annotation above disables both proxy_buffering and proxy_request_buffering, ensuring the request body streams directly to the upstream without Nginx attempting to buffer it. This aligns with the solution described in the Stack Overflow thread “NGINX proxy_read_timeout not applied for long‑running POST”.

Q: Verification Functional Test

# Simulate a long‑running webhook from a training pod curl -X POST https://webhook.example.com/v1/metrics \ -H "Content-Type: application/json" \ --data @large_payload.json \ -o /dev/null -w "%{http_code}\n" --max-time 900 Expected output: 200 (or the success code defined by the observability service) even after >10 minutes of processing.

In this article

Table of Contents

Problem Description

During multi‑GPU training jobs running in a Kubernetes cluster, webhook callbacks that report progress to an external observability platform (e.g., Datadog, Prometheus remote‑write) start failing after a few minutes of computation. The Nginx ingress controller logs errors such as:

[error] 12345#0: *6789 upstream timed out (110: Connection timed out) while reading response header from upstream
504 Gateway Timeout: no live upstreams

Symptoms observed:

Missing health‑check alerts during long training epochs.
Periodic POST requests from training pods receive HTTP 504 responses.
Training pods continue to run, but the external monitoring service reports “webhook timeout”.
Increasing the training batch size or GPU utilization correlates with earlier timeouts.

Root Cause Analysis

The Nginx ingress controller acts as a reverse proxy between the training pods and the external webhook endpoint. By default, the following timeout directives apply:

proxy_connect_timeout: 60 s (time to establish upstream connection).
proxy_send_timeout: 60 s (time to transmit request body to upstream).
proxy_read_timeout: 60 s (time to wait for a response from upstream) – see NGINX Docs – Proxy Timeout Settings.
keepalive_timeout: 75 s (idle keep‑alive connection timeout) – see NGINX Docs – HTTP Keepalive.

Long‑running GPU training loops block the HTTP/2 stream or keep the request body open for the duration of an epoch (often >5 minutes). Nginx therefore reaches proxy_read_timeout before the upstream service sends a response, triggering the “upstream timed out” error.

Additional factors that exacerbate the issue:

Insufficient client_header_buffer_size and large_client_header_buffers when training pods send large JSON payloads (see NGINX Docs – Large Client Header Buffers).
Kubernetes ingress annotations that override Nginx defaults but are left at the 30 s “nginx.ingress.kubernetes.io/proxy-read-timeout” value, as discussed in GitHub issue #6795.
GPU‑intensive jobs can starve the event loop, causing delayed flush of the request body, similar to the behavior reported in pytorch/pytorch#82457.

Investigation and Debugging Steps

1. Capture Nginx error logs

kubectl -n monitoring logs -l app=nginx-ingress-controller \
  --tail=200 | grep "upstream timed out"

Typical output:

[error] 12345#0: *6789 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.12.34.56, server: webhook.example.com, request: "POST /v1/metrics HTTP/1.1", upstream: "http://observability-svc:8080/v1/metrics"

2. Verify current timeout settings

kubectl -n ingress-nginx exec $(kubectl -n ingress-nginx get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[0].metadata.name}") -- \
  cat /etc/nginx/nginx.conf | grep -E "proxy_(connect|send|read)_timeout"

Typical output (default values):

proxy_connect_timeout 60s;
proxy_send_timeout    60s;
proxy_read_timeout    60s;

3. Inspect the webhook request payload size

kubectl exec -it training-pod-0 -- \
  cat /var/log/training/webhook_payload.json | wc -c

If the size exceeds the default client_max_body_size (1 MiB), Nginx may buffer to disk, further delaying the response.

4. Perform a packet capture on the ingress node

sudo tcpdump -i eth0 -nn -s0 -w /tmp/webhook.pcap host observability.example.com and port 443

Analysis with tshark confirms that the TCP stream remains open for >300 s without any ACK from the upstream, matching the timeout window.

Solution

The fix consists of three coordinated changes:

1. Increase proxy timeouts via ingress annotations

Apply annotations to the Ingress resource handling the webhook endpoint:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webhook-ingress
  namespace: monitoring
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_buffering off;
      proxy_request_buffering off;

2. Adjust the Nginx ConfigMap for global defaults (optional but recommended)

Before:

data:
  proxy-connect-timeout: "60s"
  proxy-send-timeout:    "60s"
  proxy-read-timeout:    "60s"
  keepalive-timeout:     "75s"

After:

data:
  proxy-connect-timeout: "300s"
  proxy-send-timeout:    "300s"
  proxy-read-timeout:    "600s"
  keepalive-timeout:     "120s"
  client-header-buffer-size: "16k"
  large-client-header-buffers: "4 32k"

Apply the ConfigMap and reload the ingress controller:

kubectl apply -f nginx-configmap.yaml
kubectl -n ingress-nginx rollout restart deployment/ingress-nginx-controller

3. Disable request buffering for large webhook payloads

The configuration-snippet annotation above disables both proxy_buffering and proxy_request_buffering, ensuring the request body streams directly to the upstream without Nginx attempting to buffer it. This aligns with the solution described in the Stack Overflow thread “NGINX proxy_read_timeout not applied for long‑running POST”.

Verification

Functional Test

# Simulate a long‑running webhook from a training pod
curl -X POST https://webhook.example.com/v1/metrics \
  -H "Content-Type: application/json" \
  --data @large_payload.json \
  -o /dev/null -w "%{http_code}\n" --max-time 900

Expected output: 200 (or the success code defined by the observability service) even after >10 minutes of processing.

Log Confirmation

kubectl -n monitoring logs -l app=nginx-ingress-controller \
  | grep "upstream timed out" | wc -l

Result should be 0 for the period after the configuration change.

Metrics

Check Prometheus metrics for Nginx:

nginx_ingress_controller_requests{status="504"} == 0

And verify that the webhook latency metric reported by the external observability platform no longer spikes during training epochs.

Prevention and Best Practices

Set generous timeouts at ingress creation. Use annotations that reflect the maximum expected training epoch duration.
Disable buffering for streaming or large JSON payloads. This avoids Nginx’s internal 60 s read buffer limit.
Monitor Nginx timeout metrics. Alert on nginx_ingress_controller_upstream_response_time_seconds exceeding a threshold.
Version lock Nginx. Ensure you run a version >= 1.21.0 where proxy_read_timeout is reliably applied to HTTP/2 streams (see the GitHub issue kubernetes/kubernetes#106123).
Test with realistic payload sizes. Use a CI job that sends a payload comparable to production webhook size and verifies that no timeout occurs.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does increasing only proxy_read_timeout sometimes not fix the issue? The request body may still be buffered, causing the upstream connection to stall. Disabling proxy_request_buffering ensures the body streams directly, and proxy_send_timeout must also be increased to cover the transmission period.
Can HTTP/2 be the cause of the timeout? Yes. Nginx’s default timeout handling for HTTP/2 streams can be stricter than HTTP/1.1. Setting keepalive_timeout higher and disabling buffering mitigates the problem.
Do I need to adjust the Kubernetes service timeout? No. The service object forwards traffic unchanged; all timeout handling resides in the ingress controller.
What is the impact of disabling buffering on memory usage? Each connection streams data directly, reducing memory pressure on the ingress pod but increasing CPU usage for TLS encryption. Ensure the ingress node has sufficient CPU resources.
How can I verify which timeout triggered the failure? Nginx error logs include the specific directive in the message. For example, “proxy_read_timeout” appears when the read timeout expires, while “client timed out (110: Connection timed out) while waiting for request body” indicates a client‑side timeout.