502 Bad Gateway audio processing failure after Nginx proxy to inference service

Problem

During model evaluation, client applications send audio files to the inference endpoint. When the request passes through the Nginx reverse proxy, the service returns 502 Bad Gateway and the audio payload received by the inference service is corrupted (truncated or contains null bytes). The symptom appears only when Nginx is in the request path; direct calls to the inference service succeed.

Observed Symptoms

• HTTP 502 response from Nginx:

2026/09/24 12:34:56 [error] 12#12: *12345 upstream sent unexpected EOF while reading response header from upstream, client: 10.2.3.45, server: inference.example.com, request: "POST /v1/evaluate HTTP/1.1", upstream: "http://127.0.0.1:8000/v1/evaluate", host: "api.example.com"
2026/09/24 12:34:56 [error] 12#12: *12345 upstream prematurely closed connection while reading response header from upstream, client: 10.2.3.45, server: inference.example.com, request: "POST /v1/evaluate HTTP/1.1", upstream: "http://127.0.0.1:8000/v1/evaluate", host: "api.example.com"

• Inference service logs show malformed audio data:

2026-09-24 12:34:55,123 ERROR audio_processor - Failed to decode audio: unexpected EOF while reading 44100 bytes (got 31234)
Traceback (most recent call last):
  File "/app/audio_processor.py", line 87, in decode
    audio = np.frombuffer(raw, dtype=np.int16)
  File ".../numpy/core/_methods.py", line 45, in frombuffer
    raise ValueError("buffer size is not a multiple of element size")
ValueError: buffer size is not a multiple of element size

Root Cause

The 502 error and corrupted payload stem from Nginx’s default handling of request bodies larger than client_max_body_size and its buffering behavior for proxying binary streams. Two misconfigurations interact:

1. Request body size limit – Nginx was configured with client_max_body_size 1m; while audio files commonly exceed 2 MiB. When the limit is hit, Nginx discards the excess bytes and forwards an incomplete body to the upstream, causing the inference service to see a truncated stream.
2. Proxy buffering – The proxy_request_buffering off; directive was missing. With buffering enabled (default), Nginx reads the entire request into memory before sending it upstream. For large multipart uploads, the buffer can overflow, leading to partial writes and the “unexpected EOF” seen in the upstream logs.

Both issues result in the upstream receiving a malformed audio payload, which then triggers a 502 response because the upstream closes the connection abruptly after failing to process the request.

Debug

Step‑by‑step investigation that isolated the problem:

1. Verify request size limits

$ curl -v -F "audio=@large.wav" http://api.example.com/v1/evaluate
*   Trying 203.0.113.10...
* Connected to api.example.com (203.0.113.10) port 80 (#0)
> POST /v1/evaluate HTTP/1.1
> Host: api.example.com
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Type: multipart/form-data; boundary=------------------------d74496d66958873e
> Content-Length: 3145728
...
< HTTP/1.1 502 Bad Gateway

Note the Content-Length: 3145728 (≈3 MiB) which exceeds the default client_max_body_size.

2. Inspect Nginx error log for body size warnings

2026/09/24 12:34:55 [error] 12#12: *12345 client intended to send too large body: 3145728 bytes, client: 10.2.3.45, server: inference.example.com

3. Check proxy buffering settings

$ nginx -T | grep -i proxy_request_buffering
# No explicit setting found – default is "on"

4. Capture the request as it reaches the backend

# On the inference host
sudo tcpdump -i any -s 0 -w /tmp/req.pcap port 8000 and host 127.0.0.1
# After reproducing the failure, inspect with Wireshark:
# The captured HTTP request body ends at byte 1048576 (1 MiB)

5. Compare behavior with direct request (bypassing Nginx)

$ curl -v -F "audio=@large.wav" http://127.0.0.1:8000/v1/evaluate
...
< HTTP/1.1 200 OK

The backend processes the full payload correctly, confirming the issue is introduced by Nginx.

Solution

Adjust Nginx configuration to allow larger request bodies and disable request buffering for the inference endpoint.

Before (problematic) configuration

server {
    listen 80;
    server_name api.example.com;

    location /v1/evaluate {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        # client_max_body_size not set (defaults to 1m)
        # proxy_request_buffering left at default (on)
    }
}

After (corrected) configuration

server {
    listen 80;
    server_name api.example.com;

    # Allow up to 20 MiB audio uploads
    client_max_body_size 20m;

    location /v1/evaluate {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;

        # Stream the request directly to the upstream without buffering
        proxy_request_buffering off;

        # Optional: increase proxy buffer size for response headers
        proxy_buffer_size 16k;
        proxy_buffers 8 32k;
    }
}

After updating the configuration, reload Nginx:

$ sudo nginx -t && sudo systemctl reload nginx

Verify

Confirm that the 502 error no longer occurs and that the backend receives the full audio payload.

1. Successful request through Nginx

$ curl -v -F "audio=@large.wav" http://api.example.com/v1/evaluate
...
< HTTP/1.1 200 OK
...

2. Backend log shows full payload processing

2026-09-24 12:45:02,001 INFO audio_processor - Decoded 3145728 bytes (44100 Hz, 2 channels)
2026-09-24 12:45:02,003 INFO inference_engine - Model inference completed in 124ms

3. Nginx error log is clean

2026/09/24 12:45:02 [notice] 12#12: *12456 client closed connection while waiting for request, client: 10.2.3.45, server: inference.example.com
# No "client intended to send too large body" entries.

Prevent

• Set appropriate client_max_body_size for all endpoints that accept binary uploads. Align the limit with the maximum expected file size plus a safety margin.
• Disable proxy buffering for streaming APIs where the request body is large and should be passed through unchanged. Use proxy_request_buffering off; on a per‑location basis.
• Monitor request size metrics (e.g., nginx_http_client_body_bytes_sent) and alert when values approach the configured limit.
• Document upload size expectations in API contracts so clients can respect limits and avoid accidental oversized payloads.
• Test Nginx configuration changes in a staging environment with payloads at the limit boundary before promoting to production.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does the 502 error appear only for audio files larger than 1 MiB?
   Because the default client_max_body_size in Nginx is 1 MiB. Requests exceeding this limit are truncated, causing the upstream to close the connection and Nginx to return 502.
2. Can I keep proxy buffering enabled and still avoid the truncation?
   Buffering can be kept if you increase proxy_buffer_size and proxy_buffers enough to hold the entire request, but this consumes memory per connection and defeats the purpose of streaming large files. Disabling buffering is the simpler, more scalable approach for binary uploads.
3. Do I need to adjust client_body_timeout as well?
   Only if clients have very slow upload speeds. The default 60 s is usually sufficient, but for low‑bandwidth environments you may raise it to avoid premature timeout errors.
4. How can I verify that Nginx is no longer buffering the request?
   Enable debug log level temporarily and look for the line “*proxy request body: off” or inspect the proxy_request_buffering directive in the effective configuration via nginx -T. Additionally, a packet capture will show the request body flowing directly to the upstream without a pause.
5. Will increasing client_max_body_size affect other services behind the same Nginx instance?
   The directive can be scoped per server or location. Apply a larger limit only to the inference endpoint to avoid unintentionally allowing oversized uploads to unrelated services.