Black cables and wires connected to the back of modular LED display panels

Embedding dimension mismatch error after Nginx proxies inference requests

by

Problem – Embedding Dimension Mismatch After Nginx Proxies Inference Requests

In a Docker‑based deployment, an Nginx container fronts a Python FastAPI model service that expects an input embedding vector of a fixed size (e.g., 768 floats). After routing inference requests through Nginx, the service starts returning:

Traceback (most recent call last):
  File "/app/inference.py", line 42, in predict
    embedding = np.array(payload["embedding"], dtype=np.float32)
  File "/usr/local/lib/python3.8/site-packages/numpy/core/_asarray.py", line 96, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: Expected embedding size 768 but got 512

The same payload sent directly to the model service (bypassing Nginx) succeeds. The error manifests as HTTP 500 responses, breaking downstream pipelines.

Root Cause – Nginx Alters the Request Body

The model expects a JSON object with an "embedding" array of length 768. Nginx was configured with proxy_buffering on and gzip on. When the request body exceeds the default proxy_buffer_size (4 KB), Nginx streams the body through a temporary file, applying gzip compression if the client sent Accept-Encoding: gzip. The Python service receives the compressed payload, which after decompression yields a truncated array (512 elements) because the gzip stream is cut off by the client_max_body_size limit (1 MB by default, but the upstream buffer was smaller). The truncation changes the JSON structure, leading to a shorter embedding vector.

Debug – Investigation Steps

1. Reproduce the error with curl

curl -X POST http://nginx.example.com/v1/predict \
     -H "Content-Type: application/json" \
     -d @payload.json -v

Response:

< HTTP/1.1 500 Internal Server Error
< content-type: application/json
< content-length: 123
...
{
  "detail": "ValueError: Expected embedding size 768 but got 512"
}

2. Compare service logs with and without Nginx

# Direct request (bypass Nginx)
2026-10-31 12:01:12,345 INFO  inference.py:45 Received embedding size: 768
2026-10-31 12:01:12,350 INFO  inference.py:78 Prediction succeeded

# Through Nginx
2026-10-31 12:02:07,112 ERROR inference.py:45 Expected embedding size 768 but got 512

3. Inspect Nginx request/response headers

# Enable debug logging in nginx.conf
error_log /var/log/nginx/error.log debug;

# Sample log excerpt
2026/10/31 12:02:07 [debug] 12#12: *5 http request body length: 24576
2026/10/31 12:02:07 [debug] 12#12: *5 http request body in memory: 0
2026/10/31 12:02:07 [debug] 12#12: *5 http request body in a temporary file: /var/cache/nginx/proxy_temp/1/00/0000000012

The body is being spooled to a temporary file, indicating buffering.

4. Verify gzip handling

# Capture raw request with tcpdump
sudo tcpdump -i any -s 0 -w request.pcap port 80

# Decode with tshark
tshark -r request.pcap -Y http.request -T fields -e http.file_data | head

The captured payload shows gzip‑compressed JSON, confirming Nginx’s compression.

5. Check Nginx configuration relevant to request bodies

# /etc/nginx/conf.d/model-proxy.conf (original)
server {
    listen 80;
    location /v1/predict {
        proxy_pass http://model-service:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_buffering on;
        gzip on;
        client_max_body_size 1m;
    }
}

Solution – Align Request Handling with Model Expectations

1. Disable gzip for JSON payloads

Compression is unnecessary for small JSON bodies and interferes with the downstream parser.

# Updated /etc/nginx/conf.d/model-proxy.conf
server {
    listen 80;
    location /v1/predict {
        proxy_pass http://model-service:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;

        # Turn off buffering for small JSON bodies
        proxy_buffering off;

        # Disable gzip for this endpoint
        gzip off;

        # Raise the limit to accommodate large embeddings
        client_max_body_size 5m;
    }
}

2. Ensure the request body is passed unchanged

Explicitly forward the Content-Length header and disable request body modification.

# Additional directives
proxy_set_header Content-Length $content_length;
proxy_pass_request_body on;

3. Restart Nginx and verify the new configuration

docker exec -it nginx-container nginx -t
docker exec -it nginx-container nginx -s reload

4. (Optional) Add a request‑size guard in the FastAPI service

Reject malformed embeddings early, providing a clearer error.

# app/main.py
from fastapi import HTTPException

MAX_EMBEDDING_DIM = 768

def validate_embedding(vec):
    if len(vec) != MAX_EMBEDDING_DIM:
        raise HTTPException(
            status_code=400,
            detail=f"Embedding size mismatch: expected {MAX_EMBEDDING_DIM}, got {len(vec)}"
        )

Verify – Confirm the Fix Works

1. Smoke test with curl

curl -X POST http://nginx.example.com/v1/predict \
     -H "Content-Type: application/json" \
     -d @payload.json -s -o /dev/null -w "%{http_code}"

Expected output: 200

2. Check service logs for correct embedding size

2026-10-31 13:15:02,112 INFO  inference.py:45 Received embedding size: 768
2026-10-31 13:15:02,118 INFO  inference.py:78 Prediction succeeded

3. Verify no gzip compression on the request path

# Capture a new request
sudo tcpdump -i any -s 0 -w verify.pcap port 80
# Decode
tshark -r verify.pcap -Y http.request -T fields -e http.file_data | head

The payload appears as plain JSON, confirming gzip is disabled.

Prevent – Operational Guardrails

  • Schema validation at the API gateway: Use an OpenAPI validator (e.g., nginx-mod-http-openapi) to reject payloads with incorrect dimensions before they reach the model.
  • Monitoring: Emit a Prometheus metric model_embedding_mismatch_total on validation failures; set an alert threshold of >5 per minute.
  • Configuration linting: Include Nginx config checks in CI to ensure gzip off for /v1/predict and proxy_buffering off for JSON endpoints.
  • Capacity planning: Set client_max_body_size based on the maximum embedding vector size plus JSON overhead (e.g., 5 MiB for 1024‑dim vectors).
  • Testing: Add integration tests that send a correctly sized embedding through the full Nginx → model path and assert a 200 response.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does the error disappear when I call the model service directly?
    Because the direct call bypasses Nginx, so the JSON payload is delivered unchanged. Nginx’s buffering and gzip settings were truncating the body.
  2. Can I keep gzip enabled for other endpoints?
    Yes. Place gzip off; inside the location /v1/predict block only, leaving global gzip enabled for static assets.
  3. What if my embeddings are larger than 5 MiB?
    Increase client_max_body_size accordingly and ensure the upstream service’s request‑body parser can handle the size. Also consider streaming the vector in chunks if it becomes prohibitively large.
  4. Is proxy buffering the only cause of body truncation?
    No. Other factors include client_body_buffer_size, upstream timeouts, and intermediate proxies that may enforce their own size limits. Always audit the full request path.
  5. How can I detect similar mismatches for other model inputs?
    Standardize input validation in the model service and surface explicit HTTP 400 errors. Coupled with metric alerts on validation failures, you’ll catch mismatches early.