Nginx upstream request body empty for LLM after adding context injection

Nginx upstream request body empty for LLM after adding context injection

Problem

After deploying a retrieval‑augmented generation (RAG) pipeline, the Nginx reverse proxy stopped sending the retrieved documents (or their metadata) to the LLM backend. The upstream request body received by the LLM is empty, resulting in generic or nonsensical completions. The failure appeared only after the context_injection module was added to the Nginx configuration.

Root Cause

The context_injection Lua handler reads the documents from an internal cache (/var/run/rag_cache.sock) and appends them to the original JSON payload. Two independent issues combined to produce an empty body:

• Incorrect request body buffering: The proxy_request_buffering off; directive was left enabled while the Lua script attempted to modify ngx.req.get_body_data(). When buffering is disabled, Nginx streams the client body directly to the upstream and does not store it in memory, causing ngx.req.get_body_data() to return nil.
• Premature read of the request body: The Lua code called ngx.req.read_body() after the proxy_pass phase, i.e., in the log_by_lua_block. At that point the request body has already been consumed by the upstream socket, so the script cannot append the context.

Both conditions lead to the final request body being the original (empty) payload, so the LLM receives no context.

Debug

Typical symptoms observed in the logs:

2026/10/02 14:12:07 [error] 12#12: *12345 upstream sent no valid HTTP/1.1 header while reading response header from upstream, client: 10.0.2.15, server: rag-proxy, request: "POST /v1/completions HTTP/1.1", upstream: "http://127.0.0.1:8000/v1/completions", host: "api.example.com"
2026/10/02 14:12:07 [debug] 12#12: *12345 lua request body: nil
2026/10/02 14:12:07 [debug] 12#12: *12345 context injection: retrieved 0 documents

Key debugging steps:

1. Enable error_log at debug level for the http block.
2. Inspect ngx.var.request_body and ngx.req.get_body_data() inside the Lua handler.
3. Capture the raw request sent to the LLM using tcpdump or ngrep -d any -W byline 'POST /v1/completions' to confirm the body is empty.
4. Check the order of Nginx phases in the configuration (e.g., access_by_lua_block vs log_by_lua_block).

Solution

Two changes are required:

1. Enable request body buffering

Remove proxy_request_buffering off; (or set it to on) so that Nginx stores the client payload in memory and makes it available to Lua.

2. Move the Lua injection to the access_by_lua_block phase

Inject the context before the request is proxied. This guarantees the body is still mutable.

Below is the before configuration (problematic):

server {
    listen 443 ssl;
    server_name api.example.com;

    location /v1/completions {
        proxy_pass http://127.0.0.1:8000;
        proxy_request_buffering off;   # <-- disables body buffering
        log_by_lua_block {
            ngx.req.read_body()
            local body = ngx.req.get_body_data()
            ngx.log(ngx.DEBUG, "lua request body: ", body)

            local ctx = retrieve_context()
            ngx.req.set_body_data(body .. ctx)
        }
    }
}

And the after configuration that resolves the issue:

server {
    listen 443 ssl;
    server_name api.example.com;

    location /v1/completions {
        # Keep default buffering (on)
        proxy_pass http://127.0.0.1:8000;

        access_by_lua_block {
            -- Ensure the request body is fully read into memory
            ngx.req.read_body()
            local body = ngx.req.get_body_data()
            if not body then
                ngx.log(ngx.ERR, "failed to read request body")
                return ngx.exit(ngx.HTTP_BAD_REQUEST)
            end

            ngx.log(ngx.DEBUG, "original request body size: ", #body)

            -- Retrieve RAG documents (JSON array of strings)
            local ctx = retrieve_context()
            if ctx then
                -- Append a newline and the context JSON
                local new_body = body:gsub("%}$", ", \"context\": " .. ctx .. "}")
                ngx.req.set_body_data(new_body)
                ngx.log(ngx.DEBUG, "injected context, new body size: ", #new_body)
            else
                ngx.log(ngx.WARN, "no context retrieved, forwarding original payload")
            end
        }
    }
}

Note the helper function retrieve_context() is unchanged; only the phase and buffering matter.

Verify

After reloading Nginx (nginx -s reload), perform the following checks:

1. Send a test RAG request with a known prompt:

curl -k -X POST https://api.example.com/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Explain quantum entanglement","max_tokens":200}'

Inspect the upstream logs (LLM service) for the received payload:

2026-10-02 14:20:13 INFO Received request: {"prompt":"Explain quantum entanglement","max_tokens":200,"context":[{"doc_id":"123","text":"Entanglement is ..."}]}

Confirm the response now contains the expected context‑aware completion.

Additional validation with tcpdump:

sudo tcpdump -i any -s 0 -A 'port 8000 and tcp[tcpflags] & (tcp-push|tcp-ack) != 0' | grep -i "context"

The captured HTTP body should include the "context" field.

Prevent

• Always keep proxy_request_buffering enabled when downstream handlers need to read or modify the request body.
• Place body‑modifying Lua code in access_by_lua_block or rewrite_by_lua_block, never in log_by_lua_block or later phases.
• Add a health‑check endpoint that echoes back the request body after injection; automate this in CI to catch regressions.
• Instrument metrics:
  • nginx_context_injection_success_total
  • nginx_context_injection_failure_total

  Alert when the failure counter spikes.

• Document the required Nginx phase ordering in the run‑book for any future Lua extensions.

FAQ

1. Why does the request body appear empty only after adding the Lua module?

   The module switched the processing phase to log_by_lua_block while also disabling buffering. In that phase the body is already streamed, so ngx.req.get_body_data() returns nil.

2. Can I keep proxy_request_buffering off and still inject context?

   No. Without buffering Nginx never stores the full body in memory, making it unavailable to Lua. The only alternative is to read the raw client stream via ngx.req.read_body() in a custom C module, which is far more complex and error‑prone.

3. Is there a way to detect at runtime whether the body is buffered?

   Yes. In Lua you can check ngx.var.request_body or the return value of ngx.req.get_body_data(). If it is nil, buffering is disabled or the body size exceeds client_body_buffer_size and has been spooled to disk.

4. My LLM still receives an empty context array after the fix. What else should I check?

   Verify that retrieve_context() actually returns a JSON string. Log its output before concatenation. Also ensure the original JSON payload ends with a closing brace; the gsub("%}$", ...) pattern assumes that format.

5. Do I need to adjust client_max_body_size for large document sets?

   When injecting many retrieved documents the request body can exceed the default 1 MiB limit. Increase client_max_body_size accordingly and monitor upstream memory usage.

Related Topic Hub: Distributed Systems Troubleshooting Hub