# Nginx access log should now contain the charset 2026/06/08 13:00:12 10.1.2.3 - - [08/Jun/2026:13:00:12 +0000] "POST /tokenize HTTP/1.1" 200 256 "application/json; charset=utf-8" "..." # Downstream FastAPI logs Raw body: b'{"text":"Привет мир 🌐"}' Tokenization succeeded, 6 tokens returned. 3. Monitoring metrics Increase in tokenizer_success_total (Prometheus counter) after deployment. Zero occurrences of UnicodeDecodeError in application logs over a full CI run. Prevention – Guardrails for Future Deployments Configuration as code: Store the Nginx block in a version‑controlled file and run nginx -t in CI to validate syntax before rollout. Automated header checks: Add a test stage that sends a sample JSON request through the proxy and asserts the presence of charset=utf-8 and correct Content-Length. Metrics & alerts: Alert on nginx_http_response_status{code=5xx} spikes and on any occurrence of UnicodeDecodeError in downstream logs. Explicit gzip policy: Keep gzip on only for static assets; disable it for API endpoints by using gzip_disable "msie6"; and gzip_proxied off;. Document request size expectations: Align client_max_body_size with the maximum tokenization input size used by your pipelines. FAQ – Common Follow‑Up Questions Why does the error appear only in the CI pipeline and not locally? CI agents often send larger or multi‑language payloads that exceed the default client_body_buffer_size. Locally you may be testing short ASCII strings, which survive the default buffering. Do I need to set Accept-Charset as well as Content-Type? Setting Accept-Charset utf-8 is optional for most frameworks, but it prevents downstream services from negotiating a different charset when the client omits it. Can I keep gzip on for static files and still avoid corruption? Yes. Use gzip_types to limit compression to static MIME types (e.g., text/css, application/javascript) and set gzip_proxied off; so API responses are never gzipped. What if I must use HTTP/1.0 for legacy upstreams? When HTTP/1.0 is required, ensure proxy_set_header Content-Length $content_length and disable proxy_request_buffering to avoid body truncation. However, tokenization services typically work fine with HTTP/1.1, which is the recommended configuration. How do I verify that Nginx is not altering the request body in production? Deploy a health‑check endpoint that echoes the received JSON (e.g., /debug/echo) and periodically curl it through the proxy, comparing the response to the original payload. Any mismatch indicates a proxy‑side transformation. Related Topic Hub: Distributed Systems Troubleshooting Hub

Nginx Reverse Proxy Causing Tokenizer Encoding Errors

Q: Debug – Systematic Investigation Steps 1. Capture the raw request as it arrives at Nginx

# Enable request logging (add to http{} block) log_format request_body '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_content_type" "$request_body"'; access_log /var/log/nginx/request_body.log request_body; Inspect the log for a sample payload:

Q: 2. Verify that the upstream service receives the same body

# In the FastAPI app, log the raw body @app.post("/tokenize") async def tokenize(request: Request): raw = await request.body() logger.info(f"Raw body: {raw!r}") # ... tokenization logic ... Typical corrupted log output:

Q: 3. Check Nginx buffering and HTTP version

# Show current proxy settings nginx -T | grep -E "proxy_http_version|proxy_pass_request_body|proxy_request_buffering" Typical output (problematic):

Q: 4. Test with curl while forcing headers

curl -v -X POST http://nginx-proxy/tokenize \ -H "Content-Type: application/json; charset=utf-8" \ -d '{"text":"Café 😊"}' Observe the Transfer-Encoding and Content-Length headers in the verbose output. Missing Content-Length or presence of Transfer-Encoding: chunked without proper buffering can be a clue.

Q: Solution – Correct Nginx Configuration for Safe Tokenizer Payloads 1. Enforce UTF‑8 charset on inbound requests

# In the server block handling /tokenize location /tokenize { charset utf-8; proxy_set_header Content-Type "application/json; charset=utf-8"; proxy_set_header Accept-Charset "utf-8"; } 2. Preserve the raw request body # Ensure request body is passed unchanged proxy_pass_request_body on; proxy_set_header Content-Length $content_length; proxy_http_version 1.1; # Enable chunked support if needed proxy_request_buffering on; # Default, but make explicit 3. Disable gzip for JSON payloads # Global gzip settings gzip on; gzip_types text/plain application/json; gzip_proxied off; # Prevent gzip on proxied responses # Or limit to safe content types only gzip_proxied no-cache no-store private; 4. Increase client body buffers for large tokenization inputs # Increase buffer size to accommodate typical request sizes (e.g., up to 1 MiB) client_body_buffer_size 256k; client_max_body_size 2m; 5. Full before/after comparison AspectBefore (faulty)After (fixed) Charset handling None (default) charset utf-8; + proxy_set_header Content-Type "application/json; charset=utf-8" Proxy HTTP version proxy_http_version 1.0; proxy_http_version 1.1; Request body buffering proxy_pass_request_body off; proxy_pass_request_body on; Gzip handling gzip on; (global) – compressed JSON bodies gzip_proxied off; (or limited to safe types) Body size limits client_max_body_size 1m; – truncated large payloads client_max_body_size 2m; + larger buffer Verification – Confirming the Fix Works 1. Functional test via curl curl -s -X POST http://nginx-proxy/tokenize \ -H "Content-Type: application/json; charset=utf-8" \ -d '{"text":"Привет мир 🌐"}' | jq . Expected output (excerpt):

In this article

Table of Contents

Problem – Tokenizer Encoding Errors Behind an Nginx Reverse Proxy

During automated model deployment in a CI/CD pipeline (GitLab CI, Jenkins, Azure DevOps, or GitHub Actions), the inference service receives POST requests containing raw text that must be tokenized. In several incidents the service raised exceptions such as:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 23: invalid start byte
Invalid token: unexpected character
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

All failures trace back to the same symptom: the JSON payload arriving at the downstream FastAPI / Flask tokenizer is corrupted or truncated. The root of the problem is the Nginx reverse proxy modifying the request body encoding or applying compression/buffering that is incompatible with the binary‑safe JSON expected by the tokenizer.

Root Cause – How Nginx Alters the Request Body

Three Nginx features interact in ways that break UTF‑8 payloads:

Charset handling – The charset module defaults to iso-8859-1 for responses and does not set Content-Type for inbound requests. When the upstream service relies on application/json; charset=utf-8, Nginx may drop the charset parameter, causing the downstream framework to assume the wrong encoding. (See NGINX Core Documentation – charset.)
Request body buffering & HTTP version – By default Nginx uses proxy_http_version 1.0. HTTP/1.0 does not support Transfer‑Encoding: chunked, so Nginx buffers the body, may truncate it when client_body_buffer_size is too small, and can drop bytes if proxy_pass_request_body is off. (Reference: NGINX Proxy Settings.)
Gzip compression – When gzip on is enabled globally, Nginx will attempt to compress any response, but it can also mistakenly compress request bodies that pass through proxy_pass if gzip_proxied is not limited to safe content types. Multi‑byte UTF‑8 characters become corrupted, leading to the “Invalid token: unexpected character” error reported in the FastAPI tokenizer (GitHub issue #8459).

Combined, these defaults cause the downstream inference service to receive a payload that is either:

Missing the charset=utf-8 directive, so the JSON parser treats bytes as ISO‑8859‑1.
Truncated because Nginx stopped buffering at client_body_buffer_size (default 8k) for large inputs.
Gzip‑altered, turning valid UTF‑8 sequences into invalid byte streams.

Debug – Systematic Investigation Steps

1. Capture the raw request as it arrives at Nginx

# Enable request logging (add to http{} block)
log_format request_body '$remote_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_content_type" "$request_body"';
access_log /var/log/nginx/request_body.log request_body;

Inspect the log for a sample payload:

2026/06/08 12:34:56 10.1.2.3 - - [08/Jun/2026:12:34:56 +0000] "POST /tokenize HTTP/1.1" 200 123 "application/json; charset=utf-8" "{"text":"Hello world 🌍"}"

If the charset=utf-8 part is missing, Nginx stripped it.

2. Verify that the upstream service receives the same body

# In the FastAPI app, log the raw body
@app.post("/tokenize")
async def tokenize(request: Request):
    raw = await request.body()
    logger.info(f"Raw body: {raw!r}")
    # ... tokenization logic ...

Typical corrupted log output:

Raw body: b'{"text":"Hello\x80world"}'

The \x80 byte indicates a lost UTF‑8 continuation byte.

3. Check Nginx buffering and HTTP version

# Show current proxy settings
nginx -T | grep -E "proxy_http_version|proxy_pass_request_body|proxy_request_buffering"

Typical output (problematic):

proxy_http_version 1.0;
proxy_pass_request_body off;
proxy_request_buffering off;

4. Test with curl while forcing headers

curl -v -X POST http://nginx-proxy/tokenize \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{"text":"Café 😊"}'

Observe the Transfer-Encoding and Content-Length headers in the verbose output. Missing Content-Length or presence of Transfer-Encoding: chunked without proper buffering can be a clue.

Solution – Correct Nginx Configuration for Safe Tokenizer Payloads

1. Enforce UTF‑8 charset on inbound requests

# In the server block handling /tokenize
location /tokenize {
    charset utf-8;
    proxy_set_header Content-Type "application/json; charset=utf-8";
    proxy_set_header Accept-Charset "utf-8";
}

2. Preserve the raw request body

# Ensure request body is passed unchanged
proxy_pass_request_body on;
proxy_set_header Content-Length $content_length;
proxy_http_version 1.1;        # Enable chunked support if needed
proxy_request_buffering on;    # Default, but make explicit

3. Disable gzip for JSON payloads

# Global gzip settings
gzip on;
gzip_types text/plain application/json;
gzip_proxied off;               # Prevent gzip on proxied responses
# Or limit to safe content types only
gzip_proxied no-cache no-store private;

4. Increase client body buffers for large tokenization inputs

# Increase buffer size to accommodate typical request sizes (e.g., up to 1 MiB)
client_body_buffer_size 256k;
client_max_body_size 2m;

5. Full before/after comparison

Aspect	Before (faulty)	After (fixed)
Charset handling	None (default)	`charset utf-8;` + `proxy_set_header Content-Type "application/json; charset=utf-8"`
Proxy HTTP version	`proxy_http_version 1.0;`	`proxy_http_version 1.1;`
Request body buffering	`proxy_pass_request_body off;`	`proxy_pass_request_body on;`
Gzip handling	`gzip on;` (global) – compressed JSON bodies	`gzip_proxied off;` (or limited to safe types)
Body size limits	`client_max_body_size 1m;` – truncated large payloads	`client_max_body_size 2m;` + larger buffer

Verification – Confirming the Fix Works

1. Functional test via curl

curl -s -X POST http://nginx-proxy/tokenize \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{"text":"Привет мир 🌐"}' | jq .

Expected output (excerpt):

{
  "tokens": [12345, 6789, 1011, 2022],
  "input_ids": [101, 12345, 6789, 1011, 2022, 102]
}

2. Log inspection

# Nginx access log should now contain the charset
2026/06/08 13:00:12 10.1.2.3 - - [08/Jun/2026:13:00:12 +0000] "POST /tokenize HTTP/1.1" 200 256 "application/json; charset=utf-8" "..."

# Downstream FastAPI logs
Raw body: b'{"text":"Привет мир 🌐"}'
Tokenization succeeded, 6 tokens returned.

3. Monitoring metrics

Increase in tokenizer_success_total (Prometheus counter) after deployment.
Zero occurrences of UnicodeDecodeError in application logs over a full CI run.

Prevention – Guardrails for Future Deployments

Configuration as code: Store the Nginx block in a version‑controlled file and run nginx -t in CI to validate syntax before rollout.
Automated header checks: Add a test stage that sends a sample JSON request through the proxy and asserts the presence of charset=utf-8 and correct Content-Length.
Metrics & alerts: Alert on nginx_http_response_status{code=5xx} spikes and on any occurrence of UnicodeDecodeError in downstream logs.
Explicit gzip policy: Keep gzip on only for static assets; disable it for API endpoints by using gzip_disable "msie6"; and gzip_proxied off;.
Document request size expectations: Align client_max_body_size with the maximum tokenization input size used by your pipelines.

FAQ – Common Follow‑Up Questions

Why does the error appear only in the CI pipeline and not locally?
CI agents often send larger or multi‑language payloads that exceed the default client_body_buffer_size. Locally you may be testing short ASCII strings, which survive the default buffering.
Do I need to set Accept-Charset as well as Content-Type?
Setting Accept-Charset utf-8 is optional for most frameworks, but it prevents downstream services from negotiating a different charset when the client omits it.
Can I keep gzip on for static files and still avoid corruption?
Yes. Use gzip_types to limit compression to static MIME types (e.g., text/css, application/javascript) and set gzip_proxied off; so API responses are never gzipped.
What if I must use HTTP/1.0 for legacy upstreams?
When HTTP/1.0 is required, ensure proxy_set_header Content-Length $content_length and disable proxy_request_buffering to avoid body truncation. However, tokenization services typically work fine with HTTP/1.1, which is the recommended configuration.
How do I verify that Nginx is not altering the request body in production?
Deploy a health‑check endpoint that echoes the received JSON (e.g., /debug/echo) and periodically curl it through the proxy, comparing the response to the original payload. Any mismatch indicates a proxy‑side transformation.

Related Topic Hub: Distributed Systems Troubleshooting Hub