HAProxy multimodal alignment error in Kubernetes cluster

Problem: HAProxy “multimodal alignment” errors in a mixed‑protocol Kubernetes ingress

In a GKE‑based AI platform the same HAProxy ingress is used to expose:

• TensorFlow Serving (gRPC over HTTP/2)
• FastAPI inference endpoints (HTTP/1.1)
• WebSocket streams (HTTP/1.1 upgrade)

After a rolling update the following symptoms appeared:

[WARNING] 2026/07/08 10:12:45 HA-Proxy[WARN] : content mismatch, header size exceeds payload
[ERROR]   2026/07/08 10:12:46 HA-Proxy[ERR]  : HTTP/2: malformed request line
[ALERT]  2026/07/08 10:12:47 HA-Proxy[ALERT]: invalid HTTP/2 frame received
[ERROR]  2026/07/08 10:12:48 HA-Proxy[ERR]  : HTTP/2: stream error: PROTOCOL_ERROR

Resulting impact:

• 502/503 responses for gRPC inference calls
• Health‑check probes (HTTP/1.1) falsely reported backend failure, causing pod restarts
• Intermittent “header size exceeds payload” errors during peak load for WebSocket streams

Root Cause Analysis

HAProxy parses inbound traffic according to the mode of the frontend. When a single frontend is configured with mode http and receives both HTTP/1.1 and HTTP/2 (gRPC) streams, HAProxy must decide which protocol parser to apply. The decision is driven by:

• ALPN negotiation during TLS hand‑shake (HAProxy docs: “gRPC support – HTTP/2 handling, ALPN”)
• Presence of the Content-Type: application/grpc header (content switching rules)

In the incident the frontend reused the same TCP connection for both protocols because:

1. ALPN was not forced; some clients fell back to HTTP/1.1 while others used HTTP/2.
2. Content‑switching rules based on :path or :authority were missing, so HAProxy fell back to the default backend after the first request on the connection.
3. A bug in HAProxy 2.8 (see GitHub issue #8453) caused the HTTP/2 parser state to become corrupted when an HTTP/1.1 request arrived on a connection that had already been upgraded to HTTP/2.

When the parser state diverged, HAProxy attempted to interpret an HTTP/1.1 payload as an HTTP/2 frame, leading to the “invalid HTTP/2 frame” and “header size exceeds payload” messages – the classic “multimodal alignment” failure described in the HAProxy Enterprise “Multimodal Alignment” troubleshooting chapter.

Investigation and Debugging Steps

1. Capture HAProxy logs with full request details

docker exec -it haproxy \
  /bin/sh -c "cat /var/log/haproxy.log | grep -E 'ERR|WARN|ALERT'"

2. Verify TLS ALPN negotiation

openssl s_client -connect ingress.example.com:443 -alpn h2 -servername ingress.example.com
# Expected output includes "ALPN protocol: h2"

3. Inspect active connections

echo "show sess" | socat stdio /var/run/haproxy.sock
# Look for sessions with "proto=HTTP/2" followed by "proto=HTTP/1.1"

4. Reproduce with curl and grpcurl

# HTTP/1.1 request (FastAPI)
curl -v http://ingress.example.com/v1/predict

# gRPC request (TensorFlow Serving)
grpcurl -plaintext -proto tensorflow.proto \
  ingress.example.com:80 tensorflow.serving.PredictionService/Predict

5. Check frontend configuration

frontend ai-ingress
    bind *:443 ssl crt /etc/haproxy/certs/ingress.pem alpn h2,http/1.1
    mode http
    option http-use-htx
    # Existing rule (problematic)
    default_backend http-backend

6. Review content‑switching rules from the HAProxy Kubernetes Ingress Controller guide

The guide recommends separate use_backend clauses that inspect :path or :authority to route gRPC vs. HTTP/1.1 traffic.

Solution: Separate Frontends or Explicit ALPN‑Based Routing

Two proven approaches:

Approach A – Split frontends by protocol

Define one frontend that only accepts HTTP/2 (gRPC) and another that only accepts HTTP/1.1. This eliminates any state sharing on a single TCP connection.

# Frontend for gRPC (HTTP/2)
frontend grpc-in
    bind *:8443 ssl crt /etc/haproxy/certs/ingress.pem alpn h2
    mode http
    option http-use-htx
    use_backend tf-serving-backend if { req.hdr(Content-Type) -i application/grpc }
    default_backend grpc-404

# Frontend for REST/WebSocket (HTTP/1.1)
frontend http-in
    bind *:8080 ssl crt /etc/haproxy/certs/ingress.pem alpn http/1.1
    mode http
    option http-use-htx
    acl is_ws hdr(Upgrade) -i websocket
    use_backend fastapi-backend if { path_beg /v1/ }
    use_backend websocket-backend if is_ws
    default_backend http-404

Update the Kubernetes Service to expose both ports (8443 and 8080) and adjust the Ingress spec.rules accordingly.

Approach B – Single frontend with ALPN‑based content switching

If exposing a single port is mandatory, add explicit use_backend rules that test the negotiated protocol via the HAProxy variable ssl_fc_alpn (available since HAProxy 2.4).

frontend ai-ingress
    bind *:443 ssl crt /etc/haproxy/certs/ingress.pem alpn h2,http/1.1
    mode http
    option http-use-htx

    # Route gRPC when ALPN is h2 and Content-Type is gRPC
    acl is_grpc ssl_fc_alpn h2
    acl grpc_ct hdr(Content-Type) -i application/grpc
    use_backend tf-serving-backend if is_grpc grpc_ct

    # Route HTTP/1.1 REST endpoints
    acl is_rest path_beg /v1/
    use_backend fastapi-backend if is_rest

    # Route WebSocket upgrades
    acl is_ws hdr(Upgrade) -i websocket
    use_backend websocket-backend if is_ws

    default_backend http-404

Key changes compared to the original configuration:

• Added alpn h2,http/1.1 on the bind line (ensures client‑side ALPN is advertised).
• Enabled option http-use-htx to force HAProxy to use the new HTX parsing engine, which is required for reliable HTTP/2 handling (HAProxy docs: “Mode http” and “option http-use-htx”).
• Introduced explicit acl checks for ssl_fc_alpn and Content-Type to prevent the parser from mixing protocols on the same connection.

Verification

Functional tests

# Verify gRPC endpoint
grpcurl -proto tensorflow.proto -insecure \
  ingress.example.com:443 tensorflow.serving.PredictionService/Predict
# Expected: SUCCESS response, no 502/503

# Verify REST endpoint
curl -k https://ingress.example.com/v1/predict
# Expected: 200 OK with JSON payload

# Verify WebSocket upgrade
websocat -n -t wss://ingress.example.com/ws/stream
# Expected: connection established, no “header size exceeds payload”

Log inspection

grep -E 'ERR|WARN|ALERT' /var/log/haproxy.log
# Should return no entries related to “invalid HTTP/2 frame” or “header size exceeds payload”

Metrics

Prometheus HAProxy exporter should show zero increase in haproxy_frontend_http_response_errors_total for the affected frontends.

Prevention and Operational Best Practices

• Enforce ALPN negotiation: always include both h2 and http/1.1 in the bind line; reject connections that do not present ALPN.
• Use separate frontends for divergent protocols when the traffic mix is heavy or when latency‑sensitive services (e.g., video streaming) share the same ingress.
• Enable option http-use-htx on all HTTP frontends to leverage HAProxy’s modern HTTP/2 parser.
• Monitor HAProxy alerts for the specific error strings listed in the “Common errors” evidence set; set up alerting on haproxy_frontend_http_response_errors_total.
• Run integration tests in CI that simulate concurrent HTTP/1.1 and gRPC calls on the same port, asserting that no “multimodal alignment” log entries appear.
• Pin HAProxy version to a release that includes the fix for issue #8453 (≥ 2.8.2) or apply the back‑ported patch if using an older version.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does HAProxy sometimes treat an HTTP/1.1 request as HTTP/2?
   Because the frontend is configured in mode http without explicit ALPN routing. If the first request on a TCP connection negotiates HTTP/2, HAProxy keeps the HTTP/2 parser state for the whole connection. A subsequent HTTP/1.1 request on the same socket is then parsed as HTTP/2, producing “malformed request line” errors.
2. Can I keep a single port and still avoid multimodal alignment errors?
   Yes, by using ALPN‑based ACLs (ssl_fc_alpn) and option http-use-htx to force HAProxy to switch parsers per request. The “single‑frontend” solution shown in Approach B demonstrates this.
3. Do health‑check probes need special handling?
   Health probes are HTTP/1.1 by default. Ensure they are routed to an HTTP/1.1 backend (e.g., a dedicated health‑check backend) or add an ACL that forces use_backend health‑check-backend if { path /health }. This prevents a probe from hitting a gRPC backend and causing a 503.
4. Is option http-use-htx mandatory for gRPC?
   Starting with HAProxy 2.4 it is recommended for any HTTP/2 traffic. It enables the HTX transaction engine, which correctly separates HTTP/2 frames from HTTP/1.1 payloads and eliminates the alignment issue described in the HAProxy Enterprise “Multimodal Alignment” chapter.
5. What version of HAProxy contains the fix for the parser state bug?
   The bug reported in GitHub issue #8453 was fixed in HAProxy 2.8.2. If you cannot upgrade, back‑port the patch from the HAProxy repository or isolate protocols using separate frontends.