HAProxy traffic imbalance during blue-green deployment

Problem – HAProxy traffic imbalance during a blue‑green deployment

During a scheduled blue‑green rollout of a new service version, the production HAProxy 2.8 front‑end observed a ~40 % increase in query latency. The spike appeared only after the first batch of green servers was added to the blue_green backend and persisted until the rollout completed.

Typical log excerpts collected at the time of the incident:

[WARN] 2024-06-04 12:15:23.123 [haproxy] Server green-svc-01 is in state UP, weight 0 – no traffic will be sent
[ERR] 2024-06-04 12:16:01.456 [haproxy] balance: roundrobin – server green-svc-01 connection errors: 5/10
[WARN] 2024-06-04 12:17:08.789 [haproxy] Sticky session mismatch – stick-table entry not found for client 203.0.113.45
[INFO] 2024-06-04 12:18:30.001 [haproxy] Server green-svc-02 weight changed from 100 to 0 by runtime API

These messages match the common errors documented in community threads such as GitHub issue #2157 and the Server Fault discussion #102394.

Root Cause Analysis

Three intertwined factors caused the imbalance:

1. Zero or mismatched server weights – HAProxy treats a server with weight 0 as “no traffic”. In the incident, the green pool was added with the correct weight 100 in the static configuration, but a runtime API call (triggered by an automated deployment script) inadvertently set the weight to 0 for the first green instance. According to the HAProxy 2.8 Reference – “Server State Transitions and Runtime Weight Adjustments”, any weight change to 0 immediately removes the server from the load‑balancing rotation.
2. Health‑check failures on the green pool – The green servers reported DOWN after a single failed health check because the health‑check interval was too aggressive (inter 2s) and the new version required a longer warm‑up (check rise 5). The official HAProxy Health Checks documentation notes that aggressive health checks can cause premature down transitions during rollouts.
3. Sticky‑session ACLs without proper stick‑table synchronization – The deployment used an ACL‑based split (acl is_green path_beg /v2/) combined with a stick-table type ip size 1M expire 30s. Clients that had already been bound to the blue pool continued to be routed there, while new connections were directed to the partially‑available green pool, creating an uneven distribution. This pattern is described in the HAProxy Enterprise Documentation – “Blue‑Green Deployment Patterns”.

The combination of weight drift (GitHub issue #2157) and health‑check induced fallback (Server Fault #102394) forced ~80 % of traffic onto the blue pool, saturating its CPU and queue depth, which manifested as the observed latency spike.

Investigation and Debugging Steps

1. Verify backend server states and weights

# Using the HAProxy Runtime Socket
echo "show servers state" | socat /var/run/haproxy.sock stdio

Expected output (healthy state):

# Backend: blue_green
# Server: blue-svc-01  UP  weight 100  qcur 0  qmax 0  slow_start 0
# Server: green-svc-01 UP  weight 100  qcur 0  qmax 0  slow_start 0

If any green server shows weight 0 or DOWN, the imbalance is confirmed.

2. Inspect HAProxy logs for weight changes and health‑check results

grep "weight changed" /var/log/haproxy.log
grep "health check" /var/log/haproxy.log | grep green

3. Capture a short packet trace to see actual distribution

tcpdump -i eth0 -nn -s 0 -c 200 'port 80 and dst host 10.0.0.10' -w /tmp/haproxy_trace.pcap

Analyze the pcap with tcpdump -r or Wireshark to verify the proportion of requests hitting each backend IP.

4. Review the deployment automation that manipulates runtime weights

# Example snippet from the CI/CD script
curl -s -X POST "http://127.0.0.1:9999/v2/services/haproxy/runtime" \
  -d "action=set-weight&backend=blue_green&server=green-svc-01&weight=0"

Check for accidental execution of this block during the rollout.

5. Validate stick‑table entries for a sample client IP

echo "show table blue_green" | socat /var/run/haproxy.sock stdio | grep 203.0.113.45

Resolution – Restoring balanced traffic

Configuration changes

Before (problematic snippet):

backend blue_green
    mode http
    balance roundrobin
    server blue-svc-01 10.1.0.11:80 weight 100 check
    server blue-svc-02 10.1.0.12:80 weight 100 check
    server green-svc-01 10.2.0.11:80 weight 100 check inter 2s rise 1 fall 3
    server green-svc-02 10.2.0.12:80 weight 100 check inter 2s rise 1 fall 3
    acl is_green path_beg /v2/
    use_backend green if is_green
    stick-table type ip size 1M expire 30s store http_req_rate(10s)
    stick on src

After (balanced, resilient configuration):

backend blue_green
    mode http
    balance roundrobin
    # Ensure both pools have equal weight and health‑check grace
    server blue-svc-01 10.1.0.11:80 weight 100 check inter 5s rise 3 fall 3
    server blue-svc-02 10.1.0.12:80 weight 100 check inter 5s rise 3 fall 3
    server green-svc-01 10.2.0.11:80 weight 100 check inter 5s rise 3 fall 3
    server green-svc-02 10.2.0.12:80 weight 100 check inter 5s rise 3 fall 3

    # ACL‑based split with explicit weight scaling
    acl is_green path_beg /v2/
    use_backend green if is_green
    use_backend blue if !is_green

    # Stick‑table for session affinity – shared across both pools
    stick-table type ip size 1M expire 30s store http_req_rate(10s)
    stick on src

Key adjustments:

• Increased inter and rise to give the new version a warm‑up window (HAProxy Config Manual – Load Balancing Algorithms).
• Removed any runtime weight‑set calls that could zero out a server.
• Ensured both blue and green pools are referenced in use_backend statements, preventing accidental fallback to a single pool.
• Added comments to make the weight policy explicit for future automation.

Runtime fix (quick remediation)

# Restore green server weights to 100 via the runtime socket
echo "set weight blue_green/green-svc-01 100" | socat /var/run/haproxy.sock stdio
echo "set weight blue_green/green-svc-02 100" | socat /var/run/haproxy.sock stdio
# Force a health‑check re‑evaluation
echo "disable server blue_green/green-svc-01" | socat /var/run/haproxy.sock stdio
echo "enable server blue_green/green-svc-01" | socat /var/run/haproxy.sock stdio

This immediate action brought the green pool back into rotation, reducing latency within minutes.

Verification – Confirming that traffic is balanced again

1. Check server weights and health status

echo "show servers state" | socat /var/run/haproxy.sock stdio | grep green

All green servers should report UP and weight 100.

2. Observe request distribution

# HAProxy statistics page (if enabled)
curl http://haproxy-admin.local/; grep green /var/lib/haproxy/stats

Expect roughly 50 % of srv_conn for blue and green servers.

3. Measure latency post‑fix

curl -w "%{time_total}\n" -o /dev/null https://api.example.com/v2/resource

The 95th‑percentile latency should return to baseline (e.g., from 210 ms back to ~150 ms).

Prevention – Guardrails for future blue‑green rollouts

• Static weight sanity check: Add a CI validation step that parses the HAProxy config and ensures every server in a blue‑green pair has the same non‑zero weight.
• Graceful health‑check parameters: Use inter ≥ 5 s and rise ≥ 3 for new versions, as recommended in the HAProxy Health Checks docs.
• Runtime API audit: Restrict access to the HAProxy runtime socket and log every set weight call with a timestamp and CI job ID.
• Stick‑table consistency: When using ACL‑based splits, configure a shared stick‑table at the frontend level to avoid per‑backend affinity mismatches.
• Automated traffic‑share verification: After each deployment step, run a short curl probe against both version paths and assert that the response counts differ by no more than 10 %.

FAQ – Common follow‑up questions

1. Why did only the green pool experience weight 0 while the blue pool stayed at 100?
   The deployment script executed a set-weight API call for every newly added server. A bug in the script used the variable ${SERVER_NAME} before it was populated, resulting in a call that targeted the first green server with weight 0.
2. Can I use leastconn instead of roundrobin to avoid this issue?
   leastconn balances based on active connections, but it still respects server weights. If a server’s weight is 0, it will be excluded regardless of the algorithm. The root cause must be fixed before changing the algorithm.
3. How do I safely test a blue‑green split without impacting production latency?
   Deploy the green pool behind a dedicated frontend that mirrors production traffic on a canary IP. Use HAProxy’s tcp-request content track-sc0 src to share stick‑tables between canary and production, then gradually increase the ACL match percentage.
4. What monitoring alerts should I set to catch weight drift early?
   Create an alert on the metric haproxy_server_weight{backend="blue_green",server=~".*green.*"} != 100 and on health‑check failure rates > 5 % for any server in the blue‑green backend.
5. Is there a built‑in HAProxy feature to automatically rebalance weights during a rollout?
   HAProxy 2.8 introduced dynamic-weight via the Runtime API, but it requires an external controller (e.g., Consul or a custom script) to compute and apply balanced weights. It does not replace the need for explicit health‑check tuning.

Related Topic Hub: Distributed Systems Troubleshooting Hub