HAProxy rate limit exceeded 429 during local development

Problem – HAProxy returns HTTP 429 “Too Many Requests” during local development

Developers running a local HAProxy instance as a reverse proxy for several micro‑services observe intermittent 429 Too Many Requests responses. The errors appear even though traffic is generated only by integration tests, IDE live‑share sessions, or manual curl loops. Typical log lines look like:

[WARN] sc0_rate exceeded for src 127.0.0.1 on table ip_rate
[INFO] 127.0.0.1:54321 [06/Jun/2026:14:23:11.123] http-in~ test_service/0/0/0/30 200 1234 - - ---- 1/1/0/0/0 0/0 "GET /api/v1/resource HTTP/1.1"
[WARN] http-request deny status 429 if { sc0_http_req_rate(60) gt 100 }

The symptom manifests as:

• Sudden bursts of 429 for otherwise healthy endpoints.
• Log entries containing status=429 and sc0_rate exceeded.
• No corresponding failures in the backend services themselves.

Root Cause Analysis

HAProxy implements rate limiting using stick‑tables together with http-request track-sc* and conditional http-request deny status 429 rules (see HAProxy Configuration Manual §4.2 and §9.3). In the development setup the configuration typically resembles:

frontend http-in
    bind *:80
    default_backend services

    # Track per‑IP request rate over a 60‑second window
    stick-table type ip size 1m expire 5m store http_req_rate(60)

    http-request track-sc0 src
    http-request deny status 429 if { sc0_http_req_rate(60) gt 100 }

Key points that lead to the 429 spikes:

1. Burst traffic is counted per connection. When developers run scripts that open many short‑lived connections (e.g., hey with keep‑alive disabled), each TCP connection increments the http_req_rate counter, quickly exceeding the gt 100 threshold.
2. Localhost IP aggregation. All services share the same 127.0.0.1 source address, so requests from different test suites, VS Code Live Share, or CI pipelines aggregate into a single stick‑table entry. This amplifies the apparent per‑IP rate.
3. Default thresholds are tuned for production traffic. The example threshold of 100 requests per 60 seconds (≈1.6 rps) is appropriate for production but far too low for parallel test execution that can generate 200 rps (see real incident “Local CI pipeline runs parallel curl loops…”).
4. Health‑check spikes during service restarts. When Docker‑compose restarts a container, HAProxy’s health‑check requests accumulate, temporarily inflating the rate counter and triggering the deny rule (see “Docker‑compose environment restarts services…”).

In short, the http-request deny … if { sc0_http_req_rate(60) gt 100 } rule is firing because the stick‑table’s per‑IP rate limit does not reflect the characteristics of a local development workload.

Investigation and Debugging

1. Verify the stick‑table contents

Use the HAProxy runtime socket to dump the table while the issue reproduces:

echo "show table ip_rate" | socat stdio /var/run/haproxy.sock

Typical output during a burst:

# ip_rate
  key         rate  used  expire  last_seen
  127.0.0.1   215   1     300s    2026-06-29 14:23:10.000

The rate column shows the request rate over the configured window (60 s). Values > 100 confirm the rule is being triggered.

2. Correlate logs with table dumps

Search the HAProxy access log for the warning pattern:

grep "sc0_rate exceeded" /var/log/haproxy.log

Sample log entry (from the evidence package):

[WARN] sc0_rate exceeded for src 127.0.0.1 on table ip_rate

3. Identify traffic sources

List active connections to HAProxy:

ss -tnp | grep ':80' | awk '{print $5}' | cut -d':' -f1 | sort | uniq -c

This reveals that most connections originate from 127.0.0.1, confirming the aggregation problem.

4. Check health‑check configuration

If health‑checks are frequent, they contribute to the rate. Verify the health‑check interval in the backend definition:

backend services
    option httpchk GET /healthz
    http-check expect status 200
    # default check interval is 2s

5. Review the official documentation

HAProxy’s manual sections provide the exact syntax used:

• §9.3 “http-request deny status 429” – how the 429 response is generated.
• §4.2 “stick-table and http-request track-sc*” – how request counters are stored.
• §9.2 “http-request deny with track-sc0 and sc_http_req_rate” – threshold definition.
• §5 “Logging and Monitoring” – fields like status=429 and sc0_rate in logs.

Resolution – Adjusting Rate‑Limit Configuration for Development

Before (problematic configuration)

frontend http-in
    bind *:80
    default_backend services

    stick-table type ip size 1m expire 5m store http_req_rate(60)
    http-request track-sc0 src
    http-request deny status 429 if { sc0_http_req_rate(60) gt 100 }

After (development‑friendly configuration)

frontend http-in
    bind *:80
    default_backend services

    # Increase table size for many short‑lived connections
    stick-table type ip size 10m expire 5m store http_req_rate(60),conn_rate(60)

    # Track both request rate and connection rate
    http-request track-sc0 src
    http-request track-sc1 src

    # Raise the request‑rate threshold for localhost
    http-request deny status 429 if { src 127.0.0.1 } { sc0_http_req_rate(60) gt 500 }
    # Keep a stricter limit for external IPs
    http-request deny status 429 if { src -f /etc/haproxy/blocked_ips.lst } { sc0_http_req_rate(60) gt 100 }

    # Optional: use a separate table for internal testing
    stick-table type ip size 1m expire 5m store http_req_rate(60) name dev_rate
    http-request track-sc2 src table dev_rate if { src 127.0.0.1 }
    http-request deny status 429 if { sc2_http_req_rate(60) gt 1000 }

Key changes explained:

• Higher threshold for localhost – the rule now allows up to 500 rps for 127.0.0.1, covering typical CI bursts.
• Separate stick‑table for dev traffic – isolates internal testing from production‑like limits.
• Increased table size – prevents “stick‑table overflow” warnings when many connections are opened rapidly.
• Conditional deny based on source IP list – preserves stricter limits for external callers.

Validation – Confirming the Fix Works

1. Reload HAProxy with the new config:

haproxy -f /etc/haproxy/haproxy.cfg -sf $(pidof haproxy)

2. Re‑run the integration test that previously caused 429s (e.g., a hey run with 200 rps for 30 s).

hey -c 50 -z 30s http://localhost/api/v1/resource

Expected outcome: No 429 responses; the summary shows a success rate of 100 %.

3. Inspect the stick‑table while the test runs:

echo "show table ip_rate" | socat stdio /var/run/haproxy.sock

Rate should stay below the new threshold (e.g., rate=420 for localhost, well under gt 500).

4. Check the HAProxy log for absence of “sc0_rate exceeded” warnings.

grep "429" /var/log/haproxy.log

Result: No lines returned, confirming that the deny rule was not triggered.

Prevention – Best Practices for Rate Limiting in Development Environments

• Separate stick‑tables per environment – use distinct tables (e.g., dev_rate, prod_rate) and bind them to different frontends.
• Parameterize thresholds – store limits in environment variables or separate files so CI can override them without modifying production config.
• Monitor stick‑table health – set up a periodic show table check in a cron job and alert on “overflow” or “rate exceeded” messages.
• Prefer connection‑rate limits for bursty scripts – conn_rate(60) captures rapid connection creation better than http_req_rate when keep‑alive is disabled.
• Exclude localhost from strict limits – add explicit conditions for src 127.0.0.1 or src ::1 in deny rules.
• Document health‑check frequency – align health‑check intervals with rate‑limit windows to avoid accidental spikes during restarts.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does HAProxy return 429 only when I run tests locally?
   Because all test traffic originates from the same IP (localhost), aggregating into a single stick‑table entry that quickly exceeds the per‑IP rate limit configured for production.
2. Can I keep the production rate limits and still avoid 429 in dev?
   Yes. Define a separate stick‑table or conditional rule that raises the threshold for 127.0.0.1 (or any IP range used for testing) while leaving production limits untouched.
3. How do I know which stick‑table entry triggered the deny?
   HAProxy logs include the warning “sc0_rate exceeded for src on table ”. Use the runtime socket command show table <name> to inspect the exact counters.
4. My health checks are causing 429 spikes after a container restart. What should I do?
   Increase the health‑check interval or exclude health‑check traffic from rate limiting by adding a condition such as if { req.hdr(User-Agent) -i haproxy-healthcheck } deny status 200 before the rate‑limit rule.
5. Is there a way to automatically adjust limits based on the environment?
   Yes. Use HAProxy’s env directive to import environment variables and reference them in the configuration, e.g., http-request deny status 429 if { sc0_http_req_rate(60) gt %{env:DEV_RATE_LIMIT} }.