Kubernetes API Server Unreachable Behind HAProxy After Health‑check Timeout

Q: Root Cause Analysis

The HAProxy configuration was using option tcp-check with the default timeout check 5s. The Kubernetes API server’s /healthz endpoint, when TLS is terminated at the API server, can take up to 8 seconds to respond under load (e.g., during model loading). When the health‑check timeout expires, HAProxy marks the backend as down, producing the 502 responses shown above.

In this article

Table of Contents

Kubernetes API server unreachable behind HAProxy after health‑check timeout

Problem Description (Symptoms and Impact)

During a routine AI inference workload deployment, the control‑plane components (controller‑manager, scheduler, kubelet) began reporting:


Failed to connect to apiserver: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

HAProxy returned 502 errors to any client trying to reach https://k8s‑lb.example.com:


SC-- 502 0/0/0 0/0 0

Systemd journal for HAProxy showed:


WARNING: backend api-server: server api-server1 is down, reason: Layer4 connection timeout

Resulting impact:

AI workload controllers (custom controllers, model‑loader operators) could not watch resources.
New inference pods failed to start because the scheduler could not place them.
Existing inference services experienced temporary outages during the health‑check storm.

Root Cause Analysis

The HAProxy configuration was using option tcp-check with the default timeout check 5s. The Kubernetes API server’s /healthz endpoint, when TLS is terminated at the API server, can take up to 8 seconds to respond under load (e.g., during model loading). When the health‑check timeout expires, HAProxy marks the backend as down, producing the 502 responses shown above.

Additional factors observed in the incident logs:

HAProxy performed SSL health checks without the check-ssl flag, leading to handshake failures for self‑signed certificates (see HAProxy SSL/TLS termination guide).
The fall count defaulted to 3, so three consecutive timeouts caused the server to be marked down, creating a 30‑second outage (edge AI deployment incident).
Health‑check interval of 2 seconds (default) generated a “health‑check storm” during a brief API server pause, as described in the Reddit r/kubernetes thread (2023‑09‑15).

In short, the mismatch between HAProxy’s Layer‑4/TCP health‑check timing and the API server’s TLS‑enabled response latency caused HAProxy to erroneously consider the API server unhealthy.

Investigation and Debugging Steps

Inspect HAProxy logs for health‑check failures.


$ journalctl -u haproxy -f | grep "Health check failed"
2024-04-12T10:15:23.123Z haproxy[1234]: Health check failed for backend api-server: SSL handshake failed
2024-04-12T10:15:25.456Z haproxy[1234]: SC-- 502 0/0/0 0/0 0

Verify the API server health endpoint directly.
```
$ curl -k https://10.0.1.10:6443/healthz
ok
```
If the request takes >5 seconds, HAProxy’s timeout check will expire.
Capture the TCP handshake to see if HAProxy is attempting an SSL health‑check without proper flags.
```
$ sudo tcpdump -i eth0 -nn -s 0 -w haproxy-health.pcap host 10.0.1.10 and port 6443
```
Analysis with Wireshark shows a TLS ClientHello from HAProxy followed by an immediate RST, confirming the missing check-ssl flag.

Check HAProxy timeout settings.


$ haproxy -c -f /etc/haproxy/haproxy.cfg
# Look for:
#   timeout check 5s
#   timeout connect 5s
#   timeout server 30s

Review Kubernetes documentation on load‑balancer health checks. The recommended endpoint is /healthz with a timeout of at least 10 seconds for TLS‑enabled backends (Kubernetes load‑balancer docs).

Resolution (Configuration Changes)

Update the HAProxy backend definition to use an HTTP health‑check against /healthz, enable TLS verification, and increase the health‑check timeout. Below is a before/after comparison.

Before


backend api-server
    mode tcp
    balance roundrobin
    option tcp-check
    timeout check 5s
    server api-server1 10.0.1.10:6443 check
    server api-server2 10.0.1.11:6443 check

After


backend api-server
    mode tcp
    balance roundrobin
    # Use HTTP health‑check over TLS
    option httpchk GET /healthz HTTP/1.0
    http-check expect status 200
    # Verify TLS but ignore cert validation (self‑signed)
    check-ssl verify none
    timeout check 12s          # > max /healthz response time
    timeout connect 5s
    timeout server 30s
    # Reduce fall count to avoid flapping during brief pauses
    fall 2
    server api-server1 10.0.1.10:6443 check ssl verify none
    server api-server2 10.0.1.11:6443 check ssl verify none

Explanation of key changes:

option httpchk GET /healthz HTTP/1.0 forces HAProxy to issue an HTTP GET, matching the API server’s health endpoint (HAProxy Configuration Manual – option tcp-check).
check-ssl verify none allows HAProxy to complete the TLS handshake without needing a trusted CA (HAProxy SSL/TLS termination guide).
Increasing timeout check to 12 seconds accommodates the worst‑case response time observed during model loading spikes.
Setting fall 2 reduces the chance of a temporary pause causing a full outage.

Verification (Validation Steps)

Reload HAProxy with the new configuration and confirm no syntax errors:


$ haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
$ systemctl reload haproxy

Observe HAProxy health‑check status:


$ echo "show stat" | socat stdio /var/run/haproxy.sock | grep api-server
api-server,api-server1,0,0,0,UP,0,0,0,0,0,0,0,0,0,0,0,0,0

Check that API server requests succeed:


$ curl -k https://k8s-lb.example.com/api/v1/nodes
{
  "items": [...]
}

Confirm controller‑manager logs no longer show connection timeouts:


$ journalctl -u kube-controller-manager -f | grep "Failed to connect"
# No output expected

Operational Experience (Lessons Learned)

Misleading symptom: The 502 error suggested a downstream application failure, but the root cause was the load balancer’s health‑check timing.
Assumption trap: Assuming the default TCP health‑check is sufficient for a TLS‑terminated API server leads to handshake failures (see GitHub issue haproxy/haproxy#2261).
Production edge case: During a spike in model loading, the API server’s /healthz response time increased to ~8 seconds, exceeding the default 5‑second HAProxy timeout (Production AI inference platform incident, 2024 Q1).
Monitoring tip: Enable HAProxy’s option httplog and export health‑check metrics to Prometheus; alert when backend_up drops below 1 for the API server.

Best Practices and Prevention

Use option httpchk GET /healthz with http-check expect status 200 for Kubernetes API servers behind HAProxy.
Set timeout check to at least twice the maximum observed /healthz latency.
When terminating TLS at the API server, add check-ssl verify none (or provide a proper CA bundle) to avoid handshake failures.
Configure fall and rise values to balance sensitivity and stability; a common pattern is fall 2, rise 3.
Export HAProxy health‑check counters (e.g., haproxy_backend_up, haproxy_server_check_fail) to a monitoring system and create alerts for sudden drops.
Periodically run a synthetic request (e.g., curl -k https://k8s-lb.example.com/healthz) from a bastion host to verify end‑to‑end connectivity.

FAQ (Related Questions)

Why does the API server become unreachable only after a workload spike?
The spike increases /healthz response latency. If HAProxy’s timeout check is shorter, health checks time out and the backend is marked down.
Can I keep TLS termination at HAProxy instead of the API server?
Yes. In that case configure HAProxy to terminate TLS and use plain TCP health checks, or use option ssl-hello-chk with a proper certificate chain.
What is the difference between option tcp-check and option httpchk for the API server?
tcp-check only verifies that a TCP connection can be established; it does not validate the HTTP health endpoint, which may be delayed by TLS handshake or application processing. httpchk performs an actual HTTP request to /healthz, providing a more accurate readiness signal.
How do I debug SSL health‑check failures in HAProxy?
Enable option ssl-hello-chk or add check-ssl verify none and inspect HAProxy logs for “SSL handshake failed”. Capturing traffic with tcpdump can also reveal missing client hello messages.
Should I increase the fall count for production clusters?
Increasing fall reduces flapping but may delay detection of genuine failures. A common practice is fall 2 and rise 3, which balances responsiveness with stability for high‑traffic API servers.

Related Topic Hub: Distributed Systems Troubleshooting Hub