Intermittent Redis Pub/Sub streaming interruptions in AI inference staging

Problem: Intermittent Redis Pub/Sub Streaming Interruptions in AI Inference Staging

The AI inference service streams tokenized responses to clients via a Redis Pub/Sub channel. In the staging environment the stream is occasionally truncated, causing the client to receive incomplete answers or a sudden “connection closed” error. The symptom manifests as:

• Log entry: socket.timeout: timed out after a few seconds of silence.
• Client error: ERR max number of clients reached during load spikes.
• Occasional TLS handshake failures: TLS handshake timed out in the Redis TLS termination proxy logs.
• Redis server log: connection reset by peer followed by “PUBSUB failed: connection closed”.

The issue is isolated to the staging environment; production, which uses a similar Redis cluster, streams without interruption.

Root Cause Analysis

1. Connection Lifecycle and Idle Timeouts

Redis Pub/Sub connections remain open for the entire lifetime of the subscription (Redis Pub/Sub documentation). If the client does not receive data for a period longer than the network idle timeout, the firewall or the TLS termination proxy silently closes the TCP socket. In staging a firewall enforces a 300‑second idle TCP timeout, which matches the longest pause between token chunks during large model responses.

2. Client‑side Keep‑Alive and Reconnection Settings

Most Redis client libraries (redis‑py, node‑redis, Redisson) default to no heartbeat on Pub/Sub sockets. Community reports (redis‑py issue, node‑redis issue) show that without explicit keep‑alive the idle timeout triggers a drop.

3. Resource Limits on the Cluster

The staging cluster runs with maxclients 1000. During model warm‑up the number of concurrent inference requests spikes, causing the server to reject new Pub/Sub connections (ERR max number of clients reached). Existing subscriber sockets are then forced to close when the server evicts idle connections to stay under the limit.

4. Memory Pressure and Eviction

When maxmemory-policy allkeys-lru is active, Redis may evict buffers that are still referenced by in‑flight Pub/Sub messages (persistence and eviction docs). This can lead to silent message loss, especially during model warm‑up when memory usage spikes.

5. TLS Proxy Renegotiation Timeout

The TLS termination proxy resets connections after a renegotiation timeout, producing the “TLS handshake timed out” error observed in the Redis Labs forum thread (forum thread).

Investigation and Debugging Steps

Step 1 – Capture Connection Lifecycle Logs

# Example of client log snippet (node‑redis)
2026-08-25T12:34:56.789Z WARN  PubSub connection lost: connection reset by peer
2026-08-25T12:35:01.002Z INFO  Reconnecting to redis://staging-redis:6379 (attempt 1)

Step 2 – Verify Firewall Idle Timeout

# On the firewall (e.g., iptables with idle timeout module)
iptables -L -v -n | grep idle_timeout
# Expected output
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 tcp timeout 300s

Step 3 – Inspect Redis Server Metrics

# Using redis-cli INFO
redis-cli -h staging-redis -p 6379 INFO clients
# Sample output
connected_clients:985
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

Step 4 – Check Memory Usage and Eviction Events

# Enable keyspace notifications for evictions
redis-cli CONFIG SET notify-keyspace-events Ex
redis-cli SUBSCRIBE __keyevent@0__:evicted
# Observe eviction logs during warm‑up

Step 5 – Test TLS Handshake Stability

# OpenSSL s_client with debug
openssl s_client -connect staging-redis:6380 -tls1_2 -servername staging-redis -debug
# Look for "handshake failure" or "renegotiation timeout"

Step 6 – Simulate Load Spike

# Simple load generator
for i in $(seq 1 2000); do
  redis-cli -h staging-redis -p 6379 PUBLISH inference:token "token-$i"
done

Solution: Harden Pub/Sub Streaming in Staging

1. Enable TCP Keep‑Alive and Client Heartbeats

Configure the Redis client to send periodic ping frames. Example for node-redis (v4):

// Before (default)
const client = createClient({ url: 'redis://staging-redis:6379' });

// After – enable keepAlive and ping interval
const client = createClient({
  url: 'redis://staging-redis:6379',
  socket: {
    keepAlive: true,
    keepAliveInitialDelay: 10000, // 10 s
    reconnectStrategy: retries => Math.min(retries * 100, 3000)
  }
});
client.on('error', err => console.error('Redis error', err));
await client.connect();

2. Increase Firewall Idle Timeout or Disable It for Redis Ports

Adjust the firewall rule to a value larger than the maximum expected pause (e.g., 900 s) or add an exception for the Redis port.

# iptables example
iptables -R INPUT 3 -p tcp --dport 6379 -j ACCEPT -m conntrack --ctstate ESTABLISHED,RELATED -m timeout --timeout 900

3. Raise maxclients and Monitor Connection Usage

Set maxclients to a comfortable headroom (e.g., 5000) and add a Prometheus alert when connected_clients exceeds 80 % of the limit.

# redis.conf
maxclients 5000

4. Switch to Redis Streams for Reliable Delivery (Optional)

If occasional message loss is unacceptable, replace Pub/Sub with a consumer‑group based Stream. Streams persist messages and survive reconnects (Redis Streams documentation).

// Producer (Python)
import redis
r = redis.Redis(host='staging-redis', port=6379)
r.xadd('inference:stream', {'token': 'token-123'}, maxlen=1000, approximate=True)

# Consumer (Node.js)
const { createClient } = require('redis');
const client = createClient({ url: 'redis://staging-redis:6379' });
await client.connect();
const group = 'inference';
await client.xgroupCreate('inference:stream', group, '$', { MKSTREAM: true });
while (true) {
  const msgs = await client.xreadGroup(group, 'consumer-1', { key: 'inference:stream', id: '>' }, { COUNT: 10, BLOCK: 5000 });
  // Process tokens
}

5. Adjust TLS Proxy Settings

Increase the renegotiation timeout or disable renegotiation for the Redis endpoint.

# Example for HAProxy
frontend redis_tls
    bind *:6380 ssl crt /etc/ssl/redis.pem
    timeout client 10m
    timeout server 10m
    # Disable renegotiation
    ssl-reuse

Verification: Confirm the Fix Works

1. Run a long‑running inference request that yields >5 minutes of token streaming.
2. Monitor client logs for absence of socket.timeout or connection reset by peer.
3. Validate that redis-cli INFO clients shows stable connected_clients below the new threshold.
4. Check firewall counters to ensure no idle‑timeout drops:

# iptables -L -v -n | grep 6379
# Expected: no packets dropped due to timeout

Optionally, enable a health‑check endpoint that subscribes to a test channel and verifies receipt of a known message every 30 seconds.

Prevention and Operational Guardrails

• Monitoring: Export connected_clients, evicted_keys, and TLS handshake error counters to Prometheus. Alert on spikes.
• Connection Hygiene: Enforce client‑side reconnection back‑off and automatic resubscription (e.g., autoResubscribe flag in redis‑py).
• Capacity Planning: Periodically run load tests that simulate peak inference concurrency to verify maxclients headroom.
• Configuration Audits: Keep staging and production Redis configurations in version‑controlled files; diff them before promotion.
• Network Stability: Verify MTU consistency across Docker overlay networks to avoid fragmented packets that trigger ERR Protocol error.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why does the stream only break after a few minutes of inactivity?
   Because the staging firewall enforces a 300‑second idle TCP timeout. When no token is sent during that window the socket is closed, causing the client to see a timeout.
2. Can increasing maxclients cause other issues?
   A higher limit consumes more file descriptors on the Redis process. Ensure the OS ulimit -n is raised accordingly (e.g., 100 000) before increasing maxclients.
3. Do Redis Streams guarantee ordering compared to Pub/Sub?
   Yes. Streams preserve insertion order and support consumer groups, which also survive client reconnects, eliminating the “message loss after reconnect” problem described in the Stack Overflow thread.
4. What client settings should I use to automatically resubscribe after a reconnect?
   In redis‑py set auto_resubscribe=True. In node‑redis enable disableOfflineQueue: false and provide a reconnectStrategy. Redisson offers retryAttempts and retryInterval configuration.
5. How can I detect TLS handshake timeouts before they affect the stream?
   Enable debug logging on the TLS proxy and monitor for “TLS handshake timed out” messages. Additionally, configure the client’s tlsHandshakeTimeout (if supported) to a lower value and alert on failures.