Grafana inference queue backlog after microservice scaling surge

Problem – Inference Queue Backlog After Microservice Scaling Surge

During a sudden traffic spike, the microservices that generate model inference results were scaled out from 4 to 20 replicas. After the scale‑up, Grafana dashboards that display inference latency and success rates became stale: metric points were delayed by 5–15 minutes, and alert rules based on rate(inference_success[1m]) stopped firing. The underlying symptom was a growing backlog in the central inference queue (e.g., a Redis list or Kafka topic) that feeds Grafana’s prometheus_remote_write exporter.

Root Cause – Why the Queue Filled Up

The backlog originated from a combination of three interacting factors:

  1. Producer overload: Each new microservice replica opened a persistent TCP connection to the queue broker and began publishing inference results at a rate that exceeded the broker’s configured max.inflight.messages (default 1000). The broker started throttling writes, causing the client library to buffer messages locally.
  2. Consumer throttling: The Grafana data source relied on a single prometheus_remote_write worker that pulled batches from the queue every 30 seconds. After scaling, the producer rate increased ~5×, but the consumer’s batch size and pull interval were unchanged, creating a supply‑demand mismatch.
  3. Back‑pressure misconfiguration: The queue’s maxmemory-policy was set to noeviction. When memory usage approached the limit, the broker started rejecting new writes with ERR maxmemory limit reached, which the client logged as “write failed, retrying”. Retries accumulated, further inflating the backlog.

In short, the scaling event amplified the producer side without a corresponding increase in consumer capacity or proper back‑pressure handling, leading to a persistent queue backlog and stale Grafana visualizations.

Debug – Investigation Steps

1. Observe Queue Metrics

# Redis example
redis-cli INFO stats | grep instantaneous_ops_per_sec
redis-cli LLEN inference_queue
redis-cli INFO memory | grep used_memory_human

Typical output during the backlog:

instantaneous_ops_per_sec:12000
127   # length of inference_queue
used_memory_human:1.9GiB

2. Inspect Producer Logs

2026-11-14T10:23:41.112Z WARN  inference-producer-7
    Write failed: ERR maxmemory limit reached, retrying (attempt 3)
2026-11-14T10:23:41.115Z INFO  inference-producer-7
    Buffered messages: 842

3. Check Consumer (Grafana Remote Write) Logs

2026-11-14T10:24:02.004Z INFO  remote_write_worker
    Pulling 200 messages from inference_queue
2026-11-14T10:24:02.056Z WARN  remote_write_worker
    Batch send failed: 429 Too Many Requests
2026-11-14T10:24:02.056Z INFO  remote_write_worker
    Backoff for 10s

4. Verify Broker Configuration

# Redis config snippet
maxmemory 2gb
maxmemory-policy noeviction
maxclients 10000

5. Correlate Scaling Events

Using Kubernetes events:

kubectl get events --field-selector involvedObject.kind=Deployment,involvedObject.name=inference-producer -n ml

Shows a burst of ScalingReplicaSet events at 10:15 UTC, matching the queue length spike.

Solution – Eliminating the Backlog

1. Increase Consumer Throughput

Adjust the remote‑write worker to pull larger batches more frequently.

# Before (remote_write.yaml)
batch_size: 200
pull_interval: 30s
# After
batch_size: 800
pull_interval: 5s
max_concurrent_workers: 4

2. Enable Producer Back‑Pressure

Configure the client library to respect ERR maxmemory limit reached and pause publishing.

// Go client example
producer, _ := redis.NewClient(&redis.Options{
    Addr: "queue:6379",
    // Enable automatic back‑off
    MaxRetries: 5,
    MinRetryBackoff: 2 * time.Second,
    MaxRetryBackoff: 30 * time.Second,
})

3. Adjust Broker Memory Policy

Switch from noeviction to allkeys-lru so stale inference results are evicted when memory pressure occurs.

# redis.conf
maxmemory 4gb
maxmemory-policy allkeys-lru

4. Scale Consumer Horizontally

Deploy two additional remote_write_worker pods behind a Service, each with the updated batch settings.

# Kubernetes deployment snippet
apiVersion: apps/v1
kind: Deployment
metadata:
  name: remote-write-worker
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: worker
        image: grafana/remote-write:latest
        env:
        - name: BATCH_SIZE
          value: "800"
        - name: PULL_INTERVAL
          value: "5s"

5. Tune Producer Rate Limiting

Introduce a token‑bucket limiter in each microservice to cap the publish rate at 500 msg/s per replica.

// Python example using ratelimit
from ratelimit import limits, sleep_and_retry

@sleep_and_retry
@limits(calls=500, period=1)
def publish_inference(result):
    redis_client.rpush('inference_queue', json.dumps(result))

Verify – Confirming the Fix

  1. Monitor queue length:
  2. watch -n 5 "redis-cli LLEN inference_queue"
    

    Expected: steady value < 50.

  3. Check remote‑write worker logs for successful batch sends:
  4. kubectl logs -l app=remote-write-worker -c worker --tail=20
    2026-11-14T10:45:12.003Z INFO  remote_write_worker
        Pulled 800 messages, sent 800 to Prometheus (200ms)
    
  5. Validate Grafana dashboards:
  6. Metric inference_latency_seconds should now show sub‑second latency with no gaps.

  7. Confirm alert firing:
  8. curl -G 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=rate(inference_success[1m]) > 0.9'
    

    Response should contain a non‑zero vector.

Prevent – Operational Guardrails

  • Autoscaling policies: Couple microservice replica scaling with a HorizontalPodAutoscaler that also scales the remote‑write worker deployment based on queue_length custom metric.
  • Back‑pressure monitoring: Alert on redis_memory_used_percent > 80 and on producer log pattern “Write failed: ERR maxmemory”.
  • Queue health dashboard: Add a Grafana panel that charts redis_queue_length and remote_write_batch_success_rate side‑by‑side.
  • Rate limiting defaults: Embed a library‑wide limiter (e.g., 300 msg/s per instance) in the inference SDK to prevent runaway publishing.
  • Graceful shutdown hooks: Ensure pods flush their local buffers before termination to avoid sudden spikes.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

  1. Why did the backlog appear only after scaling? Scaling increased the aggregate publish rate while the consumer configuration remained static, creating a supply‑demand mismatch.
  2. Can I keep noeviction and still avoid the backlog? Only if you provision enough memory and increase the consumer throughput; otherwise the broker will reject writes, leading to the same symptom.
  3. Is increasing maxmemory sufficient? It postpones the problem but does not address the root cause of mismatched producer/consumer rates; the backlog will reappear under higher load.
  4. How do I determine the optimal batch size for the remote‑write worker? Benchmark by gradually increasing batch_size until the per‑batch latency plateaus; monitor Prometheus remote_write_batch_duration_seconds for diminishing returns.
  5. What if I cannot modify the producer code? Deploy a sidecar that rate‑limits outbound traffic to the queue or use a proxy (e.g., nginx with limit_req) to enforce back‑pressure.