a group of blue squares

Kubernetes inference queue backlog causing batch job timeouts

by

Problem

Batch processing jobs that invoke a TensorFlow Serving inference service on a Kubernetes cluster are experiencing timeouts. The underlying symptom is a steadily growing inference request queue inside the service pod, eventually causing the kubectl logs output to show a backlog of pending requests and the batch job to fail with deadline exceeded errors.

Root Cause

The backlog is caused by a mismatch between the concurrency limits of the inference pod and the parallelism of the batch job. Specifically:

  • The inference service is configured with --max-concurrent-requests=8 (default for the Docker image) while the model uses a GPU that can handle only 4 simultaneous in‑process inference calls.
  • The batch job creates a Job with parallelism: 20 and each pod launches 10 curl workers that continuously POST to the inference endpoint.
  • Kubernetes Service load‑balancing (ClusterIP) distributes traffic round‑robin across the single replica of the inference pod, but the pod cannot keep up, so the internal request queue (implemented by gRPC server’s grpc::ServerBuilder::SetMaxReceiveMessageSize) grows unchecked.
  • Because the pod does not expose back‑pressure (HTTP 429) and the client code retries with exponential back‑off, the retry storm further inflates the queue.

The combination of over‑provisioned client concurrency and under‑provisioned inference capacity creates a self‑reinforcing backlog that eventually exhausts the pod’s memory, triggers OOM kills, and forces the batch job to timeout.

Debug

1. Observe queue length in pod logs


2026-10-28T14:12:03.421Z INFO  request_queue.go:45] Queue size: 124 (max: 200)
2026-10-28T14:12:03.422Z INFO  request_queue.go:45] Queue size: 126 (max: 200)
...
2026-10-28T14:12:10.001Z WARN  request_queue.go:78] Queue overflow, dropping request

2. Inspect Job pod logs for retry patterns


2026-10-28T14:12:04.112Z ERROR worker.go:102] POST https://inference-svc:8500/v1/models/my-model:deadline_exceeded
2026-10-28T14:12:04.113Z INFO  retry.go:57] retrying in 2.3s (attempt 3/5)

3. Verify Service and Deployment configuration


# inference deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inference
  template:
    metadata:
      labels:
        app: inference
    spec:
      containers:
      - name: tf-serving
        image: tensorflow/serving:2.12.0
        args:
        - --model_name=my-model
        - --model_base_path=/models/my-model
        - --port=8500
        - --rest_api_port=8501
        - --max_concurrent_requests=8   # <-- default
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            cpu: "4"
            memory: "8Gi"

4. Check Job parallelism


apiVersion: batch/v1
kind: Job
metadata:
  name: batch-inference
spec:
  parallelism: 20
  completions: 200
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: myorg/batch-worker:latest
        command: ["./run-inference.sh"]

5. Metrics inspection

Using Prometheus, query the request queue gauge:


inference_queue_size{pod="inference-deployment-abc123"}

Values consistently above 100 indicated sustained overload.

Solution

1. Align concurrency limits with GPU capacity

Set --max_concurrent_requests to a value that matches the GPU's optimal throughput (4 in this case) and enable back‑pressure via --grpc_max_concurrent_streams to return HTTP 429 when the queue exceeds a safe threshold.

Before:

- --max_concurrent_requests=8

After:

- --max_concurrent_requests=4
- --grpc_max_concurrent_streams=4
- --enable_batching=true
- --batch_timeout_micros=5000

2. Scale inference service horizontally

Deploy two replicas behind a ClusterIP Service with session‑affinity disabled, allowing the load balancer to spread traffic.

Before (replicas: 1):

spec:
  replicas: 1

After (replicas: 2):

spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

3. Reduce Job parallelism and add client‑side throttling

Adjust the batch Job to launch at most 8 workers per pod and introduce a token‑bucket limiter in run-inference.sh to cap outbound request rate.

Before (parallelism: 20):

spec:
  parallelism: 20

After (parallelism: 8):

spec:
  parallelism: 8

4. Implement exponential back‑off with jitter and respect HTTP 429

Update the client library to stop retrying immediately after a 429 response and to back‑off for at least the Retry-After header.

Verify

  1. Deploy the updated inference Deployment and Job manifests.
  2. Monitor the queue size metric for at least 10 minutes:

inference_queue_size{pod=~"inference-deployment-.*"} == 0

The gauge should stay below 20, well under the max_concurrent_requests limit.

  1. Check Job pod logs for successful responses:

2026-10-28T15:02:01.332Z INFO  worker.go:45] inference succeeded (latency: 42ms)
  1. Confirm that no Job pod exits with deadline exceeded errors.
  2. Validate overall batch completion time meets SLA (e.g., < 30 minutes for 200 items).

Prevent

  • Capacity planning: Benchmark the model on the target GPU and set max_concurrent_requests accordingly.
  • Horizontal pod autoscaling (HPA): Enable CPU/GPU‑based HPA on the inference Deployment to add replicas when inference_queue_size exceeds a threshold.
  • Back‑pressure handling: Configure the inference server to emit HTTP 429 and ensure client libraries honor it.
  • Job throttling: Use a ConfigMap to expose a MAX_REQUESTS_PER_SECOND value that can be tuned without redeploying the worker image.
  • Observability: Create a Prometheus alert on inference_queue_size > 0.8 * max_concurrent_requests and on Job pod failures with deadline_exceeded.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does increasing the number of inference pod replicas sometimes make the backlog worse?

    When the Service uses round‑robin without session affinity, each new replica receives the same traffic pattern, but if the client side continues to overload each pod (e.g., by opening many connections per pod), the total request rate may exceed the combined capacity, leading to a larger aggregate queue.

  2. Can I keep max_concurrent_requests high and rely on GPU multiplexing?

    TensorFlow Serving does not multiplex GPU kernels efficiently beyond the hardware's native concurrency. Over‑provisioning the request limit typically results in kernel queueing inside the driver, higher latency, and OOM errors.

  3. How do I expose the internal request queue size to Prometheus?

    Add the --monitoring_config_file flag pointing to a YAML that enables the tensorflow_serving.metrics collector, then scrape the /metrics endpoint of the container.

  4. What is the recommended back‑off strategy for handling HTTP 429 from the inference service?

    Use exponential back‑off with jitter, start with 500 ms, double on each retry, and respect the Retry-After header if present. Cap the maximum delay at 10 seconds to avoid excessive latency.

  5. Is it safe to set --enable_batching=true on a GPU‑bound model?

    Batching can improve throughput if the model supports it and the batch size is tuned to the GPU memory. However, it adds latency for individual requests; test with realistic batch sizes before enabling in production.