Inference Queue Backlog On EC2 During CI/CD Pipeline Execution

Q: Root Cause

The backlog originates from a mismatch between the burst traffic generated by the CI/CD pipeline and the static capacity of the inference fleet:

Q: 2. Increase request concurrency on each instance

Switch Gunicorn to use a worker count based on CPU cores (workers = 2 * CPU + 1) and enable async workers.

Q: 4. Throttle CI/CD‑generated inference bursts

Introduce a simple rate‑limiter in the pipeline using bash sleep or a token bucket implementation, ensuring no more than 300 requests per second are emitted.

In this article

Table of Contents

Problem

During CI/CD pipeline runs that automatically deploy new model versions, the inference service running on a fleet of Amazon EC2 instances experiences a rapid increase in request latency. The inference_queue length grows to several thousand entries, causing time‑outs and failed inference jobs. Typical symptoms observed in the pipeline logs are:


[2026-09-22T14:03:12Z] INFO  inference_worker: Queue size = 842
[2026-09-22T14:03:13Z] WARN  inference_worker: Request timeout after 30s (job_id=42f7c1)
[2026-09-22T14:03:14Z] ERROR inference_worker: Failed to process job_id=42f7c1 (queue depth=1024)
...
[2026-09-22T14:05:01Z] INFO  pipeline: Deploy step completed, triggering 1200 inference requests

Impact includes:

Increased end‑to‑end pipeline duration (from ~5 min to >20 min).
Failed integration tests that depend on successful inference.
Transient spikes in CPU and memory usage on the EC2 instances.

Root Cause

The backlog originates from a mismatch between the burst traffic generated by the CI/CD pipeline and the static capacity of the inference fleet:

The pipeline launches n parallel inference jobs immediately after a model is deployed, ignoring the current processing rate of the service.
EC2 instances are provisioned with a fixed t2.large (2 vCPU, 8 GiB) size and a static DesiredCapacity=3 Auto Scaling Group (ASG). No scaling policies are attached, so the fleet cannot grow to absorb the burst.
The inference server uses a single‑threaded uvicorn worker behind gunicorn with workers=1. Even though the instance has two vCPUs, only one core is utilized for request handling, limiting throughput to ~150 RPS.
Load balancing is performed by an Application Load Balancer (ALB) with a target_group health check interval of 30 s. During the burst, health checks start failing, causing the ALB to route traffic to unhealthy targets, further worsening the queue.

In summary, the system assumes a steady‑state load while the CI/CD pipeline creates a short‑lived high‑frequency load spike that exceeds both compute capacity and request‑handling concurrency, leading to a growing queue.

Debug

Step‑by‑step investigation performed in the affected environment:

Inspect ALB target health

aws elbv2 describe-target-health \
    --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/inference-tg/abcd1234efgh


{
    "TargetHealthDescriptions": [
        {
            "Target": {"Id":"i-0a1b2c3d4e5f6g7h8","Port":8080},
            "TargetHealth": {"State":"unhealthy","Reason":"Target.ResponseCodeMismatch"}
        },
        ...
    ]
}

Collect EC2 metrics during a pipeline run

aws cloudwatch get-metric-statistics \
    --namespace AWS/EC2 \
    --metric-name CPUUtilization \
    --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f6g7h8 \
    --statistics Average \
    --period 60 \
    --start-time 2026-09-22T14:00:00Z \
    --end-time   2026-09-22T14:10:00Z


Timestamp                Average
2026-09-22T14:00:00Z     92.3
2026-09-22T14:01:00Z     95.1
2026-09-22T14:02:00Z     97.8
...

Examine inference server logs for queue growth

journalctl -u inference.service -f | grep Queue


Sep 22 14:02:10 ip-10-0-1-23 inference[1234]: INFO Queue size = 256
Sep 22 14:02:20 ip-10-0-1-23 inference[1234]: INFO Queue size = 512
Sep 22 14:02:30 ip-10-0-1-23 inference[1234]: WARN Queue size = 1024 (approaching limit)

Validate Gunicorn worker configuration

cat /etc/systemd/system/inference.service


[Service]
ExecStart=/usr/local/bin/gunicorn -w 1 -k uvicorn.workers.UvicornWorker app:app
...

Check CI/CD pipeline concurrency settings

cat .gitlab-ci.yml | grep parallel


parallel:
  matrix:
    - MODEL_VERSION: ["v1.2.0","v1.2.1","v1.2.2"]

The matrix expands to 3 parallel jobs, each spawning 400 inference requests, resulting in ~1200 concurrent requests.

Solution

The fix consists of three coordinated changes:

1. Enable horizontal scaling for the inference fleet

Add a target‑tracking scaling policy that adds instances when CPUUtilization exceeds 70 % for two consecutive minutes.

# Before (static ASG)
resource "aws_autoscaling_group" "inference" {
  name                      = "inference-asg"
  launch_configuration      = aws_launch_configuration.inference.id
  min_size                  = 3
  max_size                  = 3
  desired_capacity          = 3
}

# After (dynamic ASG with scaling policy)
resource "aws_autoscaling_group" "inference" {
  name                      = "inference-asg"
  launch_configuration      = aws_launch_configuration.inference.id
  min_size                  = 2
  max_size                  = 10
  desired_capacity          = 3

  tag {
    key                 = "Role"
    value               = "inference"
    propagate_at_launch = true
  }
}

resource "aws_autoscaling_policy" "cpu_target_tracking" {
  name                   = "cpu-target-tracking"
  autoscaling_group_name = aws_autoscaling_group.inference.name
  policy_type            = "TargetTrackingScaling"

  target_tracking_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ASGAverageCPUUtilization"
    }
    target_value = 70.0
    cooldown     = 300
  }
}

2. Increase request concurrency on each instance

Switch Gunicorn to use a worker count based on CPU cores (workers = 2 * CPU + 1) and enable async workers.

# Before (single sync worker)
ExecStart=/usr/local/bin/gunicorn -w 1 -k uvicorn.workers.UvicornWorker app:app

# After (dynamic async workers)
ExecStart=/usr/local/bin/gunicorn -w $(($(nproc) * 2 + 1)) -k uvicorn.workers.UvicornWorker --worker-class uvicorn.workers.UvicornH11Worker app:app

3. Adjust ALB health‑check and connection draining

Reduce health‑check interval to 10 s and enable connection draining of 30 s so that new instances become healthy faster and existing connections are gracefully closed.

# Before (default health check)
HealthCheckIntervalSeconds = 30
HealthyThresholdCount      = 3
UnhealthyThresholdCount    = 3

# After (tuned health check)
HealthCheckIntervalSeconds = 10
HealthyThresholdCount      = 2
UnhealthyThresholdCount    = 2
DeregistrationDelay.timeout_seconds = 30

4. Throttle CI/CD‑generated inference bursts

Introduce a simple rate‑limiter in the pipeline using bash sleep or a token bucket implementation, ensuring no more than 300 requests per second are emitted.

# Example snippet in .gitlab-ci.yml
script:
  - |
    for i in $(seq 1 1200); do
      curl -s -X POST http://inference.example.com/predict -d @payload.json &
      if (( i % 300 == 0 )); then
        wait   # allow current batch to finish
        sleep 2 # back‑off
      fi
    done

Verify

After applying the changes, perform the following validation steps:

Run a pipeline execution and capture the queue metric.

kubectl logs -l app=inference -c inference --tail=100 | grep Queue

Expected output shows the queue never exceeding 200 entries.

Check ALB target health during the burst.

aws elbv2 describe-target-health --target-group-arn ...

All targets should remain healthy throughout.

Confirm autoscaling activity in CloudWatch.

aws autoscaling describe-scaling-activities --auto-scaling-group-name inference-asg

Look for a Launch activity that adds at least one instance when CPU spikes.

Measure end‑to‑end pipeline duration.

Baseline after fix: ~6 min (vs. >20 min before).

Validate inference latency via a synthetic request.

time curl -s -X POST http://inference.example.com/predict -d @payload.json

Latency should stay under 200 ms for 95 % of requests.

Prevent

Operational safeguards to avoid recurrence:

Monitoring: Create a CloudWatch alarm on InferenceQueueLength (custom metric) > 300 for >2 min.
Alerting: Route the alarm to an SNS topic that pages the on‑call SRE.
Capacity Planning: Periodically run a load‑test (e.g., locust) simulating the maximum CI/CD burst and verify that CPUUtilization stays below 70 % with the current ASG limits.
CI/CD Guardrails: Enforce a pipeline lint rule that checks for a rate_limit variable before allowing inference stages.
Infrastructure as Code Review: Require peer review of any changes to ASG size, scaling policies, or ALB health‑check settings.

FAQ

Why does the queue grow only during CI/CD runs and not in production?
The production traffic is steady and well below the per‑instance throughput. CI/CD triggers a sudden burst of hundreds of concurrent requests, overwhelming the static fleet.
Can I keep the single‑worker Gunicorn configuration and just add more instances?
Adding instances helps, but each instance will still process ~150 RPS. For large bursts you will need both more instances and higher per‑instance concurrency to keep latency low.
Is using t2.large cost‑effective for inference workloads?
For CPU‑bound inference, a compute‑optimized family (e.g., c5.large) provides better price‑performance. Consider migrating after validating model compatibility.
How do I know the optimal number of Gunicorn workers?
Use the formula workers = 2 * CPU + 1 for sync workers. For async workers (Uvicorn), start with workers = CPU and adjust based on observed CPU utilization and request latency.
What if the pipeline still needs to run more than 10 minutes?
Introduce a dedicated “staging” inference cluster with a higher max_size (e.g., 20 instances) that is only used during deployments, keeping the production fleet untouched.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

Haystack inference queue backlog during CI/CD pipeline execution

Inference queue backlog on EC2 during CI/CD pipeline execution