Azure VM Inference Queue Backlog After Increasing API Request Load

Q: Debug – Investigation Steps 1. Collect VM‑level metrics

az monitor metrics list \ --resource /subscriptions//resourceGroups//providers/Microsoft.Compute/virtualMachineScaleSets/ \ --metric "Percentage CPU" "Disk Read Operations/Sec" "Network In Total" Expected output showed CPU consistently above 90 % and Disk Read Operations/Sec hitting the premium SSD limit.

Q: 2. Inspect inference service logs

journalctl -u tensorflow-serving -f # Sample excerpt 2024-06-27T14:12:03.421Z INFO ThreadPoolExecutor rejected execution: worker pool exhausted 2024-06-27T14:12:04.012Z WARN InferenceService timed out after 30000 ms 3. Verify queue length via internal metrics endpoint curl -s http://localhost:8501/metrics | grep tf_serving_queue_size # Example output tf_serving_queue_size{model_name="resnet50"} 124 4. Check autoscale rule evaluation az monitor autoscale rule list \ --resource-group \ --autoscale-name The rule showed a scale out threshold of CPU > 80 with a cooldown 5m, confirming the lag.

Q: 5. Capture network latency between API Gateway and VM subnet

tcpdump -i eth0 -nn host and port 8501 -c 5 -w /tmp/gateway.pcap Packet captures revealed occasional retransmissions, consistent with NSG throttling reported in the community discussion.

Q: Solution – Align Processing Capacity with Traffic Peaks 1. Resize VMs or switch to a higher‑core SKU

Upgrade from Standard_D2s_v3 (2 vCPU) to Standard_D8s_v3 (8 vCPU) to give TensorFlow Serving enough worker threads.

Q: 2. Refine autoscale rules to react to queue length

Enable a custom metric based on tf_serving_queue_size exported to Azure Monitor via the diagnostics extension.

Q: 3. Increase TensorFlow Serving thread pool

Adjust the --tensorflow_intra_op_parallelism and --tensorflow_inter_op_parallelism flags, and raise the --max_batch_size to allow larger batches.

In this article

Table of Contents

Problem – Inference Queue Backlog on Azure VMs after Load Surge

After a traffic spike on the public API gateway, the AI inference microservice running on Azure Virtual Machines (or a VM Scale Set) began to accumulate requests in its internal queue. Symptoms observed:

Latency grew from ~100 ms to > 30 seconds per prediction.
API Gateway started returning HTTP 503 Service Unavailable and occasional 504 Gateway Timeout.

Application logs showed repeated messages:


ThreadPoolExecutor rejected execution: worker pool exhausted
InferenceService timed out after 30000 ms

Azure Monitor metrics indicated sustained CPU utilization at 95 % on Standard_D2s_v3 instances, with occasional spikes in disk read latency.

Root Cause – Mismatch Between Incoming Request Rate and VM Processing Capacity

The backlog resulted from a combination of three intertwined issues:

Insufficient compute resources – The chosen VM size (2 vCPU, 8 GiB RAM) could not sustain the peak request rate. TensorFlow Serving’s default thread pool saturated, leading to the “ThreadPoolExecutor rejected execution” error, as documented in the TensorFlow Serving issue #2541.
Autoscale rule granularity – Autoscaling was configured only on CPU % > 80. The rule’s evaluation interval (5 min) missed the short‑lived burst, so no additional instances were provisioned quickly enough. This aligns with the Azure VM Scale Set autoscale guide which recommends custom rules based on queue length or request rate.
Disk I/O throttling – Premium SSD IOPS limits were hit during model loading spikes, causing temporary pauses in inference workers. The incident logs showed DiskReadOps at the quota ceiling, matching the “Disk I/O limits” case in the evidence package.

Because the inference service’s internal in‑memory queue grew faster than workers could dequeue, latency increased, health checks failed, and the load balancer marked the backend as unhealthy, triggering the 503/504 responses.

Debug – Investigation Steps

1. Collect VM‑level metrics


az monitor metrics list \
  --resource /subscriptions/_{/resourceGroups//providers/Microsoft.Compute/virtualMachineScaleSets/ \
  --metric "Percentage CPU" "Disk Read Operations/Sec" "Network In Total"}

Expected output showed CPU consistently above 90 % and Disk Read Operations/Sec hitting the premium SSD limit.

2. Inspect inference service logs


journalctl -u tensorflow-serving -f
# Sample excerpt
2024-06-27T14:12:03.421Z INFO  ThreadPoolExecutor rejected execution: worker pool exhausted
2024-06-27T14:12:04.012Z WARN  InferenceService timed out after 30000 ms

3. Verify queue length via internal metrics endpoint


curl -s http://localhost:8501/metrics | grep tf_serving_queue_size
# Example output
tf_serving_queue_size{model_name="resnet50"} 124

4. Check autoscale rule evaluation


az monitor autoscale rule list \
  --resource-group  \
  --autoscale-name

The rule showed a scale out threshold of CPU > 80 with a cooldown 5m, confirming the lag.

5. Capture network latency between API Gateway and VM subnet


tcpdump -i eth0 -nn host  and port 8501 -c 5 -w /tmp/gateway.pcap

Packet captures revealed occasional retransmissions, consistent with NSG throttling reported in the community discussion.

Solution – Align Processing Capacity with Traffic Peaks

1. Resize VMs or switch to a higher‑core SKU

Upgrade from Standard_D2s_v3 (2 vCPU) to Standard_D8s_v3 (8 vCPU) to give TensorFlow Serving enough worker threads.


# Before (VMSS model)
{
  "sku": { "name": "Standard_D2s_v3", "capacity": 2 },
  "properties": { ... }
}
# After
{
  "sku": { "name": "Standard_D8s_v3", "capacity": 2 },
  "properties": { ... }
}

2. Refine autoscale rules to react to queue length

Enable a custom metric based on tf_serving_queue_size exported to Azure Monitor via the diagnostics extension.


# diagnostics extension config (partial)
{
  "metrics": [
    {
      "category": "InferenceQueue",
      "enabled": true,
      "retentionPolicy": { "days": 7, "enabled": true }
    }
  ]
}

Then create an autoscale rule:


az monitor autoscale rule create \
  --resource-group  \
  --autoscale-name  \
  --condition "MetricName='InferenceQueueSize' > 100 avg 1m" \
  --scale out 1 --cooldown 2m

3. Increase TensorFlow Serving thread pool

Adjust the --tensorflow_intra_op_parallelism and --tensorflow_inter_op_parallelism flags, and raise the --max_batch_size to allow larger batches.


# Before (docker run)
docker run -p 8501:8501 tensorflow/serving:latest
# After (docker run with tuned params)
docker run -p 8501:8501 \
  -e TF_INTRA_OP_PARALLELISM=8 \
  -e TF_INTER_OP_PARALLELISM=8 \
  -e TF_SERVING_BATCHING_PARAMETERS_FILE=/config/batching_parameters.txt \
  tensorflow/serving:latest

4. Mitigate disk I/O bottlenecks

Pre‑load the model into memory at container start and mount the model directory on an Ultra Disk with higher IOPS, or use Azure Blob storage with read_ahead enabled.


# Example of mounting an Ultra Disk
az vmss disk attach \
  --resource-group  \
  --vmss-name  \
  --lun 0 \
  --size-gb 512 \
  --sku UltraSSD_LRS

5. Adjust API Gateway health probe timeout

Increase the probe interval and timeout to tolerate temporary queue spikes, preventing premature 503 responses.


# Application Gateway probe configuration (partial)
{
  "protocol": "Http",
  "host": "10.0.0.4",
  "path": "/v1/models/resnet50",
  "interval": 30,
  "timeout": 20,
  "unhealthyThreshold": 3
}

Verify – Confirming the Fix

Metrics check: Percentage CPU should stay below 70 % during peak load; InferenceQueueSize should not exceed the configured threshold (e.g., 50).
Log validation: No longer see “ThreadPoolExecutor rejected execution” or “InferenceService timed out”.

Functional test:


for i in {1..200}; do
  curl -s -o /dev/null -w "%{time_total}\n" http://api-gateway.example.com/predict
done | awk '{sum+=$1} END {print "Avg latency:", sum/NR}'
# Expected avg latency < 200ms

Health probe: Azure Monitor shows Healthy status for all VM instances.

Autoscale activity log:


az monitor activity-log list \
  --resource-group  \
  --filter "eventName.value eq 'Microsoft.Insights/autoscale/scale/action'"

Prevent – Operational Guardrails

Guardrail	Implementation
Capacity planning	Run load‑test scripts (e.g., `hey` or `locust`) to determine the maximum sustainable QPS per VM size; document the ratio of `QPS / vCPU`.
Autoscale thresholds	Combine CPU % with custom queue‑length metric; set a minimum cooldown of 2 min to avoid thrashing.
Disk I/O monitoring	Enable `Disk Read Operations/Sec` alerts at 80 % of the SSD limit via Azure Monitor.
Health check tuning	Configure API Gateway probes with a timeout > 2× expected max inference latency.
Resource limits	Set container memory limits slightly above the model’s resident size; use `oom_score_adj` to prioritize inference process over auxiliary services.

FAQ – Common Follow‑Up Questions

Why does the queue grow even though CPU usage is below 100 %?
CPU may be idle while the inference worker threads are blocked on I/O (model loading) or waiting for GPU/CPU kernels. Monitoring the InferenceQueueSize metric is more reliable than CPU alone.
Can I use Azure Functions instead of VMs for inference?
Functions have cold‑start latency and limited execution time, which is unsuitable for high‑throughput, low‑latency models. VMs or Azure Kubernetes Service with GPU nodes provide predictable performance.
How do I expose the custom queue metric to Azure Monitor?
Install the Azure Monitor Diagnostic Extension on the VM, configure a custom performance counter that reads the tf_serving_queue_size metric, and map it to a custom Azure Monitor metric.
What if the autoscale rule still misses sudden spikes?
Enable scale‑out based on queue length and set a lower evaluation period (e.g., 1 minute). Combine with a “burst” rule that adds a fixed number of instances when the queue exceeds a critical threshold.
Is there a way to limit the maximum backlog size?
Configure TensorFlow Serving’s max_batch_size and max_queue_delay_microseconds to reject or back‑pressure excess requests, causing the API gateway to return 429 instead of queuing indefinitely.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub