icon

Azure VM webhook timeout during JMeter load test

by

Problem – Webhook Timeout During JMeter Load Test on Azure VM

During a performance benchmark, a JMeter test simulating 500 concurrent users sends HTTP POST webhook calls to an ASP.NET Core API hosted on IIS behind an Azure Load Balancer. The backend service processes each request in under 2 minutes and returns 200 OK, yet JMeter consistently records timeout errors such as:

Response code: 0, response message: java.net.SocketTimeoutException: Read timed out

Additional symptoms observed in the environment include:

  • IIS logs showing sc-status 504, sc-substatus 0 (request timed out).
  • Azure Load Balancer health probe logs: “Probe failed for backend pool” with error code 0.
  • Azure Monitor alerts: “Outbound connection timeout” on the VM NIC.

Despite the API responding with 200 OK when queried directly (e.g., via curl), the webhook requests fail under load.

Root Cause Analysis

1. Azure Load Balancer Idle Timeout

The default idle timeout for Azure Load Balancer is 4 minutes (Microsoft Docs – Load Balancer idle timeout). When JMeter holds a connection open while waiting for the webhook response, any pause longer than the idle timeout causes the Load Balancer to terminate the TCP flow, resulting in a client‑side timeout even though the backend has already completed processing.

2. IIS Request Queue Saturation

With 500 concurrent users, IIS request queue length ( maxQueueLength ) can be exceeded. When the queue is full, IIS returns 503/504 before the request reaches Kestrel, which surfaces as a timeout in JMeter. This is documented in the IIS request queue configuration guide.

3. VM NIC Bandwidth Throttling

Azure VM sizes have defined NIC throughput limits (e.g., D2_v3 – 2 Gbps). The load test pushes outbound traffic close to this ceiling, causing packet drops and “Outbound connection timeout” alerts (Azure Monitor metrics). When the NIC saturates, TCP retransmissions increase latency, triggering the Load Balancer idle timeout earlier than expected.

4. Socket Exhaustion in ASP.NET Core HttpClient

If the webhook implementation uses HttpClient per request (instead of a shared HttpClientFactory), socket exhaustion can occur under high concurrency, leading to delayed responses and eventual timeouts (GitHub issue – dotnet/aspnetcore).

Investigation and Debugging Steps

  1. Collect Load Balancer logs

    az network lb show --name MyLb --resource-group RG \
    --query "frontendIpConfigurations[].idleTimeoutInMinutes"

    Expected output: 4 (default).

  2. Inspect IIS request queue

    appcmd list config "Default Web Site" -section:system.applicationHost/sites
# Look for maxQueueLength attribute

    Typical default: maxQueueLength="1000". Verify current value.

  3. Monitor NIC throughput

    az monitor metrics list \
  --resource /subscriptions/xxxx/resourceGroups/RG/providers/Microsoft.Network/networkInterfaces/nic1 \
  --metric "Network Out Total" \
  --interval PT1M

    Check for sustained usage > 80 % of the VM’s NIC limit.

  4. Capture TCP flow

    sudo tcpdump -i eth0 -w /tmp/jmeter.pcap host

    Analyze with Wireshark for FIN/RST packets occurring around the 4‑minute mark.

  5. Review ASP.NET Core HttpClient usage

    // Bad: new HttpClient per request
using (var client = new HttpClient())
{
    await client.PostAsync(url, content);
}

// Good: injected IHttpClientFactory
private readonly HttpClient _client;
public MyService(IHttpClientFactory factory)
{
    _client = factory.CreateClient("WebhookClient");
}
await _client.PostAsync(url, content);

Resolution – Configuration and Code Changes

1. Extend Load Balancer Idle Timeout

Increase the idle timeout to accommodate the longest expected webhook latency (e.g., 10 minutes).

az network lb update \
  --name MyLb \
  --resource-group RG \
  --frontend-ip-name FrontendIP \
  --idle-timeout 10

After the change, the Load Balancer will keep idle connections alive for up to 10 minutes.

2. Tune IIS Request Queue and Connection Timeout

Adjust maxQueueLength and connectionTimeout in applicationHost.config or via appcmd:

# Before

  


# After

  


# Increase connection timeout (default 2 minutes)

3. Scale VM or Upgrade NIC Bandwidth

Move to a VM SKU with higher NIC throughput (e.g., D4_v3 – 4 Gbps) or enable Accelerated Networking.

az vm resize \
  --resource-group RG \
  --name MyVm \
  --size Standard_D4_v3

4. Consolidate HttpClient Usage

Register a named HttpClient with appropriate timeout settings.

// Startup.cs
services.AddHttpClient("WebhookClient", client =>
{
    client.Timeout = TimeSpan.FromMinutes(5);
    client.DefaultRequestHeaders.Add("User-Agent", "MyApp");
});

5. Adjust Azure Load Balancer Outbound Rules (if applicable)

Ensure outbound rules allow sufficient SNAT ports for high concurrency.

az network lb outbound-rule create \
  --resource-group RG \
  --lb-name MyLb \
  --name OutboundRule \
  --frontend-ip-config FrontendIP \
  --protocol Tcp \
  --idle-timeout 10 \
  --enable-tcp-reset true

Verification – Confirming the Fix

  1. Re‑run JMeter test with the same 500‑user profile.

    Expected result: No timeout errors; all webhook calls return 200 OK.

  2. Check IIS logs for absence of 504 entries.

    FindStr "504 0" %SystemDrive%\inetpub\logs\LogFiles\W3SVC1\*.log
# Should return no matches

  3. Validate Load Balancer metrics – idle timeout counter should stay at zero.

    az monitor metrics list \
  --resource /subscriptions/xxxx/resourceGroups/RG/providers/Microsoft.Network/loadBalancers/MyLb \
  --metric "IdleTimeoutCount"

  4. Confirm NIC utilization stays below 70 % of the new limit during the test.

  5. Inspect HttpClient socket count via dotnet-counters to ensure no exhaustion.

    dotnet-counters monitor --process-id  System.Net.Http.HttpClient

Prevention – Operational Guardrails

  • Set Load Balancer idle timeout to a value exceeding the longest expected request duration (documented in service‑level agreements).
  • Monitor Network Out Total and IdleTimeoutCount metrics; create alerts when utilization exceeds 80 % or timeout count > 0.
  • Configure IIS maxQueueLength and connectionTimeout with headroom for peak load; automate verification via Azure Policy.
  • Standardize on IHttpClientFactory with per‑named‑client timeouts to avoid socket exhaustion.
  • Select VM SKUs with sufficient NIC bandwidth for anticipated load; consider Azure Load Balancer Standard SKU for higher throughput and configurable idle timeout defaults.

FAQ – Common Follow‑Up Questions

  1. Why does the webhook succeed when called manually but timeout under load?

    Manual calls complete quickly, staying well within the Load Balancer’s idle timeout and NIC bandwidth limits. Under load, connection queues, NIC saturation, and idle timeout thresholds are reached, causing premature termination.

  2. Can I keep the default 4‑minute idle timeout and still avoid timeouts?

    Only if every webhook response finishes in less than 4 minutes and the VM NIC can sustain the traffic. Otherwise, extending the timeout or reducing response latency is required.

  3. How do I know if socket exhaustion is happening in ASP.NET Core?

    Monitor System.Net.Sockets.SocketException events, use dotnet-counters for System.Net.Http.HttpClient sockets, and watch for “Unable to connect to remote server” errors in the application logs.

  4. Does enabling Accelerated Networking help?

    Yes. Accelerated Networking reduces latency and increases NIC throughput, which can alleviate packet drops that contribute to timeout events.

  5. Is there a way to make JMeter wait longer before reporting a timeout?

    Increase the “Response Timeout” field in the HTTP Request sampler (e.g., to 600 seconds). However, this only masks the underlying infrastructure timeout; the root cause should still be addressed.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub