Azure VM Model Loading Timeout After DR Failover To Secondary Region

Q: Root Cause – Why the Timeout Happens Only After Failover

The inference engine loads a large (~4 GB) TensorFlow model from an Azure Blob Storage account that is geo‑redundantly replicated (GRS). In the primary region the VM accesses the storage endpoint myapp.blob.core.windows.net via a VNet service endpoint, which provides low‑latency, private connectivity.

Q: Debug – Investigation Steps 1. Examine systemd and application logs

journalctl -u inference-service.service -b -e -- Logs begin at Mon 2026-10-19 01:02:13 UTC, end at Mon 2026-10-19 01:07:45 UTC. -- Oct 19 01:02:13 vm-secondary systemd[1]: Starting Inference Service... Oct 19 01:02:13 vm-secondary inference[1234]: INFO - Starting model load (4GB) Oct 19 01:02:13 vm-secondary inference[1234]: ERROR - Failed to download model: Request timed out Oct 19 01:07:13 vm-secondary inference[1234]: ERROR - Model loading timed out after 300000 ms Oct 19 01:07:13 vm-secondary systemd[1]: inference-service.service: Main process exited, code=exited, status=1/FAILURE Oct 19 01:07:13 vm-secondary systemd[1]: inference-service.service: Failed with result 'exit-code'. Oct 19 01:07:13 vm-secondary systemd[1]: Failed to start Inference Service. 2. Verify network connectivity to the storage endpoint # Attempt TCP connection from the secondary VM nc -vz myapp.blob.core.windows.net 443 Connection timed out # Check DNS resolution nslookup myapp.blob.core.windows.net Server: 168.63.129.16 Address: 168.63.129.16#53 Name: myapp.blob.core.windows.net Address: 52.239.123.45 3. Inspect storage account firewall rules az storage account network-rule list \ --resource-group rg-prod \ --account-name myapp { "virtualNetworkRules": [ { "id": "/subscriptions/xxxx/resourceGroups/rg-prod/providers/Microsoft.Network/virtualNetworks/vnet-primary/subnets/subnet-primary", "action": "Allow" } ], "ipRules": [], "defaultAction": "Deny" } 4. Measure disk read latency for the model file time dd if=/dev/sda1 of=/dev/null bs=4M count=1024 iflag=direct 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB) copied, 120.3 s, 35.7 MB/s ~2 minutes of pure I/O before the model can be streamed.

Q: 5. Confirm replication status of the storage account

az storage account show \ --name myapp \ --resource-group rg-prod \ --query "geoReplicationStats" { "statusOfPrimary": "available", "statusOfSecondary": "available", "lastSyncTime": "2026-10-18T22:45:00Z" } Solution – Fixes to Eliminate the Timeout 1. Extend the storage account network allow list to the secondary VNet Add a virtual network rule for the secondary region’s subnet.

In this article

Table of Contents

Problem – Model Loading Timeout After DR Failover to Secondary Region

During a planned disaster‑recovery (DR) failover, traffic is redirected from the primary Azure region to a secondary region where a standby Azure Virtual Machine (VM) hosts the AI inference service. The service fails to start because the model loading step exceeds the configured startup deadline, resulting in a ModelLoadTimeoutException and the VM being marked unhealthy by the orchestrator.

Typical symptoms:

Systemd reports Failed to start inference-service.service: Start request timed out.
Application logs contain ERROR - Model loading timed out after 300000 ms.
Health‑check endpoint returns 503.
Azure Monitor alerts “VM startup latency > 5 min”.

Impact:

End‑users experience inference failures.
Service Level Objective (SLO) breach.
Automatic rollback may trigger, increasing failover time.

Root Cause – Why the Timeout Happens Only After Failover

The inference engine loads a large (~4 GB) TensorFlow model from an Azure Blob Storage account that is geo‑redundantly replicated (GRS). In the primary region the VM accesses the storage endpoint myapp.blob.core.windows.net via a VNet service endpoint, which provides low‑latency, private connectivity.

During failover the secondary VM resides in a different region but still uses the same storage endpoint. Two mis‑configurations combine to cause the timeout:

Network path restriction: The storage account firewall allows traffic only from the primary region’s virtual network subnet. Requests from the secondary region are blocked, causing the first TCP SYN to be dropped and the client to retry until the application‑level timeout expires.
Cold disk access: The secondary VM boots from a managed OS disk that is not pre‑attached to a premium SSD cache. The first read of the model file triggers a remote read from the storage replica, adding ~2 minutes of I/O latency. The inference service’s hard‑coded startup deadline (5 min) does not account for this extra latency.

Both issues are invisible in the primary region because the network path is allowed and the OS disk is already cached.

Debug – Investigation Steps

1. Examine systemd and application logs

journalctl -u inference-service.service -b -e
-- Logs begin at Mon 2026-10-19 01:02:13 UTC, end at Mon 2026-10-19 01:07:45 UTC. --
Oct 19 01:02:13 vm-secondary systemd[1]: Starting Inference Service...
Oct 19 01:02:13 vm-secondary inference[1234]: INFO - Starting model load (4GB)
Oct 19 01:02:13 vm-secondary inference[1234]: ERROR - Failed to download model: Request timed out
Oct 19 01:07:13 vm-secondary inference[1234]: ERROR - Model loading timed out after 300000 ms
Oct 19 01:07:13 vm-secondary systemd[1]: inference-service.service: Main process exited, code=exited, status=1/FAILURE
Oct 19 01:07:13 vm-secondary systemd[1]: inference-service.service: Failed with result 'exit-code'.
Oct 19 01:07:13 vm-secondary systemd[1]: Failed to start Inference Service.

2. Verify network connectivity to the storage endpoint

# Attempt TCP connection from the secondary VM
nc -vz myapp.blob.core.windows.net 443
Connection timed out

# Check DNS resolution
nslookup myapp.blob.core.windows.net
Server:  168.63.129.16
Address: 168.63.129.16#53

Name:    myapp.blob.core.windows.net
Address: 52.239.123.45

3. Inspect storage account firewall rules

az storage account network-rule list \
  --resource-group rg-prod \
  --account-name myapp
{
  "virtualNetworkRules": [
    {
      "id": "/subscriptions/xxxx/resourceGroups/rg-prod/providers/Microsoft.Network/virtualNetworks/vnet-primary/subnets/subnet-primary",
      "action": "Allow"
    }
  ],
  "ipRules": [],
  "defaultAction": "Deny"
}

4. Measure disk read latency for the model file

time dd if=/dev/sda1 of=/dev/null bs=4M count=1024 iflag=direct
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB) copied, 120.3 s, 35.7 MB/s

~2 minutes of pure I/O before the model can be streamed.

5. Confirm replication status of the storage account

az storage account show \
  --name myapp \
  --resource-group rg-prod \
  --query "geoReplicationStats"
{
  "statusOfPrimary": "available",
  "statusOfSecondary": "available",
  "lastSyncTime": "2026-10-18T22:45:00Z"
}

Solution – Fixes to Eliminate the Timeout

1. Extend the storage account network allow list to the secondary VNet

Add a virtual network rule for the secondary region’s subnet.

# Before (only primary VNet)
{
  "virtualNetworkRules": [
    { "id": "/subscriptions/xxxx/resourceGroups/rg-prod/providers/Microsoft.Network/virtualNetworks/vnet-primary/subnets/subnet-primary", "action": "Allow" }
  ],
  "defaultAction": "Deny"
}

# After (add secondary VNet)
{
  "virtualNetworkRules": [
    { "id": "/subscriptions/xxxx/resourceGroups/rg-prod/providers/Microsoft.Network/virtualNetworks/vnet-primary/subnets/subnet-primary", "action": "Allow" },
    { "id": "/subscriptions/xxxx/resourceGroups/rg-dr/providers/Microsoft.Network/virtualNetworks/vnet-secondary/subnets/subnet-secondary", "action": "Allow" }
  ],
  "defaultAction": "Deny"
}

az storage account network-rule add \
  --resource-group rg-prod \
  --account-name myapp \
  --vnet-name vnet-secondary \
  --subnet subnet-secondary

2. Enable Azure Blob Storage private endpoint in the secondary region

Using a private endpoint guarantees regional traffic and removes reliance on the public endpoint.

# Create private endpoint
az network private-endpoint create \
  --resource-group rg-dr \
  --name pe-blob-secondary \
  --vnet-name vnet-secondary \
  --subnet subnet-secondary \
  --private-connection-resource-id $(az storage account show -g rg-prod -n myapp --query id -o tsv) \
  --group-id blob

3. Pre‑warm the model on the secondary VM’s local SSD

Copy the model during the VM provisioning phase using a Custom Script Extension.

# customscript.sh (executed on VM creation)
#!/bin/bash
set -euo pipefail

MODEL_URL="https://myapp.blob.core.windows.net/models/v1/model.pb"
LOCAL_PATH="/mnt/cache/model.pb"

# Use azcopy with --check-md5 to ensure integrity
azcopy copy "$MODEL_URL" "$LOCAL_PATH" --check-md5

# Verify size
if [ $(stat -c%s "$LOCAL_PATH") -ne 4294967296 ]; then
  echo "Model size mismatch"
  exit 1
fi

Update the inference service configuration to load from /mnt/cache/model.pb instead of streaming from Blob Storage.

4. Increase the service startup timeout as a safety net

Modify the systemd unit file.

# Before (default 90s)
[Service]
TimeoutStartSec=90

# After (extended to 300s)
[Service]
TimeoutStartSec=300

sudo systemctl daemon-reload
sudo systemctl restart inference-service

5. Switch the storage account to Read‑Access Geo‑Redundant Storage (RA‑GRS) if not already enabled

RA‑GRS ensures the secondary region can read from the replicated copy without additional latency.

az storage account update \
  --name myapp \
  --resource-group rg-prod \
  --sku Standard_RAGRS

Verify – Confirming the Fix

Redeploy the secondary VM or apply the script extension.
Check that the private endpoint resolves to a local IP:

nslookup myapp.blob.core.windows.net
Server:  168.63.129.16
Address: 168.63.129.16#53

Name:    myapp.blob.core.windows.net
Address: 10.1.4.5

Validate that the model file exists on the local SSD and its checksum matches:

sha256sum /mnt/cache/model.pb
d2c3e5f9a1b2c4d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0  /mnt/cache/model.pb

Start the inference service and watch the logs:

sudo systemctl start inference-service
journalctl -u inference-service -f
Oct 19 01:12:05 vm-secondary inference[5678]: INFO - Loading model from /mnt/cache/model.pb
Oct 19 01:12:07 vm-secondary inference[5678]: INFO - Model loaded successfully (4.3 GB)
Oct 19 01:12:07 vm-secondary systemd[1]: Started Inference Service.

Run a health‑check request:

curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/healthz
200

All checks pass; the service starts within the new timeout window.

Prevent – Operational Guardrails for Future Failovers

Network policy automation: Include secondary VNet rules in the storage account firewall as part of the DR infrastructure-as-code (ARM/Bicep/Terraform).
Private endpoints per region: Deploy a regional private endpoint for every storage account accessed by a DR VM.
Model caching strategy: Store a copy of the model on a premium SSD attached to the VM image or use Azure Managed Disks with ReadOnly snapshots that are attached during failover.
Startup timeout tuning: Parameterize TimeoutStartSec based on model size and expected I/O latency; expose it as a configurable environment variable.
Monitoring & alerts: Create Azure Monitor alerts for:
- Blob storage latency > 500 ms (metric AverageE2ELatency)
- VM boot time > 4 min
- Systemd service start failures
Regular DR drills: Automate a failover test that validates model load time, network connectivity, and health‑check response within the SLO window.

FAQ – Common Follow‑Up Questions

Why does the model load fine in the primary region but time out after failover?
Because the storage account firewall only allowed the primary VNet, causing the secondary VM’s requests to be dropped. Additionally, the secondary VM read the model from remote storage instead of a local cache, adding latency.
Can I keep the same storage endpoint URL after failover?
Yes, but you must expose the endpoint to the secondary VNet via either firewall rules or a regional private endpoint. The URL does not change; the connectivity path does.
Is increasing the systemd timeout sufficient?
It prevents the service from being killed prematurely but does not solve the underlying latency. Use it as a safety net while implementing network and caching fixes.
Do I need to replicate the model to a separate storage account?
Not if you enable RA‑GRS and private endpoints. Replication is handled by Azure; the key is to ensure the secondary region can read the replicated copy.
How can I verify which storage endpoint (public vs. private) the VM is using?
Run nslookup myapp.blob.core.windows.net on the VM. A private endpoint resolves to a 10.x.x.x address, whereas the public endpoint resolves to a public IP.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub