Nginx Load Balancer Replica Model Sync Lag After Deployment

Q: Debug – Systematic Investigation Steps 1. Verify volume propagation latency

# Inside a running replica stat -c %Y /models/model_v2.pt # modification epoch inotifywait -e modify /models/model_v2.pt -t 5 Expected output shows the modification event within a second. If the event is delayed, the mount is not using :delegated or :cached appropriately.

Q: 2. Inspect Nginx upstream keepalive state

# Query the Nginx status module (if enabled) curl http://localhost/nginx_status | grep keepalive Typical output:

Q: 3. Check proxy cache headers

curl -I http://staging.example.com/models/model_v2.pt Look for Age and Cache-Control headers. A non‑zero Age suggests the file is being served from cache.

Q: 4. Review container logs for missing files

docker logs inference_replica_1 2>&1 | grep "open()" Sample log from the incident:

In this article

Table of Contents

Problem – Model Weight Synchronization Lag Behind Nginx Load Balancer

In a staging environment we run three Docker‑based inference replicas behind an Nginx upstream. Model artifacts are stored on a shared NFS mount and are updated by a CI/CD pipeline that copies a new .pt file into /models. After each deployment the expectation is that every replica serves the new model within 5 seconds. Observed behavior:

Requests routed to the same replica for a few seconds continue to return predictions based on the previous model version.
Log snippets show Nginx reusing persistent upstream connections that were opened before the new file appeared:
```
nginx: [notice] 12#12: *1234 open() "/models/model_v2.pt" failed (2: No such file or directory)
```
Two of the three replicas serve the stale model while the third (restarted by the deployment script) immediately returns the new predictions.
Occasional 304 Not Modified responses from proxy_cache indicate cached static content is being reused.

The lag ranges from 20 ms to 130 ms, which is unacceptable for latency‑sensitive inference pipelines.

Root Cause – Interaction of Nginx Keepalive, Proxy Cache, and Docker Volume Propagation

Three independent mechanisms combine to produce the observed lag:

Keepalive upstream connections – Nginx’s keepalive directive (documented in the NGINX Docs – Load Balancing (upstream module)) keeps a socket open to each replica. The socket’s process space holds an open file descriptor for the model file that was read when the connection was first established. When the model file is replaced on the shared volume, the already‑opened descriptor continues to point at the old inode until the connection is closed or the process re‑opens the file.
Proxy cache serving stale files – The proxy_cache directives (see NGINX Docs – Proxy Cache and Cache Invalidation) cache static model files. Without an explicit proxy_cache_bypass rule, Nginx may return a cached 304 response even after the underlying file has changed, as illustrated by the incident log:
```
nginx: [error] 12#12: *5678 upstream sent unexpected header "304 Not Modified" while reading response header from upstream
```
Docker volume propagation delay – When the shared NFS mount is bound into the containers without a consistency flag, Docker’s default :cached behavior can delay file visibility for up to 30 ms (Docker Documentation – Bind mounts and volume propagation). In the real incident the third replica, which was restarted, saw the new file immediately because its mount was refreshed on container start.

Combined, these factors mean that a replica can continue serving the previous model until either the keepalive socket is recycled, the proxy cache expires, or the OS page cache discards the stale inode.

Debug – Systematic Investigation Steps

1. Verify volume propagation latency

# Inside a running replica
stat -c %Y /models/model_v2.pt   # modification epoch
inotifywait -e modify /models/model_v2.pt -t 5

Expected output shows the modification event within a second. If the event is delayed, the mount is not using :delegated or :cached appropriately.

2. Inspect Nginx upstream keepalive state

# Query the Nginx status module (if enabled)
curl http://localhost/nginx_status | grep keepalive

Typical output:

keepalive_requests  1024
keepalive_idle      75

High keepalive_requests indicates sockets are being reused heavily.

3. Check proxy cache headers

curl -I http://staging.example.com/models/model_v2.pt

Look for Age and Cache-Control headers. A non‑zero Age suggests the file is being served from cache.

4. Review container logs for missing files

docker logs inference_replica_1 2>&1 | grep "open()"

Sample log from the incident:

nginx: [notice] 12#12: *1234 open() "/models/model_v2.pt" failed (2: No such file or directory)

5. Correlate health‑check intervals

NGINX Plus health checks (see NGINX Plus Docs – Health Checks and Active Monitoring) default to a 5‑second interval. If the health‑check period is longer than the desired rollout window, a replica may stay “up” while serving stale content.

Solution – Align Nginx, Docker, and Deployment Pipeline for Immediate Model Rollout

1. Disable upstream keepalive for inference traffic

Because each request loads the model file from disk, the overhead of establishing a new TCP connection is negligible compared to the risk of stale descriptors.

# nginx.conf – upstream block
upstream inference_backend {
    server replica1:8080;
    server replica2:8080;
    server replica3:8080;
    # Remove keepalive directive
    # keepalive 32;
}

2. Force cache bypass on model files

Add a location block that disables caching for any request under /models/. This matches the recommendation in the Nginx proxy cache docs.

# nginx.conf – server block
location /models/ {
    proxy_pass http://inference_backend;
    proxy_cache_bypass $http_upgrade $arg_nocache;
    proxy_no_cache    $http_upgrade $arg_nocache;
    proxy_cache       off;
    add_header        Cache-Control "no-store, no-cache, must-revalidate, max-age=0";
}

3. Use delegated volume consistency for immediate visibility

Update the Docker Compose file to mount the shared NFS volume with :delegated. This tells Docker that the container’s view of the filesystem is authoritative, reducing propagation latency.

# docker-compose.yml
services:
  inference:
    image: myorg/inference:latest
    volumes:
      - type: bind
        source: /mnt/shared/models
        target: /models
        bind:
          propagation: rshared
          consistency: delegated

4. Reduce health‑check interval to 1 second

This ensures that a replica that has not yet received the new model is marked unhealthy quickly, causing Nginx to stop routing traffic to it.

# nginx_plus.conf – upstream health check
upstream inference_backend {
    server replica1:8080;
    server replica2:8080;
    server replica3:8080;
    health_check interval=1 fails=1 passes=1;
}

5. Add a post‑deployment “touch” trigger

After copying the new model file, issue a touch on a sentinel file that the inference service watches (via inotify) to force a reload of the model in memory.

# deployment script fragment
docker cp model_v2.pt replica1:/models/
docker cp model_v2.pt replica2:/models/
docker cp model_v2.pt replica3:/models/
# Signal reload
docker exec replica1 touch /models/.reload
docker exec replica2 touch /models/.reload
docker exec replica3 touch /models/.reload

Verify – Confirm All Replicas Serve the New Model Within 5 seconds

Trigger a deployment of model_v3.pt.

Immediately start a high‑frequency curl loop against each replica directly:

for i in {1..3}; do
  for n in {1..20}; do
    curl -s http://replica${i}:8080/model_version
    sleep 0.2
  done
done

Collect the responses. All replicas should report v3 after the first two iterations (≈0.4 s).
Check Nginx logs for any “open() … failed” notices. Absence of such messages confirms the file is present.
Run nginx -t to verify the configuration reload succeeded.

Prevent – Operational Guardrails for Future Deployments

Monitoring: Add a Prometheus metric that scrapes /model_version from each replica every second. Alert if any replica reports a version older than the latest deployment tag.
Alerting: Configure an alert on Nginx error logs containing “open() … failed” or “proxy_cache_bypass” warnings.
Configuration Management: Store the Nginx upstream block in version‑controlled templates; enforce keepalive removal via CI linting.
Deployment Pipeline: Include a step that validates the :delegated mount flag and runs an inotifywait sanity check before marking the deployment as successful.
Health‑Check Tuning: Keep the health‑check interval ≤ 1 second for inference services that require rapid rollout of model artifacts.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does disabling keepalive fix the stale model problem?
Because each request opens a fresh socket, forcing the application process to re‑open the model file. The stale file descriptor held by a persistent connection is discarded, guaranteeing the newest inode is read.
Can I keep keepalive and still avoid stale files?
Yes, but you must explicitly close and reopen the upstream sockets after a deployment (e.g., nginx -s reload) or use the proxy_next_upstream directive to force a new connection on a custom header.
Is :delegated safe for production NFS mounts?
It is safe when the NFS server is the source of truth and the containers are the primary consumers. Delegated mode prioritizes container‑side visibility, which is exactly what inference services need for immediate model rollout.
What if the model file is larger than the OS page cache?
Large files may be read in chunks; however, the stale‑inode issue still applies. Ensure the inference service re‑loads the model on receipt of the sentinel .reload file or on SIGUSR2.
How do I know if proxy cache is still active after the change?
Inspect the response headers for X-Cache-Status (if enabled) or check the proxy_cache_path statistics via ngx_http_cache_status_module. A status of MISS confirms the cache is bypassed.