Docker Swarm Batch Ingestion Fails With Exit Code 1 During Replication

Q: Problem – Batch Ingestion Jobs Fail with Exit Code 1 During Multi‑Region Replication

In a production Docker Swarm deployment spanning US‑East, EU‑West and AP‑Southeast, nightly batch ingestion services terminate abruptly with exit code 1. The container logs contain messages such as:

Q: Debug – Systematic Investigation Steps 1. Collect Service and Daemon Logs

# Swarm service status docker service ps ingestion_batch --no-trunc # Container logs (last 20 lines) docker logs --tail 20 $(docker ps -q -f name=ingestion_batch) # Docker daemon journal journalctl -u docker.service -b | grep -i "failed to start container" Typical output:

Q: 2. Verify Network Latency and MTU

# Ping remote storage from each manager node ping -c 5 nfs-us-east.example.com # Capture VXLAN traffic tcpdump -i eth0 udp port 4789 -c 20 -w overlay.pcap # Show current MTU ip link show docker0 Observed MTU mismatch (1500 on host, 1450 on overlay) can cause packet fragmentation and retransmission.

Q: 3. Measure NFS/EFS Latency

# Simple NFS read latency time cat /mnt/replica/healthcheck.txt # EFS CloudWatch metric (latency) aws cloudwatch get-metric-statistics --namespace AWS/EFS \ --metric-name TotalIOBytes --statistics Average \ --period 60 --start-time $(date -u -d '-5 minutes' +%FT%TZ) \ --end-time $(date -u +%FT%TZ) --dimensions Name=FileSystemId,Value=fs-12345678 4. Inspect Swarm Service Definition docker service inspect ingestion_batch --pretty Key fields to review:

Q: 2. Tune Overlay Network MTU and Health‑Check Intervals

# Increase MTU on Docker daemon (daemon.json on each manager) { "default-address-pools": [ {"base":"10.0.0.0/8","size":24} ], "mtu": 1450 } # Restart daemon systemctl restart docker # Update service health‑check to tolerate higher latency docker service update \ --health-cmd "curl -f http://localhost:8080/health || exit 1" \ --health-interval 10s \ --health-timeout 5s \ --health-retries 5 \ ingestion_batch 3. Add a Retry Wrapper Around the Ingestion Entry Point Wrap the actual job in a shell script that retries the storage connection with exponential back‑off.

Q: 4. Adjust Restart Policy

docker service update \ --restart-condition any \ --restart-delay 10s \ --restart-max-attempts 3 \ ingestion_batch This ensures transient failures trigger a container restart rather than a permanent failure.

In this article

Table of Contents

Problem – Batch Ingestion Jobs Fail with Exit Code 1 During Multi‑Region Replication

In a production Docker Swarm deployment spanning US‑East, EU‑West and AP‑Southeast, nightly batch ingestion services terminate abruptly with exit code 1. The container logs contain messages such as:


failed to connect to remote storage: timeout
rpc error: code = Unavailable desc = connection timed out
context deadline exceeded

Symptoms observed:

Containers start, attempt to mount the shared NFS/EFS volume, then exit after ~10 seconds.
Swarm service status shows State: Failed and ExitCode: 1 (Docker container exit codes).
Systemd journal entry: docker.service: Failed to start container …: Exit code 1.
Replication latency spikes (30 ms → >200 ms) on the remote storage during peak traffic.
Overlay network health‑check failures across regions.

Root Cause – Interaction Between Remote Storage Timeouts and Swarm Overlay Networking

The failure originates from two tightly coupled factors:

Remote storage mount timeout: Docker uses the volume driver (NFS/EFS) to mount the shared volume before the container entrypoint runs. The default NFS mount timeout is 10 seconds (GitHub issue 38973). During replication bursts, latency on the NFS server exceeds this threshold, causing the mount operation to abort and the daemon to report “error while mounting volume”. The daemon then exits the container with exit code 1 (Docker docs).
Overlay network latency: Swarm overlay networks rely on VXLAN encapsulation and inter‑node gossip. Cross‑region links add additional round‑trip time and occasional packet loss. Health‑check probes (default 2 seconds interval) time out, leading the service scheduler to consider the task unhealthy and terminate it (GitHub issue 2105).

When both conditions occur simultaneously, the container never reaches its application code; it fails during the initialization phase, which is why the logs only show generic “timeout” messages.

Debug – Systematic Investigation Steps

1. Collect Service and Daemon Logs


# Swarm service status
docker service ps ingestion_batch --no-trunc

# Container logs (last 20 lines)
docker logs --tail 20 $(docker ps -q -f name=ingestion_batch)

# Docker daemon journal
journalctl -u docker.service -b | grep -i "failed to start container"

Typical output:


2024-09-08T12:34:56.123Z docker[1234]: time="2024-09-08T12:34:56Z" level=error msg="failed to start container 9c1d2e3f: error while mounting volume: mount nfs4:/data/replica:/var/lib/app: timeout"
2024-09-08T12:34:57.001Z docker[1234]: time="2024-09-08T12:34:57Z" level=error msg="rpc error: code = Unavailable desc = connection timed out"

2. Verify Network Latency and MTU


# Ping remote storage from each manager node
ping -c 5 nfs-us-east.example.com

# Capture VXLAN traffic
tcpdump -i eth0 udp port 4789 -c 20 -w overlay.pcap

# Show current MTU
ip link show docker0

Observed MTU mismatch (1500 on host, 1450 on overlay) can cause packet fragmentation and retransmission.

3. Measure NFS/EFS Latency


# Simple NFS read latency
time cat /mnt/replica/healthcheck.txt

# EFS CloudWatch metric (latency)
aws cloudwatch get-metric-statistics --namespace AWS/EFS \
  --metric-name TotalIOBytes --statistics Average \
  --period 60 --start-time $(date -u -d '-5 minutes' +%FT%TZ) \
  --end-time $(date -u +%FT%TZ) --dimensions Name=FileSystemId,Value=fs-12345678

4. Inspect Swarm Service Definition


docker service inspect ingestion_batch --pretty

Key fields to review:

RestartPolicy – default is none, causing immediate failure.
EndpointSpec – overlay network name and aliases.
Mounts – NFS driver options, e.g., type=volume,source=replica_vol,target=/data,volume-driver=local,volume-opt=type=nfs,volume-opt=o=addr=10.0.0.5,rw.

Solution – Adjust Timeouts, Network Settings, and Service Resilience

1. Increase NFS/EFS Mount Timeout

Modify the volume driver options to raise the timeout from the default 10 seconds to 60 seconds.


# Before (docker-compose.yml snippet)
volumes:
  replica_vol:
    driver: local
    driver_opts:
      type: nfs
      o: addr=10.0.0.5,rw
      device: ":/data/replica"

# After – extended timeout
volumes:
  replica_vol:
    driver: local
    driver_opts:
      type: nfs
      o: addr=10.0.0.5,rw,timeo=60,retrans=3
      device: ":/data/replica"

timeo sets the NFS timeout (tenths of a second); retrans controls retry count.

2. Tune Overlay Network MTU and Health‑Check Intervals


# Increase MTU on Docker daemon (daemon.json on each manager)
{
  "default-address-pools": [
    {"base":"10.0.0.0/8","size":24}
  ],
  "mtu": 1450
}
# Restart daemon
systemctl restart docker

# Update service health‑check to tolerate higher latency
docker service update \
  --health-cmd "curl -f http://localhost:8080/health || exit 1" \
  --health-interval 10s \
  --health-timeout 5s \
  --health-retries 5 \
  ingestion_batch

3. Add a Retry Wrapper Around the Ingestion Entry Point

Wrap the actual job in a shell script that retries the storage connection with exponential back‑off.


#!/bin/sh
MAX_RETRIES=5
DELAY=5

for i in $(seq 1 $MAX_RETRIES); do
  if /app/ingest --source /data/replica; then
    exit 0
  fi
  echo "Ingestion failed, attempt $i/$MAX_RETRIES – sleeping $DELAY seconds"
  sleep $DELAY
  DELAY=$((DELAY * 2))
done

echo "All retries exhausted – exiting with code 1"
exit 1

Update the service to use this script as the entrypoint.

4. Adjust Restart Policy


docker service update \
  --restart-condition any \
  --restart-delay 10s \
  --restart-max-attempts 3 \
  ingestion_batch

This ensures transient failures trigger a container restart rather than a permanent failure.

Verification – Confirming the Fix Works

Deploy the updated stack and trigger a batch run.

Monitor service status:


docker service ps ingestion_batch --no-trunc --filter "desired-state=running"

Check that the container stays Running for the full job duration (e.g., >15 minutes).
Validate storage latency metrics remain below the new 60 second timeout.

Confirm overlay health‑checks succeed:


docker service inspect ingestion_batch --format '{{json .Spec.TaskTemplate.ContainerSpec.HealthCheck}}'

Review logs for absence of “timeout” messages.

Prevention – Operational Guardrails and Best Practices

Monitoring: Set alerts on NFS/EFS latency (>30 ms) and overlay network packet loss.
Capacity Planning: Provision remote storage with burst‑able throughput (e.g., EFS provisioned mode) to absorb replication spikes.
Network Hygiene: Keep overlay MTU consistent across all nodes; enable net.ipv4.tcp_keepalive_time and net.core.somaxconn as recommended in Stack Overflow discussion.
Resilience Patterns: Use retry wrappers for any external I/O during container start‑up.
Service Definition Hygiene: Explicitly declare restart-condition and health‑check parameters for all batch jobs.

FAQ – Common Follow‑Up Questions

Why does the failure only appear in cross‑region deployments?
Because inter‑region latency adds to both storage access time and overlay packet round‑trip, pushing the combined delay past the default mount and health‑check timeouts.
Can I keep the default NFS timeout and rely on Docker retries?
Docker does not retry volume mounts; the daemon treats a mount failure as fatal. You must increase the timeout or implement a wrapper that retries after the container starts.
Is increasing the overlay MTU safe for all cloud providers?
Most cloud VPCs support up to 1500 bytes. Reducing MTU to 1450 avoids fragmentation on the VXLAN tunnel while staying within the lowest common denominator across providers.
How do I differentiate between a storage timeout and a network timeout?
Storage timeouts appear in daemon logs as “error while mounting volume” with the NFS driver name. Network timeouts surface as “rpc error: code = Unavailable” or “context deadline exceeded” from the application layer.
Should I use --net=host for batch jobs to bypass overlay latency?
Using --net=host removes overlay encapsulation but also eliminates Swarm’s service discovery and isolation. It is only advisable for single‑node testing; production should keep overlay networking with tuned parameters.

Related Topic Hub: Distributed Systems Troubleshooting Hub