blue and red cargo ship on dock during daytime

Docker batch ingestion timeout intermittent during large file processing

by

Problem: Intermittent Docker Batch Ingestion Timeout on Large Files

In a production data pipeline, a batch ingestion service runs inside a Docker container. When processing files larger than ~2 GB, the job occasionally aborts with a timeout error and reports partial data load. The failure is non‑deterministic: the same file can succeed on one run and fail on another, making the issue hard to reproduce.

Root Cause

The timeout originates from three interacting factors:

  1. cgroup memory pressure: The container is started with --memory=4g. Large files cause the Java heap (or Python process) to grow beyond the limit, triggering the kernel OOM killer. The OOM event kills the process, which Docker reports as an exit code 137. The ingestion framework interprets the abrupt termination as a timeout.
  2. Docker default stop_timeout: When the OOM killer sends SIGKILL, Docker attempts a graceful shutdown with a default stop_timeout=10s. If the process is still writing to the network socket, Docker’s shutdown sequence can cut the connection, leading to partial data load errors.
  3. File streaming implementation: The ingestion code streams the file using a fixed‑size buffer (256 KB). Under high memory pressure, the buffer allocation fails intermittently, causing the read loop to stall. The framework’s internal watchdog then triggers a ProcessingTimeoutException after 300 seconds.

Debug: Investigation Steps

1. Capture Container Exit Information

docker ps -a --filter "name=ingest_batch" --format "{{.ID}} {{.Status}} {{.ExitCode}}"

Typical output when the issue occurs:

c3f2a1b4d5e6 Exited (137) 2 minutes ago

2. Examine Docker Daemon Logs

journalctl -u docker.service -f | grep c3f2a1b4d5e6

Relevant snippet:

Oct 15 14:32:10 host dockerd[1234]: Container c3f2a1b4d5e6 killed by OOM: memory limit exceeded
Oct 15 14:32:10 host dockerd[1234]: Stopping container c3f2a1b4d5e6 (kill)
Oct 15 14:32:10 host dockerd[1234]: Container c3f2a1b4d5e6 stopped

3. Inspect Application Logs

docker logs ingest_batch --tail 200

Sample excerpt:

2026-10-15 14:31:45,123 INFO Starting ingestion of file /data/bigfile_20261015_01.parquet
2026-10-15 14:31:45,124 DEBUG Buffer size set to 262144 bytes
2026-10-15 14:32:08,567 WARN Read buffer allocation failed, retrying...
2026-10-15 14:32:10,001 ERROR ProcessingTimeoutException: ingestion did not complete within 300 seconds
2026-10-15 14:32:10,002 INFO Partial rows written: 1,254,321

4. Verify cgroup Memory Statistics

docker exec ingest_batch cat /sys/fs/cgroup/memory/memory.stat

Typical values during failure:

cache 5242880
rss 4294967296
rss_huge 0
swap 0
hierarchical_memory_limit 4294967296
hierarchical_memsw_limit 4294967296

5. Reproduce with Controlled Memory Limit

docker run --rm -m 8g -v /data:/data my-ingest-image \
  java -jar ingest.jar --file /data/bigfile_20261015_01.parquet

Increasing the limit to 8 GB eliminates the OOM kill, confirming the memory constraint as a primary trigger.

Solution: Stabilize Ingestion Under Large Files

1. Adjust Container Memory Limits

Provision sufficient memory based on the maximum expected file size. For 2–5 GB files, a 12 GB limit provides headroom for heap expansion and OS buffers.

# Before
docker run -d --name ingest_batch --memory=4g my-ingest-image

# After
docker run -d --name ingest_batch --memory=12g --memory-swap=12g my-ingest-image

2. Increase Docker stop_timeout

Set a longer graceful shutdown period to allow the ingestion process to finish flushing data.

# daemon.json
{
  "default-stop-timeout": 30
}

Reload Docker daemon:

systemctl reload docker

3. Refactor Buffer Allocation Logic

Replace the fixed‑size buffer with a dynamically sized ByteBuffer that falls back to a smaller size on allocation failure, and surface the error to the watchdog.

// Before
private static final int BUFFER_SIZE = 256 * 1024;
byte[] buffer = new byte[BUFFER_SIZE];

// After
private static final int MAX_BUFFER_SIZE = 256 * 1024;
private static final int MIN_BUFFER_SIZE = 64 * 1024;

private byte[] allocateBuffer() {
    int size = MAX_BUFFER_SIZE;
    while (size >= MIN_BUFFER_SIZE) {
        try {
            return new byte[size];
        } catch (OutOfMemoryError e) {
            size /= 2; // halve and retry
        }
    }
    throw new IllegalStateException("Unable to allocate ingestion buffer");
}

4. Tune JVM Heap (if Java)

Set the heap to a fraction of the container limit, leaving room for native memory.

# Before
JAVA_OPTS="-Xmx4g"

# After
JAVA_OPTS="-Xmx8g -XX:MaxDirectMemorySize=2g"

5. Enable OOM Score Adjustment

Prevent the kernel from killing the ingestion process before other less critical processes.

# Docker run option
--oom-score-adj=-500

Verify: Confirm the Fix Works

  1. Deploy the updated container definition with the new memory limit and timeout.
  2. Run a controlled ingestion of a 4 GB test file.
  3. Monitor Docker events and application logs for the absence of OOM kills and timeout messages.

Sample successful run logs:

2026-10-15 15:02:12,001 INFO Starting ingestion of file /data/bigfile_test.parquet
2026-10-15 15:02:12,005 DEBUG Allocated buffer of 262144 bytes
2026-10-15 15:04:57,342 INFO Ingestion completed successfully. Rows written: 12,345,678
2026-10-15 15:04:57,345 INFO Container exited with code 0

Docker status check:

docker ps -a --filter "name=ingest_batch"

c7d8e9f0a1b2 Up 2 minutes ingest_batch

Prevent: Operational Guardrails

  • Resource Monitoring: Add a Prometheus alert on container_memory_usage_bytes approaching 80 % of the limit.
  • Health Checks: Implement a liveness probe that verifies the ingestion process can allocate the maximum buffer size.
  • CI Validation: Include a test that runs the ingestion job against a synthetic 5 GB file in a container with the production memory limit.
  • Configuration Audits: Enforce a policy that all batch jobs declare memory and memory-swap values in the deployment manifest.
  • Graceful Degradation: If memory usage exceeds 90 %, pause new file ingestion and back‑pressure upstream producers.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does the ingestion succeed on small files but fail on large ones?
    Large files increase heap and native memory usage, pushing the container over its cgroup limit. The OOM killer terminates the process, which the framework reports as a timeout.
  2. Can I keep the 4 GB memory limit and still process large files?
    Only if you reduce the JVM heap or buffer size dramatically, which will degrade throughput. The reliable approach is to increase the container memory limit to match the workload.
  3. What is the difference between docker stop timeout and the OOM kill?
    The OOM kill is immediate (SIGKILL) and bypasses Docker’s graceful shutdown. The stop_timeout only applies when Docker sends SIGTERM and waits for the process to exit cleanly.
  4. How can I detect that a buffer allocation failure caused the timeout?
    Enable debug logging around buffer allocation and watch for “Read buffer allocation failed” messages. Correlate the timestamp with the watchdog’s timeout trigger.
  5. Is there a way to automatically adjust memory limits per file size?
    Yes. Use a wrapper script that inspects the file size and launches the container with --memory set to fileSize * 2 + overhead. Ensure the host has sufficient RAM to accommodate the highest expected limit.