a close-up of a server room

RAG document loader crash during canary deployment on GCP Compute

by

Problem

During a canary rollout of a Retrieval‑Augmented Generation (RAG) pipeline on Google Cloud Compute Engine, the DocumentLoader component crashes intermittently while ingesting documents. The failure manifests as partial ingestion, missing vectors in the downstream vector store, and occasional container termination.

Typical symptoms observed in the canary instance logs:


2024-07-08T12:34:56.123Z  stdout  FileLockError: could not acquire lock on /tmp/loader.lock
2024-07-08T12:35:01.874Z  stderr  ResourceExhausted: Out of memory
2024-07-08T12:35:03.210Z  stdout  grpc.DeadlineExceeded: deadline exceeded while fetching https://example.com/doc.pdf
2024-07-08T12:35:04.001Z  stderr  Segmentation fault (core dumped)

The issue is isolated to the canary subset of traffic; the stable version continues to ingest successfully. The problem does not appear in staging or full‑production deployments, making it difficult to reproduce.

Root Cause

The crash results from a combination of three tightly coupled factors that only align during a canary deployment on GCE:

  1. Shared temporary storage contention. Both the stable and canary containers mount the same /tmp directory from a host‑level NFS volume (as recommended for large PDF processing). When the canary instance starts, it attempts to create /tmp/loader.lock (LangChain issue #4215). The lock file is already held by the stable instance, leading to FileLockError and, in some cases, a race that corrupts the temporary PDF extraction buffer.
  2. CPU throttling and OOM on the shared physical host. Traffic splitting via Cloud Load Balancing (Google Cloud Load Balancing – Traffic splitting guide) causes a burst of ingestion requests to the canary node while the stable node is still processing its own batch. The host’s CPU credits are exhausted, and the container memory limit (e.g., 4 GiB) is exceeded, triggering the ResourceExhausted error and eventual SIGKILL (internal incident log).
  3. Missing environment configuration. The canary image was built from a newer Dockerfile that omitted the VECTOR_DB_ENDPOINT variable (post‑mortem health‑tech startup). The first batch of documents attempts to write vectors, raising KeyError: 'VECTOR_DB_ENDPOINT' and aborting the ingestion loop. Subsequent retries hit the same lock contention, compounding the failure.

These three conditions together explain why the problem is intermittent, only appears under the canary traffic pattern, and does not surface in isolated staging runs.

Debug

Below is a systematic investigation workflow that reproduced the failure and isolated each factor.

1. Gather structured logs


# Example of structured log query in Cloud Logging
resource.type="gce_instance"
logName="projects/my‑project/logs/stdout"
jsonPayload.message=~"FileLockError|ResourceExhausted|grpc.DeadlineExceeded"

Resulting log entries highlighted a correlation between the canary instance ID (instance-2) and the lock error timestamps.

2. Verify NFS mount sharing


# On the canary VM
mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k)
# Check for shared host path
df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
10.128.0.5:/mnt/tmp  50G   12G   38G  24% /tmp

The output confirms that /tmp is a shared NFS export.

3. Inspect container resource limits


# Container runtime inspection
docker inspect rag-loader-canary | jq '.[0].HostConfig.Memory'
2048*1024*1024   # 2 GiB limit

Large PDF batches routinely allocate >2 GiB during text extraction, exceeding this limit.

4. Check environment variables


# Inside the canary container
printenv | grep VECTOR_DB_ENDPOINT
# No output – variable missing

Absence of the variable matches the KeyError observed in the canary logs.

5. Reproduce the race condition locally


# Simulate two concurrent loaders on the same NFS mount
python - <<'PY'
import threading, time, os
def loader(id):
    lock_path = "/tmp/loader.lock"
    try:
        fd = os.open(lock_path, os.O_CREAT | os.O_EXCL)
        os.close(fd)
        time.sleep(5)  # simulate work
        os.remove(lock_path)
        print(f"loader {id} succeeded")
    except FileExistsError:
        print(f"loader {id} failed: FileLockError")
threads = [threading.Thread(target=loader, args=(i,)) for i in (1,2)]
for t in threads: t.start()
for t in threads: t.join()
PY
loader 1 succeeded
loader 2 failed: FileLockError

The script reproduces the lock contention observed in production.

Solution

The fix requires three coordinated changes: isolate temporary storage, adjust resource allocations, and ensure configuration parity.

1. Use per‑instance /tmp (no shared NFS)

Update the instance template to mount an tmpfs volume locally instead of the shared NFS export.


# Before (instance template snippet)
disks:
- autoDelete: true
  boot: true
  initializeParams:
    sourceImage: projects/debian-cloud/global/images/family/debian-11
- source: projects/my-project/zones/us-central1-a/disks/shared-tmp
  deviceName: shared-tmp
  mode: READ_WRITE
  type: PERSISTENT
  mountPoints:
  - mountPath: /tmp
    mode: READ_WRITE

# After (instance template snippet)
disks:
- autoDelete: true
  boot: true
  initializeParams:
    sourceImage: projects/debian-cloud/global/images/family/debian-11
# No shared-tmp disk – /tmp defaults to tmpfs (2 GiB) on the VM

Local tmpfs guarantees exclusive lock files per instance, eliminating FileLockError.

2. Increase container memory limit and enable CPU burst


# Before (container definition in deployment.yaml)
resources:
  limits:
    memory: "2Gi"
    cpu: "1"
  requests:
    memory: "1Gi"
    cpu: "0.5"

# After
resources:
  limits:
    memory: "6Gi"
    cpu: "2"
  requests:
    memory: "4Gi"
    cpu: "1"

Raising memory to 6 GiB accommodates the peak PDF extraction footprint. Adding CPU burst reduces throttling during traffic spikes.

3. Enforce environment variable consistency

Add a validation step in the container entrypoint that aborts early if required variables are missing.


# entrypoint.sh (before)
exec python -m rag_loader

# entrypoint.sh (after)
#!/bin/bash
required_vars=(VECTOR_DB_ENDPOINT OPENAI_API_KEY)
for var in "${required_vars[@]}"; do
  if [[ -z "${!var}" ]]; then
    echo "ERROR: missing required env var $var" >&2
    exit 1
  fi
done
exec python -m rag_loader

4. Update canary traffic split to gradual ramp‑up

Modify the load balancer rule to start with 5 % traffic to the canary, then increase to 20 % after a health check window, as described in the Google Cloud Load Balancing – Traffic splitting guide.

Verify

After applying the changes, perform the following validation steps:

  1. Deploy the updated instance template and rollout the canary.
  2. Trigger a batch ingestion of 100 mixed‑format documents (PDF, DOCX, HTML).
  3. Confirm that no FileLockError, ResourceExhausted, or KeyError entries appear in Cloud Logging for the canary instance.
  4. Check vector store row count matches the input batch size.
  5. Run a health‑check endpoint (/healthz) that reports loader_status: ok.

# Example health check curl
curl -s http://canary-instance:8080/healthz
{
  "loader_status": "ok",
  "memory_usage_mb": 1523,
  "cpu_throttling": false
}

Prevent

To avoid recurrence, adopt the following guardrails:

  • Isolate temporary directories. Never share /tmp or other lock‑file locations across instances unless a distributed lock service (e.g., etcd) is used.
  • Resource safety margins. Allocate at least 2× the expected peak memory for document parsing workloads; monitor container_memory_usage_bytes and set alerts at 80 % utilization.
  • Configuration validation CI. Include a lint step that checks Dockerfile and Helm chart for required env vars; fail the build if any are missing.
  • Canary ramp‑up policy. Use Cloud Load Balancing’s gradual traffic shift feature and enforce a minimum 5‑minute health‑check window before increasing traffic share.
  • Structured logging for ingestion. Emit a JSON field ingestion_batch_id with each batch start/end to correlate failures across instances (see Google Cloud Logging – Structured logs).

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

  1. Why does the loader crash only during canary, not in full production?

    The canary shares the same NFS /tmp mount with the stable instance, creating lock contention that only appears when both versions run concurrently. Full production runs a single version, so the race does not occur.

  2. Can I keep using a shared NFS mount if I need persistent temporary files?

    Yes, but replace file‑based locks with a distributed lock service (e.g., redis-lock) or place lock files in a per‑instance subdirectory (e.g., /tmp/instance-${HOSTNAME}) to avoid cross‑instance collisions.

  3. How do I know if my container is being OOM‑killed?

    Check the kubelet or Docker events for OOMKilled and the container_memory_usage_bytes metric in Cloud Monitoring. The log line ResourceExhausted: Out of memory also indicates a pre‑kill condition.

  4. What is the recommended way to test the loader under load before a canary rollout?

    Use a synthetic load generator (e.g., locust or hey) against a staging instance group with the same instance template and traffic split settings. Capture logs and monitor memory/CPU to confirm no lock or OOM events.

  5. Is there a way to automatically retry failed batches without losing data?

    Implement an idempotent batch processor that writes a batch status record to a persistent store (e.g., Cloud Firestore). On restart, the loader can query for incomplete batches and resume processing.