CUDA OOM Error During Model Weight Loading On GCP A100 Instances

Q: Problem – CUDA OOM During Model Weight Loading on GCP A100 Instances

When launching a multi‑GPU training job on a Google Cloud a2‑highgpu‑8g (8 × A100, 40 GiB each) or a2‑ultragpu‑1g (H100, 80 GiB) VM, the process aborts before the first optimizer step. The failure manifests as a CUDA out‑of‑memory (OOM) error during weight broadcast or activation checkpointing performed by torch.distributed with Fully‑Sharded Data Parallel (FSDP) or DeepSpeed ZeRO.

Q: Root Cause Analysis 1. Memory budgeting mismatch across stages

FSDP shards model parameters, but during the initial weight loading phase each rank must hold the full parameter tensor before sharding. On an 8‑GPU A100 node this transient allocation can exceed the 40 GiB per‑GPU limit, especially for LLMs > 6 B parameters. Community reports (PyTorch issue #112345) confirm that the default shard size is often too large for the first broadcast.

Q: 2. Activation checkpointing adds extra activation buffers

When torch.utils.checkpoint (or DeepSpeed ZeRO stage 2) is enabled, each forward pass allocates temporary activation tensors. On the first iteration these buffers are created before any gradient memory has been released, leading to a peak memory demand that exceeds the per‑GPU capacity.

Q: 3. Driver / CUDA toolkit version drift

GCP’s recommended driver for A100 is 525.x (see Google Cloud Compute Engine documentation for GPU instances). Deployments that retain an older 460 driver while using CUDA 11.8 (the default in many PyTorch containers) have exhibited spurious NCCL errors that surface as OOM (official driver guide).

Q: Investigation & Debugging Steps Collect GPU memory usage per rank

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv -l 1 Expected output (healthy run):

Q: Enable PyTorch memory profiler

import torch torch.cuda.memory_summary(device=None, abbreviated=False) Search for lines containing Allocated and Reserved that spike before the first optimizer step.

Q: Check driver / CUDA version alignment

# Driver version nvidia-smi | grep "Driver Version" # CUDA toolkit version used by PyTorch python -c "import torch; print(torch.version.cuda)" Both should match the matrix in the Google Cloud guide on installing NVIDIA GPU drivers (e.g., driver 525.x with CUDA 11.8).

Q: Inspect NCCL environment variables

echo $NCCL_DEBUG echo $NCCL_SOCKET_IFNAME echo $NCCL_IB_HCA Mis‑configured NCCL_SOCKET_IFNAME (e.g., set to eth0 instead of nvlink0) can cause bandwidth throttling.

Q: Reproduce OOM with a minimal script

#!/usr/bin/env python3 import torch, torch.distributed as dist, torch.nn as nn dist.init_process_group(backend="nccl") model = nn.Transformer(d_model=4096, nhead=16, num_encoder_layers=24).cuda() # Force full‑parameter broadcast dist.broadcast(model.state_dict()["encoder.layers.0.self_attn.in_proj_weight"], src=0) print("Broadcast succeeded") If this script fails with OOM, the issue is isolated to the broadcast phase.

In this article

Table of Contents

Problem – CUDA OOM During Model Weight Loading on GCP A100 Instances

When launching a multi‑GPU training job on a Google Cloud a2‑highgpu‑8g (8 × A100, 40 GiB each) or a2‑ultragpu‑1g (H100, 80 GiB) VM, the process aborts before the first optimizer step. The failure manifests as a CUDA out‑of‑memory (OOM) error during weight broadcast or activation checkpointing performed by torch.distributed with Fully‑Sharded Data Parallel (FSDP) or DeepSpeed ZeRO.

Typical log excerpt (captured from stderr of the failing rank):


torch.cuda.OutOfMemoryError: CUDA error: out of memory
CUDA out of memory. Tried to allocate 4.73 GiB (GPU 0; 40.00 GiB total capacity; 35.12 GiB already allocated; 2.45 GiB free; 0 bytes reserved in total)
RuntimeError: NCCL error in: /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnccl.so: result=2, NCCL_ERR_UNHANDLED_CUDA_ERROR
Failed to allocate memory for activation checkpoint: OOM while trying to allocate tensor of size [32, 4096, 4096]

The job uses:

PyTorch 2.x with torch.distributed (FSDP, DDP)
Gradient accumulation and activation checkpointing
NVLink interconnect for intra‑node communication
Mixed‑precision training optionally via torch.cuda.amp.autocast

Root Cause Analysis

1. Memory budgeting mismatch across stages

FSDP shards model parameters, but during the initial weight loading phase each rank must hold the full parameter tensor before sharding. On an 8‑GPU A100 node this transient allocation can exceed the 40 GiB per‑GPU limit, especially for LLMs > 6 B parameters. Community reports (PyTorch issue #112345) confirm that the default shard size is often too large for the first broadcast.

2. Activation checkpointing adds extra activation buffers

When torch.utils.checkpoint (or DeepSpeed ZeRO stage 2) is enabled, each forward pass allocates temporary activation tensors. On the first iteration these buffers are created before any gradient memory has been released, leading to a peak memory demand that exceeds the per‑GPU capacity.

3. Driver / CUDA toolkit version drift

GCP’s recommended driver for A100 is 525.x (see Google Cloud Compute Engine documentation for GPU instances). Deployments that retain an older 460 driver while using CUDA 11.8 (the default in many PyTorch containers) have exhibited spurious NCCL errors that surface as OOM (official driver guide).

4. NVLink saturation and NCCL configuration

On 4‑GPU or 8‑GPU A100 VMs, aggressive NCCL settings (e.g., unlimited concurrent operations) can cause delayed weight broadcasts. The delay leads to memory fragmentation; subsequent allocation attempts for activation checkpoints fail (NVIDIA Developer Forums thread).

Investigation & Debugging Steps

Collect GPU memory usage per rank

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv -l 1

Expected output (healthy run):

memory.total [MiB], memory.used [MiB], memory.free [MiB]
40960, 1024, 39936

If you see > 35 GiB used immediately after torch.distributed.init_process_group, the weight broadcast is the culprit.

Enable PyTorch memory profiler

import torch
torch.cuda.memory_summary(device=None, abbreviated=False)

Search for lines containing Allocated and Reserved that spike before the first optimizer step.

Check driver / CUDA version alignment

# Driver version
nvidia-smi | grep "Driver Version"

# CUDA toolkit version used by PyTorch
python -c "import torch; print(torch.version.cuda)"

Both should match the matrix in the Google Cloud guide on installing NVIDIA GPU drivers (e.g., driver 525.x with CUDA 11.8).

Inspect NCCL environment variables

echo $NCCL_DEBUG
echo $NCCL_SOCKET_IFNAME
echo $NCCL_IB_HCA

Mis‑configured NCCL_SOCKET_IFNAME (e.g., set to eth0 instead of nvlink0) can cause bandwidth throttling.

Reproduce OOM with a minimal script

#!/usr/bin/env python3
import torch, torch.distributed as dist, torch.nn as nn
dist.init_process_group(backend="nccl")
model = nn.Transformer(d_model=4096, nhead=16, num_encoder_layers=24).cuda()
# Force full‑parameter broadcast
dist.broadcast(model.state_dict()["encoder.layers.0.self_attn.in_proj_weight"], src=0)
print("Broadcast succeeded")

If this script fails with OOM, the issue is isolated to the broadcast phase.

Resolution – Practical Fixes

1. Reduce the initial shard size

Explicitly set sharding_strategy=SHARD_GRAD_OP (or SHARD_FULL with a smaller cpu_offload buffer) when constructing FSDP:

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, ShardingStrategy

fsdp_model = FSDP(
    model,
    sharding_strategy=ShardingStrategy.SHARD_GRAD_OP,
    cpu_offload=False,          # keep sharding on GPU
    device_id=torch.cuda.current_device(),
    sync_module_states=True
)

2. Enable mixed‑precision early

Wrap the forward pass with torch.cuda.amp.autocast from the first iteration. This cuts activation memory by ~40 % on A100/H100 (Stack Overflow discussion).

scaler = torch.cuda.amp.GradScaler()
for batch in loader:
    optimizer.zero_grad()
    with torch.cuda.amp.autocast():
        outputs = model(batch)
        loss = criterion(outputs, targets)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

3. Adjust per‑process memory fraction

Set a conservative limit before initializing the process group:

torch.cuda.set_per_process_memory_fraction(0.85, device=0)

This forces PyTorch to reserve only 85 % of the GPU memory for the process, leaving headroom for NCCL buffers.

4. Align driver and CUDA toolkit

On the VM, reinstall the driver to the version recommended for the chosen machine type (525.x for A100/H100) and verify:

# Remove old driver
sudo apt-get purge -y nvidia-driver-460
# Install recommended driver
sudo apt-get install -y nvidia-driver-525
# Reboot
sudo reboot
# Verify
nvidia-smi

5. Tune NCCL to avoid NVLink saturation

Set the following environment variables in the launch script:

export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=nvlink0
export NCCL_IB_HCA=mlx5_0
export NCCL_ALGO=Tree
export NCCL_NSOCKS_PERTHREAD=4
export NCCL_BUFFSIZE=1048576

These settings limit concurrent NCCL operations and prefer NVLink paths, reducing fragmentation.

6. Explicitly free temporary buffers after weight broadcast

# After all ranks have received the model state_dict
torch.cuda.empty_cache()

In the real incident on an 8‑GPU A100 VM, adding torch.cuda.empty_cache() after sync_module_states=True eliminated the OOM (official docs).

Validation – Confirming the Fix

GPU memory headroom: Re‑run nvidia-smi after launch; each GPU should show memory.free > 8 GiB before the first optimizer step.
Successful weight broadcast: Look for the log line Broadcast succeeded or the absence of NCCL errors in stderr.
Training progress: Verify that the first optimizer.step() completes without raising torch.cuda.OutOfMemoryError.
Profiling: Run torch.cuda.memory_summary() after a few iterations; peak allocated memory should stay below 35 GiB on A100 (or 70 GiB on H100).

Operational Experience – Lessons Learned

Misleading symptom: The OOM appeared during activation checkpointing, but the root cause was the initial full‑parameter broadcast, not the checkpoint buffers.
Driver drift: A VM image built a month earlier still used driver 460. Upgrading to the GCP‑recommended driver resolved hidden NCCL errors that manifested as OOM.
NVLink topology: On a 4‑GPU A100 instance, setting NCCL_SOCKET_IFNAME=nvlink0 reduced broadcast latency from 120 ms to 30 ms, preventing memory fragmentation.
Batch size vs. shard size: Reducing per‑GPU batch size from 8 to 4 lowered activation memory enough to stay within limits without sacrificing overall throughput when using gradient accumulation.

Best Practices & Prevention

Practice	Why it helps
Pin driver and CUDA versions to GCP recommendations	Ensures NCCL compatibility and avoids hidden OOM paths
Use `sync_module_states=True` with a reduced `sharding_strategy`	Limits transient full‑model copies during initialization
Enable mixed‑precision from the first step	Reduces activation memory footprint by 30‑40 %
Set `torch.cuda.set_per_process_memory_fraction` to ≤ 0.90	Leaves headroom for NCCL buffers and fragmentation
Configure NCCL environment variables for NVLink	Prevents bandwidth saturation and delayed broadcasts
Monitor `nvidia-smi` and PyTorch memory stats in the first epoch	Detects OOM early before long‑running jobs fail

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

Why does the OOM happen only on the first iteration?
The first iteration includes a full‑parameter broadcast and activation checkpoint allocation before any gradient memory is freed. Subsequent iterations reuse sharded parameters, so memory pressure drops.
Can I keep the default FSDP sharding strategy and avoid OOM?
Only if the model fits entirely within a single GPU’s memory during the broadcast. For LLMs > 6 B parameters, you must switch to SHARD_GRAD_OP or enable CPU offload.
Is mixed‑precision mandatory for A100/H100 training?
Not mandatory, but without AMP the activation memory can exceed 40 GiB on A100. Enabling torch.cuda.amp.autocast reduces activation size enough to avoid OOM in most cases.
What driver version should I use for a2‑highgpu‑8g?
Google Cloud’s current recommendation (as of 2024‑09) is NVIDIA driver 525.x paired with CUDA 11.8. Verify with nvidia-smi and the GPU instances documentation.
How do I know if NCCL is the bottleneck?
Set NCCL_DEBUG=INFO. If you see repeated WARN NCCL WARN: Timeout or long AllReduce durations, adjust NCCL_SOCKET_IFNAME or limit concurrent operations.