Problem
During training of large transformer models (e.g., BERT‑large, GPT‑2) on NVIDIA A100 GPUs, the training job intermittently aborts with a CUDA out‑of‑memory (OOM) error:
Traceback (most recent call last):
File "train.py", line 112, in
trainer.train()
File "/opt/torch/torch/nn/modules/module.py", line 1190, in __call__
return self.forward(*input, **kwargs)
File "trainer.py", line 237, in train_step
loss.backward()
File "/opt/torch/torch/cuda/__init__.py", line 123, in _lazy_init
torch._C._cuda_init()
RuntimeError: CUDA out of memory. Tried to allocate 2.14 GiB (GPU 0; 40.00 GiB total capacity; 31.12 GiB already allocated; 0.00 GiB free; 31.12 GiB reserved in total)
Despite nvidia-smi showing ~38 GiB free before the failure, the same batch size runs successfully on the same hardware when the job is restarted. The environment is an air‑gapped data center; drivers and CUDA toolkit are installed from a local repository, and the Linux kernel includes a custom patch that modifies the mmu_notifier subsystem.
Root Cause
The intermittent OOM originates from a regression introduced by the custom kernel patch. The patch disables the mmu_notifier callback that CUDA’s UVM driver uses to release page‑table entries when the GPU memory manager frees a block. When the callback is suppressed, the kernel’s page‑fault handling path retains stale GPU page mappings, causing the UVM driver to believe that memory is still in use. Over time, the driver’s internal accounting diverges from the actual hardware allocation, and subsequent cudaMalloc calls see less free memory than physically available, triggering the OOM error.
Key observations that point to this cause:
- OOM occurs only after the kernel has been patched; a clean kernel boots the same workload without error.
- GPU memory usage reported by
nvidia-smijumps from ~31 GiB to ~38 GiB without any new allocation, indicating that the driver’s internal free‑list is out of sync. - Kernel logs contain repeated
UVM: failed to release page table entrymessages after the patch.
Debug
1. Capture GPU memory state before failure
# Check GPU memory before the training step that fails
nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv
# Example output
GPU, memory.total [MiB], memory.used [MiB], memory.free [MiB]
0, 40960, 31200, 9760
2. Inspect kernel logs for UVM warnings
$ sudo journalctl -k -g "UVM"
Nov 08 10:15:32 host kernel: [UVM] failed to release page table entry: -22
Nov 08 10:15:32 host kernel: [UVM] mmu_notifier callback disabled by custom patch
3. Verify the presence of the custom patch
$ grep -R "mmu_notifier" /usr/src/linux-*/patches/custom/
--- a/mm/mmu_notifier.c
+++ b/mm/mmu_notifier.c
@@ -45,7 +45,7 @@
- register_mmu_notifier(&uvm_mmu_notifier);
+ /* disabled for security compliance */
4. Reproduce the issue with a minimal script
import torch
torch.cuda.init()
for i in range(100):
a = torch.randn(1024, 1024, device='cuda')
del a
torch.cuda.empty_cache()
Running the script on the patched kernel triggers the same OOM after ~70 iterations, while the same script runs to completion on an unpatched kernel.
5. Correlate driver version with kernel patch
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 525.85.12 Thu Sep 14 20:00:00 UTC 2023
The driver version is recent enough to rely on mmu_notifier. Disabling the notifier therefore breaks the driver’s memory reclamation path.
Solution
Option A – Restore the original mmu_notifier callback (recommended)
Remove the offending lines from the custom patch and rebuild the kernel.
# Original patch fragment (to be removed)
- /* disabled for security compliance */
+ register_mmu_notifier(&uvm_mmu_notifier);
Rebuild steps (example for Ubuntu 22.04 with DKMS):
$ sudo apt-get install build-essential libncurses-dev bc
$ cd /usr/src/linux-5.15.0
$ make olddefconfig
$ make -j$(nproc)
$ sudo make modules_install install
$ sudo update-grub
$ sudo reboot
After reboot, verify that the notifier is active:
$ sudo dmesg | grep "mmu_notifier"
[ 0.123456] UVM: mmu_notifier enabled
Option B – Patch the NVIDIA driver to tolerate missing callbacks
If the kernel cannot be changed (e.g., compliance constraints), apply NVIDIA’s “UVM fallback” patch (available in the driver source tarball) and rebuild the driver:
$ tar -xf NVIDIA-Linux-x86_64-525.85.12.run --strip-components=1 -C /tmp/nvidia-driver
$ cd /tmp/nvidia-driver
$ patch -p1 < patches/uvm_mmu_notifier_fallback.patch
$ sudo ./nvidia-installer --no-cc-version-check --silent
This patch makes the driver fall back to a slower, reference‑count based release path when mmu_notifier is unavailable. Performance impact is typically <5 % for transformer workloads.
Option C – Reduce GPU memory pressure as a temporary workaround
Adjust the training script to use gradient checkpointing and smaller micro‑batch sizes. This avoids hitting the faulty reclamation path but does not address the root cause.
# Before (full backpropagation)
batch_size = 64
# After (gradient checkpointing)
batch_size = 64
torch.utils.checkpoint.checkpoint_sequential(model, segments=4, input=inputs)
Verify
After applying Option A (restoring the notifier), repeat the minimal script:
$ python test_uvm.py
Iteration 1/100 – OK
...
Iteration 100/100 – OK
Run the full training job and monitor GPU memory:
$ watch -n 1 nvidia-smi
Expected outcome: memory usage stabilizes around the peak allocation (e.g., 31 GiB) without sudden drops or OOM errors. The training completes without interruption.
Prevent
- Kernel‑driver compatibility matrix: Maintain a documented matrix that records which kernel patches are safe for the installed NVIDIA driver version. Automate a CI check that runs
nvidia-smi -qand validatesmmu_notifierpresence. - Automated health check at node startup:
#!/bin/bash if ! dmesg | grep -q "UVM: mmu_notifier enabled"; then echo "Critical: UVM mmu_notifier disabled – node will be cordoned" kubectl cordon $(hostname) exit 1 fi - Monitoring: Add a Prometheus metric that alerts when
nvidia_smi_memory_used_bytesexceeds 90 % of total memory for more than 5 minutes, and a separate alert for repeatedUVM: failed to release page table entrykernel messages. - Patch review process: Require that any kernel modification touching memory‑management subsystems be reviewed by a GPU‑aware engineer and tested against the driver’s
selftests/uvmsuite.
Related Topic Hub: GPU Infrastructure Troubleshooting Hub
FAQ
- Why does the OOM appear only intermittently? The driver’s internal free‑list gradually diverges from the hardware state. Early allocations succeed; after enough stale page tables accumulate, the driver believes memory is exhausted, causing a sudden OOM.
- Can I detect the missing
mmu_notifierwithout rebooting? Yes. Rundmesg | grep "UVM: mmu_notifier". If the output contains “disabled” or is absent, the notifier is not active. - Does disabling the notifier affect other CUDA applications? Any workload that relies on UVM (e.g., multi‑process service, unified memory) will experience similar reclamation issues. Purely statically allocated kernels may continue to work.
- Is the driver fallback patch safe for production? It restores functionality but incurs a modest performance penalty. It should be used only as a temporary measure while the kernel is being corrected.
- How can I test my kernel changes for GPU compatibility before deployment? Build a minimal Docker image with the target kernel headers, compile the NVIDIA driver inside the container, and run the
selftests/uvmsuite. Any failures indicate incompatibility.