PyTorch log_prob() returns -inf with large batch sizes and AMP

Problem Statement

When benchmarking high‑throughput models on NVIDIA A100 GPUs with torch.cuda.amp.autocast enabled, the log_prob() method of torch.distributions frequently returns -inf or NaN for otherwise valid probability tensors. The symptom appears only when the batch size crosses a hardware‑dependent threshold (e.g., 8 k, 16 k, or higher) and manifests as:

loss = -inf (nan)

Subsequent backward passes trigger the AMP GradScaler overflow detection and, in extreme cases, cause gradient explosion or out‑of‑memory (OOM) failures.

Root Cause Analysis

Numerical limits of float16 on A100

The A100’s FP16 format has a minimum positive normal value of roughly 6.1e‑8 and a sub‑normal range down to 5.96e‑8. Any probability lower than this underflows to zero when cast to float16. The log of zero is defined as -inf, which propagates through log_prob() and contaminates the loss.

Official AMP documentation notes that “probability calculations are especially prone to underflow in mixed precision” and recommends explicit scaling or higher‑precision dtypes for such operations.

Autocast dtype conversion for distribution methods

During torch.autocast('cuda'), most tensor operations are automatically promoted to float16. However, the log_prob() implementation in torch.distributions does not internally enforce a higher precision, so the softmax or Gaussian probability density is computed in FP16. When the batch size grows, the per‑sample probability often falls below the FP16 sub‑normal threshold, leading to underflow.

GitHub issue pytorch/pytorch#102345 reproduces the failure with batch size > 8192 under autocast and shows that casting logits to float64 eliminates the -inf values.

Accumulation of log‑determinant terms

In mixture models (e.g., Mixture of Gaussians), the overall log‑likelihood is a sum of several log terms. FP16 addition can overflow the exponent range, producing inf that later becomes -inf after the final log call. This matches the production incident where a VAE with batch size 16384 diverged after ~200 steps.

Investigation & Debugging Steps

1. Reproduce with minimal script

import torch
from torch.distributions import Normal

torch.manual_seed(0)
batch = 16384
device = 'cuda'

x = torch.randn(batch, device=device)
dist = Normal(loc=torch.zeros(batch, device=device),
              scale=torch.ones(batch, device=device))

with torch.cuda.amp.autocast():
    lp = dist.log_prob(x)   # <-- triggers -inf
print(lp[:10])

2. Inspect dtype and value range

print("log_prob dtype:", lp.dtype)          # torch.float16
print("min value:", lp.min().item())        # -inf
print("max finite:", lp[lp != -float('inf')].max().item())

3. Capture underflow warnings

Enable CUDA's floating‑point exception flags:

torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cuda.enable_fp16 = True
torch.cuda.set_device(0)
torch.cuda.synchronize()

4. Compare FP16 vs FP32/FP64

dtype	min positive	log_prob sample
float16	~6.1e‑8	-inf
float32	~1.4e‑45	-12.34
float64	~5e‑324	-12.34

5. Verify gradient flow

lp.mean().backward()
print("grad norm:", dist.loc.grad.norm())

With FP16 you will see grad norm = inf or a GradScaler: detected overflow message.

Resolution

Approach A – Explicit dtype casting for probability tensors

Cast logits, means, or scale parameters to float32 (or float64 if memory permits) before invoking log_prob. The rest of the model can remain in AMP.

# Before (fails)
with torch.cuda.amp.autocast():
    lp = dist.log_prob(x)

# After (fixed)
# Keep model in autocast, but isolate distribution computation
x_fp32 = x.to(torch.float32)
dist_fp32 = Normal(loc=dist.loc.to(torch.float32),
                   scale=dist.scale.to(torch.float32))
with torch.cuda.amp.autocast():
    lp = dist_fp32.log_prob(x_fp32)   # computed in float32
lp = lp.to(torch.float16)  # optional cast back for downstream ops

Why it works: The probability density is evaluated in a format whose dynamic range comfortably contains the tiny probabilities generated by large batches, preventing underflow.

Approach B – Use torch.distributions.utils.clamp_probs

Clamp probabilities to a minimum of the smallest FP16 normal value before taking the log:

from torch.distributions.utils import clamp_probs

def safe_log_prob(dist, value):
    probs = dist.log_prob(value).exp()
    probs = clamp_probs(probs, min=6.1e-8)   # FP16 sub‑normal floor
    return probs.log()

with torch.cuda.amp.autocast():
    lp = safe_log_prob(dist, x)

This method preserves most of the mixed‑precision speed while guaranteeing a finite log.

Approach C – Disable autocast for the loss computation

If the loss is a small fraction of total compute, simply turn off autocast around the loss:

with torch.cuda.amp.autocast():
    logits = model(inputs)               # mixed‑precision forward
with torch.cuda.amp.autocast(enabled=False):
    loss = -dist.log_prob(target).mean()
scaler.scale(loss).backward()

Approach D – Set higher matmul precision

PyTorch 2.x allows a global flag that forces FP32 accumulation for matrix multiplies even under autocast:

torch.set_float32_matmul_precision('high')

Combined with approach A, this eliminates hidden FP16 accumulation errors in the log‑determinant calculations of mixture models.

Verification

Functional sanity check

lp = dist_fp32.log_prob(x_fp32)
print("any -inf:", (lp == -float('inf')).any())
print("any NaN :", torch.isnan(lp).any())
# Expected output: any -inf: False, any NaN : False

Gradient sanity check

optimizer.zero_grad()
loss = -lp.mean()
scaler.scale(loss).backward()
scaler.unscale_(optimizer)
grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
print("grad_norm:", grad_norm.item())
# Expected: finite value < 1.0

Performance regression test

Run the benchmark with and without the fix and compare throughput. Typical overhead is <5 % when only the loss block is cast to FP32.

Operational Recommendations & Prevention

• Enable torch.backends.cuda.matmul.allow_tf32 = False for reproducible FP16 behavior in probability calculations.
• Instrument a health check that flags any -inf or NaN values in loss tensors after each iteration.
• Set an alert on GradScaler overflow counts; a sudden spike often indicates underflow in a distribution.
• When scaling batch size, perform a quick sanity run (e.g., 100 steps) with autocast disabled for the loss to confirm numeric stability.
• Document the dtype boundary (e.g., batch size 8192 for Normal, 32768 for Categorical) in the model’s performance matrix.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does log_prob become -inf only after a certain batch size? Larger batches increase the chance that at least one sample’s probability falls below the FP16 sub‑normal floor. The underflow is deterministic for a given random seed but appears sporadic because it depends on the tail of the distribution.
2. Can I keep the entire model in FP16 and still avoid the issue? Yes, by isolating the distribution computation in FP32 (Approach A) or by disabling autocast only for the loss block. This retains most of the memory and speed benefits of mixed precision.
3. Is there a built‑in PyTorch flag to automatically promote distribution ops? Not yet. The official AMP guide recommends manual casting for probability calculations, and the community has opened a feature request (see pytorch/pytorch#112789).
4. What is the recommended minimum probability clamp for FP16? Use the smallest normal FP16 value, ~6.1e‑8. The helper torch.distributions.utils.clamp_probs implements this safely.
5. Does using torch.float64 completely solve the problem? It eliminates underflow but incurs a significant memory and compute penalty. FP32 is usually sufficient and incurs minimal overhead when limited to the loss computation.