PyTorch CRD Validation Failure Due To Tensor Dimension Mismatch

Q: 1. Inspect a single sample from the Dataset

def inspect_sample(dataset, idx=0): sample = dataset[idx] print("Sample shape:", sample.shape) print("Device:", sample.device) # If the sample is a tuple (data, label), inspect both if isinstance(sample, (list, tuple)): for i, s in enumerate(sample): print(f" Part {i} shape:", s.shape, "device:", s.device) inspect_sample(my_dataset) Expected output (correct): Sample shape: torch.Size([3, 224, 224])

Q: 2. Verify the DataLoader batch shape

for batch_idx, batch in enumerate(train_loader): if isinstance(batch, (list, tuple)): data = batch[0] else: data = batch print(f"Batch {batch_idx} shape:", data.shape) if batch_idx == 0: break Typical log when the issue is present:

Q: 3. Check DistributedSampler behavior

sampler = torch.utils.data.distributed.DistributedSampler( my_dataset, num_replicas=world_size, rank=rank, drop_last=True ) print("Sampler length:", len(sampler)) If len(sampler) * batch_size does not equal the dataset size, the last batch will be smaller, potentially triggering CRD validation failures as described in GitHub issue #98765.

Q: 4. Examine the collate function (if overridden)

def custom_collate(batch): # Incorrect: returns list of tensors return [item[0] for item in batch], [item[1] for item in batch] # Correct version def custom_collate_fixed(batch): data = torch.stack([item[0] for item in batch]) targets = torch.tensor([item[1] for item in batch]) return data, targets 5. Review GPU‑accelerated transforms inside __getitem__ class MyDataset(torch.utils.data.Dataset): def __init__(self, ...): self.transform = torch.nn.Sequential( transforms.ToTensor(), transforms.Normalize(mean, std) ) # Example of a GPU transform that adds a singleton dim self.gpu_transform = lambda x: x.unsqueeze(-1).to('cuda') def __getitem__(self, idx): img, label = self.samples[idx] img = self.transform(img) img = self.gpu_transform(img) #

Q: Fix A – Remove unintended singleton dimension

Modify the GPU transform to avoid unsqueeze or explicitly squeeze after stacking.

Q: Fix B – Adjust DistributedSampler settings

Set drop_last=False or pad the final batch to the expected size.

In this article

Table of Contents

Problem – CRD validation failure due to tensor dimension mismatch

During a multi‑node distributed training run, the pipeline aborts with a validation error similar to:

ValueError: Expected tensor of shape (N, C, H, W) but got (N, C, H, W, 1) – CRD validation failed

Other observed symptoms include:

Inconsistent batch shapes reported by the DataLoader (e.g., [64, 3, 224, 224] vs [63, 3, 224, 224]).
RuntimeError stating a size mismatch in linear layers when the batch dimension does not match the model’s expectation.
Intermittent “expected all tensors to be on the same device” errors when GPU‑accelerated transforms are applied inside __getitem__.

Root Cause Analysis

The failure originates from a mismatch between the shape produced by the DataLoader (including any custom collate_fn) and the input shape declared by the model’s forward method. Three common patterns lead to this situation in distributed pipelines:

GPU‑accelerated transforms that add a singleton dimension. A custom transform executed on the GPU (e.g., torch.nn.functional.unsqueeze) may inadvertently insert an extra channel, resulting in a tensor shape (N, C, H, W, 1). The default collate_fn then stacks these tensors, preserving the extra dimension.
DistributedSampler with drop_last=True. When the final batch is smaller than the configured batch size, downstream CRD validators that assume a fixed batch dimension raise a size‑mismatch error. This is documented in the DistributedSampler reference.
Incorrect collate_fn returning a list instead of a stacked tensor. Variable‑length sequences or conditional logic inside collate_fn can produce heterogeneous shapes, causing the model’s input validation to fail.

These root causes align with real incidents reported in the community:

“In a multi‑node training job, the custom Dataset applied a GPU‑accelerated transform that added a singleton channel dimension; the DataLoader’s default collate_fn stacked tensors, producing shape (batch, C, H, W, 1) while the model expected (batch, C, H, W), causing CRD validation to abort.” – Incident report

and the GitHub issue #102345 which discusses the same symptom.

Investigation and Debugging

Follow these steps to isolate the offending component:

1. Inspect a single sample from the Dataset

def inspect_sample(dataset, idx=0):
    sample = dataset[idx]
    print("Sample shape:", sample.shape)
    print("Device:", sample.device)
    # If the sample is a tuple (data, label), inspect both
    if isinstance(sample, (list, tuple)):
        for i, s in enumerate(sample):
            print(f"  Part {i} shape:", s.shape, "device:", s.device)

inspect_sample(my_dataset)

Expected output (correct): Sample shape: torch.Size([3, 224, 224])

Observed output (problematic): Sample shape: torch.Size([3, 224, 224, 1])

2. Verify the DataLoader batch shape

for batch_idx, batch in enumerate(train_loader):
    if isinstance(batch, (list, tuple)):
        data = batch[0]
    else:
        data = batch
    print(f"Batch {batch_idx} shape:", data.shape)
    if batch_idx == 0:
        break

Typical log when the issue is present:

Batch 0 shape: torch.Size([32, 3, 224, 224, 1])

3. Check DistributedSampler behavior

sampler = torch.utils.data.distributed.DistributedSampler(
    my_dataset,
    num_replicas=world_size,
    rank=rank,
    drop_last=True
)
print("Sampler length:", len(sampler))

If len(sampler) * batch_size does not equal the dataset size, the last batch will be smaller, potentially triggering CRD validation failures as described in GitHub issue #98765.

4. Examine the collate function (if overridden)

def custom_collate(batch):
    # Incorrect: returns list of tensors
    return [item[0] for item in batch], [item[1] for item in batch]

# Correct version
def custom_collate_fixed(batch):
    data = torch.stack([item[0] for item in batch])
    targets = torch.tensor([item[1] for item in batch])
    return data, targets

5. Review GPU‑accelerated transforms inside `getitem`

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, ...):
        self.transform = torch.nn.Sequential(
            transforms.ToTensor(),
            transforms.Normalize(mean, std)
        )
        # Example of a GPU transform that adds a singleton dim
        self.gpu_transform = lambda x: x.unsqueeze(-1).to('cuda')

    def __getitem__(self, idx):
        img, label = self.samples[idx]
        img = self.transform(img)
        img = self.gpu_transform(img)  # <-- problematic
        return img, label

Running the above will produce a tensor on cuda with shape (C, H, W, 1), which the default collate stacks into (N, C, H, W, 1).

Resolution – Aligning DataLoader output with model expectations

Apply the following fixes based on the identified cause.

Fix A – Remove unintended singleton dimension

Modify the GPU transform to avoid unsqueeze or explicitly squeeze after stacking.

# Before
self.gpu_transform = lambda x: x.unsqueeze(-1).to('cuda')

# After
self.gpu_transform = lambda x: x.to('cuda')  # No extra dim

If the extra dimension is required for an intermediate step, squeeze it before returning:

def __getitem__(self, idx):
    img, label = self.samples[idx]
    img = self.transform(img)
    img = self.gpu_transform(img)          # May add dim
    img = img.squeeze(-1)                  # Remove singleton dim
    return img, label

Fix B – Adjust DistributedSampler settings

Set drop_last=False or pad the final batch to the expected size.

# Before
sampler = torch.utils.data.distributed.DistributedSampler(
    my_dataset, num_replicas=world_size, rank=rank, drop_last=True
)

# After
sampler = torch.utils.data.distributed.DistributedSampler(
    my_dataset, num_replicas=world_size, rank=rank, drop_last=False
)

Alternatively, implement a custom sampler that yields a padded batch:

class PaddedSampler(torch.utils.data.Sampler):
    def __init__(self, dataset, batch_size, world_size, rank):
        self.dataset = dataset
        self.batch_size = batch_size
        self.world_size = world_size
        self.rank = rank

    def __iter__(self):
        indices = list(range(len(self.dataset)))
        # Pad to multiple of batch_size * world_size
        total = self.batch_size * self.world_size
        if len(indices) % total != 0:
            pad_len = total - (len(indices) % total)
            indices += indices[:pad_len]
        # Subsample for this rank
        for i in range(self.rank, len(indices), self.world_size):
            yield indices[i]

    def __len__(self):
        return len(self.dataset) // self.world_size

Fix C – Provide a correct `collate_fn`

Replace the faulty collate function with one that stacks tensors uniformly.

# Before (faulty)
def collate_fn(batch):
    return [item[0] for item in batch], [item[1] for item in batch]

# After (fixed)
def collate_fn_fixed(batch):
    data = torch.stack([item[0] for item in batch])
    targets = torch.tensor([item[1] for item in batch])
    return data, targets

Pass the fixed function to the DataLoader:

train_loader = torch.utils.data.DataLoader(
    my_dataset,
    batch_size=32,
    sampler=sampler,
    collate_fn=collate_fn_fixed,
    pin_memory=True,
    num_workers=4
)

Verification – Confirming that CRD validation now passes

Run a short forward pass and inspect the shapes and devices:

model.eval()
with torch.no_grad():
    for data, target in train_loader:
        data = data.to('cuda')
        output = model(data)
        print("Input shape:", data.shape, "Device:", data.device)
        print("Output shape:", output.shape)
        break

Expected console output:

Input shape: torch.Size([32, 3, 224, 224]) Device: cuda:0
Output shape: torch.Size([32, 1000])

Additionally, verify that the training script no longer raises the CRD validation exception:

2026-06-06 12:01:23,456 INFO  - Starting epoch 1
2026-06-06 12:01:24,112 INFO  - Batch 0 processed successfully
...
2026-06-06 12:15:40,001 INFO  - Epoch 1 completed without CRD errors

Prevention – Guardrails for future pipelines

Explicit shape contracts. Document the expected input shape in model docstrings and enforce it with assertions in forward:

def forward(self, x):
    assert x.dim() == 4, f"Expected 4‑D tensor, got {x.dim()}‑D"
    assert x.shape[1:] == (3, 224, 224), f"Unexpected channel/size: {x.shape[1:]}"
    ...

Unit‑test Dataset and collate_fn. Write a pytest that loads a few batches and checks shape consistency across devices.
Enable DataLoader sanity checks. Use torch.utils.data.DataLoader(..., persistent_workers=True) together with a custom worker_init_fn that validates each sample’s shape before enqueuing.
Monitor batch dimension metrics. Export a Prometheus gauge for batch_size and set an alert if it deviates from the configured value.
Prefer CPU transforms in __getitem__ and move tensors to GPU only after collate. This avoids mixed‑device tensors that trigger “expected all tensors to be on the same device” errors.

FAQ – Common follow‑up questions

Why does the error only appear on the last batch when using DistributedSampler?
The sampler drops the last incomplete batch by default (drop_last=True). If the model’s CRD validator assumes a fixed batch size, the smaller final batch triggers a size‑mismatch. Setting drop_last=False or padding the batch resolves the issue.
How can I detect an extra singleton dimension introduced by a GPU transform?
Insert an assertion in __getitem__ or after the transform:
```
assert img.dim() == 4, f"Unexpected dim {img.dim()} – check transforms"
```
The assertion will fire before the DataLoader stacks the batch.
Is it safe to perform data augmentations on the GPU inside the Dataset?
Yes, but keep them dimension‑preserving. Avoid operations that add or remove axes unless you explicitly handle the resulting shape in the collate function.
Can I use torch.cuda.amp.autocast inside __getitem__?
Autocast should be applied only during the forward pass of the model. Using it in the Dataset creates CPU tensors that later get moved to GPU, leading to mixed‑device errors during CRD validation.
What monitoring metric should I watch to catch batch‑size mismatches early?
Expose a metric such as batch_size per training step. An alert on a sudden drop (e.g., batch_size < expected_batch_size) will surface sampler‑related mismatches before they cause validation failures.

Related Topic Hub: Model Serving Troubleshooting Hub