PyTorch Tensor Shape Mismatch In RAG Context Injection

Q: Embedding dimension corruption

In a few incidents (see Real incidents), the padding routine exited early due to Docker resource limits, leaving some rows partially padded. When these rows are later passed through the embedding layer, the hidden dimension (e.g., 768) is preserved, but the effective sequence length is inconsistent, causing a runtime error like “expected hidden size 768 but got 1024”. This is a secondary effect of the same mis‑padding bug.

In this article

Table of Contents

Problem – Tensor Shape Mismatch in RAG Context Injection

When using a Retrieval‑Augmented Generation (RAG) pipeline inside a Docker‑based Jupyter sandbox, the model often drops or truncates tokens from the retrieved documents. The symptom manifests as missing context during generation and degraded answer quality.

Typical log excerpt:


context_input_ids.shape: torch.Size([3, 210])
context_attention_mask.shape: torch.Size([3, 128])
RuntimeError: Expected tensor for query to have shape [batch_size, seq_len, hidden_dim] but got [3, 210, 768]

Other observed errors include:

IndexError: index out of range in self‑attention (attention_mask size [batch, seq_len] does not match input size [batch, padded_len])
UserWarning: Token indices sequence length is longer than the specified maximum sequence length for this model (truncating)
ValueError: Mismatched dimensions between context_input_ids (torch.Size([batch, max_context_len])) and context_attention_mask (torch.Size([batch, original_max_len]))

Root Cause Analysis

Dynamic padding misalignment

The RAGRetriever returns a variable‑length list of tokenized documents per batch. A common pattern is to use torch.nn.utils.rnn.pad_sequence to create a tensor of shape (batch, max_len). However, the corresponding attention_mask is often built from the original (pre‑padding) lengths and left unchanged. This leads to a mismatch where the mask still reflects the shorter original length while the input IDs have been padded to a longer size.

According to the PyTorch MultiheadAttention documentation, the mask must have the same seq_len dimension as the query/key/value tensors. If the mask is shorter, the attention module silently drops the excess tokens, which is why the generation step appears to work but the context is truncated.

Embedding dimension corruption

In a few incidents (see Real incidents), the padding routine exited early due to Docker resource limits, leaving some rows partially padded. When these rows are later passed through the embedding layer, the hidden dimension (e.g., 768) is preserved, but the effective sequence length is inconsistent, causing a runtime error like “expected hidden size 768 but got 1024”. This is a secondary effect of the same mis‑padding bug.

Investigation and Debugging Steps

Reproduce the shape mismatch locally. Insert debug prints right after padding:


# After retrieving documents
doc_ids = [torch.tensor(d) for d in retrieved_ids]  # list of 1‑D tensors
padded_ids = pad_sequence(doc_ids, batch_first=True, padding_value=tokenizer.pad_token_id)
print("padded_ids.shape:", padded_ids.shape)  # e.g. torch.Size([3, 210])

# Build attention mask from original lengths
orig_lengths = torch.tensor([len(d) for d in doc_ids])
attention_mask = (torch.arange(padded_ids.size(1))[None, :] < orig_lengths[:, None]).long()
print("attention_mask.shape:", attention_mask.shape)  # should be torch.Size([3, 210])

If the second print shows (3, 128) while the first shows (3, 210), the mask is stale.

Validate mask broadcasting with MultiheadAttention. Run a minimal forward pass:


self_attn = torch.nn.MultiheadAttention(embed_dim=768, num_heads=12)
query = key = value = embedding_layer(padded_ids)  # shape [batch, seq, hidden]
output, _ = self_attn(query.transpose(0,1), key.transpose(0,1),
                     value.transpose(0,1), key_padding_mask=~attention_mask.bool())

If a RuntimeError about shape mismatch appears, the mask is the culprit.

Check Docker resource limits. Inspect container logs for OOM or CPU throttling that could abort the padding loop.


docker stats 
journalctl -u docker.service | grep -i oom

Resolution – Correct Padding and Mask Alignment

Unified padding utility

Replace ad‑hoc mask construction with a helper that guarantees shape parity:


def pad_and_mask(sequences, pad_token_id, device='cpu'):
    """
    Returns:
        padded_ids (Tensor): [batch, max_len]
        attention_mask (Tensor): [batch, max_len] (1 for real tokens, 0 for padding)
    """
    # Pad sequences
    padded_ids = torch.nn.utils.rnn.pad_sequence(
        sequences, batch_first=True, padding_value=pad_token_id
    ).to(device)

    # Create mask from padded shape
    mask = (padded_ids != pad_token_id).long()
    return padded_ids, mask

Integration into RAG pipeline

Before feeding context into the generator, invoke the utility:


# Retrieve and tokenize documents
retrieved_ids = [torch.tensor(tokenizer.encode(doc)) for doc in docs]

# Apply unified padding
context_input_ids, context_attention_mask = pad_and_mask(
    retrieved_ids,
    pad_token_id=tokenizer.pad_token_id,
    device='cuda'
)

# Verify shapes
assert context_input_ids.shape == context_attention_mask.shape, \
    f"Shape mismatch: {context_input_ids.shape} vs {context_attention_mask.shape}"

Before / After Comparison

Aspect	Before Fix	After Fix
Input IDs shape	(3, 210)	(3, 210)
Attention mask shape	(3, 128) ❌	(3, 210) ✅
Generation output	Missing 82 tokens, degraded relevance	Full context retained, higher BLEU/ROUGE
Runtime errors	IndexError, RuntimeError	None

Verification – Ensuring Correct Behavior

Shape assertion. The assert line above will raise immediately if a mismatch reappears.
Log the first few token IDs. Confirm that padding tokens (tokenizer.pad_token_id) appear only after the original sequence length.
Run a sanity generation. Compare generated text with and without context injection to ensure the retrieved information is reflected.
Monitor metrics. Track context_token_coverage (ratio of non‑pad tokens to max_len) in your monitoring system; it should stay at 1.0 for all batches.

Prevention – Guardrails for Future Development

Encapsulate padding logic. Keep a single source of truth for both IDs and masks; avoid manual mask construction.
Unit tests. Add tests that feed sequences of varying lengths (including edge cases like length 0 and max length) and assert shape equality.
Container resource budgeting. Allocate sufficient memory and CPU to the Docker sandbox; configure --shm-size and --ulimit to prevent premature termination of the padding routine.
Runtime checks. Enable torch.autograd.set_detect_anomaly(True) during development to catch shape‑related errors earlier.
Logging standards. Emit both context_input_ids.shape and context_attention_mask.shape at INFO level before each generation call.

FAQ – Common Follow‑Up Questions

Why does the model still generate output when the mask is shorter? MultiheadAttention silently ignores positions where the mask is absent, effectively treating them as padding. The forward pass succeeds, but any tokens beyond the mask are never attended to, causing silent truncation.
Can I use torch.nn.functional.pad instead of pad_sequence? Yes, but you must manually compute the target length and ensure the mask is padded to the same length. pad_sequence handles variable lengths automatically and reduces the risk of mismatch.
How do I debug similar issues in other transformer models? Follow the same pattern: print shapes of input_ids, attention_mask, and the tensors fed to the attention module. Verify that all three share identical seq_len dimensions.
What if my retrieved documents exceed the model’s maximum sequence length? Truncate documents before padding or increase model.config.max_position_embeddings if the architecture permits. Always log a warning when truncation occurs.
Is there a way to automate mask generation inside Hugging Face’s DataCollator? Implement a custom DataCollatorForRAG that calls the unified pad_and_mask function and returns a dictionary with both input_ids and attention_mask. This ensures every batch respects shape consistency.

Related Topic Hub: Model Serving Troubleshooting Hub