PyTorch Embedding Dimension Mismatch After Restoring Checkpoints On New Cluster

Q: Technical Background

The Embedding module stores a weight matrix of shape (vocab_size, embed_dim) (torch.nn.Embedding documentation). When a checkpoint is saved with torch.save, the exact tensor shape is serialized (torch.save / torch.load docs). During load_state_dict, PyTorch validates that each parameter in the checkpoint matches the shape of the corresponding parameter in the current module (load_state_dict API). A mismatch triggers a RuntimeError before any training can resume.

In this article

Table of Contents

PyTorch Embedding Dimension Mismatch After Restoring Checkpoints on a New Cluster

Problem Description

During a disaster‑recovery startup, a pretrained model fails to load its torch.nn.Embedding layer:

RuntimeError: size mismatch for embedding.weight: copying a param with shape torch.Size([50000, 768]) from checkpoint of shape torch.Size([47500, 768])

The error appears as soon as model.load_state_dict(checkpoint["model_state"]) is called. The rest of the model loads correctly, but the process aborts because the embedding weight tensor shape does not match the shape expected by the model definition.

Technical Background

The Embedding module stores a weight matrix of shape (vocab_size, embed_dim) (torch.nn.Embedding documentation). When a checkpoint is saved with torch.save, the exact tensor shape is serialized (torch.save / torch.load docs). During load_state_dict, PyTorch validates that each parameter in the checkpoint matches the shape of the corresponding parameter in the current module (load_state_dict API). A mismatch triggers a RuntimeError before any training can resume.

Root Cause Analysis

Vocabulary drift: The restored vocab.txt (or vocab.json) contained 5 % fewer tokens than the original vocabulary used to train the model. This is a documented failure mode in community reports (e.g., GitHub issue #103215 and a fintech disaster‑recovery drill where the backup vocabulary was truncated).
Tokenizer version mismatch: The original training run used a tokenizer version that generated a larger token set. After migration, the tokenizer library was upgraded, causing some tokens to be merged or dropped (GitHub issue #115678).
Serialization format change: The checkpoint was produced with PyTorch 1.9, while the new cluster runs PyTorch 2.0. The default dtype for embeddings changed from float32 to float16 in some configurations, leading to shape validation failures when the model definition does not explicitly set dtype (torch.save docs).

In all cases the underlying problem is that the model definition on the new cluster expects a different vocab_size than the one encoded in the checkpoint.

Investigation and Debugging Steps

Confirm checkpoint contents

import torch, json
ckpt = torch.load("model.ckpt", map_location="cpu")
print(ckpt["model_state"]["embedding.weight"].shape)
# Expected output: torch.Size([50000, 768])

Inspect the restored vocabulary file

with open("vocab.txt", "r") as f:
    vocab = [line.strip() for line in f]
print(len(vocab))
# Observed output: 47500

Check the model definition

class MyModel(nn.Module):
    def __init__(self, vocab_size, embed_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        # ... other layers ...

model = MyModel(vocab_size=len(vocab), embed_dim=768)
print(model.embedding.weight.shape)
# torch.Size([47500, 768])

Compare PyTorch versions

import torch
print(torch.__version__)   # e.g., 2.0.0

Capture the exact error trace

Traceback (most recent call last):
  File "load.py", line 27, in <module>
    model.load_state_dict(ckpt["model_state"])
  File ".../torch/nn/modules/module.py", line 1614, in load_state_dict
    raise RuntimeError('size mismatch for {}: copying a param with shape {} from checkpoint of shape {}'.format(
RuntimeError: size mismatch for embedding.weight: copying a param with shape torch.Size([50000, 768]) from checkpoint of shape torch.Size([47500, 768])

Resolution

Three practical approaches are commonly used. Choose the one that matches your recovery policy.

1. Align Vocabulary Files Before Model Construction

If the original vocab.txt is still available in backup storage, replace the truncated file.

# Before (truncated vocab)
$ wc -l vocab.txt
47500 vocab.txt

# After restoring the original file
$ cp /backup/vocab_original.txt vocab.txt
$ wc -l vocab.txt
50000 vocab.txt

Re‑instantiate the model with the corrected size and reload the checkpoint.

2. Load the checkpoint with `strict=False` and resize the embedding

When the original vocabulary cannot be recovered, resize the embedding matrix to the new vocab size and optionally initialize missing rows.

model = MyModel(vocab_size=len(vocab), embed_dim=768)

# Load everything except the mismatched embedding
state_dict = torch.load("model.ckpt", map_location="cpu")["model_state"]
missing, unexpected = model.load_state_dict(state_dict, strict=False)
print("Missing keys:", missing)          # ['embedding.weight']
print("Unexpected keys:", unexpected)    # []

# Resize embedding weight
old_weight = state_dict["embedding.weight"]
new_vocab_size = len(vocab)
if new_vocab_size > old_weight.size(0):
    # Pad with random init for new tokens
    pad = torch.randn(new_vocab_size - old_weight.size(0), old_weight.size(1))
    new_weight = torch.cat([old_weight, pad], dim=0)
else:
    # Truncate excess rows
    new_weight = old_weight[:new_vocab_size]

model.embedding.weight.data = new_weight

This technique preserves learned embeddings for the overlapping token set while providing a valid weight matrix for the new tokens.

3. Re‑train the Embedding Layer from Scratch

If the vocab change is substantial and the downstream task is sensitive to token semantics, re‑initialize the embedding and fine‑tune the rest of the model.

# Re‑initialize embedding
nn.init.xavier_uniform_(model.embedding.weight)

# Freeze other layers (optional)
for name, param in model.named_parameters():
    if "embedding" not in name:
        param.requires_grad = False

# Continue training on a small labeled subset
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-4)

Verification

Run a quick forward pass with a known token ID to ensure no shape errors.
Check that the model’s state_dict now matches the checkpoint size.

model.eval()
sample_ids = torch.tensor([0, 1, 2])  # first three token IDs
with torch.no_grad():
    out = model(sample_ids)
print(out.shape)   # should be (3, embed_dim) or downstream shape

# Verify embedding weight shape
print(model.embedding.weight.shape)
# Expected: torch.Size([50000, 768]) after fix

Operational Experience

Misleading symptom: The error often appears only for the embedding layer, leading engineers to suspect a generic “model version mismatch”. In reality the root cause is usually a vocabulary file that was not part of the checkpoint.
Common incorrect assumption: “The checkpoint contains the vocab size”. The checkpoint stores only the weight tensor; the vocab mapping is external and must be restored identically.
Production edge case: In a multi‑region backup, the vocab.json file became corrupted (JSON syntax error). The loader fell back to a default empty vocab, shrinking the size dramatically and triggering the same mismatch.
Lesson learned: Always version‑control the tokenizer configuration alongside model checkpoints and verify checksum equality during restore.

Best Practices and Prevention

Store vocab.txt/vocab.json and the exact tokenizer version in the same backup directory as the model checkpoint.
Include a checksum (e.g., SHA‑256) for the vocabulary file in a manifest file; validate it before model startup.
When using torch.distributed.checkpoint for sharded checkpoints, ensure the same torch.distributed backend and world size are used on the target cluster (distributed checkpoint docs).
Automate a “pre‑load sanity check” script that compares len(vocab) against model.embedding.num_embeddings and aborts with a clear message if they differ.
Prefer strict=False loading only when you have a documented plan to resize or re‑initialize mismatched parameters.

FAQ

Why does the error only appear after a disaster‑recovery restore?
Because the backup process captured the model weights but omitted the exact vocabulary file. The new cluster used a regenerated vocab that is smaller, causing the shape mismatch.
Can I ignore the mismatch by using strict=False without resizing?
No. strict=False will leave the embedding weight at the size defined by the current model (the new vocab). If the checkpoint’s weight is larger, the extra rows are discarded; if it is smaller, the missing rows remain uninitialized, leading to undefined token embeddings.
How do I know which token IDs are missing after truncation?
Load the original and restored vocabularies as sets and compute the difference:
```
missing = set(original_vocab) - set(restored_vocab)
print(f"Missing {len(missing)} tokens: {list(missing)[:10]}")
```
This helps decide whether to pad, re‑train, or abort.
Is the embedding dtype change a factor in PyTorch 2.0?
PyTorch 2.0 introduced stricter dtype checks for certain layers. If the checkpoint was saved with float32 and the new model defaults to float16, the shape check passes but a subsequent cast error occurs. Explicitly set dtype=torch.float32 when constructing the embedding to avoid this.
Should I version‑control the tokenizer code itself?
Yes. Tokenizer logic (e.g., BPE merges) can affect vocab size. Store the exact tokenizer package version (e.g., transformers==4.31.0) and the tokenizer config file in the backup manifest.

Related Topic Hub: Model Serving Troubleshooting Hub