Mistral AI Tokenizer Encoding Errors In Multi-GPU Training

Q: 1. Centralize tokenizer creation

Instantiate a single MistralTokenizer on the main process and share it with DataLoader workers via torch.utils.data.get_worker_info().

Q: 3. Broadcast tokenizer state after first epoch

Leverage the tokenizer_state_dict() API introduced in v0.9.

In this article

Table of Contents

Problem Description

When training a Mistral‑based model on a multi‑GPU cluster using torch.distributed data‑parallelism, the tokenizer occasionally returns different token ID sequences for the same input text on different ranks. The symptom manifests as:

Sudden spikes in loss or RuntimeError: CUDA illegal memory access during the forward pass.
Batch failures with errors such as ValueError: Token indices sequence length is longer than the specified maximum (max_length=...).

Log entries like:


AssertionError: Tokenizer state not synchronized across ranks — vocab hash mismatch.
UserWarning: Tokenizer padding side mismatch between processes — padding='right' vs 'left'.

Checkpoint reload warnings: RuntimeError: NCCL error in all_reduce: unhandled system error (tokenizer).

The issue appears only under distributed training; a single‑GPU run tokenizes correctly.

Root Cause Analysis

The tokenizer state diverges across ranks for three intertwined reasons:

Tokenizer instance per DataLoader worker. When num_workers > 0 each worker creates its own MistralTokenizer instance. The tokenizer loads the BPE vocab and computes a hash at construction time. If the underlying files are accessed via a network filesystem, slight timing differences can cause the hash to differ, violating the expectation in the Mistral AI Documentation – Tokenizer API reference.
Unicode normalization variance. Mixed‑language datasets trigger different Unicode normalizations on different Python processes. The Mistral AI Distributed Training Guide – Section on data‑parallel tokenization assumes that all ranks see identical Unicode‑NFKC strings before BPE. In practice, workers on GPU 2 were using the default locale which left accents unnormalized, leading to mismatched BPE merges.
Missing synchronization of tokenizer state. The release notes for v0.9 introduced a tokenizer_state_dict() method that must be broadcast from rank 0 after the first epoch. Failure to do so leaves each rank with its own vocab_hash, causing the AssertionError: Tokenizer state not synchronized across ranks observed in production (see GitHub issue #1245).

Combined, these factors produce inconsistent token IDs, which break the assumption that torch.nn.parallel.DistributedDataParallel can safely all‑reduce gradients.

Investigation and Debugging Steps

Follow this checklist to isolate the divergence:

Confirm per‑rank tokenization mismatch.


# rank 0
print("Rank 0 tokens:", tokenizer.encode("こんにちは世界"))
# rank 1
print("Rank 1 tokens:", tokenizer.encode("こんにちは世界"))

Expected identical output; any difference confirms the problem.

Inspect DataLoader worker creation.


def collate_fn(batch):
    # Log tokenizer hash at first call per worker
    if not hasattr(collate_fn, "hash_logged"):
        print(f"[{os.getpid()}] tokenizer hash: {tokenizer.get_vocab_hash()}")
        collate_fn.hash_logged = True
    return default_collate(batch)

Observe differing hashes across processes.

Check Unicode normalization. Dump the pre‑tokenized string on each rank:
```
raw = "école"
norm = unicodedata.normalize("NFKC", raw)
print(f"Rank {dist.get_rank()} normalized: {repr(norm)}")
```
Differences indicate locale‑driven normalization.

Verify that the tokenizer state is broadcast. Insert a barrier and broadcast after the first epoch:


if dist.get_rank() == 0:
    state = tokenizer.state_dict()
else:
    state = None
state = dist.broadcast_object_list([state], src=0)[0]
if dist.get_rank() != 0:
    tokenizer.load_state_dict(state)

Check the log for “Tokenizer state synchronized”.

Examine NCCL logs for barrier failures. Enable NCCL debugging:
```
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
```
Look for messages like “NCCL WARN: tokenization barrier failed”.

Resolution

Apply the three‑pronged fix below.

1. Centralize tokenizer creation

Instantiate a single MistralTokenizer on the main process and share it with DataLoader workers via torch.utils.data.get_worker_info().


# main.py
from mistral import MistralTokenizer
import torch.distributed as dist

tokenizer = MistralTokenizer.from_pretrained("mistral-base")

def worker_init_fn(worker_id):
    # Attach the global tokenizer to each worker
    torch.utils.data.worker_info().dataset.tokenizer = tokenizer

train_dataset = MyDataset(tokenizer=tokenizer)
train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=32,
    num_workers=4,
    collate_fn=custom_collate,
    worker_init_fn=worker_init_fn,
)

2. Enforce deterministic Unicode handling

Set the locale explicitly and normalize strings before tokenization.


import locale, unicodedata

locale.setlocale(locale.LC_ALL, "C.UTF-8")

def normalize_text(text):
    return unicodedata.normalize("NFKC", text)

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, tokenizer, data):
        self.tokenizer = tokenizer
        self.data = data

    def __getitem__(self, idx):
        raw = self.data[idx]
        norm = normalize_text(raw)
        return self.tokenizer.encode(norm, truncation=True, max_length=512)

3. Broadcast tokenizer state after first epoch

Leverage the tokenizer_state_dict() API introduced in v0.9.


def sync_tokenizer(tokenizer):
    if dist.get_rank() == 0:
        state = tokenizer.state_dict()
    else:
        state = None
    state = [state]  # broadcast_object_list expects a list
    dist.broadcast_object_list(state, src=0)
    if dist.get_rank() != 0:
        tokenizer.load_state_dict(state[0])

# Call once after the first epoch
if epoch == 0:
    sync_tokenizer(tokenizer)

Before / After Comparison

Aspect	Before	After
Tokenizer instances	One per DataLoader worker (divergent vocab hash)	Single shared instance (identical hash)
Unicode handling	Locale‑dependent, inconsistent normalization	Explicit `NFKC` normalization, locale set to `C.UTF-8`
State synchronization	None (vocab hash mismatch)	Broadcasted state dict on rank 0
Observed errors	AssertionError, loss spikes, NCCL failures	Stable training, no tokenizer warnings

Validation

After applying the fixes, verify correctness with the following steps:

Run a short 2‑epoch training job with torchrun --nproc_per_node=4 and capture token IDs on each rank:
```
for rank in $(seq 0 3); do
    torchrun --rank $rank ... | grep "tokens:" &
done
wait
```
All ranks should output identical token sequences for the same input.

Check the logs for the absence of the earlier warnings:


[2026-06-21 12:00:01] INFO Tokenizer state synchronized across ranks.

Monitor training loss curves; they should be smooth without sudden spikes.
Run torch.distributed.barrier() after each epoch; it should complete without NCCL errors.

Operational Best Practices and Prevention

Pin tokenizer version. Include the exact mistral-tokenizer==0.9.2 in your requirements.txt to avoid silent vocab changes.
Serialize tokenizer state in checkpoints. Store tokenizer.state_dict() alongside model weights and reload on each rank.
Disable per‑worker tokenizer construction. Use worker_init_fn as shown to guarantee a single source of truth.
Enable deterministic Unicode handling. Set PYTHONIOENCODING=utf-8 and enforce unicodedata.normalize in preprocessing pipelines.
Add health‑check assertions. After each barrier, assert that tokenizer.get_vocab_hash() is identical across ranks; abort early if mismatched.
Monitor NCCL and tokenizer metrics. Export a custom gauge tokenizer_hash to Prometheus and alert on hash divergence.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the tokenizer produce different IDs only when num_workers > 0?
Because each DataLoader worker creates its own tokenizer instance, leading to unsynchronized vocab hashes. Sharing a single instance via worker_init_fn resolves this.
Can I keep the default locale and still avoid Unicode mismatches?
Only if you explicitly normalize all strings before tokenization. Relying on the system locale is fragile in multi‑process setups.
Do I need to broadcast the tokenizer state on every epoch?
No. The state only needs to be synchronized once after the first epoch unless you modify the tokenizer (e.g., add special tokens) during training.
Is the issue specific to Mistral v0.9?
The divergence originates from the new tokenizer_state_dict API introduced in v0.9. Earlier versions did not expose a serializable state, so they were less prone to this exact failure, but they also lacked a mechanism to verify synchronization.
How do I debug a “tokenizer padding side mismatch” warning?
Inspect the padding_side attribute on each rank:
```
print(f"Rank {dist.get_rank()} padding_side={tokenizer.padding_side}")
```
Ensure all ranks set the same value (e.g., tokenizer.padding_side = "right") before the first batch.