PyTorch RAG Citation Formatting Inconsistent After Model Update

Q: 1. Reproduce the symptom

python run_rag.py --model checkpoint_v2 --retriever faiss # Sample output: Answer: The capital of France is Paris. [<cite>] [<cite>] 2. Inspect the retrieved documents retrieved = rag.retriever(query="What is the capital of France?") print([doc.metadata['source_id'] for doc in retrieved]) # Expected: ['doc_12', 'doc_45'] # Actual: [] Empty source_id list confirms loss during batching.

Q: 3. Verify the collate function

def rag_collate_fn(batch): # Original implementation (pre‑upgrade) docs = [item['docs'] for item in batch] source_ids = [doc.metadata['source_id'] for docs in batch for doc in docs] return { 'input_ids': torch.stack([item['input_ids'] for item in batch]), 'source_ids': source_ids, # … } After the recent change, source_ids was inadvertently omitted:

Q: 4. Check tokenizer special tokens

print(tokenizer.added_tokens_encoder) # Expected entry: {'<cite>': 50265} # Actual: {} The missing entry explains the ValueError: Invalid citation format logged by format_citations.

Q: 5. Review logs for post‑processing errors

2026-07-01 14:32:10,212 | ERROR | rag.pipeline | KeyError: 'citation_id' 2026-07-01 14:32:10,215 | ERROR | rag.pipeline | ValueError: Invalid citation format – expected pattern \[\\d+\] Solution – Restoring Consistent Citation Formatting The fix consists of two parts: (1) restore metadata propagation in the collate function and (2) re‑inject the <cite> token into the tokenizer.

In this article

Table of Contents

Problem – Inconsistent Citation Formatting After Model Update

During development of a custom Retrieval‑Augmented Generation (RAG) pipeline built on PyTorch and Hugging Face Transformers, the generated answers increasingly contain malformed citations. Typical symptoms include:

Missing source identifiers, e.g. [?] or plain text cite tags.
Reference numbers that do not correspond to the retrieved document list, producing duplicate or out‑of‑order brackets such as [1][1][3].
Runtime errors during post‑processing, for example KeyError: 'citation_id' or IndexError: list index out of range while formatting citations.

The issue appears after upgrading the Transformers library from 4.30 to 4.34 (see GitHub issue #32257) and after a recent change to the custom collate function that batches retrieved documents.

Root Cause – How the Pipeline Loses Citation Metadata

The RAG pipeline relies on a tight coupling between three components:

Retriever output: a list of Document objects each carrying a source_id (or doc_id) field.
Collate function: packs the retrieved documents into a batch tensor while preserving a parallel source_ids list (see PyTorch torch.utils.data documentation).
Generator post‑processing: the format_citations helper scans the generated token stream for the special <cite> token (added via the tokenizer’s added_tokens_encoder) and replaces it with [n] where n indexes into the source_ids list.

Two intertwined regressions break this contract:

Metadata loss in the collate step: The custom collate function introduced in the CI change omitted the source_ids field when constructing the batch (see GitHub issue #115237). Consequently, the generator receives a tensor without a matching list, causing the formatter to fallback to placeholders ([?]).
Tokenizer regression: Upgrading to Transformers 4.34 removed the custom <cite> token from added_tokens_encoder. Without this token, the generated text contains raw citations (e.g., “cite 12”) that the regex \[\d+\] cannot match, triggering ValueError: Invalid citation format – expected pattern \[\\d+\] (see the Stack Overflow example).

Both failures violate the expectations documented in the official PyTorch torch.nn.Module forward‑pass guidelines for preserving auxiliary metadata across batches.

Debug – Investigation Steps

Below is a reproducible debugging workflow that isolates the missing citation IDs.

1. Reproduce the symptom

python run_rag.py --model checkpoint_v2 --retriever faiss
# Sample output:
Answer: The capital of France is Paris. [<cite>] [<cite>]

2. Inspect the retrieved documents

retrieved = rag.retriever(query="What is the capital of France?")
print([doc.metadata['source_id'] for doc in retrieved])
# Expected: ['doc_12', 'doc_45']
# Actual: []

Empty source_id list confirms loss during batching.

3. Verify the collate function

def rag_collate_fn(batch):
    # Original implementation (pre‑upgrade)
    docs = [item['docs'] for item in batch]
    source_ids = [doc.metadata['source_id'] for docs in batch for doc in docs]
    return {
        'input_ids': torch.stack([item['input_ids'] for item in batch]),
        'source_ids': source_ids,
        # …
    }

After the recent change, source_ids was inadvertently omitted:

def rag_collate_fn(batch):
    # Faulty version
    return {
        'input_ids': torch.stack([item['input_ids'] for item in batch]),
        # 'source_ids' missing!
    }

4. Check tokenizer special tokens

print(tokenizer.added_tokens_encoder)
# Expected entry: {'<cite>': 50265}
# Actual: {}

The missing entry explains the ValueError: Invalid citation format logged by format_citations.

5. Review logs for post‑processing errors

2026-07-01 14:32:10,212 | ERROR | rag.pipeline | KeyError: 'citation_id'
2026-07-01 14:32:10,215 | ERROR | rag.pipeline | ValueError: Invalid citation format – expected pattern \[\\d+\]

Solution – Restoring Consistent Citation Formatting

The fix consists of two parts: (1) restore metadata propagation in the collate function and (2) re‑inject the <cite> token into the tokenizer.

1. Update the collate function

Before:

def rag_collate_fn(batch):
    return {
        'input_ids': torch.stack([item['input_ids'] for item in batch]),
        # 'source_ids' missing!
    }

After (preserving source IDs and aligning batch dimensions):

def rag_collate_fn(batch):
    # Preserve input tensors
    input_ids = torch.stack([item['input_ids'] for item in batch])

    # Extract and pad source IDs per document
    max_docs = max(len(item['docs']) for item in batch)
    source_ids = []
    for item in batch:
        ids = [doc.metadata.get('source_id', f'unknown_{i}') for i, doc in enumerate(item['docs'])]
        # Pad with a sentinel to keep tensor shapes consistent
        ids += [''] * (max_docs - len(ids))
        source_ids.append(ids)

    # Convert to tensor of strings (requires torch>=2.0)
    source_ids_tensor = torch.tensor(source_ids, dtype=torch.object)

    return {
        'input_ids': input_ids,
        'source_ids': source_ids_tensor,
        'attention_mask': torch.stack([item['attention_mask'] for item in batch]),
    }

2. Reinstate the citation token

Before (token missing):

tokenizer.added_tokens_encoder
# {}

After (register token and resize model embeddings):

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained('facebook/rag-token-base')
citation_token = '<cite>'
if citation_token not in tokenizer.added_tokens_encoder:
    tokenizer.add_tokens([citation_token])
    # Resize model embeddings to accommodate new token
    model = AutoModelForSeq2SeqLM.from_pretrained('facebook/rag-token-base')
    model.resize_token_embeddings(len(tokenizer))

3. Adjust post‑processing hook (optional)

If downstream code still expects a specific field name, add a compatibility shim:

def postprocess_citations(output_ids, source_ids):
    # Guard against missing source_ids
    if source_ids is None or len(source_ids) == 0:
        source_ids = [''] * len(output_ids)
    return format_citations(output_ids, source_ids)

Verification – Confirming the Fix

Run the same query after applying the patches:

python run_rag.py --model checkpoint_v2 --retriever faiss
# Expected output:
Answer: The capital of France is Paris. [1] [2]

Additional checks:

Log inspection: No KeyError or ValueError lines related to citations.

Tokenizer dump:

print(tokenizer.added_tokens_encoder['<cite>'])
# 50265

Unit test:

def test_citation_alignment():
    output, source_ids = rag.generate("Who wrote '1984'?")
    assert '[1]' in output
    assert source_ids[0] == 'doc_7'

Prevention – Guardrails and Best Practices

Schema‑enforced collate: Use a dataclass or TypedDict for batch items and assert presence of source_ids before returning from the collate function.
Tokenizer version lock: Pin the Transformers version that introduced the <cite> token (>= 4.30, < 4.34) or add a migration script that re‑adds the token on startup.
Continuous integration test: Include a regression test that generates a RAG answer and asserts the presence of at least one correctly formatted citation.
Monitoring: Emit a custom metric rag.citation_format_errors whenever format_citations raises an exception; alert on spikes.
Metadata integrity check: After each retrieval step, log the count of documents and their IDs; mismatch with the generator batch size should trigger a warning.

FAQ – Common Follow‑Up Questions

Why do citations disappear only after upgrading Transformers?
The upgrade removed the custom <cite> token from the tokenizer’s added_tokens_encoder. Without the token the formatter cannot recognize citation markers, resulting in missing brackets.
Can I use a different special token instead of <cite>?
Yes. Define a new token (e.g., <ref>), add it to the tokenizer, and update the regex in format_citations accordingly. Ensure the token is added before model loading so the embedding matrix is resized.
How do I debug a “KeyError: ‘citation_id’” in production?
Check that the collate function returns a source_ids field and that each document’s metadata includes a unique source_id. Logging the batch dict right before the forward pass will reveal missing keys.
What causes duplicate citation numbers?
Duplicate source_id values in the retrieved set (e.g., due to non‑unique IDs in the FAISS index) lead the formatter to map multiple markers to the same index. Ensure the retriever de‑duplicates IDs or assign a globally unique identifier per document.
Is there a way to validate citation alignment automatically?
Implement a post‑generation sanity check that parses all \[\d+\] tokens, compares the highest index to len(source_ids), and raises an exception if they diverge.

Related Topic Hub: Model Serving Troubleshooting Hub