RAG document loader crash during backup restoration

In short, the DocumentLoader crashes because it cannot locate or correctly read the required index and metadata files, or because the runtime environment cannot process them.

Investigation and Debugging Steps

Follow this systematic checklist to isolate the failure mode.

1. Verify backup completeness

   # List objects in the backup bucket (AWS S3 example)
   aws s3 ls s3://rag-backup/2024-06-15/ --recursive
   
   # Expected artefacts
   index.faiss
   doc_id_map.pkl
   documents.jsonl
   tokenizer/
   model/
   

   If any file is missing, the backup process must be corrected.

2. Inspect file integrity

   # Download a sample file and compute checksum
   aws s3 cp s3://rag-backup/2024-06-15/documents.jsonl ./tmp.jsonl
   sha256sum ./tmp.jsonl
   
   # Compare with checksum stored at backup time
   cat ./tmp.jsonl.sha256
   

   A mismatched checksum indicates corruption.

3. Check encoding and line endings

   # Detect non‑UTF8 bytes
   python - <<'PY'
   import codecs, sys
   with open('tmp.jsonl', 'rb') as f:
       data = f.read()
   try:
       data.decode('utf-8')
   except UnicodeDecodeError as e:
       print('Decode error:', e)
   PY
   

   If a UnicodeDecodeError appears, re‑encode the file:

   iconv -f utf-16 -t utf-8 tmp.jsonl -o tmp_utf8.jsonl
   dos2unix tmp_utf8.jsonl   # normalize line endings
   

4. Validate FAISS index compatibility

   # Print FAISS version used to create the index
   python - <<'PY'
   import faiss, pickle
   with open('index.faiss', 'rb') as f:
       header = f.read(8)
   print('FAISS index header bytes:', header)
   PY
   

   Cross‑check with the torch and faiss versions in the current environment:

   pip list | grep -E 'torch|faiss'

5. Run the loader in isolation with verbose logging

   python -X faulthandler - <<'PY'
   import logging, os
   from transformers import RagRetriever, RagTokenizer, RagSequenceForGeneration
   from transformers.models.rag.document_loader import DocumentLoader
   
   logging.basicConfig(level=logging.DEBUG)
   
   loader = DocumentLoader(
       index_path="index.faiss",
       doc_id_map_path="doc_id_map.pkl",
       source_path="documents.jsonl",
       batch_size=128,
       num_workers=4,
   )
   documents = loader.load()
   print(f'Loaded {len(documents)} documents')
   PY
   

   Observe the debug output for the exact point of failure.

6. Check for permission or network errors

   # Example of an S3 permission failure
   aws s3 cp s3://rag-backup/2024-06-15/index.faiss ./index.faiss
   # Expected error:
   # An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
   

   Ensure the service role has s3:GetObject for the backup bucket.

Resolution

Apply the fixes that correspond to the identified root cause.

1. Restore missing artefacts

If index.faiss or doc_id_map.pkl are absent, re‑run the backup script with the --include-index flag (as documented in the official RAG guide).

# Corrected backup command (AWS example)
aws s3 sync ./rag_artifacts/ s3://rag-backup/2024-06-15/ \
    --exclude "*" \
    --include "index.faiss" \
    --include "doc_id_map.pkl" \
    --include "documents.jsonl" \
    --include "tokenizer/**" \
    --include "model/**"

2. Re‑encode corrupted source files

Normalize encoding and line endings before loading.

# Batch re‑encoding script
for f in *.jsonl; do
    iconv -f utf-16 -t utf-8 "$f" | dos2unix > "${f}.utf8"
    mv "${f}.utf8" "$f"
done

3. Align runtime dependencies

Pin torch and faiss-cpu versions to those used during indexing. Example requirements.txt:

torch==2.1.0
faiss-cpu==1.7.4
transformers==4.38.2
datasets==2.16.1

Re‑install in the recovery environment:

pip install -r requirements.txt

4. Add retry and timeout handling for cloud streams

Wrap the loader’s file opening logic with boto3 retries (or Azure SDK equivalents). Example patch:

import boto3
from botocore.config import Config
import time

s3 = boto3.client('s3', config=Config(retries={'max_attempts': 5, 'mode': 'standard'}))

def download_with_retry(bucket, key, dest):
    for attempt in range(5):
        try:
            s3.download_file(bucket, key, dest)
            return
        except Exception as e:
            if attempt == 4:
                raise
            time.sleep(2 ** attempt)

5. Verify loader configuration

Ensure the DocumentLoader is instantiated with correct paths (absolute or mounted) and matching batch_size/num_workers for the restored hardware.

# Before (incorrect relative path)
loader = DocumentLoader(
    index_path="index.faiss",
    doc_id_map_path="doc_id_map.pkl",
    source_path="documents.jsonl",
)

# After (explicit mount point)
loader = DocumentLoader(
    index_path="/mnt/backup/index.faiss",
    doc_id_map_path="/mnt/backup/doc_id_map.pkl",
    source_path="/mnt/backup/documents.jsonl",
    batch_size=256,
    num_workers=8,
)

Verification

After applying the fixes, run the loader in a controlled test run and confirm successful ingestion.

# Smoke test
python - <<'PY'
from transformers.models.rag.document_loader import DocumentLoader
loader = DocumentLoader(
    index_path="/mnt/backup/index.faiss",
    doc_id_map_path="/mnt/backup/doc_id_map.pkl",
    source_path="/mnt/backup/documents.jsonl",
    batch_size=128,
    num_workers=4,
)
docs = loader.load()
print(f"✅ Loaded {len(docs)} documents")
PY

Additional validation steps:

• Check that index.faiss size on disk matches the size recorded before the outage.
• Run a quick retrieval query:

  python - <<'PY'
  from transformers import RagRetriever, RagTokenizer, RagSequenceForGeneration
  tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
  retriever = RagRetriever.from_pretrained(
      "facebook/rag-token-nq",
      index_name="custom",
      passages_path="/mnt/backup/documents.jsonl",
      index_path="/mnt/backup/index.faiss",
  )
  model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
  input_ids = tokenizer("What is the capital of France?", return_tensors="pt").input_ids
  outputs = model.generate(input_ids)
  print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
  PY

  If the answer is returned without exception, the restoration succeeded.

Prevention and Best Practices

• Atomic backup snapshots: Use storage‑level snapshotting (e.g., S3 versioning, Azure Blob snapshots) to guarantee that all artefacts are captured together.
• Checksum verification: Store SHA‑256 hashes alongside each artefact and validate them during restore.
• Version pinning: Record the exact torch, faiss, and transformers versions in a requirements.txt file stored with the backup.
• Health‑check endpoint: Expose a lightweight endpoint that loads a single document via DocumentLoader and returns 200 OK only if the index and metadata are present.
• Retry policies for cloud I/O: Wrap all S3/Azure/GCS interactions with exponential back‑off and idempotent retries (see GitHub issue #12487 for a reference implementation).
• Automated integration test: After each backup, run a CI job that restores the artefacts to a temporary environment and executes the smoke‑test query above.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the loader crash with FileNotFoundError: index.faiss only after a restore?

   The backup process omitted the FAISS index file. The loader expects the index to exist locally; without it, it raises a FileNotFoundError. Verify that the backup script includes --include-index or equivalent.

2. How can I confirm which FAISS version created the index?

   Inspect the first few bytes of the index file (as shown in the debugging step) and compare them with the version string printed by faiss.__version__ in the environment that performed the indexing.

3. Is it safe to re‑index documents instead of restoring the original index?

   Re‑indexing guarantees compatibility with the current runtime but can be time‑consuming for large corpora. Use it as a fallback when the original index is corrupted or missing.

4. What causes UnicodeDecodeError on restored JSONL files?

   During transfer the files were compressed or saved with a non‑UTF‑8 codec (e.g., UTF‑16) and line endings were converted to Windows style. Normalizing encoding with iconv and running dos2unix resolves the issue.

5. Can I use a different cloud provider for the backup without changing the loader code?

   Yes, as long as the files are accessible via a local mount or a compatible SDK (boto3, azure-storage-blob, google-cloud-storage). The loader only requires local file paths; the download step must handle provider‑specific authentication and retries.