RAG query decomposition error on air-gapped server after model cache update

RAG Query Decomposition Error on an Air‑Gapped Server After Model Cache Update

Problem

After updating the local cache of a Hugging Face Transformers model used for Retrieval‑Augmented Generation (RAG), every call to RagRetriever fails with a ValueError during query decomposition. The failure occurs only on the isolated, air‑gapped host that has no outbound network access and runs with restricted filesystem permissions.

Typical traceback:

Traceback (most recent call last):
  File "rag_inference.py", line 42, in <module>
    answers = rag_model.generate(**inputs)
  File ".../transformers/models/rag/modeling_rag.py", line 587, in generate
    query_input_ids = self.retriever.question_encoder(**question_encodings).input_ids
  File ".../transformers/models/rag/retrieval_rag.py", line 312, in __call__
    raise ValueError(f"Tokenizer for {self.question_encoder_name} not found in cache.")
ValueError: Tokenizer for facebook/rag-tokenizer not found in cache.

The error prevents any downstream generation, causing service outages for chat‑bot endpoints that rely on RAG.

Root Cause

The RAG pipeline depends on three separate artifacts:

  • Question encoder model (e.g., facebook/rag-tokenizer).
  • Generator model (e.g., facebook/rag-sequence-nq).
  • Retriever index files (FAISS or Elasticsearch).

When the model cache was refreshed, the new version of the question encoder introduced a change in the tokenizer configuration file (tokenizer_config.json) and added a new vocab.txt. Because the air‑gapped server only mirrors the pytorch_model.bin and config.json files, the updated tokenizer assets were never copied. The library therefore raises a ValueError when it cannot locate a matching tokenizer directory.

Compounding the issue, the transformers.utils.hub fallback logic attempts to download missing files from huggingface.co. With no internet connectivity, the request hangs for several seconds before the exception propagates, adding latency to the failure.

Debug

Step‑by‑step investigation that reproduced the problem:

  1. Confirm missing assets by inspecting the local cache directory ($HOME/.cache/huggingface/hub).
$ ls $HOME/.cache/huggingface/hub/facebook__rag-tokenizer
config.json
pytorch_model.bin
# tokenizer files are absent
  1. Check the library’s lookup path using the internal helper.
>>> from transformers import AutoTokenizer
>>> AutoTokenizer.from_pretrained("facebook/rag-tokenizer")
Traceback (most recent call last):
  ...
ValueError: Tokenizer for facebook/rag-tokenizer not found in cache.
  1. Inspect the model revision that triggered the update.
$ cat $HOME/.cache/huggingface/hub/facebook__rag-tokenizer/commit_hash.txt
b7c9f3e7a2d5c9e1a0c6d2f8e4b9a1c3
$ git ls-remote https://huggingface.co/facebook/rag-tokenizer b7c9f3e7a2d5c9e1a0c6d2f8e4b9a1c3
# shows that tokenizer files were added in this commit
  1. Validate filesystem permissions. The service runs under the ragsvc user, which only has read access to the cache root.
$ ls -l $HOME/.cache/huggingface/hub/facebook__rag-tokenizer
drwxr-x--- 2 ragsvc ragsvc 4096 Sep 12 10:02 .

The directory is readable, but the missing files are the actual blocker.

Solution

Copy the missing tokenizer assets from a machine with internet access, then adjust the cache layout to match the expected structure.

Step 1 – Export the complete tokenizer directory from a connected host

# On a machine with internet
$ git clone https://huggingface.co/facebook/rag-tokenizer
$ cd rag-tokenizer
$ ls
config.json  tokenizer_config.json  vocab.txt  merges.txt  special_tokens_map.json
$ tar czf rag-tokenizer.tar.gz .

Step 2 – Transfer the archive to the air‑gapped server

# Using scp, USB, or any approved transfer method
$ scp rag-tokenizer.tar.gz ragsvc@airgap-host:/tmp/

Step 3 – Extract into the cache path

# On the air‑gapped host (as ragsvc)
$ mkdir -p $HOME/.cache/huggingface/hub/facebook__rag-tokenizer
$ tar xzf /tmp/rag-tokenizer.tar.gz -C $HOME/.cache/huggingface/hub/facebook__rag-tokenizer
$ ls $HOME/.cache/huggingface/hub/facebook__rag-tokenizer
config.json  tokenizer_config.json  vocab.txt  merges.txt  special_tokens_map.json

Step 4 – Verify that the tokenizer can be loaded

>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-tokenizer")
>>> tokenizer.encode("test query")
[101, 2775, 102]

Optional – Pin the revision to avoid future silent updates

Modify the model loading code to include an explicit revision argument that points to the cached commit hash.

from transformers import RagTokenizer, RagRetriever, RagSequenceForGeneration

tokenizer = RagTokenizer.from_pretrained(
    "facebook/rag-tokenizer",
    revision="b7c9f3e7a2d5c9e1a0c6d2f8e4b9a1c3",  # pinned to known good version
    local_files_only=True
)

retriever = RagRetriever.from_pretrained(
    "facebook/rag-sequence-nq",
    index_name="custom",
    use_dummy_dataset=True,
    local_files_only=True
)

model = RagSequenceForGeneration.from_pretrained(
    "facebook/rag-sequence-nq",
    retriever=retriever,
    local_files_only=True
)

Verify

Run a minimal inference request and confirm that the pipeline no longer raises a tokenizer error.

python rag_inference.py --question "What is Retrieval‑Augmented Generation?"

Expected console output (truncated):

Input question: What is Retrieval‑Augmented Generation?
Generated answer: Retrieval‑Augmented Generation (RAG) combines a parametric language model with a non‑parametric knowledge base...

Additional verification steps:

  • Check the process logs for the absence of ValueError: Tokenizer for ... not found in cache.
  • Inspect the Prometheus metric rag_query_success_total (if exported) – it should increase.
  • Run a health‑check endpoint that performs a dry‑run of RagRetriever and returns HTTP 200.

Prevent

Implement safeguards to avoid recurrence:

Preventive Measure Implementation Detail
Enforce local_files_only=True in all production code paths. Wrap model loading in a helper that injects the flag automatically.
Pin model revisions. Store the desired commit hash in a configuration file (e.g., models.yml) and reference it at runtime.
Automate cache synchronization. Use a CI job that runs transformers-cli download for the exact set of assets and ships a tarball to the air‑gapped environment.
Validate cache completeness during deployment. Run a pre‑flight script that attempts to load each model/tokenizer with local_files_only=True and aborts on failure.
Monitor for silent cache updates. Track the transformers.file_utils._CACHE_REPO_ROOT modification timestamps and alert if they change unexpectedly.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the error appear only after a cache update?
    The new model revision introduced additional tokenizer files that were not present in the previous cache snapshot. The library expects those files and fails when they are missing.
  2. Can I disable the automatic download attempt?
    Yes. Pass local_files_only=True to every from_pretrained call. This forces the library to raise an error immediately if assets are absent, avoiding hidden network retries.
  3. What if the air‑gapped host has read‑only filesystem permissions?
    Copy the required assets to a directory that the service user can read (e.g., /opt/models/rag-tokenizer) and set the TRANSFORMERS_CACHE environment variable to point there.
  4. How do I know which files are required for a given model?
    Run transformers-cli download <repo_id> --revision <hash> on a connected machine; it prints the full list of files that will be cached.
  5. Is there a way to test the entire RAG pipeline offline before deployment?
    Create a unit test that loads the tokenizer, encoder, generator, and retriever with local_files_only=True and runs a single generate call. Include this test in the CI pipeline that runs in the isolated environment.