Milvus ingestion fails with tokenizer encoding error in GitHub Actions

Problem: Milvus ingestion fails with tokenizer encoding error in GitHub Actions

During automated integration tests run on GitHub Actions (or similar CI/CD runners), the data‑loading stage aborts with an exception originating from the tokenizer used to generate text embeddings. Typical log excerpts look like:

UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 45
Traceback (most recent call last):
  File "/home/runner/work/project/project/tests/test_ingest.py", line 78, in test_ingest
    vectors = tokenizer.encode(texts)
  File ".../site-packages/transformers/tokenization_utils_base.py", line 1234, in encode
    return self._tokenizer.encode(text, **kwargs)
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 45
MilvusException: failed to insert vectors: tokenizer encoding error

Similar failures appear on Windows agents:

UnicodeDecodeError: 'cp1252' codec can't decode byte 0x80 in position 12: ordinal not in range(256)
MilvusException: failed to insert vectors: tokenizer encoding error

The error stops the pymilvus insert() call (see Milvus Documentation – Python SDK) and the CI job exits with a non‑zero status.

Root Cause Analysis

The underlying cause is a mismatch between the runtime locale/encoding expected by the HuggingFace tokenizer and the environment provided by the CI runner. The tokenizer operates on str objects that must be UTF‑8 encoded before being sent to Milvus. In CI environments the following conditions are common:

• Linux runners default to C.UTF-8 or have LANG unset, which does not guarantee full Unicode support for surrogate pairs (e.g., emoji \ud83d).
• Windows runners use the legacy code page cp1252, causing UnicodeDecodeError when the tokenizer reads raw bytes.
• Docker containers launched in CI may inherit an empty LANG variable, leading to the generic MilvusException: tokenizer encoding error observed in the Milvus logs.

Milvus’ Data Import Guide states that string fields must be UTF‑8 encoded. The Compatibility Matrix also notes that supported HuggingFace tokenizer versions expect the Python process to run with utf-8 as the default encoding.

Investigation and Debugging Steps

1. Reproduce locally with the same environment variables

# Simulate CI environment on a local machine
export LANG=C.UTF-8
export PYTHONIOENCODING=utf-8
python -c "from transformers import AutoTokenizer; \
t = AutoTokenizer.from_pretrained('distilbert-base-uncased'); \
print(t.encode('test 😊'))"

If the command fails with UnicodeEncodeError, the locale is the culprit.

2. Inspect CI runner locale

# GitHub Actions step
- name: Show locale
  run: locale

Typical output on a mis‑configured runner:

LANG=
LC_CTYPE="C"
LC_NUMERIC="C"
...

3. Check Python’s default encoding

python - <<'PY'
import sys
print(sys.getdefaultencoding())
PY

If the result is ascii or cp1252, the process will reject non‑ASCII characters.

4. Verify tokenizer version compatibility

pip show transformers
# Expected: version >= 4.30.0 (per Milvus compatibility matrix)

Version mismatches can surface as ValueError: tokenization failed: unknown encoding, as reported in the nightly GitHub Actions workflow.

Solution: Enforce UTF‑8 Locale and Explicit Encoding in CI

Step‑by‑step remediation

1. Set locale variables explicitly. Add a step before any Python execution that exports LANG and LC_ALL to en_US.UTF-8 (or at least UTF-8).
2. Force Python I/O encoding. Export PYTHONIOENCODING=utf-8 so that any implicit read/write uses UTF‑8.
3. Pin a compatible tokenizer version. In requirements.txt specify a version that matches the Milvus compatibility matrix (e.g., transformers==4.31.0).
4. Wrap tokenization calls with explicit encode/decode. This guards against accidental fallback to the system default.

Before

# .github/workflows/ci.yml
- name: Run integration tests
  run: |
    pytest tests/integration

After

# .github/workflows/ci.yml
- name: Set UTF‑8 locale
  run: |
    sudo apt-get update && sudo apt-get install -y locales
    sudo locale-gen en_US.UTF-8
    export LANG=en_US.UTF-8
    export LC_ALL=en_US.UTF-8
    export PYTHONIOENCODING=utf-8
    echo "Locale set to $(locale)"
- name: Install dependencies
  run: |
    pip install -r requirements.txt
- name: Run integration tests
  env:
    LANG: en_US.UTF-8
    LC_ALL: en_US.UTF-8
    PYTHONIOENCODING: utf-8
  run: |
    pytest tests/integration

Code change: explicit encoding in tokenization helper

# utils/tokenizer.py
from transformers import AutoTokenizer

_tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def encode_texts(texts):
    # Ensure each string is a proper Unicode object
    safe_texts = [t if isinstance(t, str) else str(t) for t in texts]
    # Explicitly encode to UTF‑8 before passing to tokenizer (required on Windows)
    encoded = [_tokenizer.encode(t.encode("utf-8", errors="ignore").decode("utf-8")) for t in safe_texts]
    return encoded

By normalising the input to UTF‑8, the tokenizer no longer raises UnicodeEncodeError, and pymilvus.insert() receives clean byte strings.

Verification

• Locale check: The CI step locale should output LANG=en_US.UTF-8 and LC_ALL=en_US.UTF-8.
• Python encoding: python -c "import sys; print(sys.getdefaultencoding())" must print utf-8.
• Successful tokenization: Running the helper script locally should produce a list of integer token IDs without exceptions.
• Milvus insert success: CI logs should now contain a line similar to Milvus insert succeeded: 1000 vectors inserted and the job exits with status 0.

Prevention and Best Practices

Practice	Why it matters	Implementation
Enforce UTF‑8 locale in all CI runners	Guarantees consistent Unicode handling across Linux, macOS, and Windows agents	Set LANG, LC_ALL, and PYTHONIOENCODING in workflow files
Pin tokenizer and Milvus SDK versions	Prevents silent breakage after upstream releases	Specify exact versions in requirements.txt and monitor the Milvus Compatibility Matrix
Validate input strings before tokenization	Filters out malformed surrogate pairs that can trigger encoding errors	Use str.encode('utf-8', errors='ignore') or a sanitisation utility
Add health‑check step for locale	Detects mis‑configured runners early	Include a tiny script that fails the job if locale is not UTF‑8
Monitor Milvus insertion metrics	Surface encoding‑related failures in production	Track milvus_insert_errors_total and alert on spikes

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

1. Why does the error appear only in CI and not locally?
   Local machines usually have a user‑configured locale (e.g., en_US.UTF-8). CI runners start with a minimal environment where LANG may be unset or set to C, causing the tokenizer to fall back to the system default encoding.
2. Can I avoid changing the CI workflow and just modify the Python code?
   You can wrap token strings with explicit UTF‑8 encode/decode as shown, but without fixing the environment you may still hit other Unicode‑related libraries (e.g., logging, file I/O). Setting the locale is the more robust, system‑wide solution.
3. What if I need to ingest non‑UTF‑8 data (e.g., legacy ISO‑8859‑1 files)?
   Convert those files to UTF‑8 before they reach the tokenizer. Use iconv or Python’s .encode('iso-8859-1').decode('utf-8') pattern, then proceed with the UTF‑8 pipeline.
4. Is there a way to detect the encoding error before calling insert()?
   Wrap the tokenization step in a try/except UnicodeError block and surface a custom exception. This allows the CI job to fail fast with a clear message.
5. Do newer Milvus versions eliminate the need for these locale settings?
   Later Milvus releases still rely on the Python process’s default encoding for string fields. The documentation (CI/CD Best Practices) continues to recommend explicit UTF‑8 locale configuration.