Haystack text generation ignoring stop sequences in Kubernetes

Problem Description

In a Kubernetes‑deployed Haystack microservice stack, the generate endpoint returns text that continues past the user‑specified stop sequence. The symptom is observable both in API responses and in logs such as:

GenerationError: stop token not found in generated sequence – stop='\nAnswer:'
Traceback (most recent call last):
  File "/app/pipelines/generation.py", line 112, in _run_generation
    generated = model.generate(**generation_kwargs)
ValueError: stop sequence token id not in tokenizer vocab
Warning: pipeline.generate: max_new_tokens reached before stop criteria; output may be truncated

Clients receive malformed answers where the expected delimiter (e.g., \nAnswer:) is missing, causing downstream parsing failures in the Q&A workflow.

Root Cause Analysis

Haystack forwards the stop list supplied in generation_kwargs to the underlying Hugging Face generate method. The method relies on stopping_criteria, which expects token IDs that match the tokenizer used at inference time. The following factors broke this contract in the reported deployment:

• Tokenizer mismatch: The inference container (haystack-inference) was built with a custom HF model whose tokenizer uses a different byte‑pair encoding (BPE) than the default tokenizer bundled with the Haystack API. As a result, tokenizer.encode(stop_seq) returned IDs that do not exist in the inference tokenizer’s vocabulary, leading to the ValueError shown above. This aligns with the production incident on GKE where “the model’s tokenizer used a different byte‑pair encoding than the one assumed by Haystack”.
• Missing return_full_text=False: By default Haystack returns the full prompt plus generated text. When return_full_text is true, the stop‑criteria check is performed on the generated portion only, but the concatenated output can hide the stop token, causing the pipeline to think the stop condition was never met. The GitHub issue #2451 highlights this exact behavior.
• Environment variable misconfiguration: The container was started with --max-new-tokens=0 due to an incorrectly set MAX_NEW_TOKENS env var. This forced the generation loop to bypass stop‑criteria evaluation and rely solely on the model’s internal max length, as observed in the Docker‑compose to Kubernetes migration incident.
• Stale tokenizer after pod restart: An autoscaling event caused the inference pod to restart without re‑initialising the tokenizer configuration. The in‑memory token IDs for stop sequences remained from the previous version, leading to “stop token not found” errors (see issue #2198).

Investigation and Debugging

1. Inspect API request payload to confirm the stop sequence is being sent:

curl -X POST http://haystack-api:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
        "query": "What is the capital of France?",
        "generation_kwargs": {
          "max_new_tokens": 50,
          "stop": ["\nAnswer:"]
        }
      }'

2. Verify that the inference container receives the same generation_kwargs: Enable debug logging in haystack-inference:

export LOG_LEVEL=DEBUG
journalctl -u haystack-inference -f | grep generation_kwargs

3. Check tokenizer consistency: Run a one‑off tokenization test inside both containers.

# In haystack-api container
python - <<'PY'
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("my-model")
print(tokenizer.encode("\\nAnswer:"))
PY

# In haystack-inference container
python - <<'PY'
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("my-model")
print(tokenizer.encode("\\nAnswer:"))
PY

If the printed ID lists differ, the vocab mismatch is confirmed.

4. Confirm environment variables:

kubectl exec -it $(kubectl get pod -l app=haystack-inference -o jsonpath='{.items[0].metadata.name}') -- printenv | grep MAX_NEW_TOKENS

Look for a value of 0 or an empty string.

5. Examine pod restart behavior: Review the pod’s initContainer logs to ensure the tokenizer files are re‑loaded after a restart.

kubectl logs $(kubectl get pod -l app=haystack-inference -o jsonpath='{.items[0].metadata.name}') -c init-tokenizer

Resolution

1. Align Tokenizers Across Services

Package the same tokenizer files with both haystack-api and haystack-inference images, or mount a shared ConfigMap.

# Dockerfile snippet for both services
COPY tokenizer/ /opt/model/tokenizer/
ENV HF_TOKENIZER_PATH=/opt/model/tokenizer/

Rebuild and redeploy:

docker build -t myrepo/haystack-api:latest .
docker build -t myrepo/haystack-inference:latest .
kubectl rollout restart deployment/haystack-api
kubectl rollout restart deployment/haystack-inference

2. Pass Explicit EOS Token ID via generation_kwargs

Map the stop string to the correct token ID at request time:

import os
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(os.getenv("HF_TOKENIZER_PATH"))
stop_seq = "\nAnswer:"
stop_id = tokenizer.encode(stop_seq, add_special_tokens=False)[0]

payload = {
    "query": "What is the capital of France?",
    "generation_kwargs": {
        "max_new_tokens": 50,
        "eos_token_id": stop_id,
        "return_full_text": False,
        "skip_special_tokens": True
    }
}

3. Disable Full‑Prompt Echo

Set return_full_text=False globally in the pipeline configuration (YAML example):

# haystack-pipeline.yaml
components:
  - name: generator
    type: TransformersGenerator
    params:
      model_name_or_path: my-model
      return_full_text: false
      generation_kwargs:
        max_new_tokens: 100

4. Correct Environment Variable for Token Limits

Ensure MAX_NEW_TOKENS is set to a positive integer or omitted to use the default.

# values.yaml snippet for Helm chart
haystackInference:
  env:
    - name: MAX_NEW_TOKENS
      value: "150"

5. Reload Tokenizer on Pod Restart

Add a postStart hook that forces a fresh tokenizer load:

# deployment.yaml excerpt
spec:
  containers:
    - name: haystack-inference
      image: myrepo/haystack-inference:latest
      lifecycle:
        postStart:
          exec:
            command: ["/bin/sh", "-c", "python -c 'from transformers import AutoTokenizer; AutoTokenizer.from_pretrained(\"$HF_TOKENIZER_PATH\")'"]

Validation

1. Send a test request with the updated payload and verify that the response ends exactly at \nAnswer::

curl -X POST http://haystack-api:8000/generate -d @payload.json

Expected response snippet:

What is the capital of France?
Answer: Paris

2. Check the inference logs for a line confirming stop‑criteria hit:

2026-06-14 10:22:31,124 INFO pipeline.generate - Stopping criteria met at token 23 (eos_token_id=50257)

3. Run a token‑ID verification script to ensure the stop ID exists in the tokenizer vocab:

python - <<'PY'
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained("my-model")
assert "\\nAnswer:" in t.get_vocab()
PY

4. Monitor the generation_success metric in Prometheus; it should show a spike in successful generations without “max_new_tokens reached” warnings.

Prevention and Best Practices

• Version‑pin tokenizer and model together: Use the same Docker image tag for both API and inference services.
• Validate stop sequences at startup: Add an init script that encodes each configured stop string and aborts the container if any ID is missing.
• Expose generation metrics: Track stop_criteria_hits and max_new_tokens_exceeded to detect regressions early.
• Keep environment variables explicit: Avoid defaulting MAX_NEW_TOKENS to 0; enforce a minimum via Helm values.yaml schema validation.
• Use health checks that include a short generation probe:

# livenessProbe snippet
livenessProbe:
  exec:
    command: ["python", "-c", "import requests; r=requests.post('http://localhost:8000/generate', json={'query':'test','generation_kwargs':{'max_new_tokens':5,'stop':['.']}}); assert r.ok"]
  initialDelaySeconds: 30
  periodSeconds: 60

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

1. Why does the stop sequence work locally but not in Kubernetes?
   Because the local environment uses the same tokenizer binary as the model, while the Kubernetes deployment had a mismatched tokenizer image. Aligning the tokenizer files resolves the discrepancy.
2. Do I need to set both eos_token_id and stop?
   Haystack translates stop into eos_token_id internally. When the tokenizer vocab differs, supplying the explicit eos_token_id ensures the correct token ID is used.
3. Can I use multiple stop strings?
   Yes. Encode each string to its token ID and pass a list to eos_token_id (or use stopping_criteria directly). Ensure all IDs exist in the shared tokenizer.
4. Is skip_special_tokens required?
   Setting skip_special_tokens=True prevents special tokens (e.g., <pad>) from being emitted after the stop token, which can otherwise appear as stray characters in the response.
5. How do I debug tokenization mismatches without redeploying?
   Run the tokenization test shown in the investigation steps inside the running pods. The printed ID arrays reveal whether the stop string maps to a known token.