Qwen model hallucinated descriptions during concurrent image-text inference

Problem Description

During a high‑traffic period the Qwen multimodal endpoint started returning unrelated or fabricated image captions. The issue manifested only when the API gateway routed dozens to thousands of concurrent image‑text requests through a load‑balanced pool of inference pods.

Typical symptoms observed in logs and client responses:

• JSON response field caption contains text unrelated to the supplied image (e.g., “A beach sunset” for an invoice image).
• Intermittent VisualEmbeddingShapeMismatchError and CrossAttentionIndexOutOfBounds exceptions.
• Log entry TokenStreamDesyncWarning: visual token offset 256 differs from textual offset 0 emitted by the Qwen runtime.
• Latency spikes when batch size exceeds ~32 requests; SLA violations of the 150 ms latency target.

Example error snippet (extracted from journalctl -u qwen-inference.service):

2024-07-12T14:23:07.842Z ERROR VisualEmbeddingShapeMismatchError: expected shape (batch, seq_len, dim) but got (batch, dim)
2024-07-12T14:23:07.845Z WARN TokenStreamDesyncWarning: visual token offset 384 differs from textual offset 0
2024-07-12T14:23:07.849Z ERROR CrossAttentionIndexOutOfBounds: token index 1024 exceeds sequence length 768

These symptoms match the community reports in GitHub Issue #842 and the fintech production incident of Jan 2024 where a 15 % rise in hallucinated captions was traced to a shared visual‑embedding buffer not being cleared between requests.

Root Cause Analysis

The Qwen multimodal architecture (see the Qwen‑7B‑Chat Multimodal Model Card on Hugging Face) consists of three logical stages:

1. Visual encoder – extracts a fixed‑length embedding sequence (V = (seq_len, dim)) from the input image.
2. Token synchronizer – concatenates visual tokens with the textual token stream before feeding them to the transformer.
3. Cross‑attention layers – attend jointly over visual and textual tokens.

When the inference service runs in concurrency=true mode, the SDK reuses a single visual_encoder instance across threads to reduce GPU memory churn (Qwen SDK Reference). The SDK also maintains a per‑process buffer that holds the most recent visual embedding before it is stitched into the token stream.

Under dynamic batching (described in the Qwen Inference Guide), the runtime may reshape the batch dimension on the fly. If the visual‑embedding buffer is not atomically cleared or duplicated per request, two concurrent requests can write into the same memory region:

• Request A writes its visual tokens (offset 0‑255).
• Before A’s tokens are consumed, Request B overwrites the buffer with its own visual tokens (offset 256‑511).
• The token synchronizer then concatenates a mixed visual token sequence with the textual tokens of Request A, leading to TokenStreamDesyncWarning and cross‑attention maps that point at unrelated visual content.

This race condition is amplified when the batch size grows beyond the static pre‑allocated buffer (≈32 requests), as reported in GitHub Issue #913 and observed in the e‑commerce recommendation service incident (Mar 2024).

Investigation and Debugging

The following step‑by‑step procedure reproduces the desynchronization and isolates the faulty component.

1. Enable verbose runtime logging

export QWEN_LOG_LEVEL=debug
qwen-inference --config ./config.yaml

Sample debug output shows overlapping visual‑embedding writes:

[DEBUG] VisualEncoder::encode_image: request_id=42 start
[DEBUG] VisualEncoder::encode_image: request_id=43 start
[DEBUG] VisualEncoder::write_buffer: request_id=42 wrote 256 tokens at offset 0
[DEBUG] VisualEncoder::write_buffer: request_id=43 overwrote buffer at offset 0 (previously 42)
[WARN] TokenStreamDesyncWarning: visual token offset 256 differs from textual offset 0 (request_id=42)

2. Capture the process memory layout

pid=$(pgrep -f qwen-inference)
pmap -x $pid | grep visual_buffer

The buffer is a single shared memory region (/dev/shm/qwen_visual_buffer) mapped into all worker threads.

3. Reproduce with a minimal concurrent client

import threading, requests, base64, json

def call(img_path, text):
    img_b64 = base64.b64encode(open(img_path,'rb').read()).decode()
    payload = {"image": img_b64, "text": text}
    r = requests.post("http://api.example.com/v1/qwen/multimodal", json=payload)
    print(r.json()["caption"])

threads = []
for i in range(2):
    t = threading.Thread(target=call, args=("invoice.jpg", "Describe the document"))
    threads.append(t)
    t.start()
for t in threads:
    t.join()

Running the script while the service is under a batch size of 64 consistently produces a caption from the *other* thread.

4. Verify that disabling shared buffers eliminates the issue

# In config.yaml
visual_encoder:
  shared_instance: false   # Force per‑request encoder instance

After reload, the same concurrent test yields correct, matching captions.

Resolution

The fix consists of two complementary changes:

1. Enforce per‑request visual encoder isolation

Update the inference service configuration to disable the global shared encoder and enable a lightweight per‑request lock.

Before:

visual_encoder:
  shared_instance: true
  max_concurrency: 128

After:

visual_encoder:
  shared_instance: false          # each request gets its own encoder instance
  max_concurrency: 64            # keep within GPU memory budget
  lock_mode: per_request          # SDK will acquire a mutex around encode_image

2. Patch the SDK to clear the visual‑embedding buffer atomically

Apply the community‑provided hot‑fix from GitHub Issue #842 (commit e3f9b7c).

// qwen_sdk/visual_encoder.cpp
void VisualEncoder::encode(const Image& img, RequestContext* ctx) {
    std::lock_guard<std::mutex> guard(buffer_mutex_);   // NEW: protect shared buffer
    buffer_.clear();                                   // NEW: ensure no stale tokens
    auto embedding = model_->forward(img);
    buffer_.push_back(embedding);
    ctx->set_visual_tokens(buffer_);
}

Rebuild the SDK and redeploy the inference pods.

3. Adjust dynamic batching thresholds

According to the Qwen Inference Guide, keep max_batch_size ≤ 32 when shared_instance=false to avoid unnecessary memory pressure.

batching:
  dynamic: true
  max_batch_size: 32
  latency_target_ms: 150

Validation

After applying the changes, perform the following checks:

Functional verification

# Run the concurrent client script (8 parallel threads)
python concurrent_test.py
# Expected: All captions correctly describe their respective images

Log inspection

$ grep TokenStreamDesyncWarning /var/log/qwen-inference.log
# No output – warning cleared

Metrics

Confirm that the qwen_visual_embedding_buffer_hits counter remains at zero and that the 95th‑percentile latency stays below the SLA target.

curl -s http://metrics.example.com/metrics | grep qwen_visual_embedding_buffer
# output: qwen_visual_embedding_buffer_hits 0

Cross‑attention sanity check

Run a single‑request trace using the SDK’s debug_attn_map flag and verify that visual tokens align with the expected image regions.

qwen-inference --debug_attn_map=true --image sample.jpg --text "Summarize"
# Inspect the printed attention heatmap – tokens 0‑255 correspond to visual patches

Prevention and Best Practices

• Never enable shared_instance=true in a high‑concurrency deployment. The performance gain is outweighed by the risk of token desynchronization.
• Set max_batch_size conservatively (≤ 32) when running multimodal models; larger batches should be handled by scaling out pods rather than increasing batch size.
• Instrument the service with a custom alert on TokenStreamDesyncWarning and VisualEmbeddingShapeMismatchError thresholds (e.g., alert if count > 0 in a 5‑minute window).
• Use the SDK’s visual_encoder.lock_mode=per_request or wrap calls in a process‑level mutex if you must share the encoder for memory‑constrained GPUs.
• Validate the shape of visual embeddings before concatenation:

if visual.shape != (seq_len, dim):
    raise RuntimeError(f"Invalid visual shape: {visual.shape}")

Enable the runtime flag --strict_token_sync (available from Qwen‑Inference v1.2) to abort requests that exhibit offset mismatches.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why does the problem disappear when I reduce the batch size?
   Because the shared visual‑embedding buffer is allocated for the maximum batch size. Smaller batches reduce the chance that two requests write to the same memory region before the first request finishes, effectively hiding the race condition.
2. Can I keep shared_instance=true and still avoid hallucinations?
   Only by adding an explicit lock around visual_encoder.encode_image and clearing the buffer after each call. The SDK patch in the resolution section implements exactly that.
3. Is the issue specific to Qwen‑7B‑Chat or does it affect other Qwen multimodal models?
   All Qwen multimodal variants share the same visual‑encoder implementation, so the token‑desync bug appears in any model that uses the shared encoder path under concurrency.
4. How can I monitor cross‑attention map health in production?
   Enable the debug_attn_map flag on a sampling of requests and push the resulting heatmaps to a Prometheus‑compatible histogram. Sudden spikes in the “visual‑token‑attention‑ratio” metric often precede desynchronization failures.
5. What impact does auto‑scaling have on this problem?
   When pods are added or removed, sticky sessions may be lost, causing requests that were partially processed on one pod to be resumed on another pod with a fresh visual‑encoder instance. This can surface as a “visual token desync” error, as seen in the May 2024 translation platform incident.