yellow and white 10 card

Qwen empty transcription after uploading 48kHz wav on autoscaling cluster

by

Problem

When a 48 kHz .wav file is uploaded to the Qwen transcription service running on a multi‑tenant Kubernetes cluster, the API either returns an empty transcription string or raises a ProcessingException. The issue appears only under load when the autoscaler creates new pods; lower‑sample‑rate files (16 kHz, 22.05 kHz) are processed correctly.

Root Cause

The Qwen model is compiled with a hard‑coded expectation of a 16 kHz mono PCM input. The service performs an implicit ffmpeg conversion only if the input sample rate matches the model’s sample_rate configuration (default 16000). In the production deployment the following conditions combine to cause the failure:

  • Sample‑rate mismatch: 48 kHz audio is passed directly to the model without resampling because the audio_preprocess flag is disabled in the container’s environment.
  • Pod warm‑up race: Autoscaled pods start with an empty /tmp/models volume. The model files are lazily downloaded on first request; the first request may hit a pod that has not yet completed the download, leading to a fallback path that skips preprocessing.
  • Shared storage latency: The shared PersistentVolume used for audio assets is provisioned with ReadWriteMany NFS. Under high concurrency the NFS latency causes the ffmpeg command to time out, returning a zero‑byte output file.

Consequently the model receives an empty or malformed audio buffer, producing an empty transcription or throwing an exception.

Debug

1. Log inspection


2026-10-12T14:03:27.842Z INFO  transcription-service: Received request id=7f9c3a2b, sample_rate=48000
2026-10-12T14:03:27.845Z WARN  audio-preprocess: Skipping resample, flag DISABLE_RESAMPLE=true
2026-10-12T14:03:27.950Z ERROR transcription-model: ProcessingException: Input audio buffer is empty

2. Verify pod startup sequence


kubectl get pods -l app=qwen-transcribe -w
NAME                                 READY   STATUS    RESTARTS   AGE
qwen-transcribe-5d9f7c9d8b-8kz9l      0/2     Init:0/1  0          12s
qwen-transcribe-5d9f7c9d8b-8kz9l      1/2     Running   0          5s
qwen-transcribe-5d9f7c9d8b-8kz9l      2/2     Running   0          5s

The second container (model loader) may still be pulling model files when the API container receives the first request.

3. Reproduce locally


curl -X POST http://localhost:8080/transcribe \
  -H "Content-Type: audio/wav" \
  --data-binary @sample_48k.wav -v

Response:


HTTP/1.1 200 OK
Content-Type: application/json

{"transcript":""}

4. Check ffmpeg execution


docker exec -it qwen-transcribe-5d9f7c9d8b-8kz9l -- ffmpeg -i /tmp/input.wav -ar 16000 /tmp/resampled.wav
ffmpeg version 5.1.2 ...
Input #0, wav, from '/tmp/input.wav':
  Duration: 00:00:05.00, bitrate: 1411 kb/s
  Stream #0:0: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
Output #0, wav, to '/tmp/resampled.wav':
  Stream #0:0: Audio: pcm_s16le, 16000 Hz, mono, s16, 256 kb/s

When the pod is under load, the same command times out after 30 s, leaving /tmp/resampled.wav empty.

Solution

1. Enforce resampling for all inputs

Update the deployment to set DISABLE_RESAMPLE=false and add a fallback conversion step that always runs before the model call.


# before (environment snippet)
env:
  - name: DISABLE_RESAMPLE
    value: "true"


# after
env:
  - name: DISABLE_RESAMPLE
    value: "false"
  - name: FORCE_RESAMPLE
    value: "true"

2. Add an init container to pre‑warm the model


apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen-transcribe
spec:
  replicas: 3
  template:
    spec:
      initContainers:
      - name: model-warmup
        image: qwen/model-downloader:latest
        command: ["sh", "-c", "wget -qO- https://model-repo/qwen.pt | tar -xz -C /models"]
        volumeMounts:
        - name: model-volume
          mountPath: /models
      containers:
      - name: api
        image: qwen/api:latest
        env:
        - name: DISABLE_RESAMPLE
          value: "false"
        volumeMounts:
        - name: model-volume
          mountPath: /models
      volumes:
      - name: model-volume
        persistentVolumeClaim:
          claimName: qwen-model-pvc

3. Increase NFS timeout and add retry logic for ffmpeg


# audio_preprocess.py (excerpt)
def resample(input_path, output_path):
    for attempt in range(3):
        result = subprocess.run(
            ["ffmpeg", "-y", "-i", input_path, "-ar", "16000", output_path],
            stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=60
        )
        if result.returncode == 0 and os.path.getsize(output_path) > 0:
            return output_path
        time.sleep(5)  # back‑off
    raise ProcessingException("ffmpeg resample failed after retries")

4. Adjust Horizontal Pod Autoscaler (HPA) to use a warm‑up period


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qwen-transcribe-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwen-transcribe
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 120
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

Verify

Functional test


curl -X POST http://transcribe.mycorp.com/v1 \
  -H "Content-Type: audio/wav" \
  --data-binary @sample_48k.wav | jq .

Expected JSON response:


{
  "transcript": "The quick brown fox jumps over the lazy dog."
}

Log confirmation


2026-10-12T14:45:02.112Z INFO  audio-preprocess: Resampling /tmp/input.wav -> /tmp/resampled.wav (48k -> 16k)
2026-10-12T14:45:02.219Z INFO  transcription-model: Received 5.00s audio, 16000 Hz, 80000 samples
2026-10-12T14:45:03.001Z INFO  transcription-model: Transcription completed, length=12 tokens

Metrics

Metric Before After
Empty transcription rate 12 % 0 %
ffmpeg timeout errors 8 /min 0 /min
HPA scaling latency 45 s (cold start) 15 s (warm start)

Prevent

  • Validate audio sample rate at API gateway; reject >24 kHz with a clear error.
  • Pin the model container image to a version that includes the resampling flag enabled by default.
  • Monitor ffmpeg_exit_code and audio_resample_duration_seconds via Prometheus alerts.
  • Use a ReadWriteOnce block storage for model files to avoid NFS latency spikes.
  • Run a periodic “warm‑up” job that loads the model and performs a dummy transcription.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the issue only appear after the autoscaler creates new pods? New pods start with an empty model cache. The first request may hit a pod that has not yet completed the lazy model download, causing the preprocessing path to be bypassed and the 48 kHz audio to be sent directly to the model.
  2. Can I keep 48 kHz audio without resampling? The current Qwen checkpoint was trained on 16 kHz data; higher sample rates produce undefined behavior. Either resample to 16 kHz or retrain the model with 48 kHz inputs.
  3. What error should I look for in logs to detect a failed ffmpeg conversion? Look for ffmpeg: error while processing stream or a non‑zero exit code, followed by ProcessingException: ffmpeg resample failed.
  4. Is there a way to enforce resampling at the ingress level? Yes, you can add an Envoy filter or an API gateway Lambda that runs ffmpeg -ar 16000 before forwarding the payload to the service.
  5. How do I test that the warm‑up init container works correctly? Exec into a pod after deployment and verify that /models/qwen.pt exists and that python -c "import torch; torch.load('/models/qwen.pt')" succeeds without downloading.