A person holding a smart phone in their hand

Mistral speech-to-text pod crashes with segmentation fault after codec removal

by

Mistral speech‑to‑text pod crashes with segmentation fault after codec removal

Problem

The mistral-stt StatefulSet runs on GPU‑enabled nodes (Kubernetes v1.27). After a routine image update that removed unused audio codec packages, the pod starts but crashes during the first inference request. The container logs end with a segmentation fault (SIGSEGV) inside the audio preprocessing library.

Typical log excerpt


2024-09-27T14:12:03.421Z ERROR [audio_preprocess] Loading audio file: /tmp/input.wav
2024-09-27T14:12:03.423Z FATAL [audio_preprocess] libavcodec.so.58: symbol not found: avcodec_find_decoder_by_name
2024-09-27T14:12:03.424Z INFO  [main] Received request id=7f3c2a1b
2024-09-27T14:12:03.425Z SIGSEGV (core dumped) at 0x7f8c3d2a1b90
2024-09-27T14:12:03.426Z Stack trace:
2024-09-27T14:12:03.426Z   #0  0x7f8c3d2a1b90 in avcodec_open2
2024-09-27T14:12:03.426Z   #1  0x7f8c3d2b3c20 in AudioDecoder::Init()
2024-09-27T14:12:03.426Z   #2  0x7f8c3d2c5e10 in AudioPreprocessor::Load()
2024-09-27T14:12:03.426Z   #3  0x7f8c3d2d7a00 in InferenceHandler::Run()
2024-09-27T14:12:03.426Z   #4  0x7f8c3d2e1b30 in main

Impact: All speech‑to‑text requests fail, causing downstream transcription pipelines to stall. The pod restarts repeatedly, exhausting the pod disruption budget.

Root Cause

The image build script removed libavcodec58 and related ffmpeg libraries to reduce surface area. The Mistral inference code relies on libavcodec for WAV/MP3 decoding. When the shared library is missing, the dynamic linker resolves symbols at runtime, but the missing symbol avcodec_find_decoder_by_name causes the loader to return a NULL pointer. The subsequent call to avcodec_open2 dereferences this NULL pointer, leading to a segmentation fault.

Why it manifested only after the image change:

  • The service previously used the system‑installed ffmpeg packages from Ubuntu 22.04.
  • The sidecar container does not provide the codecs; the main container expects them to be present in the same filesystem namespace.
  • During the image rebuild, apt-get purge libavcodec58 was added to the Dockerfile to “trim unused libs”.

Debug & Investigation

Step‑by‑step diagnostics performed on a failing pod (mistral-stt-0):

  1. Inspect the container filesystem for the missing library.

    
kubectl exec -it mistral-stt-0 -c mistral -- bash
ls -l /usr/lib/x86_64-linux-gnu/libavcodec*

    Expected output (when present):

    
-rw-r--r-- 1 root root 1.8M Jan 12 2023 /usr/lib/x86_64-linux-gnu/libavcodec.so.58

    Actual output:

    
ls: cannot access '/usr/lib/x86_64-linux-gnu/libavcodec*': No such file or directory

  2. Check dynamic linker dependencies of the binary that loads the codec.

    
ldd /app/bin/audio_preprocess | grep avcodec

    Result:

    
libavcodec.so.58 => not found

  3. Confirm that the segmentation fault originates from the missing symbol.

    
gdb -batch -ex "run /tmp/input.wav" -ex "bt" /app/bin/audio_preprocess

    Backtrace matches the stack trace shown in the pod logs, confirming the NULL dereference.

  4. Validate that the sidecar does not provide the library.

    
kubectl exec -it mistral-stt-0 -c logger -- bash -c "ls /usr/lib/x86_64-linux-gnu/libavcodec*"

    Output: same “No such file or directory”.

Solution

Two viable approaches were evaluated:

Option Pros Cons
Re‑install libavcodec58 in the main container Minimal code change; restores original runtime behavior. Increases image size; re‑introduces previously removed packages.
Bundle a static build of ffmpeg with the application Self‑contained binary; no external dependencies. Requires custom build pipeline; larger binary.

The production team chose the first option for speed of remediation.

Before (Dockerfile snippet)


FROM ubuntu:22.04 AS base
RUN apt-get update && apt-get install -y \
    python3 \
    python3-pip \
    && apt-get purge -y libavcodec58 libavformat58 \
    && apt-get clean && rm -rf /var/lib/apt/lists/*
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
CMD ["python3", "-m", "mistral.stt.server"]

After (Dockerfile snippet)


FROM ubuntu:22.04 AS base
RUN apt-get update && apt-get install -y \
    python3 \
    python3-pip \
    libavcodec58 \
    libavformat58 \
    libavutil58 \
    && apt-get clean && rm -rf /var/lib/apt/lists/*
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
CMD ["python3", "-m", "mistral.stt.server"]

Re‑building and redeploying the image:


docker build -t registry.example.com/mistral-stt:20240927 .
docker push registry.example.com/mistral-stt:20240927
kubectl set image statefulset/mistral-stt mistral=registry.example.com/mistral-stt:20240927

After the rollout, the pod starts without crashing, and inference succeeds.

Verification

Validate both the absence of crashes and correct transcription output.

  1. Check pod status:
    2. 
kubectl get pods -l app=mistral-stt -w

    All pods should reach Running with READY 2/2.

  2. Run a health‑check endpoint:
    3. 
curl -s http://mistral-stt-0.mistral.svc.cluster.local:8080/healthz

    Expected JSON:

    
{"status":"ok","codec":"libavcodec58"}
  3. Submit a test audio file and compare the transcription:
    4. 
curl -X POST -H "Content-Type: audio/wav" \
     --data-binary @samples/hello.wav \
     http://mistral-stt-0.mistral.svc.cluster.local:8080/transcribe

    Expected output (example):

    
{"text":"hello world"}
  4. Inspect logs for absence of SIGSEGV:
    5. 
kubectl logs mistral-stt-0 -c mistral | grep -i segv

    No output indicates the fault is resolved.

Prevention & Operational Guardrails

  • Dependency lock file. Keep a apt‑list or dpkg‑query snapshot in source control to detect accidental removals.
  • Container image scanning. Integrate a CI step that runs ldd on all binaries and fails if any required shared library is missing.
  • Runtime health probe. Extend the /healthz endpoint to verify that critical shared libraries are loadable (e.g., attempt dlopen("libavcodec.so.58")).
  • Pod disruption budget. Set a maxUnavailable of 1 for the StatefulSet to avoid total service outage during rollouts.
  • Version pinning. Use explicit package versions (e.g., libavcodec58=7:4.4.2-0ubuntu0.22.04.1) to avoid upstream regressions.

FAQ

  1. Why does the pod crash only after the image rebuild? The rebuild removed libavcodec58. The binary loads the library at runtime; without it, the loader returns NULL, leading to a segmentation fault.
  2. Can I use a static ffmpeg build instead of installing the library? Yes. Building ffmpeg with --enable-static and linking it statically into the application removes the runtime dependency, but it requires a custom build pipeline and larger binaries.
  3. Is there a way to detect missing codecs before deployment? Add a CI check that runs ldd on the compiled binary and fails if any required libav* entries are “not found”.
  4. Do sidecar containers share the same filesystem namespace? No. Each container has its own root filesystem; shared libraries must be present in the container that executes the code, not just in a sidecar.
  5. Will upgrading Ubuntu to 24.04 solve the issue? Upgrading may bring newer library versions, but the root cause (removing the package) would still exist. Ensure the required packages are installed regardless of the base OS version.

Related Topic Hub: LLM Systems Troubleshooting Hub