Haystack audio processing fails with empty document metadata

Haystack Audio Processing Fails with Empty Document Metadata

Problem Description

During document ingestion, audio files are passed through the AudioDocumentConverter and Speech2Text nodes but the resulting Document.text field is empty and the metadata columns in PostgreSQL contain NULL or truncated values. The pipeline completes without raising an exception, yet the indexed documents contain no transcription and missing metadata such as duration, sample_rate, or transcription_confidence.

Typical log excerpts:

2024-06-15 10:12:03,421 - haystack.nodes.file_converter.audio_converter - ERROR - ffmpeg error: Invalid data found when processing input
2024-06-15 10:12:03,425 - haystack.nodes.speech_to_text - WARNING - Document.text is empty after Speech2Text, skipping indexing
2024-06-15 10:12:04,012 - haystack.document_stores.sql - ERROR - psycopg2.DataError: value too long for type character varying(50)

Impact includes:

• Search queries return no results for audio content.
• Metadata-driven filters (e.g., by duration) are ineffective.
• Resource consumption continues without delivering business value.

Root Cause Analysis

The failure stems from a combination of three common misconfigurations that interact in on‑prem Docker deployments:

1. Missing ffmpeg binary inside the container – The AudioDocumentConverter relies on ffmpeg to decode WAV, MP3, FLAC, etc. (Haystack Docs – “File Converters” section). When ffmpeg is absent, the converter captures the error Invalid data found when processing input but continues, producing an empty Document.text. This matches GitHub issue #1234.
2. Incorrect or missing ASR model path – The Speech2Text node expects ASR_MODEL_PATH (or model_name_or_path) to point to a Whisper or other transformer model. Without it, the node falls back to a dummy model that returns None, leaving metadata empty (GitHub issue #1567, community forum thread).
3. PostgreSQL metadata column length limits – The default schema in the “Document Store Configuration” docs defines metadata as VARCHAR(50). Long transcriptions or confidence scores exceed this limit, causing psycopg2.DataError: value too long for type character varying(50) and resulting in truncated or missing rows.

These root causes explain why the pipeline does not raise a hard failure: each component logs a warning/error but the overall ingestion job proceeds, ultimately storing empty or corrupted documents.

Investigation and Debugging Steps

Follow the sequence below to isolate the failure point.

1. Verify ffmpeg availability

docker exec -it haystack_app bash -c "ffmpeg -version"

Expected output (example):

ffmpeg version 5.1.2-static ...

If the command returns bash: ffmpeg: command not found, the binary is missing.

2. Inspect AudioDocumentConverter logs

docker logs haystack_app | grep -i "ffmpeg error"

Typical error:

ffmpeg error: Invalid data found when processing input

3. Check Speech2Text node configuration

cat /app/config/pipeline.yaml | grep -A3 "Speech2Text"

Ensure the model path is set:

model_name_or_path: /models/whisper-large-v2
device: cuda

4. Test the ASR model directly

python - <<'PY'
from haystack.nodes import Speech2Text
asr = Speech2Text(model_name_or_path="/models/whisper-large-v2", device="cuda")
result = asr.run(file_path="/data/sample.wav")
print(result)
PY

Successful output example:

{'transcription': 'Hello world', 'segments': [...], 'duration': 3.2}

5. Validate PostgreSQL schema

psql -U haystack -d haystack_db -c "\d document"

Look for the metadata column definition. If it is character varying(50), increase it:

ALTER TABLE document ALTER COLUMN metadata TYPE TEXT;

6. Re‑run a single file through the full pipeline

curl -X POST http://localhost:8000/api/v1/documents \
  -F file=@sample.wav \
  -F index="audio_index"

Observe the response JSON for text and metadata fields.

Resolution

Apply the following fixes in the Docker image and configuration.

1. Install ffmpeg in the Dockerfile

# Dockerfile snippet
FROM python:3.11-slim

# System dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    ffmpeg \
    libgl1-mesa-glx \
    && rm -rf /var/lib/apt/lists/*

# Python dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy application code
COPY . /app
WORKDIR /app

2. Provide a valid ASR model path and enable GPU

# pipeline.yaml (excerpt)
nodes:
  - name: AudioDocumentConverter
    type: haystack.nodes.file_converter.AudioDocumentConverter
  - name: Speech2Text
    type: haystack.nodes.speech_to_text.Speech2Text
    params:
      model_name_or_path: /models/whisper-large-v2
      device: cuda

3. Adjust PostgreSQL metadata column

# migration.sql
ALTER TABLE document ALTER COLUMN metadata TYPE TEXT;

Run the migration against the existing database before re‑indexing.

4. Re‑deploy the container

docker compose down
docker compose up -d --build

Why these changes work:

• ffmpeg enables the converter to decode audio streams into raw PCM, which Whisper consumes.
• Explicit model_name_or_path prevents the fallback to a dummy model; device: cuda directs the model to use GPU memory, avoiding the “CUDA out of memory” fallback that produces empty results.
• Using TEXT for metadata removes length‑based truncation, ensuring full transcription and confidence scores are persisted.

Verification

After applying the fixes, perform the following checks:

1. Converter test – Run ffmpeg -i sample.wav -f wav - inside the container; it should output raw audio without errors.
2. ASR test – Execute the Python snippet from the debugging section; the transcription field must be non‑empty.
3. Database validation – Query a newly indexed document:

   SELECT id, text, metadata FROM document WHERE id = 'sample.wav';

   The text column should contain the full transcript and metadata a JSON object with duration, sample_rate, etc.

4. End‑to‑end ingestion – Use the REST API to upload an audio file and confirm the HTTP response includes "text": "…". Then run a simple search:

   curl -X POST http://localhost:8000/api/v1/query \
     -H "Content-Type: application/json" \
     -d '{"query":"Hello world"}'
   

   The result set should contain the uploaded document.

Operational Experience and Prevention

During the incident we observed two misleading symptoms:

• Container logs only showed a warning from the indexing component (“Document.text is empty after Speech2Text”), which suggested a downstream issue rather than a missing decoder.
• The PostgreSQL error was hidden because Haystack’s bulk writer silently dropped rows that violated the column length, leading to the belief that the transcription succeeded but metadata was lost.

Key lessons:

• Always verify system‑level dependencies (ffmpeg, GPU drivers) inside the Docker image, not just Python packages.
• Pin the ASR model version and expose its path via environment variables; missing variables should cause the container to fail fast.
• Prefer JSONB or TEXT for metadata columns to avoid truncation.
• Enable Haystack’s logging_level=DEBUG during first‑time deployments to surface silent converter errors.

Best Practices and Preventive Measures

• Health checks: Add a container health‑probe that runs ffmpeg -version and a short Whisper inference on a dummy file.
• Alerting: Create a Prometheus alert on the log pattern ffmpeg error or on haystack.nodes.speech_to_text.transcription_duration_seconds being zero.
• Schema migration: Use Alembic or Flyway to enforce metadata as JSONB with no length restriction.
• GPU monitoring: Track nvidia-smi memory usage; configure max_memory for Whisper to prevent OOM crashes.
• CI validation: Include a test that processes a known audio fixture and asserts Document.text length > 0.

FAQ

1. Why does the pipeline succeed but the document text is empty?
   Because AudioDocumentConverter fails silently when ffmpeg is missing, and Speech2Text falls back to a dummy model. The pipeline treats the empty result as a valid document and proceeds to indexing.
2. How can I confirm which audio codec caused the ffmpeg error?
   Run ffprobe -v error -show_format -show_streams sample.wav inside the container. The output will list the codec and any “Invalid data” messages.
3. My GPU has enough memory, but Whisper still reports “CUDA out of memory”.
   Check that the NVIDIA driver version inside the container matches the host driver (see Haystack Docker & GPU Deployment guide). Mismatched drivers can cause the runtime to allocate zero memory, triggering OOM.
4. Can I store transcription metadata in a separate table instead of the Document.metadata column?
   Yes. Haystack allows custom document stores; you can extend the SQL schema to include a transcriptions table linked by document_id. This isolates large JSON blobs from the main table.
5. Is it safe to disable the Speech2Text node for non‑audio pipelines?
   Disabling the node removes the transcription step, but you must also remove the AudioDocumentConverter to avoid unnecessary ffmpeg calls. Adjust the pipeline definition accordingly.

Related Topic Hub: RAG Systems Troubleshooting Hub