Problem – Empty Audio Feature Panels in Grafana
After the nightly audio‑processing batch job runs, several Grafana dashboards that visualise audio feature metrics (e.g., audio_feature_pitch_mean, audio_feature_snr_histogram) display either blank panels or the error “No data” despite the dashboards being correctly configured. The issue appears only after the batch job completes; the same panels show data when the job is manually triggered and succeeds.
Root Cause – Metrics Not Exported by Failed Batch Jobs
The nightly batch job consists of three stages:
- Ingest raw audio files from the
s3://audio‑rawbucket. - Run feature extraction (Python
librosapipeline) and write results to a PostgreSQL table. - Expose Prometheus metrics via a
/metricsendpoint using theprometheus_clientlibrary.
When the job fails during stage 2 (e.g., due to a corrupted file or a missing codec), the process exits before reaching stage 3. Consequently, the prometheus_client exporter never updates the metric values, and Prometheus scrapes a series of 0 or NaN samples. Grafana, which relies on those samples, renders empty panels.
Key observations that point to this root cause:
- Prometheus scrape logs contain “
target returned HTTP 500” for the batch exporter during the failure window. - The batch job’s log ends with a stack trace from
ffmpegindicating an unsupported audio codec. - Manual re‑run of the job on a single file succeeds, and metrics appear instantly on the dashboards.
Debug – Investigation Steps
1. Verify Prometheus Scrape Errors
journalctl -u prometheus -u grafana | grep "audio_batch_exporter"
# Example output
2026-11-07T02:00:12.345Z prometheus[1234]: scrape target "audio_batch_exporter:9100" failed: HTTP 500 Internal Server Error
2. Inspect Batch Job Logs
cat /var/log/audio_batch/job_20231107.log
2026-11-07 01:58:00 INFO Starting audio batch job
2026-11-07 01:58:01 INFO Processing file s3://audio-raw/track_001.wav
2026-11-07 01:58:03 ERROR ffmpeg error: Unknown codec: 'aac'
Traceback (most recent call last):
File "/opt/audio_batch/extract.py", line 87, in main
subprocess.run(ffmpeg_cmd, check=True)
File "/usr/lib/python3.11/subprocess.py", line 511, in run
raise CalledProcessError(retcode, cmd, output=output, stderr=stderr)
subprocess.CalledProcessError: Command '['ffmpeg', '-i', 's3://audio-raw/track_001.wav', ...]' returned non-zero exit status 1
2026-11-07 01:58:03 INFO Job aborted after 2 files processed
3. Confirm Metric Exporter State
curl -s http://audio_batch_exporter:9100/metrics | grep audio_feature
# Expected output when job succeeds
audio_feature_pitch_mean{job="audio_batch"} 220.5
audio_feature_snr_histogram_bucket{le="10",job="audio_batch"} 15
# Output during failure
# (no lines returned)
4. Check Prometheus Time‑Series Gaps
In Grafana, open the “Explore” view for audio_feature_pitch_mean and set the time range to the failure window. The graph shows a flat line at 0 with a “No data” annotation.
5. Identify the Faulty Input
Cross‑reference the failed file list from the batch log with the S3 inventory. The problematic file uses the .aac codec, which the current ffmpeg build does not support.
Solution – Make the Batch Job Resilient and Ensure Metric Export
1. Guard Against Processing Errors
Wrap the feature extraction step in a try/except block that logs the error but continues processing remaining files. Ensure the exporter is started before any processing begins.
# before (faulty)
for file in files:
extract_features(file) # aborts on first error
# after (robust)
from prometheus_client import start_http_server, Counter, Gauge
# Exporter metrics
files_processed = Counter('audio_batch_files_processed', 'Number of files processed')
files_failed = Counter('audio_batch_files_failed', 'Number of files that failed processing')
last_success_timestamp = Gauge('audio_batch_last_success_timestamp', 'Unix timestamp of last successful run')
def main():
start_http_server(9100) # start exporter early
for file in files:
try:
extract_features(file)
files_processed.inc()
except Exception as e:
files_failed.inc()
logger.error("Failed processing %s: %s", file, e)
continue
last_success_timestamp.set_to_current_time()
2. Deploy a Fixed ffmpeg Build
Install an ffmpeg package with AAC support or compile from source with --enable-libfdk_aac. Example for Ubuntu:
sudo apt-get purge ffmpeg
sudo add-apt-repository ppa:jonathonf/ffmpeg-4
sudo apt-get update
sudo apt-get install ffmpeg libavcodec-extra
ffmpeg -codecs | grep aac
# aa... D A aac Advanced Audio Coding (AAC) (decoders: aac )
3. Ensure Exporter Remains Alive on Partial Failures
The exporter is now started before any processing. Even if all files fail, the last_success_timestamp gauge will still be exported, allowing Grafana to display “0” instead of a missing series.
Verify – Confirm the Fix Works
1. Run a Controlled Nightly Job
./run_audio_batch.sh --date 2023-11-08
# Wait for completion, then:
curl -s http://audio_batch_exporter:9100/metrics | grep audio_feature
Expected output (sample):
# HELP audio_batch_files_processed Number of files processed
# TYPE audio_batch_files_processed counter
audio_batch_files_processed 1245
# HELP audio_batch_files_failed Number of files that failed processing
# TYPE audio_batch_files_failed counter
audio_batch_files_failed 2
# HELP audio_feature_pitch_mean Audio pitch mean (Hz)
# TYPE audio_feature_pitch_mean gauge
audio_feature_pitch_mean 215.3
2. Check Grafana Panels
Refresh the affected dashboards. Panels now show the latest values; the “No data” banner disappears. The “Failed files” counter can be added as a separate panel for operational visibility.
3. Validate Prometheus Scrape Success
promtool query instant http://localhost:9090/api/v1/query 'audio_batch_files_failed'
# Should return a non‑zero value if failures occurred, otherwise 0
Prevent – Operational Guardrails
- Metric Exporter Healthcheck: Add a
/healthzendpoint that returns 200 only when the exporter has started. Configure Prometheusscrape_intervalto 30s and setscrape_timeoutto 10s. - Alert on Zero Increments: Create a Prometheus alert that fires if
audio_batch_files_processeddoes not increase for two consecutive scrapes. - File Validation Pipeline: Run a pre‑flight S3 inventory check that rejects files with unsupported codecs. Use
ffprobeto verify codec compatibility before processing. - Rollback Strategy: Keep the previous stable ffmpeg version in a separate Docker layer; if a new codec is introduced, test the binary in a staging environment before rolling out.
- Logging Hygiene: Emit structured JSON logs with fields
file,stage,error. This enables log aggregation tools (e.g., Loki) to filter failures quickly.
Related Topic Hub: Observability Troubleshooting Hub
FAQ
- Why do the panels stay empty even after fixing the codec?
If the exporter process was not restarted after the fix, Prometheus may still be scraping a stale endpoint that returns HTTP 500. Restart the exporter container or service to re‑establish a healthy scrape. - Can I see which specific files caused metric gaps?
Add a label to theaudio_batch_files_failedcounter, e.g.,audio_batch_files_failed{file="track_001.wav"}. Then query Prometheus for the label values to identify problematic inputs. - Why does the “No data” message appear only for some panels?
Panels that query a gauge (e.g.,audio_feature_pitch_mean) will show “No data” when the series is missing. Panels that aggregate counters (e.g.,sum(audio_batch_files_processed)) may still display a zero value, which looks like a valid metric. - Is it safe to ignore the
audio_batch_files_failedmetric?
No. Ignoring failures can mask data quality issues and cause downstream analytics to drift. Treat any non‑zero value as a signal to investigate the input data. - How can I test the exporter locally before deploying?
Run the exporter in a Docker container with--network host, feed it a small set of audio files, and curlhttp://localhost:9100/metrics. Verify that all expected metric lines appear and that the HTTP status is 200.