Docker Container Audio/video Drift After Scaling Inference Pods In Kubernetes

Q: 1. Verify container clock vs. host clock

# Inside a running pod kubectl exec -it infer-pod-1 -- date +%s%N # On the node ssh node-01 "date +%s%N" Expected output: difference < 10 ms. In the drifting pods the offset was ~150 ms.

Q: 2. Check CPU throttling metrics

# Using cAdvisor metrics exposed by kubelet curl -s http://localhost:10255/stats/summary | jq '.pods[] | select(.podRef.name=="infer-pod-3") | .cpu.throttlingData' Typical output for a healthy pod:

Q: 3. Capture GStreamer pipeline timestamps

gst-launch-1.0 -v filesrc location=sample.mp4 ! decodebin name=dec \ dec. ! queue ! audioconvert ! audioresample ! autoaudiosink \ dec. ! queue ! videoconvert ! autovideosink \ 2>&1 | grep "pts" Log snippet from a drifting pod:

Q: 4. Inspect node time synchronization

# On each worker node timedatectl status chronyc tracking Nodes missing NTP daemonset reported “System clock not synchronized”.

Q: Resolution

The fix combines three actions: enforce host time sync, eliminate CPU throttling for multimedia workloads, and adjust Docker runtime for real‑time scheduling.

Q: 1. Deploy an NTP daemonset

Apply the official chrony daemonset to all worker nodes so that container clocks stay within 5 ms of the host.

Q: 2. Remove restrictive CPU quotas and enable real‑time runtime

Update the pod spec to request exclusive CPUs and add the --cpu-rt-runtime flag via the runtimeClassName “runc‑rt”. This aligns with Docker’s best practices for multimedia workloads (Docker Engine – real‑time scheduling).

In this article

Table of Contents

Problem Description

The multimodal inference service runs inside Docker containers on a Kubernetes cluster. Each pod receives a live video feed with an accompanying audio track, processes them with an AI model, and re‑encodes the synchronized stream for downstream consumption.

After enabling the Horizontal Pod Autoscaler (HPA) and scaling from 2 to 8 inference pods, operators observed a gradual drift where audio leads video by several seconds. The drift worsens over time, eventually causing noticeable lip‑sync errors and buffer underflow warnings in the playback client.

Typical log excerpts from a drifting pod:


[2026-08-29 12:14:03.112] gst:warning: pipeline.c:1234: Audio/video sync lost: pts not monotonic
[2026-08-29 12:14:03.115] ffmpeg: error: pts not monotonic, resetting timestamps
[2026-08-29 12:14:04.021] docker: warning: CPU quota exceeded, throttling may cause jitter
[2026-08-29 12:14:05.003] kubelet: warning: clock skew detected, offset=+152ms

Impact includes:

Degraded user experience for real‑time streaming (sports, live translation).
Increased retransmission due to audio buffer starvation.
Potential SLA violations for latency‑sensitive pipelines.

Root Cause Analysis

Three inter‑related factors cause the temporal misalignment when pods are scaled:

Container clock drift: Docker containers inherit the host’s clock but, when CPU limits are applied, the cgroup scheduler can pause the container thread, causing the kernel’s monotonic clock to lag behind real time. The Docker Engine documentation on resource limits and CPU throttling notes that throttling “may affect time‑keeping accuracy”. In the production incident, newly created pods ran on nodes without an NTP daemonset, leading to offsets >100 ms (Kubernetes SIG Architecture discussion on clock skew).
CPU throttling under load: The inference containers are CPU‑bound. When the HPA adds pods, the per‑pod CPU quota is reduced (e.g., cpu: "500m"). Docker emits “CPU quota exceeded” warnings, and the kernel’s real‑time scheduler cannot guarantee a steady tick rate. GStreamer timestamps become jittery, as described in the Live sports broadcast service incident.
Network latency spikes during scaling: Adding pods triggers service mesh re‑routing and temporary packet loss. While network latency alone does not cause drift, it amplifies timestamp divergence when combined with the above clock issues (Docker networking latency considerations).

In summary, the drift originates from non‑monotonic presentation timestamps (PTS) generated inside containers whose clocks are unsynchronized and throttled.

Investigation and Debugging

Below is a reproducible debugging workflow that isolates each factor.

1. Verify container clock vs. host clock


# Inside a running pod
kubectl exec -it infer-pod-1 -- date +%s%N
# On the node
ssh node-01 "date +%s%N"

Expected output: difference < 10 ms. In the drifting pods the offset was ~150 ms.

2. Check CPU throttling metrics


# Using cAdvisor metrics exposed by kubelet
curl -s http://localhost:10255/stats/summary | jq '.pods[] | select(.podRef.name=="infer-pod-3") | .cpu.throttlingData'

Typical output for a healthy pod:


{
  "throttledPeriods": 0,
  "throttledTime": 0
}

Drifting pods showed thousands of throttled periods and non‑zero throttledTime.

3. Capture GStreamer pipeline timestamps


gst-launch-1.0 -v filesrc location=sample.mp4 ! decodebin name=dec \
  dec. ! queue ! audioconvert ! audioresample ! autoaudiosink \
  dec. ! queue ! videoconvert ! autovideosink \
  2>&1 | grep "pts"

Log snippet from a drifting pod:


... pts=12345678 (0x00bc614e) dts=12345670 (0x00bc6146) ... 
... pts=12345679 (0x00bc614f) dts=12345671 (0x00bc6147) ... 
... pts=12345680 (0x00bc6150) dts=12345672 (0x00bc6148) ...

Notice non‑monotonic increments when the container is throttled.

4. Inspect node time synchronization


# On each worker node
timedatectl status
chronyc tracking

Nodes missing NTP daemonset reported “System clock not synchronized”.

5. Review Docker run options

The deployment manifest used:


resources:
  limits:
    cpu: "500m"
  requests:
    cpu: "250m"

and omitted any real‑time scheduling flags.

Resolution

The fix combines three actions: enforce host time sync, eliminate CPU throttling for multimedia workloads, and adjust Docker runtime for real‑time scheduling.

1. Deploy an NTP daemonset

Apply the official chrony daemonset to all worker nodes so that container clocks stay within 5 ms of the host.


apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: chrony
spec:
  selector:
    matchLabels:
      name: chrony
  template:
    metadata:
      labels:
        name: chrony
    spec:
      containers:
      - name: chrony
        image: docker.io/library/chrony:latest
        securityContext:
          privileged: true
        volumeMounts:
        - name: host-etc
          mountPath: /etc/chrony
      volumes:
      - name: host-etc
        hostPath:
          path: /etc/chrony

2. Remove restrictive CPU quotas and enable real‑time runtime

Update the pod spec to request exclusive CPUs and add the --cpu-rt-runtime flag via the runtimeClassName “runc‑rt”. This aligns with Docker’s best practices for multimedia workloads (Docker Engine – real‑time scheduling).


apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
spec:
  replicas: 4
  template:
    spec:
      runtimeClassName: runc-rt
      containers:
      - name: infer
        image: myrepo/infer:latest
        resources:
          limits:
            cpu: "2000m"      # allocate whole CPUs
          requests:
            cpu: "2000m"
        securityContext:
          privileged: false
        command: ["sh", "-c"]
        args:
          - |
            exec /usr/local/bin/infer \
              --cpu-rt-runtime=950000   # 95% of a CPU in µs

3. Verify GStreamer timestamps are monotonic

After redeploy, run the same pipeline test. Expected log:


... pts=12345678 (0x00bc614e) dts=12345678 (0x00bc614e) ...
... pts=12345679 (0x00bc614f) dts=12345679 (0x00bc614f) ...

Before / After Comparison

Aspect	Before	After
Container clock offset	~150 ms	<5 ms
CPU throttling	throttledPeriods=8423	throttledPeriods=0
GStreamer log	“Audio/video sync lost: pts not monotonic”	No sync warnings
Observed drift	Audio leads video by 3 s after 10 min	Stable sync <30 ms

Verification

Validate the solution with the following steps:

Deploy the updated manifest and wait for the HPA to scale to the target replica count.
Run a continuous health probe that checks PTS monotonicity:


while true; do
  gst-launch-1.0 -v videotestsrc num-buffers=100 ! \
    timeoverlay ! fakesink 2>&1 | grep "pts not monotonic" && exit 1
  sleep 30
done

Confirm that kubectl logs no longer contain “pts not monotonic” or “CPU quota exceeded”.
Measure end‑to‑end latency with ffprobe on the output stream:


ffprobe -show_entries frame=pts_time -select_streams a -i rtmp://...
ffprobe -show_entries frame=pts_time -select_streams v -i rtmp://...
# Compare timestamps; difference should stay < 0.05 s

Check node time sync status:


kubectl get nodes -o wide
kubectl exec -n kube-system daemonset/chrony -- chronyc tracking

Prevention and Best Practices

Enforce host time synchronization: Run an NTP/Chrony daemonset on all nodes and monitor clock_offset_seconds via Prometheus.
Reserve whole CPUs for multimedia pods: Use cpu: "2000m" limits and avoid fractional quotas that trigger cgroup throttling.
Leverage Docker real‑time runtime: Add --cpu-rt-runtime and --cpu-rt-period via a custom RuntimeClass for any pipeline that depends on precise timestamps.
Monitor jitter metrics: Export container_cpu_cfs_throttled_seconds_total and set alerts for spikes > 10 ms.
Validate pipeline timestamps in CI: Include a test that runs a short GStreamer/FFmpeg job and asserts monotonic PTS.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does the drift appear only after scaling? Scaling introduces new pods on nodes that may lack NTP sync and applies tighter CPU quotas. The combination creates divergent container clocks and throttling‑induced jitter, which accumulate as drift.
Can I keep CPU limits and still avoid drift? Yes, but you must allocate whole CPUs (no fractional limits) and enable real‑time scheduling. Fractional limits cause the kernel to periodically pause the container, breaking monotonic time.
Do I need --privileged to sync clocks? No. The privileged flag is unnecessary; proper time synchronization is achieved with a daemonset and real‑time runtime flags.
How do I detect clock skew before it impacts A/V sync? Export the node metric node_clock_offset_seconds (from the Chrony daemonset) and set an alert threshold of 20 ms.
Will enabling --cpu-rt-runtime increase cost? It reserves CPU cycles for the container, potentially reducing overall node packing density. Evaluate the trade‑off based on SLA requirements for real‑time media.