Kafka Video/LiDAR Timestamp Drift After Intermittent Network On Edge Node

Q: Root Cause Analysis 1. Timestamp Types and Out‑of‑Order Detection

Kafka stores a message timestamp that can be either CreateTime (producer supplied) or LogAppendTime (broker assigned). The broker validates monotonicity per partition (see Apache Kafka Documentation – Message Timestamp). When a producer reconnects after a network outage, batches that were buffered during the outage are flushed with their original CreateTime. If the system clock on the edge node has drifted or the producer’s linger.ms exceeds the reconnection interval, those timestamps become older than the last committed record, triggering the “out of order” warning.

Q: 2. Intermittent Network + Producer Buffering

The producer configuration defaults (linger.ms=5, batch.size=16384) cause records to be accumulated before a send. When the Wi‑Fi link drops, the delivery.timeout.ms timer keeps running; once connectivity returns the buffered batch is sent in one large request. If the batch contains video frames captured over several seconds, their CreateTime values are far behind the current wall‑clock, producing a timestamp gap.

Q: Investigation and Debugging Steps Collect broker and producer logs around the outage. journalctl -u kafka -f | grep -i "out of order" Expected output: 2023-10-12 14:03:21,456 WARN org.apache.kafka.clients.producer.internals.RecordAccumulator - Record is out of order: current timestamp 1698451234567 is less than last timestamp 1698451240123 Verify timestamp type configuration. # Broker config grep log.message.timestamp.type /etc/kafka/server.properties # Producer config grep timestamp.type /etc/kafka/producer.properties Typical output: log.message.timestamp.type=CreateTime timestamp.type=CreateTime Check system clock drift. timedatectl status ntpstat If timedatectl shows System clock synchronized: no, NTP is not keeping pace. Capture a packet trace during reconnection. sudo tcpdump -i wlan0 -w /tmp/kafka-reconnect.pcap port 9092 Inspect the capture for large batches sent immediately after the SYN/ACK handshake. Inspect consumer lag and watermarking. (Kafka Streams) kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group inference-pipeline Look for partitions whose CURRENT-OFFSET jumps backward. Reproduce the scenario in a controlled test. Disable the network for 10 s, then restore, and observe timestamp ordering in the topic using kafka-console-consumer with --property print.timestamp=true. kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic video-frames --from-beginning --property print.timestamp=true | head -n 20 Resolution – Aligning Multimodal Timestamps 1. Enforce Monotonic Timestamps at the Producer

Implement a custom TimestampExtractor that guarantees non‑decreasing timestamps per partition. The extractor can fall back to the last emitted timestamp if the sensor clock moves backward.

Q: 2. Switch Broker to LogAppendTime for Sensor Topics

Set log.message.timestamp.type=LogAppendTime for video-frames and lidar-points topics. This removes reliance on the producer’s clock and guarantees that the broker assigns a monotonically increasing timestamp at append time.

Q: 4. Stabilize System Clock

Deploy a lightweight NTP client (e.g., chrony) and monitor drift.

In this article

Table of Contents

Problem – Timestamp Drift Between Video and LiDAR Streams on an Edge Kafka Node

In a low‑power edge deployment a local Kafka broker aggregates two high‑rate topics:

video‑frames – 30 fps H.264 payloads, timestamps generated by the camera firmware.
lidar‑points – 10 Hz point‑cloud packets, timestamps generated by the LiDAR driver.

During intermittent Wi‑Fi outages the following symptoms were observed:

Video frames appear ahead of LiDAR points by several seconds when consumed by the inference pipeline.

Kafka logs contain warnings such as:

WARN org.apache.kafka.clients.producer.internals.RecordAccumulator - Record is out of order: current timestamp 1698451234567 is less than last timestamp 1698451240123

Consumers occasionally throw OffsetOutOfRangeException because fetched records have timestamps outside the current log range.
Real‑time inference latency spikes, and downstream sensor fusion modules report mis‑aligned timestamps.

Root Cause Analysis

1. Timestamp Types and Out‑of‑Order Detection

Kafka stores a message timestamp that can be either CreateTime (producer supplied) or LogAppendTime (broker assigned). The broker validates monotonicity per partition (see Apache Kafka Documentation – Message Timestamp). When a producer reconnects after a network outage, batches that were buffered during the outage are flushed with their original CreateTime. If the system clock on the edge node has drifted or the producer’s linger.ms exceeds the reconnection interval, those timestamps become older than the last committed record, triggering the “out of order” warning.

2. Intermittent Network + Producer Buffering

The producer configuration defaults (linger.ms=5, batch.size=16384) cause records to be accumulated before a send. When the Wi‑Fi link drops, the delivery.timeout.ms timer keeps running; once connectivity returns the buffered batch is sent in one large request. If the batch contains video frames captured over several seconds, their CreateTime values are far behind the current wall‑clock, producing a timestamp gap.

3. Unsynchronized System Clock

Many ARM‑based edge devices run an unsynchronized RTC. In one incident the node’s NTP daemon lagged by >5 s, so LogAppendTime differed from the sensor‑embedded timestamps. When the broker elected a new leader after a partition reassignment, the follower replayed records with the stale LogAppendTime, interleaving them with fresh producer records and breaking multimodal alignment (see the real incident description in the evidence package).

4. CPU Starvation and GC Pauses

High‑throughput video ingestion saturates the limited CPU. GC pauses (>200 ms) delay the producer’s internal clock, causing the next batch to be stamped with a timestamp that appears earlier than the previous batch’s last record. This manifests as the same “Record is out of order” warning.

Investigation and Debugging Steps

Collect broker and producer logs around the outage.

journalctl -u kafka -f | grep -i "out of order"

Expected output:

2023-10-12 14:03:21,456 WARN org.apache.kafka.clients.producer.internals.RecordAccumulator - Record is out of order: current timestamp 1698451234567 is less than last timestamp 1698451240123

Verify timestamp type configuration.

# Broker config
grep log.message.timestamp.type /etc/kafka/server.properties
# Producer config
grep timestamp.type /etc/kafka/producer.properties

Typical output:

log.message.timestamp.type=CreateTime
timestamp.type=CreateTime

Check system clock drift.
```
timedatectl status
ntpstat
```
If timedatectl shows System clock synchronized: no, NTP is not keeping pace.
Capture a packet trace during reconnection.
```
sudo tcpdump -i wlan0 -w /tmp/kafka-reconnect.pcap port 9092
```
Inspect the capture for large batches sent immediately after the SYN/ACK handshake.
Inspect consumer lag and watermarking. (Kafka Streams)
```
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group inference-pipeline
```
Look for partitions whose CURRENT-OFFSET jumps backward.
Reproduce the scenario in a controlled test. Disable the network for 10 s, then restore, and observe timestamp ordering in the topic using kafka-console-consumer with --property print.timestamp=true.
```
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic video-frames --from-beginning --property print.timestamp=true | head -n 20
```

Resolution – Aligning Multimodal Timestamps

1. Enforce Monotonic Timestamps at the Producer

Implement a custom TimestampExtractor that guarantees non‑decreasing timestamps per partition. The extractor can fall back to the last emitted timestamp if the sensor clock moves backward.

public class MonotonicTimestampExtractor implements TimestampExtractor {
    private volatile long lastTimestamp = -1L;

    @Override
    public long extract(ConsumerRecord<Object, Object> record, long partitionTime) {
        long sensorTs = ((SensorRecord) record.value()).getEmbeddedTimestamp();
        long ts = Math.max(sensorTs, lastTimestamp + 1);
        lastTimestamp = ts;
        return ts;
    }
}

2. Switch Broker to `LogAppendTime` for Sensor Topics

Set log.message.timestamp.type=LogAppendTime for video-frames and lidar-points topics. This removes reliance on the producer’s clock and guarantees that the broker assigns a monotonically increasing timestamp at append time.

Before After

Before	After
`# topic creation (default) kafka-topics.sh --create --topic video-frames --partitions 3 --replication-factor 2`	`# topic creation with LogAppendTime kafka-topics.sh --create --topic video-frames \ --partitions 3 --replication-factor 2 \ --config log.message.timestamp.type=LogAppendTime`

# topic creation (default)
kafka-topics.sh --create --topic video-frames --partitions 3 --replication-factor 2

# topic creation with LogAppendTime
kafka-topics.sh --create --topic video-frames \
  --partitions 3 --replication-factor 2 \
  --config log.message.timestamp.type=LogAppendTime

3. Tighten Producer Buffering Parameters

Reduce the window in which out‑of‑order timestamps can accumulate.

# producer.properties
linger.ms=0
batch.size=32768
delivery.timeout.ms=30000
max.in.flight.requests.per.connection=1
enable.idempotence=true

Setting max.in.flight.requests.per.connection=1 forces strict ordering, while enable.idempotence=true protects against duplicates during reconnection.

4. Stabilize System Clock

Deploy a lightweight NTP client (e.g., chrony) and monitor drift.

# /etc/chrony/chrony.conf
pool pool.ntp.org iburst
maxdelay 0.1

5. Guard Against GC‑Induced Pauses

Use the G1GC with tuned pause targets, or move the video producer to a separate JVM with a larger heap.

# JVM options for video producer
-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -Xmx512m

Verification – Confirming Alignment

Consume both topics with timestamps printed.

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic video-frames --from-beginning \
  --property print.timestamp=true | grep -E "timestamp|frame_id" | tail -n 10

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic lidar-points --from-beginning \
  --property print.timestamp=true | grep -E "timestamp|scan_id" | tail -n 10

Verify that timestamps are monotonically increasing and that the delta between corresponding video and LiDAR records stays within the expected ±100 ms window.

Check Kafka Streams watermarks.
```
curl -s http://localhost:8080/streams/metrics | jq '.watermark'
```
Watermarks should advance without sudden regressions.

Monitor broker metrics.

# JMX query for out‑of‑order timestamps
jmxterm -n -l localhost:9999 -v silent \
  -i <<EOF
get kafka.log:type=Log,name=NumOutOfOrderTimestamps,topic=video-frames,partition=*
quit
EOF

The metric should stay at 0 after the fix.

Run an end‑to‑end inference test. Feed a known synchronized video/LiDAR capture and assert that the fusion module reports timestampDiff < 30 ms.

Prevention – Operational Guardrails

Enforce timestamp type policy. Require LogAppendTime for all sensor streams via a cluster‑wide topic.creation.enable policy.

Alert on out‑of‑order warnings. Create a Prometheus rule:

- alert: KafkaOutOfOrderTimestamp
  expr: sum by (topic) (kafka_log_num_out_of_order_timestamps) > 0
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "Out‑of‑order timestamps detected on {{ $labels.topic }}"
    description: "Check producer buffering and clock sync."

Network health checks. Deploy a watchdog that restarts the producer if the network interface flaps more than three times in a minute.
Clock drift monitoring. Use chrony tracking and alert when RMS offset > 100 ms.
Resource reservation. Pin the video producer to a dedicated CPU core and set cpu.cfs_quota_us to avoid starvation.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the “Record is out of order” warning appear only after a Wi‑Fi outage?
Because the producer buffers records during the outage; when the connection returns the buffered batch carries old CreateTime values that are earlier than the last timestamp already written to the partition.
Can I keep CreateTime and still avoid drift?
Yes, but you must guarantee monotonicity yourself, e.g., by using a custom TimestampExtractor that never returns a timestamp lower than the previous one.
What is the impact of switching to LogAppendTime on downstream consumers?
Consumers receive broker‑assigned timestamps, which are always increasing. If downstream logic relies on sensor‑embedded timestamps, embed the original sensor time in the payload and use it for domain‑specific alignment.
Is max.in.flight.requests.per.connection=1 required?
It eliminates reordering caused by retries. When combined with enable.idempotence=true it provides exactly‑once semantics without sacrificing ordering.
How do I differentiate between clock drift and genuine sensor timestamp gaps?
Correlate the sensor‑embedded timestamp (payload field) with the broker timestamp. If the broker timestamp jumps forward while the embedded timestamp stays continuous, the issue is clock drift; otherwise it is a sensor‑side gap.