Qwen controller manager crash during high‑throughput event spikes

Problem – Qwen Controller Manager Crashes Under High‑Throughput Event Spikes

The controller manager component of the Qwen event‑driven platform intermittently terminates with a SIGSEGV when the system processes a sudden surge of inbound events (e.g., 50 k events/s for a few seconds). The crash propagates to the control plane, causing:

• Loss of in‑flight control messages
• Stalled reconciliation loops for custom resources
• Temporary outage of downstream pipelines
• Memory usage spikes from 1.2 GiB to >4 GiB before termination

Typical log excerpt captured at the moment of failure:

2026-09-23T14:12:07.842Z ERROR controller_manager.go:312] panic: runtime error: invalid memory address or nil pointer dereference
goroutine 27 [running]:
runtime.throw(0x10c3a0c, 0x1)
	/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc0000f3f68 sp=0xc0000f3f40 pc=0x44f1b2
runtime.sigpanic()
	/usr/local/go/src/runtime/sigpanic.go:84 +0x1 fp=0xc0000f3f78 sp=0xc0000f3f68 pc=0x44f5d5
github.com/qwen/controllers/event_controller.go:187 +0x2b5 fp=0xc0000f3fd0 sp=0xc0000f3f78 pc=0x1234ab
github.com/qwen/controllers.(*EventController).processBatch(0xc0000a4000, 0x0, 0x0, 0x0)
	/event_controller.go:254 +0x1c4 fp=0xc0000f4030 sp=0xc0000f3fd0 pc=0x1234f2
...

The crash occurs only when the event ingress rate exceeds a configurable threshold (default maxBatchSize = 5000) and the system is under memory pressure.

Root Cause – Unhandled Concurrency Exceptions and Memory Leak in Batch Processing

Two intertwined defects were identified:

1. Race condition in the shared eventBuffer slice. The controller spawns a goroutine per incoming connection that appends to a global eventBuffer without proper synchronization. Under load, multiple goroutines concurrently resize the slice, leading to a corrupted underlying array. The subsequent read in processBatch dereferences a nil pointer, triggering the panic shown above.
2. Leaked context.CancelFunc objects. Each batch processing call creates a child context with context.WithCancel but never invokes the cancel function when the batch completes successfully. During spikes, thousands of contexts remain alive, retaining references to the entire event payload and inflating the heap.

Both defects stem from the assumption that the event ingestion path is single‑threaded and that batch processing always fails fast, which is false in a high‑throughput, multi‑client environment.

Debug – Investigation Steps

1. Reproduce the Spike in a Staging Cluster

# Generate 60k events over 10 seconds
kubectl run loadgen --image=busybox --restart=Never -- \
  sh -c "i=0; while [ $i -lt 60000 ]; do \
    curl -s -X POST http://controller-manager.qwen.svc:8080/event -d '{\"id\":$i}'; \
    i=$((i+1)); done"

Observed the controller manager pod restart after ~12 seconds.

2. Capture a Heap Profile at the Moment of Failure

kubectl exec -it $(kubectl get pod -l app=controller-manager -o jsonpath='{.items[0].metadata.name}') \
  -- /usr/bin/pprof -http=:6060 http://localhost:6060/debug/pprof/heap

The profile showed a rapid increase in eventBuffer allocations and a large number of live context.CancelFunc objects.

3. Race Detector Run

go test -race ./...

The race detector reported:

WARNING: DATA RACE
Read at 0x00c0000a4000 by goroutine 27:
  github.com/qwen/controllers/event_controller.go:187 +0x2b5

Previous write at 0x00c0000a4000 by goroutine 12:
  github.com/qwen/controllers/event_controller.go:102 +0x1a3

4. Review the Batch Processing Loop

func (c *EventController) processBatch() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel() // <-- missing in production code

    batch := c.eventBuffer[:c.maxBatchSize] // race on c.eventBuffer
    // processing logic ...
}

Solution – Synchronized Buffer and Proper Context Cancellation

Before (v1.4.2)

type EventController struct {
    eventBuffer []Event
    maxBatchSize int
    // no mutex
}

func (c *EventController) ingest(e Event) {
    c.eventBuffer = append(c.eventBuffer, e) // unsynchronized
}

func (c *EventController) processBatch() {
    ctx, cancel := context.WithCancel(context.Background())
    // cancel never called
    batch := c.eventBuffer[:c.maxBatchSize]
    // ...process batch...
}

After (v1.5.0)

type EventController struct {
    eventBuffer []Event
    maxBatchSize int
    mu sync.Mutex               // protects eventBuffer
}

func (c *EventController) ingest(e Event) {
    c.mu.Lock()
    c.eventBuffer = append(c.eventBuffer, e)
    c.mu.Unlock()
}

func (c *EventController) processBatch() {
    c.mu.Lock()
    // copy slice to avoid holding lock during processing
    batchSize := min(len(c.eventBuffer), c.maxBatchSize)
    batch := make([]Event, batchSize)
    copy(batch, c.eventBuffer[:batchSize])
    // shrink original buffer
    c.eventBuffer = c.eventBuffer[batchSize:]
    c.mu.Unlock()

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel() // guarantees release

    // ...process batch using ctx...
}

Key changes:

• Introduced sync.Mutex to serialize access to eventBuffer.
• Copied the batch out of the shared slice before processing, eliminating concurrent reads/writes.
• Added defer cancel() to ensure every context is cancelled, preventing heap retention.
• Adjusted maxBatchSize to a dynamic value based on available memory (optional).

Verify – Confirming the Fix

1. Run the Same Load Generator

kubectl delete pod -l app=controller-manager
# wait for new pod
kubectl logs -f $(kubectl get pod -l app=controller-manager -o jsonpath='{.items[0].metadata.name}')

Expected outcome: No panic; controller manager remains healthy for the entire 60 k event burst.

2. Monitor Memory Usage

kubectl top pod -l app=controller-manager

Memory should stay within 1.5 GiB (baseline) instead of spiking to >4 GiB.

3. Verify No Leaked Contexts

go tool pprof -http=:8080 http://localhost:6060/debug/pprof/heap

Search for context.CancelFunc – count should be near zero after batch completion.

4. Run Race Detector Again

go test -race ./...

All race warnings resolved.

Prevent – Operational Guardrails and Best Practices

• Enable Go race detector in CI. Fail builds on any data race.
• Enforce context cancellation. Wrap any context.WithCancel or WithTimeout in a helper that logs if cancel() is not called.
• Instrument buffer size metrics. Export event_buffer_len and event_buffer_capacity to Prometheus; alert when length approaches capacity.
• Back‑pressure at the ingress layer. Return HTTP 429 when eventBuffer exceeds a high‑water mark (e.g., 80 % of maxBatchSize * 10).
• Periodic heap inspection. Run a sidecar that triggers /debug/pprof/heap every 5 minutes and checks for growth trends.
• Load‑testing policy. Include spike patterns (burst of 10× baseline for < 30 s) in performance test suites.

FAQ – Common Follow‑Up Questions

1. Why does the crash only happen under spikes and not during steady load?
   Because the race condition manifests when multiple ingestion goroutines concurrently resize the shared slice. Under steady load the contention window is small enough that the race rarely triggers.
2. Can I keep the original unsynchronized buffer and just increase maxBatchSize?
   No. Increasing the batch size only amplifies the race window and memory pressure. The underlying concurrency bug must be fixed.
3. Is the mutex a performance bottleneck for high‑throughput ingestion?
   The mutex protects only the append operation, which is O(1). In practice the lock contention remains below 1 ms even at 100 k events/s. If needed, a lock‑free ring buffer can be introduced, but the mutex solution satisfies current SLAs.
4. How do I detect leaked contexts in production without a heap dump?
   Instrument a counter that increments on context.WithCancel creation and decrements in the deferred cancel(). Alert when the counter diverges from zero for more than a configurable interval.
5. Do I need to restart the controller manager after deploying the fix?
   A rolling restart is recommended to clear any corrupted buffers and leaked contexts that may remain in memory from prior crashes.

Related Topic Hub: LLM Systems Troubleshooting Hub