The last hop: alert → human

The rule engine (L12) emits an AlertEvent. The alert-processor (cmd/alert-processor, pkg/alertprocessor) is the final hop: it persists that alert and delivers it to a human — reliably, and exactly once even when the machinery retries. The delightful thing is you'll recognise almost every mechanism. The system has a small set of patterns and reuses them; this lesson is the proof.

1 · The pipeline (you've seen its shape before)

Alerts travel the same way blocks do: a Redis stream + consumer group, consumed at-least-once with an XACK after success — exactly Lesson 7's backbone, reused for the alert path:

// pkg/alertprocessor/processor.go — Start
ap.rdb.XGroupCreateMkStream(ctx, cfg.AlertStream, cfg.AlertConsumerGroup, "0")
// tolerate BUSYGROUP (group already exists) — idempotent setup

Worker-pool fan-out (a clean Go concurrency pattern)

One readLoop goroutine pulls messages and feeds a channel; a pool of worker goroutines drains it concurrently:

// readLoop → msgCh → N workers
msgCh := make(chan redis.XMessage)
go ap.readLoop(ctx, msgCh)            // producer: XREADGROUP → channel
for i := 0; i < numWorkers; i++ {
    go ap.worker(ctx, i, msgCh)        // consumers: process concurrently
}

This is the canonical Go "fan-out": one reader, many workers sharing a channel. It parallelises delivery (notifications can be slow) while the stream's consumer group guarantees each message is handled once.

2 · processMessage — four steps

Each worker runs the same four steps (processor.go processMessage):

3 · Exactly-once delivery from at-least-once transport ⭐

Here's the elegant part. The stream is at-least-once — a crash before XACK means the alert is redelivered (L7). Naïvely that double-notifies the customer. The fix is the same idempotency move you saw in Lesson 4, now applied to alerts:

4 · A masterclass in fail-loud ops ⭐

This subsystem contains one of the best defensive-ops decisions in the repo, and it's worth studying (cmd/alert-processor/main.go ensureAlertIndexOrFatal). On startup it ensures the OpenSearch index mapping — and if that fails, it deliberately FATAL-crashes the pod rather than warning and continuing. Why?

5 · The product loop, finally closed 🔁

You can now trace a single fact from chain to human with no gaps:

on-chain event            (L1)
  → decoded                 (L3)
  → filtered + written      (L4)   ← monitored set, atomic commit
  → enriched + discovered   (L5)
  → risk fields computed    (L6·L13)   ← max / noisy-OR / HHI
  → rule condition met      (L12)  ← firing state machine emits AlertEvent
  → AlertStream             (L7 idiom)
  → persisted (dedup) +
    delivered to a human     (L15)   ← exactly-once, fail-loud

That is the entire system, end to end. Every box you can now open.

Check yourself

What you can now do

• Trace an alert from the rule-engine through the Redis AlertStream to delivery — and recognise it as L7's idiom reused.
• Explain how at-least-once transport + dedup-by-message-ID yields effectively-once alerting.
• Explain persist-then-deliver and the Delivered flag for no-loss, retryable delivery.
• Articulate the fail-loud-on-misconfig decision and connect it to L8/L9's "stop, don't corrupt."
• Read the worker-pool fan-out Go pattern.