How blocks travel between the binaries

Since Lesson 1 we've waved at "the Redis Stream" as the thing connecting block-ingest to the indexer. Time to open it. The three binaries are separate processes — they can't call each other's functions. They communicate by passing blocks through a durable, ordered log in Redis. Understanding this backbone explains how the system stays correct when a consumer is slow, restarts, or dies mid-block — which is the deep part you came for.

1 · block-ingest, the producer

One instance per chain. Its job: turn the RPC into an ordered stream of RawBlock payloads. The interesting engineering is all about not being slow or wasteful:

Your first Go channel

The feed hands blocks to the publisher over a channel — Go's typed, thread-safe pipe between concurrent goroutines. It's buffered to the prefetch size:

// pkg/feed/rpc.go — Subscribe
ch := make(chan *types.RawBlock, prefetch)   // buffered channel, capacity = prefetch
// producer goroutine: ch <- block   (send; blocks if buffer full)
// consumer side:      block := <-ch (receive; blocks if buffer empty)

That "blocks if buffer full" is not a bug — it's flow control. If nobody drains the channel, the fetcher naturally pauses. Hold that thought; it connects to backpressure below.

Source: pkg/feed/rpc.go (RPCFeed, Subscribe, next, WithPrefetch/WithCache/WithFinalityLag), cmd/block-ingest/main.go.

2 · The stream: a durable log, not pub/sub

The producer appends each block to a Redis Stream keyed blocks:{chainID} with XADD. The critical mental model: a Redis Stream is an append-only, persistent, ordered log — not fire-and-forget pub/sub.

// pkg/queue/queue.go — Publish
q.rdb.XAdd(ctx, &redis.XAddArgs{
    Stream: "blocks:1",           // the durable log for chain 1
    MaxLen: q.maxLen, Approx: true,  // soft cap (defence-in-depth trim)
    Values: ...,                       // the serialized RawBlock
})

3 · Consumer groups: the indexer's read position

The indexer reads with XREADGROUP as part of a consumer group. A consumer group is Redis's mechanism for "a team of workers sharing a stream, where each entry is handed to exactly one member and the group remembers what's been delivered and acknowledged."

// pkg/queue/queue.go
q.rdb.XGroupCreateMkStream(ctx, streamKey, group, startID)  // once: create group (+ stream)

q.rdb.XReadGroup(ctx, &redis.XReadGroupArgs{
    Group: group, Consumer: name,
    Streams: []string{streamKey, ">"},  // ">" = give me entries never delivered to this group
    Count:   batch,
})

Three properties fall out of this, and they're the heart of the lesson:

① At-least-once delivery (read → process → ack)

An entry delivered to a consumer enters that consumer's PEL (Pending Entries List) — "delivered but not yet acknowledged." The indexer only calls XACK after it has committed the block to Memgraph. So the cycle is:

If the indexer crashes between steps 1 and 3, the entry stays in the PEL — unacked. It will be redelivered. That means a block can be processed more than once: at-least-once, not exactly-once.

② Reclaiming a dead consumer's work

What if a consumer doesn't just pause but dies with entries still in its PEL? Another consumer reclaims them with XAUTOCLAIM, so no block is stranded:

q.rdb.XAutoClaim(ctx, ...)  // take over entries idle in a dead consumer's PEL

③ Backpressure: the producer slows when the consumer falls behind

If block-ingest raced ahead unbounded, it would pile up unread blocks in Redis (memory!) or trigger MAXLEN trimming that silently drops unread blocks. So Publish samples the consumer backlog and blocks before XADD when the consumer is too far behind:

// pkg/queue/queue.go — backpressure
// backlog = max over consumer groups of (lag + pending)
if backlog >= threshold { awaitBackpressure(ctx) }   // pause producer until it drains

The flow-control chain, end to end

Now the buffered channel from §1 clicks into place. SubscribeToStream calls Publish synchronously, so a blocked Publish (waiting on backpressure) stops draining the prefetch channel → the channel buffer fills → the RPC fetcher's ch <- block blocks → fetching pauses. One stalled consumer applies smooth back-pressure all the way back to the RPC calls. No config, no coordination — just blocking primitives composing.

4 · The other Redis (don't confuse them)

Recall from Lesson 1 there are two Redis instances. Keep them straight:

Check yourself

What you can now do

• Explain how the binaries communicate: a durable, ordered Redis Stream — not calls, not pub/sub.
• Describe block-ingest's producer tricks (round-robin, prefetch, 7-day RPC cache, finality lag) and read a buffered Go channel.
• Explain consumer groups, the PEL, at-least-once delivery (read→process→XACK), and reclaim (XAUTOCLAIM).
• Explain backpressure — including the lag+pending vs XLEN deadlock — and how it propagates back to RPC.
• Connect at-least-once delivery to why the write path is idempotent. The two designs are one story.