Why every write funnels through one pod

Back in Lesson 4 I told a small lie of omission: "the indexer commits a batch in one transaction." The real architecture is subtler, and it exists because of a production disaster. This lesson is the true machinery — and it retroactively explains two things you've already met: the mysterious "recorder" (L4) and the two cursors (L8).

1 · The disaster that forced the design

A 2026-05-16 stage bootstrap was meant to seed hundreds of thousands of pools and tokens into Memgraph. Validation found ~93% of pools and ~82% of tokens simply missing. The v4 pool seeder?

1 / 600,000
pools written. The rest silently vanished.

2 · The fix: serialize all bulk writes through one pod

OCC aborts require concurrent transactions. So remove the concurrency: route every bulk write through a single writer. Producers stop calling the database directly. Instead they publish a graphwrite.Request to a Redis stream graph-writes:{chainID}, and one leased graph-writer pod per chain applies them serially.

The "one pod" guarantee is a lease — a singleton election. Exactly one graph-writer per chain holds lease:graph-writer:{chainID} (Redis, with a TTL); a second pod blocks at AcquireLease until the first releases or its lease expires. No two writers ever run at once, so OCC conflicts are structurally impossible.

Source: docs/single-writer.md (motivation + topology), pkg/graphwrite/ (Client, Consumer), pkg/statetracker (AcquireLease). The full disaster write-up is in that doc — read it; it's one of the best things in the repo.

3 · The Request envelope — and how atomicity survives

The wire format (pkg/graphwrite/envelope.go) is deliberately simple:

type Request struct {
    Version  uint8       // protocol version; consumer rejects newer
    Source   string      // producer slug: "price_refresher", "indexer", …
    ChainID  uint64
    GraphID  string
    IdemKey  string      // "{source}:{unique id}" — for idempotent replay
    BlockNum uint64
    Cypher   *CypherOp   // exactly one of Cypher…
    Delta    *types.GraphDelta  // …or Delta — never both, never neither
}

A CypherOp can carry many statements, and the consumer applies all of them in a single ExecuteWrite transaction. That's how multi-statement atomicity is preserved even though the producer no longer holds the transaction.

The invariants that keep it safe

4 · This is why there are two cursors (L8 explained)

Now Lesson 8's two-cursor model isn't arbitrary — it's a direct consequence of this topology:

Between publish and consume there's an async window where Memgraph is at block N but the Redis balance cache is still at N−1. That's exactly the drift L8's reconciler heals on restart — the window just widened from "tx-commit duration" to "tx-commit + publish-to-consume latency." Same recovery path, longer fuse.

5 · One writer, one problem: the firehose drowns the cursor

A single FIFO writer does ~58 applies/sec. But enrichment is ~98% of write volume. So the rare, critical source="indexer" cursor-advancing envelopes got stuck behind an ~80k backlog of enrichment writes. The consequence chain is a greatest-hits of this course:

The fix — lanes (GRAPHWRITE_LANE_SPLIT_ENABLED, FORTA-2920): split the stream into a priority state lane (cursor-advancing) and a bulk meta lane (the firehose) — but keep still one pod, one lease, one in-flight Memgraph writer (you must not reintroduce concurrency!). The writer drains the state lane preferentially so the cursor never starves behind enrichment. Each lane carries its own DLQKey and HaltKey (pkg/graphwrite/lane_params.go).

6 · DLQ + halt: failure preserves order

What if a Request keeps failing to apply? The writer must not just skip it — skipping would violate contiguity (blocks applied in order, no gaps) and break the determinism invariant. Instead: a repeatedly-failing entry goes to a dead-letter queue (DLQMaxLen=10_000), and the writer can halt the lane (the HaltKey). A halted writer stops advancing Memgraph's cursor — which (full circle) is exactly what trips L8's watchdog to pause the indexer. The system would rather stop than apply out of order.

7 · What stays direct — and the lint guard

Not everything goes through the writer. Sequential, one-shot writers never collide, so they keep direct ExecuteWrite access:

• pkg/genesis/loader.go — one-shot bootstrap, runs alone before any other writer.
• One-shot operator backfills (cmd/holds-backfill, …) — main() + os.Exit, no loop.

The principle: OCC needs concurrency, so only concurrent fan-out producers must be serialized.

Check yourself

What you can now do

• Explain the OCC-vs-MVCC root cause and why the Memgraph migration forced a single-writer design.
• Describe the topology: producers publish Requests → one leased graph-writer applies serially.
• Connect it back: the "recorder" (L4) buffers into a Request; the two cursors (L8) are publish vs apply.
• Explain lanes (cursor priority without reintroducing concurrency), DLQ + halt (stop rather than reorder), and the lint guard that enforces it all.