Lesson 29 · Chain-Reference Quality Harness · New Subsystem

How the system audits itself

Re-checking the derived graph against on-chain truth — and routing durable drift to action. ~14 min.

Builds on: L23 · L26 · L24 Anchor: balance conservation, market state New: gap / excess / drift New: streak → ticket / heal

Everything you've studied builds the graph from derived signals — RPC probes, heuristics, decoders. Any of them can be wrong: a stale balance, a missed admin, an edge that no longer reflects the chain. So the system does what a careful engineer would — it re-reads the chain and checks its own work. That's the chain-reference harness, the home of the "freshness verifier" L26 kept pointing at, and the closed loop that makes discovery's honest gaps actionable.

Your anchor: conservation laws you can check on-chain
You know facts that must hold if the graph is right: the sum of every HOLDS edge for a token should equal its totalSupply(); an Aave market's collateral in the graph should match the market's actual on-chain state; a Safe's owners in the graph should match getOwners() today. Each is a re-derivable invariant. The harness ships one Verifier per class that re-computes the truth from chain and compares.
Two different "is it right?" axes — don't conflate them
Parity (L23/L28) asks "does Go match the Python batch pipeline?" — same inputs, same output. Chainref (this lesson) asks "does the graph match the blockchain?" — the deeper question, since both Go and Python could agree and still both be stale. Parity guards the port; chainref guards reality. A node can pass one and fail the other.

1 · The Verifier: a set comparison against truth

The interface (verifier.go) is strikingly small. Each verifier supplies the chain truth-set, the stored set, and a per-instance recheck:

type Verifier interface {
    Class() string                                          // e.g. "HOLDS", "LENDING_COLLATERAL"
    EnumerateOnChain(ctx, block) ([]Ref, error)             // C — the canonical truth set
    EnumerateGraph(ctx) ([]Ref, error)                      // G — what we currently store
    VerifyInstance(ctx, ref, block) (pass bool, []Diff)     // recheck one ref's VALUES
}

From C and G the runner derives three kinds of finding — and that taxonomy is the whole mental model:

gap · C \ G

On chain, missing from the graph. We failed to discover something. This is the machine form of L19/L25's discovery_gap.

excess · G \ C

In the graph, gone on chain. A stale edge/node we should prune (e.g. an owner removed, a market closed).

drift · C ∩ G

Present both sides, but the stored value ≠ chain value beyond tolerance. The right shape, the wrong number.

The drift tolerance — two bounds, both required
A drift is flagged only when AbsErr > AbsTolerance AND RelErr > RelTolerance. Requiring both means a tiny absolute wiggle on a huge balance, or a large relative swing on a dust amount, won't trip a finding — only a mismatch that's big in both senses counts. Same "distinguish a real miss from rounding noise" discipline as L19's ×1.001 cap slack and L16's minDelta.

2 · ~30 verifiers, three jobs

The verify_*.go files (one per class) sort into three jobs — and you've met all three before:

JobAsksExamples
Coverage"did we find everything?" (gaps)verify_at_risk_coverage, verify_multisig_coverage, verify_oracle_discovery_coverage
Correctness"is what we stored right?" (drift)verify_balance_conservation, verify_lending_aave / compound_v3 / morpho / euler_v2, verify_admin
Freshness"is it recent enough?"verify_lending_freshness, the NavlinkFreshnessVerifier from L26
This closes loops from earlier lessons
Coverage verifiers are the audit side of discovery (L24–L28): a verify_multisig_coverage gap means a Safe whose owners the probes (L25) missed. The NavlinkFreshnessVerifier is the safety net L26 said replaced the whitelist gate — here's where it lives. And chainref's per-instance VALUE checks are the on-chain cousin of L23's in-process CheckInvariants: same instinct, different reference (the chain vs. internal logic).

3 · The actuator loop — what makes it more than a test

A unit test fails loudly and stops. A production graph drifts constantly and partially — you can't page a human on every transient blip. So findings flow through a streak-based actuator:

verifieremits finding
OpenSearchupsert + streak++
streak ≥ N?FetchPromotable
Linear ticketauto-filed work
or
healerauto-fix
The constraint you've seen everywhere: RPC budget
Re-reading the entire graph from chain every cycle is impossible at 1.5M nodes — same tension as L24/L25. Each verifier carries a Config with sampling/budget knobs, and the runner verifies a sampled subset per cycle (runner_sampling.go). Coverage trends over many cycles rather than a full sweep each time — quality is measured statistically, not exhaustively.

4 · Where the output surfaces

Each run writes a ClassReport (coverage %, gap/excess/drift counts) persisted as a :QualityReport node — read by the admin-panel /quality page (L17). So an operator sees, per class, "HOLDS coverage 99.2%, 14 drifts, 3 gaps" at a glance, with the durable findings behind it in OpenSearch and the escalated ones linked to Linear. The harness turns "is our graph any good?" from a vibe into a dashboard.

Check yourself

1. What question does the chainref harness answer that the parity harness (L23) does not?
2. A verifier exposes EnumerateOnChain (C) and EnumerateGraph (G). What is a "gap" finding?
3. A "drift" finding is flagged only when AbsErr > AbsTolerance AND RelErr > RelTolerance. Requiring both bounds means…
4. The verify_balance_conservation verifier checks which on-chain invariant?
5. Why are findings tracked with a streak counter rather than acted on immediately?
6. A finding's streak reaches the promotion threshold. What happens?
7. Besides ticketing, what's the other outcome a finding can have?
8. Why does the harness sample a subset per cycle rather than re-verify the whole graph?
↳ Ask your teacher
Try: "Show me verify_balance_conservation's EnumerateOnChain." · "What's the streak threshold for promotion, and is it per-class?" · "How does a healer decide it can auto-fix vs. must escalate?" · "What's PartialEnumerator and the excess-suppression rule about?" · "How does the /quality admin page read ClassReport?"

What you can now do

The self-correction loop is now visible
Discovery (L24–L28) builds the graph; chainref audits it against the chain, escalates durable drift to Linear, and auto-heals what it can. Together they're a closed control loop: derive → measure against truth → correct. That loop is why a graph assembled from fuzzy heuristics can still be trusted enough to price billions in risk.

Grounded in: pkg/quality/chainref/verifier.go (Verifier interface — EnumerateOnChain C / EnumerateGraph G / VerifyInstance, Kind gap/excess/drift, Tolerance Abs∧Rel, Config sampling, ClassReport:QualityReport), findings.go (OpenSearch FindingDoc + streak upsert/reset, FetchPromotable streak≥threshold), linear_promoter.go (LinearPromoter + linear_issue_id write-back), runner_healer.go/healers/, runner_sampling.go, the ~30 verify_*.go classes. Verify against source — the code is the truth.