Engine Architecture

Status — design proposal, not shipped

Expands Deliverable 3 of the research pass (ec-workspace/research/world-model-data-flywheel/03-flywheel-design-brief.md). The quest execution side (entities, DSL, enforcement, CRM authoring) is already specified in Quests — Technical Design; this page specifies the layer that decides which quests exist, at what bounty, and how we know they worked. The novelty-engine prototype (LLM proposes goal + verifier code, novelty scored against an archive) is the seed of the Quest Compiler below.

The one-sentence version

An eval/attribution layer maintains a live map of where models are weak and data is thin; a quest compiler turns the worst gaps into priced quests; the fleet fills them under QC; curation feeds the corpus; retraining closes the loop — and every component copies a mechanism that already worked somewhere else.

The loop

---

1 · Eval & attribution layer

Why it exists: the flywheel's premise is that knowing what's missing is worth more than collecting more. Waymo quantified it: ~3% targeted additions → +31% on rare slices. The layer answers two questions continuously: where are the models weak? and where is the data thin? — and keeps them distinct, because they have different fixes.

How — a two-tier valuation stack (the key lesson from production practice: frontier labs do not run influence functions over their corpora):

Tier	Mechanism	Cadence	Borrowed from
Slow / ground truth	Annealing ablations: mix a candidate data slice at high weight into a partially-trained model, measure benchmark delta on held-out slices	Per release / per major quest campaign	Llama 3's production data-valuation method
Fast / full stream	Distilled influence scorer: a small model trained to imitate the slow tier's judgments, cheap enough to score every incoming clip; refreshed as the main model evolves	Continuous	MATES (NeurIPS 2024): ~2× selection gains, ~half the FLOPs
Spatial	Coverage atlas: embedding-space density model over the corpus; cell priority = density gap × learnability signal (Klindt's broken-TV caveat: reward the derivative of prediction improvement, not raw novelty — the C5 annealing loop supplies the signal). World-space layer: the realtime-spatial-coverage voxel map is the per-title instantiation, incl. angle-diversity (axis E)	Continuous	Waymo Rare Example Mining — rareness ≠ difficulty; B254 ("undirected diversity actively harms transfer") vs Klindt's see-everything — an open tension the learnability weight arbitrates empirically
Forensic only	Gradient attribution (EK-FAC/TRAK) on small query sets: "which clips drove this failure?"	On demand	Anthropic influence-functions work — never as a routine pass

Design rule: rareness (a data-support gap) gets a quest; difficulty ( ambiguity) gets a model/label fix, not more data. Conflating them is how datasets bloat without models improving (Cruise's anti-goal: stop collecting solved scenarios).

2 · Quest compiler

Why it exists: hand-authoring quests doesn't scale past a catalogue, and the Quests as a Data Product triage (axes A–E, PREMIUM/COMMODITY verdicts) needs a supply of candidates ranked by measured gap value, not intuition.

How:

1. Input: the gap list from layer 1, each gap tagged with its atlas cell, affected eval slices, and estimated leverage.
2. Generation-first gate (Wayve's GAIA pattern): before questing humans, check whether the current world model or a video generator can synthesize the gap. Only human-quest what generation fails at. This keeps quest spend concentrated where human data has monopoly value — and plan for that share to shrink (SIMA 2 trend), which is fine: the durable asset is the coverage intelligence, not the quest volume.
3. LLM proposes goal + verifier code — the existing novelty-engine prototype, pointed at gaps instead of open-ended novelty. The verifier compiles into the quests requirements DSL (Quests — Technical Design §3–5) plus post-hoc checks run server-side against the recording.
4. Triage against the A–E axes — a generated quest still has to pass the buyer-value verdict rule: COMMODITY and fake-transfer candidates are discarded regardless of gap size.
5. Bounty pricing = rareness × leverage. This plugs directly into the Reward Economy principle that a quest's cash reward is tuned to data value — the atlas and scorer make that tuning measurable instead of vibes. Pay per verified coverage unit, never per hour (incentive literature: hourly pay invites idle farming; salient performance bonuses work for effort-responsive tasks, and gameplay is one). No public precedent prices bounties by marginal model improvement — this is a differentiator, and per-contribution influence scores double as a royalty rail (contributor lock-in).
6. Anti-bias machinery — not optional. Instruction-bias findings from NLP predict quest-following play will be stylistically narrower than organic play (speedrun-to-objective, ignore distractors). Mitigations: many phrasings per quest, per-contributor caps per quest family, disjoint contributors between train and eval slices, and a permanent organic:directed mix in every shipped dataset.

3 · Fleet capture

Why client-side intelligence: Tesla's core lesson — the fleet is a distributed queryable sensor network, and the expensive asset is the trigger library (each trigger encodes a known model weakness). Mobileye's lesson — ship features, not pixels (~10 KB/km): compute compact coverage signatures in the overlay; upload full video on quest match or trigger hit, signatures always.

How:

• A versioned trigger library ships with the overlay: coverage predicates (atlas cell membership), verifier hooks, and model-disagreement detectors. Iterate trigger → retrain → remeasure per weakness (Tesla ran ~7 rounds per network).
• Quests ride the existing spine — lobbies, quest_id attribution, requirements JSON, server-side enforcement — exactly as specified in the quests thread. This page adds nothing to that design except the upstream source of quests and the downstream measurement.
• The organic stream is a first-class input, never fully displaced by quests: it is the naturalness baseline (§4), the atlas's density prior, and the source of serendipitous coverage no compiler would propose.
• Capture spec (locked in Phase 0, Roadmap): ms-aligned K/M/controller streams, 1080p60 master video with 720p30 working copies (masters keep the 3D-reconstruction leg alive; fidelity is not the premium axis — WHAM shipped on 300×180@10Hz), mic-only audio (never incoming voice channels), session metadata, per-clip consent reference.

4 · Curation & QC

Why aggressive curation first: Cosmos's production pipeline kept ~2–8% of raw video. Expect similar. Never run valuation machinery on uncurated volume.

How:

• Automated curation: dedup (semantic), quality filters, taxonomy balancing across the game catalogue and atlas cells.
• Fraud & QC — assume adversaries from day one (botted play, replayed sessions, emulator spoofing; there is no public QC literature for gameplay fraud specifically, so we build it): gold quests (some visible, some not), verifier replay against recordings, input-stream statistical fingerprinting (human vs scripted cadence), reputation with graduated trust, and collusion modeling from the crowdsourcing canon.
• Naturalness slice: a monitored eval slice comparing directed vs organic play distributions. Nobody has measured quest-induced distribution shift in gameplay — measuring and publishing it (either way) is cheap credibility and protects the product from quietly teaching world models quest-shaped behavior.
• Consent & provenance ledger — per-clip: contributor consent version, title/EULA whitelist status at capture time, processing lineage. This is a product feature (EU AI Act Art. 53-exposed buyers must survive public disclosure of sources), not compliance overhead. It also implements withdrawal/redaction (Ego4D-grade hygiene) — see Risk & Compliance.

5 · Training & the measured delta

The engine's output is not "a corpus" but a measured claim: this slice, added to this model, moved these eval slices by this much. Internally that powers the founding experiment (Roadmap); commercially it is the product for directed-collection customers, and the per-slice influence scores are the pricing input for everything else.

What each precedent contributed (summary)

Precedent	The mechanism we copy
Tesla data engine	Client-side trigger library; selective upload; trigger→retrain→remeasure loop
Waymo REM	Rareness-by-density gap definition; rareness ≠ difficulty; expect ~30% rare-slice gains from single-digit-% targeted additions
Cruise CLM	Auto-mine current-model errors; retire solved scenarios (anti-bloat)
Wayve GAIA	Generation-first gate; world model as evaluator
Mobileye REM	Signatures-not-pixels bandwidth economics
VPT	Leverage-based valuation; locked-down capture settings
WHAM→WHAMM	Directed curation beats passive volume for world models (n=1, ours would be n=2 and controlled)
SIMA 2	Human share of the loop shrinks; plan the business for it
Ego4D	Scenario-list coverage design; consent/withdrawal as product
MATES + Llama 3	Two-tier valuation: distilled scorer + annealing ground truth
Cosmos	Curation-first; ~2–8% survival expectation
Mercor/Poseidon	Attribution as the payment rail

Where prior art is silent (our open research questions)

1. No public system closes eval → quests → human fleet → measured delta.
2. Attribution for video/world models at production scale is unproven (first sighting: ATHENA, 2026 — watch, don't depend).
3. No quantified directed-vs-organic ablation exists for gameplay data — the founding experiment fills this.
4. Quest-induced distribution shift in gameplay is unmeasured — the naturalness slice fills this.
5. Dynamic bounty pricing tied to marginal model improvement has no public precedent — our rareness × leverage pricing is novel ground.
6. Gameplay-specific fraud (botted/replayed/spoofed sessions) has no dedicated public QC literature — we build detection early and treat it as IP.