Track A — The Founding Experiment
The company's first flywheel evidence. The Klindt POC shows our data moves a model (data value); this experiment shows direction is what makes our data special (flywheel value):
Claim under test: at equal hours and equal compute, coverage-directed (quest-driven) fleet data beats passive fleet data for world-model fine-tuning — with the largest deltas on rare/OOD slices.
Prior art predicts success but has never tested it for cross-game world models: WHAMM (1 week curated ≈ beat 7 years passive), Waymo (+31% rare-slice gains from 3% targeted additions), VPT (35:1 leverage) — Technical Evidence §B. A positive result is the pricing evidence for Product 1, the pitch deck, and a publishable first.
Step 1 — Build coverage atlas v1
What. An embedding-space map of the existing corpus that makes "thin" quantifiable (Engine Architecture §1).
How. Embed fixed-length video+action windows (an off-the-shelf video encoder is fine for v1 — the atlas needs relative density, not perfection; upgrading the embedder later re-draws the map without invalidating the method). Fit a density model (kNN-distance or normalizing flow — Waymo used flows) over the embeddings, per title and pooled. Define cells via clustering; rank by density × downstream-relevance (does any eval slice touch this cell?).
Why density, not difficulty. Rareness is a data-support gap and is fixable with data; difficulty is ambiguity and often isn't (Waymo's core distinction). The quest engine targets rareness; difficulty findings route to labeling/model work instead.
Why learnability-weighted, not raw density (vault revision 2026-07-08). Klindt's explicit caveat (2026-06-26 call): reward the first derivative of how much you know, not raw novelty — a pure low-density hunt is his "broken TV" failure mode. And B254 in Guido's wiki warns "undirected diversity actively harms transfer" (in tension with Klindt's "see the pen everywhere" — name it, don't hide it). Atlas v1 cell priority = density gap × learnability signal (did prediction improve after data from this cell? — exactly what the C5 annealing loop measures, so the two calibrate each other).
Output. The ranked thin-cell list that the directed arm will target; atlas snapshot frozen for the experiment.
Step 2 — Pre-register eval slices
What. All evaluation defined before any collection — Antonio's benchmark-first principle, applied at experiment scale.
The slices.
| Slice | Contents | What it tests |
|---|---|---|
| Common | Held-out sessions from high-density cells, per title | Directed data doesn't hurt the bread-and-butter |
| Rare | Held-out sessions from the targeted thin cells | The headline claim |
| OOD | One entire held-out title (from the Phase-1 set, never trained on) | Generalization, the PhyWorld concern |
| Naturalness | Organic-play sessions, statistically profiled | Step 7 — quest-induced distribution shift |
How. Fixed metrics per slice (action-conditioned prediction error, rollout consistency horizon, task-completion where verifiable); success thresholds and analysis plan written down and internally pre-registered (a dated doc in this repo). No metric added after data exists. Contributors appearing in eval slices are excluded from both training arms (annotator-artifact hygiene).
Step 3 — Author the directed arm
How. Quest compiler v1 (Track C) pointed at the thin-cell list: LLM proposes goal + verifier per cell, compiled to the requirements DSL; every quest triaged by the A–E axes (COMMODITY candidates discarded even if the cell is thin); many phrasings per quest and per-contributor caps per quest family (instruction-bias mitigations); bounties per verified coverage unit through the Reward Economy spine.
Guardrail. Minimal scaffolding — specify the what, never the how (the Klindt-POC lesson: execution variance is the value).
Step 4 — Collect both arms concurrently
Design. Same fleet, same weeks — controls for contributor mix, game patches, and meta shifts. Passive arm: organic play, no quest exposure for enrolled sessions. Directed arm: quest-driven. Contributors randomized to arms per window (not self-selected); nobody contributes to both arms in the same window.
Sizing — against fleet reality (revised 2026-07-08). The organic fleet is ~1–4 recording contributors/day (~80–120 gross hours captured in the program's entire history), so this experiment is a dedicated paid-operator program, not fleet collection. Pilot ~50 h/arm ≈ 6 operator-days/arm — feasible now. Full N (planning anchor 200–500 h/arm, bounded by what a Matrix-Game-class fine-tune can detect) ≈ 25–60 operator-days/arm — a staffing + budget decision (alignment review, team decision #2), likely phased behind Phase-2 revenue. Power analysis on the pilot sets final N; if the rare-slice delta needs >500 h/arm to detect, that is itself an effect-size finding — stop and reassess.
Cost sanity. At VPT-benchmark contributor economics (~$20/hr all-in),
collection is roughly $8K–$20K/arm — cheap relative to what the number is
worth. Constraint: bounty awards flow through the existing payout machinery
(manual Altitude rail, €500/contributor pilot caps, ≤€5,000
prestazione-occasionale ceilings) — high-frequency small awards at full N
force the Wise-API phase earlier and bound per-operator earnings; "cost per
verified coverage unit" must include payout-ops overhead. Coverage-unit
bounties also need schema: hour_basis_types is deliberately time-only, so
either a new basis type or routing entirely through quest awards (team
decision #6). Model the arms as a new program_phase + two cohorts — the
phases/policies schema was built for exactly this.
Step 5 — Train matched fine-tunes
How. One open base world model (Matrix-Game 2.0 / MineWorld class — reuse the Klindt POC harness where possible); identical base checkpoint, compute budget, recipe, and stopping rule across arms. Three arms minimum:
- Passive — N hours organic.
- Directed — N hours quest-driven.
- Mixed (50/50) — because production will always blend, and the mixed number is the sellable configuration.
Seeds: ≥2 runs per arm if budget allows — single-seed deltas on small fine-tunes are noise magnets.
Step 6 — Evaluate
How. Pre-registered slices only; report every metric, including the ones that didn't move. Success = directed ≥ passive on common slices AND directed ≫ passive on rare/OOD slices. Attribute honestly: if mixed ≈ directed, say so (it means a cheaper product mix).
Step 7 — Measure quest-induced distribution shift
Why. Instruction-bias literature predicts quest-following play is stylistically narrower than organic play; nobody has measured it for gameplay (Engine Architecture §4). Whatever the answer, publishing it is cheap credibility — and if the shift is large, it changes the product (mandatory organic mixing ratios).
How. Compare directed vs organic session distributions on behavioral statistics (action entropy, trajectory diversity within the same cells, time-to-objective profiles) and on model behavior (does a directed-data-trained model over-predict goal-seeking?). Report alongside the headline result.
Step 8 — Write the two numbers on the wall
Outputs.
- The delta: pp improvement on rare slices at equal hours — Product 1's pricing evidence.
- The unit cost: cost per verified coverage unit — the denominator of every future bounty and contract. VCU definition (unified with the existing quality vocabulary, 2026-07-08): VCU = Anto-kept hours × learnability-weighted coverage-cell weight, expressed as quest requirements + eval-schema fields in the shipped L0/L1/L2 validation stack — not a parallel taxonomy. Since "Anto-kept" has never been operationally defined (a hole the vault has flagged since May), the pre-registered probes become the first concrete "kept" definition.
Plus: the public writeup (the marketing is the science), the internal post-mortem, and the calibrated influence scorer (Track C) validated against this experiment's ground truth.
Reading failure
If directed ≈ passive everywhere: check, in order — (1) were the targeted cells actually thin and eval-relevant (atlas validity)? (2) did quests actually land in the cells (verifier validity)? (3) was N sufficient (power)? Only if all three hold does the result read as "the flywheel is a nice-to-have" → tripwire T4 in the Roadmap: weight shifts to publisher-partnered capture + Product 3, where the fleet has value without the coverage thesis. That answer, learned at pilot scale for five figures, is the cheapest strategic information available to the company.