Technical Evidence

Status — evidence review, July 2026

Condensed from Deliverable 2 of the research pass (ec-workspace/research/world-model-data-flywheel/02-technical-evidence-review.md), which carries full citations. Grades: STRONG (quantified, peer-reviewed or flagship, verified against the primary source), SUGGESTIVE (credible but single-source or indirect), SPECULATIVE (inference). The section most likely to be misquoted in a pitch is §C — read it before claiming anything about robotics.

A. Do real action labels beat latent actions?

The honest answer: labels win on grounding and precision; latent actions win on pretraining breadth per dollar; and no controlled head-to-head exists at world-model scale. The claim "real labels matter" is assembled from converging indirect evidence:

The leverage pattern — labels as catalyst, not corpus

• VPT (OpenAI 2022) — STRONG. 1,962 contractor hours with real K/M logs trained an inverse-dynamics model (90.6% keypress accuracy, 0.97 mouse R²) that pseudo-labeled ~70,000 hours of scraped video — a 35:1 leverage ratio. The 70K scraped hours were worthless for agent training until the 2K labeled hours existed. This is why Quests as a Data Product treats frame-synced actions as IDM-seed material and prices sync precision over volume.
• D2E (ICLR 2026) — STRONG for the pattern. 259 hours of desktop-gameplay demos (31 games, real K/M events) train a "Generalist-IDM" that pseudo-labels 1,000+ hours of online video. The leverage pattern replicates a generation later, on exactly Playroll-shaped data.
• Consequence: value quests by leverage (does this data train a labeler, ground a model, fill a coverage hole?) — never by hours. Also the warning: labeled-hour demand saturates once an IDM is accurate in-distribution. The durable asset is labels in games/genres/input regimes where existing IDMs fail.

The controllability pattern — every playable system needed actions

Every working playable world model used real or agent-generated action labels: GameNGen (RL-agent actions), Oasis (the VPT corpus — it literally reuses VPT's camera quantizer), WHAM (1B+ image–controller pairs), Matrix-Game 2.0 (~800–1,200 action-annotated hours), Hunyuan-GameCraft (scraped 1M+ AAA clips plus a synthetic action-annotated set built specifically to get control precision). — STRONG as a pattern. Meanwhile the no-labels lineage (Genie 1→3), three generations in, still lists "limited action space" as its own first limitation. Precision control is the empirically label-hungry axis.

The counter-evidence — take it seriously

• LAPA (ICLR 2025) — STRONG. Latent-action pretraining on actionless video beats OpenVLA pretrained on ground-truth robot actions (36.8 vs 30.8 real-world average) at >30× less pretraining compute. But: it underperforms on fine-grained motion (grasp precision) and still requires an action-labeled set at fine-tuning time to ground latent actions.
• UniVLA, AdaWorld — STRONG, same shape: latent pretraining is breadth-efficient; action-supervised grounding re-enters at the end; and action information per se (even inferred) is what makes a video model controllable at all.
• NitroGen (NVIDIA, Jan 2026) — STRONG. Gamepad inputs reconstructed from on-screen controller overlays across 40K hours / 1,000+ games (not an IDM — overlay parsing; joystick R² 0.84). Up to 52% relative improvement fine-tuning on held-out games. Input-alignment scarcity erodes for gamepad games; KBM precision and voice-aligned intent resist longest.

The honest pitch is therefore: grounding + precision + coverage — not "latent actions don't work." And the empty quadrant is diversity-with-labels: WHAM had labels without diversity, Genie has diversity without labels, Matrix-Game/GameCraft have both at only ~10³ hours. A many-game, real-input, consented corpus at 10⁵–10⁶ hours has no published equal.

B. Diversity vs volume

• WHAM → WHAMM (Microsoft, 2025) — STRONG. The cleanest published datapoint: WHAM needed ~7 years of passively harvested telemetry; WHAMM reached a better (real-time, 2× resolution) model from one week of deliberately collected, curated pro-tester data. Microsoft's own framing: "more intentional data collection and curation." Caveat: different architectures, not a controlled ablation.
• Waymo Rare Example Mining (ECCV 2022) — STRONG. Mining ~3% of the unlabeled pool by embedding-space rareness improved rare-object detection ~31%. Key conceptual import: rareness ≠ difficulty — rareness is a data-support gap and is fixable with data; difficulty is ambiguity and often isn't. This distinction drives the coverage atlas in Engine Architecture.
• PhyWorld (ICML 2025) — STRONG result, SUGGESTIVE transfer. Scaling a toy-physics video model 30K→3M examples: near-perfect in-distribution, zero OOD improvement. Generalization behaves like retrieval — prescription: fill the combination grid, don't accumulate more of the modal trajectory.
• Cosmos curation (NVIDIA) — STRONG. 20M raw hours → ~10⁸ clips through dedup/filter/taxonomy (~2–8% survival). Production world-model practice is aggressive curation, not attribution machinery.

Net: the flywheel's core bet — directed diversity beats passive volume — is supported everywhere it has been measured, and has never been measured for cross-game world models. Playroll would generate the first direct evidence (Roadmap §founding experiment).

C. Games → robotics, skeptically

Headline: as of mid-2026 there is essentially one documented instance of game data improving a real-robot benchmark, and it is small.

• No major robot foundation model trains on game footage — STRONG, verified. V-JEPA 2's corpus is fully enumerated (zero gameplay); GR00T uses Ego4D-class egocentric + sim + teleop; Cosmos's published taxonomy has no gameplay category; Open X-Embodiment is real robot data only; Physical Intelligence, Figure, Tesla use teleop + real egocentric human video. Revealed preference: labs with 20M-hour pipelines chose not to include gameplay.
• The one direct datapoint — D2E — SUGGESTIVE. Its headline numbers (96.6% LIBERO, 83.3% CANVAS) are simulation benchmarks; it includes one real arm experiment: 70%→80% pick-and-place, single task, n=30, author-reported, unreplicated. Adding pseudo-labeled gameplay actually hurt some long-horizon sim tasks.
• Against — STRONG testimony. Sergey Levine's "Sporks of AGI" argues surrogate data (explicitly including game-style data) structurally cannot substitute for real robot data; game engines are documented to fake contact physics (NVIDIA built Newton because game-grade physics isn't good enough).
• For — STRONG fact, SUGGESTIVE inference. The Genie lineage — trained on gameplay — became Genie 3, which Waymo adapted into its World Model for AV simulation (confirmed, Feb 2026). Game-video priors are load-bearing in one physical-AI-adjacent production system.

Verdict: "gameplay data helps robotics today" — LOW confidence. "Contributes indirectly via world-model pretraining within 3–5 years" — MODERATE-SPECULATIVE. This is precisely the Quests as a Data Product position: robotics is a future high-level option (navigation/locomotion, long-horizon intent, multi-agent coordination); manipulation/force quests are fake-transfer — do not build them. Keep Leg B optionality free by biasing the catalogue toward first-person, physics-heavy, continuous-control titles; spend nothing else.

D. Neural game engines — the customer's state of the art

System	State (mid-2026)	Data dependence
Genie 3 / Project Genie (DeepMind)	Consumer preview; 720p/24fps, minutes-scale consistency; promptable world events didn't survive into product	Undisclosed video; latent actions; weak agent control
Decart Oasis → MirageLSD → Oasis 3	Real-time; pivoted to video-to-video restyling (sidesteps the action-data problem); ~$4B	VPT corpus (open) for Oasis
Microsoft Muse/WHAMM	"Gameplay ideation," not shipping games; real-time Quake II with artifacts	First-party EULA telemetry; then 1 week curated
Matrix-Game 2.0/3.0, MineWorld (open)	Real-time 25fps; long-horizon memory WIP	Hundreds of action-annotated hours — the binding ingredient

Consensus: demo, not engine. No shipped commercial game runs on a neural engine. Builders repeatedly name action-conditioned data as the bottleneck (discounting sources who sell data). Persistent blockers: minutes-scale memory, per-session cost, control fidelity — the last of which is exactly the axis our data addresses.

What this page licenses you to say (and not say)

Safe to claim	Not safe to claim
"Every playable world model to date needed action labels; ours are real, ms-synced, cross-game"	"Latent actions don't work" (LAPA beats ground-truth pretraining on breadth)
"Directed collection beat passive volume in every measured case (WHAMM, Waymo, VPT)"	"Directed collection is proven for world models" (nobody has measured it — we will)
"A small labeled seed unlocks huge scraped corpora at 35:1"	"Buyers need millions of labeled hours" (demand saturates; breadth is what's priced)
"Game-video priors are load-bearing in Waymo's production simulator"	"Game data trains robots" (one thin datapoint, sim-heavy)