Proposal: Real-Time Spatial Coverage

Status — draft for review, not shipped

Owner: Tom · 2026-06-29. The contributor-facing front end of directable, coverage-audited capture. It renders the state-estimation ladder as a live HUD and feeds the offline 3D reconstruction pipeline. An interactive mock lives in the workspace at realtime-coverage-hud/coverage-hud-mock.html.

1. What we're building and why

Let players construct spatial clouds of game worlds as they explore, with real-time feedback — they see what they've mapped, what's still fog, and they get a live signal (and reward) every time they add genuinely new ground.

This is not a cosmetic minimap. It is the contributor-facing front end of directable, coverage-audited capture — the thing that turns "a player wandered around for 25 minutes" into "a player deliberately covered the parts of the world we don't yet have." That directability is the EC moat, and a live coverage signal is the most natural way to make a human do it without it feeling like task-work.

It pays off in three places at once:

• Data quality / yield. Coverage and viewpoint-diversity are the difference between footage and reconstructable geometry. Surfacing them live lets us steer capture toward the gaps instead of discovering them after upload.
• Contributor UX / retention. Fog-of-war reveal is intrinsically motivating. "Reveal the map, earn Season EXP for new ground" is a loop players already get.
• Failure-driven capture. A coverage objective ("map the unobserved regions of region X from ≥3 angles") slots straight into the existing capture-spec schema as a directable spec.

2. The core insight: split it in two

The single most important decision is to not compute a metrically perfect dense reconstruction live on the player's machine while the game is running. Split the problem by difficulty:

Layer	Runs	Job	Fidelity needed
Real-time coverage proxy	On-device, during play	Tell the player where they've been and where the gaps are	Coarse. A voxel occupancy + viewpoint map is enough
Precise spatial cloud	Async / server-side, after the session	Build the sellable point cloud / mesh / splat	Full. This is where metric accuracy lives

The realtime layer only needs to answer "has this region been observed, and from how many angles?" — a cheap question. The expensive question (where exactly is every surface) is answered offline by the existing reconstruction pipeline (align → fuse → voxel/TSDF → mesh; see 3D representations concepts). The recorded session (MP4 + CSV inputs + JSON metadata, per pure-v2) is the same artifact either way; the realtime proxy just rides on top of it.

This split is what makes "realtime" tractable on a machine whose GPU is already rendering the game and running NVENC.

3. Where the 3D comes from — pose is the load-bearing input

Coverage is really just "where has the camera been and what has its frustum seen, with line-of-sight." Everything keys off camera pose, so the pose source decides both accuracy and which titles we can support. This maps directly onto the state-estimation ladder — we are not inventing a new spine, we are putting a live HUD on the one already documented.

Tier 0 — engine truth (the accurate anchor). For CS2 we already extract exact per-tick camera and entity world-state from the .dem. For realtime, two viable paths:

• Demo-tail: parse the demo file as the engine writes it (a few hundred ms latency — fine for a coverage HUD).
• Game State Integration (GSI): CS2 GSI emits the local player's position and forward direction live over localhost HTTP. Critical limitation to design around: while playing (not spectating), GSI exposes only the local player's state — good enough for our own camera pose and therefore ego-coverage, but it does not give other players' positions live. Full multi-entity world-state stays a post-game demo-parse job.

Tier 0 gives metric, drift-free coverage and is the calibration yardstick.

Tier 1 — input-fused visual estimation (any mouse-look game). For titles with no replay API, estimate pose from what we always capture: control inputs (mouse dx/dy ≈ camera rotation, WASD ≈ translation) + off-the-shelf monocular metric depth (UniDepth V2 / Depth Anything V2). Because CS2 gives ground truth, we calibrate the estimator against engine truth and ship estimated coverage with a bounded error budget — surfaced honestly in the HUD as a confidence state, never hidden.

4. The real-time coverage algorithm

Lightweight, runs every keyframe (not every rendered frame):

1. Sparse voxel hash. Discretize the world into voxels (~0.5–1 m). A hash map keeps it sparse — only observed space is stored. Same voxel-hashing idea the offline TSDF path uses, just coarser and occupancy-only.
2. Frustum carving with line-of-sight. Each keyframe, splat the view frustum (or depth points, in Tier 1) into voxels. A voxel counts as observed only if within range, within FOV, and with clear line-of-sight (walls occlude).
3. Observation accounting — count and angle. Per voxel store an observation count and a bitmask of viewing-angle buckets. A surface seen once from one angle is nearly useless for reconstruction; seen from 3+ angles it's reconstructable. The HUD's "% seen from 3+ angles" is what predicts yield.
4. Frontier detection. Frontier = observed voxels adjacent to unobserved-but- reachable ones — classic frontier-based exploration. This drives the on-screen guidance.
5. Guidance (the directable layer). Nearest-frontier steering is the simple version; next-best-view scoring (rank gaps by expected coverage gain) is how a capture spec points a contributor at buyer-relevant gaps.

5. Performance budget — the real constraint

The game owns the GPU and NVENC is already encoding (pure-v2: Windows + NVIDIA, capture + encoder as the two failure points). A heavy depth net + dense SLAM running concurrently is the thing most likely to hurt frame rate and break capture. Mitigations, in priority order:

• Keyframe, don't per-frame (~2–5 Hz is smooth and cuts compute 10–20×).
• Coarse proxy live, precise async (§2).
• Tier 0 is nearly free — just frustum carving, no neural net. CS2 and other replay-API titles get the full live HUD at trivial cost. Ship Tier 0 first.
• Tier 1 is the expensive tier — small/distilled depth model, reduced resolution, lower depth rate, or a second GPU for serious rigs.
• Async reconstruction handoff — the live session only flags "good coverage of X" into telemetry; the cloud build happens on our infra from the uploaded triplet.

6. Making it beautiful and effective — UX principles

Built to the canonical Playroll launcher tokens (#131312 surface; lime #f4ff7b for primary emphasis only, observing lime-restraint; green #41d67a for live recorder state and the frontier; SF Mono labels).

1. Fog-of-war, not a dashboard. The map is the feedback. Unmapped space is literally dark; exploring reveals it.
2. Reward novelty in the moment. New ground fires a toast and ticks the "New cells · Season EXP" counter — tying to the agreed economy: passive recording earns EXP, EXP feeds the Season track. Coverage is the most honest possible EXP source.
3. Quality is visible. Two bars: raw coverage and "seen from 3+ angles" — teaching that re-observing from new angles is what reconstruction needs.
4. Guidance without coercion. A single calm "unmapped ground" chip + arrow. Directable, never a military objective.
5. Honesty about confidence. The pose-source tier is shown; estimated titles display a bounded-confidence bar. On-brand: transparent data-controller, no dark patterns.

Copy discipline applied (per playroll-ui/design-md-playroll.md): no copyrighted title named in player-facing surfaces, no currency/limit framing, no military/tactical language, progress framed as mapped space + EXP.

7. What we need to build

#	Component	Where	Notes
1	Live pose tap — Tier 0	recorder (playroll-cpp)	CS2 GSI listener and/or demo-tail reader → camera pose stream
2	Coverage engine	recorder / native	Sparse voxel hash, frustum carve + LOS, count+angle accounting, frontier extraction. Pure compute
3	Overlay renderer	playroll-ui	The HUD: map, coverage bars, novelty toast, guidance, tier/confidence chip
4	Telemetry	OTEL pipeline	Per-session coverage %, multi-angle %, frontier-closure — feeds quality lens + verified-hours dashboard
5	Reconstruction handoff	infra → 3D recon	Mark strong-coverage sessions; trigger offline align/fuse on the uploaded triplet
6	Tier 1 estimator (phase 2)	research → native	Input-fused visual odometry + monocular depth, calibrated vs CS2
7	Coverage capture-spec type (phase 2)	capture-spec schema	A spec defining a coverage objective + NBV scoring

8. Recommended validation path (spikes, in order)

See the companion spike: Tier-0 GSI / demo-tail realtime pose feasibility.

1. Tier-0 realtime pose feasibility — GSI listener; confirm usable local-player pose at adequate rate while playing; demo-tail fallback. Go/no-go for the wedge.
2. Coverage-correctness vs ground truth — run the coverage engine on a recorded CS2 session; check live voxel coverage against the full demo-parsed trajectory.
3. Perf-under-load — frame-time + capture integrity with overlay + engine running during a real recording on representative hardware.
4. Pioneer UX test (Phil, with Pioneers) — does fog-of-war + EXP pull people toward gaps? Watch for the "didn't notice the signal" class of issue.
5. Tier-1 pilot (phase 2) — estimator on one engine-less title; report calibrated error vs CS2.

9. Risks & open questions

• GSI local-player-only limit — realtime = ego-coverage; multi-entity stays offline. Designed-around, not a blocker.
• GPU contention on Tier 1 — the depth model is the budget risk; keep Tier 1 strictly phase 2.
• Scale ambiguity (Tier 1 monocular) — fine for relative coverage; needs engine scale or known FOV for metric. Bounded-error framing handles the honesty.
• Teleports / respawns / map changes break naive odometry — need relocalization + cut detection (already in the ladder's failure surface). Tier 0 sidesteps it.
• Anti-cheat optics — a localhost GSI listener and post-hoc demo parsing are sanctioned, non-injecting paths. No live memory reading or overlay injection into the game process.
• Open: voxel size vs legibility vs memory; how aggressively guidance should steer; whether coverage EXP needs an anti-farm cap (likely yes — re-observing the same angle shouldn't pay).

10. Bottom line

Realtime is doable, and the cheap, accurate version (Tier 0, engine-truth coverage on replay-API titles like CS2) is a small, low-risk build we can ship as the wedge, with heavy reconstruction staying offline where it belongs. It lands on existing strategy: the visible front end of directable, coverage-audited capture; it feeds the Season-EXP economy honestly; and it slots into the capture-spec / failure-driven loop. The general version (Tier 1, any game) is a clean phase-2 follow-on reusing the same HUD and the same calibrate-on-CS2 methodology already in the ladder.