Pipeline Output Data Products

Plain-language summary. Everywhere else in this section describes the database — the rows the app reads and writes. This page describes something different: what our processing pipelines turn raw recordings into. A player uploads a raw capture (video, the input log, a transcript); a pipeline processes it and emits a data product — clean, labelled, timestamped clips with a sheet of metadata wrapped around them. That data product is the thing we sell in the catalog. This page is the contract for its shape: what every data product carries, how it names itself, and how we certify it is good.

One idea to hold onto: a data product is self-describing. It can be handed to a buyer, a teammate, or a later pipeline and it explains itself — where it came from, what is inside, how it was made, and whether it passed.

Status — envelope + eval implemented at L0

The base / provenance / tags / eval envelope described here is now produced by the L0 pipeline. A final step (step_11_eval in Data_Validation) reads the finished manifest.json, evaluates it against a quest + a catalog spec, and writes the self-describing product file data_product.json (dp_schema_version + base + eval + payload).

The path is additive: manifest.json is untouched (the review UI and every existing reader keep working) — the envelope is a new sibling file that wraps the manifest as its payload. The verdict is a pure function of metrics already in the manifest, so it is re-derivable without re-running any GPU stage (see Backtesting verdicts).

Where this sits relative to the database

The Recording Spine ends at two tables — pipeline_sessions and pipeline_runs — that record that a processing run happened and where its inputs live. This page defines the shape of what a run produces. The database points at the work; the data product is the work's output.

Processing levels

A level is a processing stage. Each level consumes the level below it (or the raw capture, for L0) and emits its own data product. Same envelope, different payload.

Level	What it is	Owner	Status
L0	Raw capture → clean, labelled, timestamped clips, captioned and tagged, certified by the eval step	Alessio (clips) · Vincenzo (eval)	Live — envelope + eval produced as data_product.json
L1 — calibrated	Derived intentionality on top of L0: fuse the transcript (verbalized intent), the input log, the game context, and the multi-POV timeline into intent / predicted-action signal — the world-model enrichment layer	Vincenzo	Reserved — envelope inherited, payload to be defined
L2 — reconstructed	3D scene reconstruction (point-cloud registration / Gaussian splat) over one or more synchronized POVs — a navigable spatial product	Marco · Antonio	Reserved — envelope inherited, payload to be defined

Each level consumes and enriches the one below it, carrying the lower level's output forward as metadata — so an L2 product traces back through its L1 and L0 (and the exact pipeline configuration that produced each), and can be regenerated on demand. We define the envelope once and fill L0 now; L1 and L2 inherit the same base / provenance / tags / eval and differ only in their payload. (L2's spatial reconstruction is the harder, less-portable step and may not always sit on the main L0→L1→L2 line — see Open items.)

Single and group products

A processing run is not always one-in, one-out. When the input is a party (a lobby — several players recording the same moment from different viewpoints), the run fans out into two kinds of product:

• Single product — one per member's recording. It carries what is visible from that seat: the clips, the player's intent (voice / transcript), and the alignment between inputs and on-screen action.
• Group product — up to one cumulative product for the whole party. It carries the facts that only exist across members: multi-viewpoint synchronization, how well the viewpoints cover the scene, and party size. Its provenance lists the member products it was built from.

So a run of N inputs produces up to N + 1 outputs: the N single products and, when the input is a party, one group product. Both kinds are still L0 — the group product is not a higher level, it is a sibling that aggregates its members. A solo recording is the simple case: one input, one single product, no group product.

Single vs group is a buyer-visible distinction

kind (single / group) is a first-class property the catalog surfaces, not an internal detail. Group products are the premium tier: at L0 they hold the multi-viewpoint and coverage value that a single seat cannot — the scarce, hardest-to-source data. Their properties (the cross-viewpoint metrics) must make that value explicit so the catalog can present and price them distinctly from single products.

The envelope

Every data product — any level, single or group — has the same four parts:

{
  "base": {
    "dataset_id": "...",        // stable, globally unique handle
    "level": "L0",
    "kind": "single",           // "single" | "group"
    "game": "gta5",             // the game (a catalog game) — first-class filter
    "provenance": { /* where it came from and how it was made */ },
    "metadata": { /* descriptive, level-specific fields */ },
    "tags": ["driving", "city", "first_person"]   // global vocabulary (see Tags)
  },
  "payload": { /* the actual data — clips for L0 */ },
  "eval":    { /* measured metrics and a pass/fail verdict (see Eval) */ }
}

base.dataset_id — identity

A data product is identified by what it was made from and how it was made: its capture source combined with the exact generation configuration that produced it (the processing variant and its resolved parameters). Two consequences follow:

• Re-running the same capture with the same configuration yields the same product — it overwrites in place, while each run's full record is retained separately for history.
• A different generation configuration is a different product — the same footage processed two ways are two distinct, separately listable products.

Identity is never the buyer: who orders a product, or with what delivery terms, has no bearing on its dataset_id. The group-product handle follows the same source-and-configuration rule once the party input mode lands (see Open items).

base.provenance — the lineage

Answers "where did this come from and how was it made," so the product is reproducible and auditable.

Field	Meaning
source_refs	The inputs: the originating session, lobby, and quest (where applicable), plus references to the raw capture files. For a group product it also lists the member products it was built from.
pipeline_version	An identifier of the exact pipeline build that produced the product.
processing_profile	Which processing variant ran and with what configuration — the reproducibility record. See Configurable processing.
produced_at	When it was produced.

base.game — the originating game

The game the recording is of. It is first-class, not buried in metadata, because the game is the single most-used buyer filter and the join to the catalog. A single recording is one game, and a party is one game by construction (a lobby is bound to one game), so a single field is enough.

base.metadata — descriptive, level-specific

Descriptive fields that are not lineage, not the game, and not a pass/fail metric — for example estimated frame rate, clip counts, and total / validated duration. New descriptive fields land here without touching the rest of the envelope.

payload — the data itself

For L0, the clip list. Each clip carries its time and frame bounds, its duration, the reconstructed input state at its start, the per-window perception output (caption, action, point-of-view, tags), and any joined transcript. Both the kept clips and the rejected ones are retained, each rejected clip annotated with why it was dropped, so the product is fully transparent. For a group product the payload references its member products and adds the cumulative cross-viewpoint facts. For L1 (calibrated intent) and L2 (3D reconstruction) the payload is reserved.

eval — the certification

Covered in its own section — it is the part that makes a product sellable rather than merely produced.

Tags — one global vocabulary

Tags are global and quest-independent. This is deliberate:

Buyers never see quests. A buyer searches the data — "footage of someone driving in a city at night" — across the entire corpus, regardless of which quest (if any) produced it. So the tag layer must be one flat vocabulary that means the same thing everywhere.

The pipeline assigns tags from a single fixed vocabulary (driving, walking, shooting, city, indoor, night, first_person, …) at a fine grain; this contract rolls them up (window → clip → product), deduplicated, so a product advertises the union of what is inside it. One list, applied uniformly, comparable across the whole catalog.

Alongside the fixed tags, each clip carries a free-text caption describing what is happening. Captions are the open-ended layer the fixed tags cannot express, and they back the free-form search described in Finding data.

The one decoupling rule

The processing may vary per quest (see below), but the tag vocabulary stays constant. A quest can change how the perception step is prompted; it must never change what a tag means. That is what keeps the catalog filter coherent across the whole corpus.

Eval — metrics and certification

eval is produced by a reusable final step, the same at every level, that runs after the payload is built. It answers two questions and keeps both answers:

1. The measurements (eval.metrics) — the raw numbers the pipeline can measure: frame rate, validated duration, perception ratios (gameplay, first-person), and, as the upstream signals become available, voice-intent density, multi-viewpoint synchronization, coverage, and party size. They are always kept, even when nothing is checking them against a requirement — they are the evidence.
2. The verdicts (eval.criteria) — each requirement checked, recording the metric it was derived from, the threshold, and pass/fail:

"eval": {
  "metrics": {
    "fps": 59.9,
    "validated_seconds": 71.2,
    "first_person_ratio": 0.93,
    "voice_intent_density_per_min": null,   // upstream signal not yet available
    "multi_pov_clock_skew_ms": null,        // group-product metric
    "coverage_score": null
  },
  "criteria": [
    { "requirement": "min_fps",     "source": "catalog_spec", "metric_ref": "fps",                "op": ">=", "threshold": 60,  "pass": false },
    { "requirement": "first_person","source": "catalog_spec", "metric_ref": "first_person_ratio",  "op": ">=", "threshold": 0.5, "pass": true  },
    { "requirement": "min_minutes", "source": "quest",        "metric_ref": "validated_seconds",   "op": ">=", "threshold": 900, "pass": false }
  ],
  "verdict": "fail"
}

eval is intrinsic to the product — measured numbers and a verdict derived from them. It carries no publication state (no certificate, no price, no license). Those are derived when the product is listed or sold, whether on its own or inside a bundle, keeping the product itself stable and free of fields that depend on whether or how it gets published.

Two properties keep this clean:

• Metrics are decoupled from thresholds. The pipeline measures once; pass/fail is re-derived from the stored metrics whenever a requirement changes — no reprocessing needed to re-judge.
• Missing signals are explicit. A metric that cannot yet be measured is null and its criterion is left unevaluated, never a silent zero. The harder signals (voice density, multi-viewpoint synchronization, coverage) appear as the capture side produces them; nothing blocks on them today.

Verdict values

verdict is derived from the criteria whose metric was actually measured:

Verdict	Meaning
pass	At least one criterion was evaluated and none failed.
fail	At least one evaluated criterion failed.
pending	No criterion could be evaluated yet (every referenced metric is null — e.g. a run that has not produced the signals its quest needs).

Unevaluated criteria never turn a pass into a fail. This is what lets the two validation tiers coexist on the same envelope:

• Numeric tier — the fast checks (fps, validated minutes, party size, frame freeze/black-screen) measured without the VLM. A numeric-only run certifies on these alone; its VLM/KPI criteria stay pending. This tier is what feeds the fast post-session feedback to the player in the consumer (B2C) reward loop.
• VLM / KPI tier — first_person_ratio and the quest's qualitative KPI ratios (see below). These fill in once the VLM stage runs, completing the verdict.

Qualitative requirements become measurable

A quest expresses a qualitative requirement as an extra VLM field + a KPI expression + a threshold. "Play near a lake for 90% of the time" becomes a boolean field near_lake, a KPI near_lake == true aggregated as a duration_ratio, and a threshold min_lake_proximity_ratio >= 0.9. The eval step evaluates the expression per kept window, duration-weights it into a single ratio metric, and checks it like any numeric criterion. The bound is not the framework — it is whether the VLM can perceive the field reliably, which is why quests mark each field's difficulty and gate only on easy fields.

Backtesting verdicts

Because the verdict re-derives from stored metrics, the whole catalogue can be re-judged in seconds with no GPU. scripts/backtest_eval.py walks every manifest.json under a root, evaluates each against a chosen quest + spec, and prints the accept/reject distribution, the accepted hours, and the top reject reasons (optionally writing data_product.json beside each manifest and a per-product CSV). Tune a threshold, re-run, get a new distribution.

Where eval lands — catalog + dashboard

When a product is published, the catalog indexer (pipeline/catalog_index.py) lifts the eval block and base.metadata from data_product.json into the catalog.data_products row — eval (jsonb: verdict + metrics + criteria), metadata, and duration_s (taken from eval.metrics.validated_seconds, so numeric-tier products with no VLM are no longer counted as zero seconds). The pipeline-KPIs dashboard reads these: eval_coverage_pct is the share of products carrying a non-empty eval, and the per-verdict / per-axis cards render from the same block. A product whose run was not bound to a quest keeps eval = {} and counts as un-evaluated coverage. The product's dataset_id is the canonical id the pipeline issues (generation_id.build_dataset_id), so the envelope and the catalog row are the same product.

Two consumers, one eval

The criteria come from two sources, and the eval block serves both the same way:

source	Where it comes from	Example
quest	The quest's requirements (minimum party size, minimum qualified minutes, required game, …) — see Quests — Technical Design.	min_minutes >= 15
catalog_spec	A buyer's acceptance criteria (minimum frame rate, input-to-action sync tolerance, on-screen-UI policy, …).	min_fps >= 60

Because the metrics map onto both, a single eval pass answers "does this satisfy the quest it was captured for?" and "does it clear what a buyer will accept?" — the bridge between why we captured the data and whether we can sell it.

Quest and free-play lanes

A recording is captured in one of two lanes: a quest lane, driven by a quest's requirements, or a free-play lane, with no quest. A free-play product simply has no quest criteria — its eval carries only the catalog_spec checks, plus its metrics, which are always kept. The eval block needs no special case: criteria from a source that does not apply are just absent, and the metrics stand on their own.

There is also a third reader of the same block: the person who recorded. Once a session is processed, the launcher can fetch that session's accept/reject outcome together with plain-English reasons derived from the failed criteria (a session not yet processed reads as pending). It is a pure projection of eval into player-facing language — no separate quality bookkeeping exists, so what the player is told and what the catalog certifies can never drift apart.

Configurable processing

Which processing runs, and how, is selectable per run. In the quest lane, the quest's requirements drive three things at once:

• The processing variant selects which steps run.
• The perception step can take a quest-specific prompt context — for example, a quest about narrated intent primes the model to attend to it — while the global tag vocabulary is unchanged (see the decoupling rule above).
• The eval uses the same requirements as its checklist.

In the free-play lane there is no quest, so a default variant runs and the eval applies only the buyer (catalog_spec) criteria.

The product records which variant and configuration ran in provenance.processing_profile, so it is reproducible regardless of how the run was configured.

Selling data

A data product is directly browsable and directly sellable. It is the atom and a unit a buyer can transact on its own — it appears in the catalog as a listing a buyer can inspect and buy. A buyer is never forced to take a pre-made package.

A bundle is simply a set of products sold together, created two ways:

• On-demand, by the buyer — the buyer selects several products and assembles them at checkout. The catalog already supports this through its order-builder flow; the bundle is created by the act of buying.
• Curated, by us — when we hold an adequate set of products and understand how to price them, we package and name a release proactively. This is a back-office step. The catalog does not currently host a release browse section; for now releases are how we organize and price supply.

Concept	What it is	Who makes it
Data Product	The atom — one self-describing pipeline output. Directly listable and sellable.	The pipeline
Bundle / Order	An on-demand set a buyer assembles at checkout.	The buyer
Release	A set we curate and name proactively once supply and pricing are clear.	Us (back-office)

The discipline that matters: no commercial or publication state lives on the product. A product carries no price, license, or certificate. A bundle references its member products without copying or mutating them, so one product can sit in many bundles. A catalog listing — whether it advertises one product or a curated release — derives its stats (hours, sessions, segments) as rollups over its member products and its quality from their eval.

Pricing is volume-based: a per-hour rate that depends on the quantity of data, applied the same way to a single product or a large bundle. Because price follows from the quantity and the buyer's terms, it is derived at sale time, not stored on the product.

Finding data

A data product's metadata only earns its keep if a buyer can find what they need fast and smart — not just quick exact-match lookups, but a search that understands what the buyer means. The model supports two query shapes, both returning data products:

1. Structured — "first-person, GTA V, ≥ 60 fps, ≥ 10 min, tagged driving and night." Predicate filtering over the structured fields (game, frame rate, duration) and the tags list. Fast and precise.
2. Free-form — "an attack on a tower." A description that no fixed field captures and whose exact words may appear nowhere in the data. It is answered by similarity of meaning between the buyer's description and each product's captions and tags, surfacing the products the description applies to even when the wording differs.

The clever part is that the two combine: a buyer can narrow by structured facts and describe an intent in their own words, and the catalog reconciles both into one ranked set of data products. The fixed tags give precise filtering; the captions give the free-form search rich, meaning-level material to match against — so the catalog behaves like an assistant that understands the request, not a keyword box. This relevance ranking and how the two shapes are weighted together is a search-quality concern tracked in Open items.

Storage — index separate from payload

Keep the heavy payload (the clips and the full product file) in object storage, and persist a lightweight, searchable index of the queryable metadata that points back at it. The index is small and queried constantly; the payload is large and fetched only on a sale. Keeping the index alongside the existing recorder data lets it reference existing entities (the game, the session, the lobby) directly rather than duplicating them.

The guiding principle for the data model: every field exists to be exploited. game, frame rate, duration, and tags drive the structured filters; caption backs the free-form search; eval is the quality gate; dataset_id is the join to bundles and to the stored payload. A field that cannot be tied to a query a buyer or a packager will run does not belong in base.

How it grows

The contract is meant to be extended, not frozen:

• New descriptive field → add it to base.metadata. Nothing else moves.
• New measurement → add it to eval.metrics; add a criterion when a requirement uses it. Older products keep their existing metric set.
• New tag → extend the one global vocabulary; it stays comparable.
• New level (L1 calibrated, L2 reconstructed) → reuse base / provenance / tags / eval and define a new payload. The envelope does not change.
• New buyer dimension → it is just another catalog_spec criterion.

The rule of thumb is wrap, don't rewrite. Every product, old or new, stays self-describing, and a reader can always fall back to base and eval even without understanding a newer payload.

Open items

• Group-product payload: the exact cumulative fields beyond the member list and the multi-viewpoint / coverage metrics — define when the party input mode lands.
• Catalog presentation of single vs group: how the catalog distinguishes and prices group (premium) products against single ones.
• Scene-level retrieval: whether individual scenes are exposed in the catalog at all, and if so to which buyers — the free-form search returns whole products for now.
• Catalog facets: add the game / frame-rate / duration filters the buyer UI does not yet expose, and reconcile the catalog's existing play-style and modality facets with the content tags defined here.
• Similarity search: the approach for meaning-level search over captions and tags. The searchable index itself lives alongside the recorder data (see the storage note above); only the similarity-search method is still open.
• Search quality: how structured and free-form queries are combined and ranked into one relevant result set, so the catalog search is clever, not just fast.
• L1 / L2 envelope reservation: confirm L1 (calibrated intent, Vincenzo) and L2 (3D reconstruction, Marco/Antonio) inherit base / eval unchanged before the wrapper is locked.
• L2 on the main line: decide whether 3D reconstruction always derives from L1 or can branch directly off L0 (it needs external inputs and is the least-portable step), so the L0→L1→L2 lineage stays accurate.