L0 — Data Pipeline

Renewal in progress — some details may be stale

This page is the L0 stage of our processing pipeline (the first level that turns raw captures into data products — see Pipeline Output Data Products). It was written earlier in the project and parts of it are being updated: repo name, folder layout, and step list have since moved on. Treat the high-level flow as correct but verify specifics against the live Data_Validation repo until this page is refreshed with a clear description of the current L0 pipeline.

The data-filtering repo (C:\EC\data-filtering) is a GPU-accelerated Python pipeline that converts raw gameplay recordings produced by playroll-cpp into clean, annotated sub-clips suitable for training AI game-playing agents.

It is an offline post-processing tool — it does not run inside the recorder. It consumes a recorded MP4 plus its companion input-event CSV and emits trimmed clips, per-clip CSVs with reconstructed input state, motion plots, and VLM-derived captions.

Inputs and outputs

Direction	Artifact	Source
Input	video.mp4	Pure-v2 capture MP4 from playroll-cpp
Input	events.csv	frame_number, event_type, input_name, value event log
Output	clip_XXXX_*.mp4	Sub-clips cut on motion boundaries
Output	clip_XXXX_*.csv	Per-clip event log with synthetic state_at_start rows
Output	manifest.json	Clip index with frame ranges, VLM scores, window timeline (single canonical manifest — rewritten in place by the VLM stage; there is no separate manifest_vlm.json)
Output	data_product.json	The sellable L0 product envelope: base + eval verdict + payload (the manifest), written by the eval step
Output	scores.png	Motion-score plot

Pipeline stages

Stage 1 — Motion scoring (pipeline/motion.py)

Reads the video via an ffmpeg pipe (no OpenCV-CUDA dependency) and fuses two per-frame signals into a [0, 1] motion score:

• Frame differencing on the luma channel, computed on GPU via PyTorch.
• Farneback dense optical flow magnitude, computed on CPU via cv2.calcOpticalFlowFarneback.

Both signals are normalised independently before weighted fusion (w_diff=0.4, w_flow=0.6 by default). The optical-flow stage downsizes frames to a 320 px short side before computation.

Stage 2 — Segmentation (pipeline/segmenter.py)

Thresholds the motion-score series and applies temporal hysteresis to emit (start_frame, end_frame) dynamic intervals. There is deliberately no Gaussian smoothing — hysteresis handles the spiky nature of gameplay-motion signals more cleanly. Static runs shorter than min_static_s do not split a clip; dynamic runs shorter than min_dynamic_s are discarded.

Stage 3 — Cutting (pipeline/cutter.py)

Cuts the source MP4 at segment boundaries using ffmpeg stream copy (-c copy) so cutting is near-instant regardless of resolution. Emits manifest.json with frame ranges, timestamps, and clip paths.

Stage 4 — CSV join (pipeline/csv_join.py)

Attaches event-based controller input data to each clip. Because the CSV is event-driven — rows only appear when an input changes — a naive frame-range slice would lose any input held across the boundary. The join therefore:

1. Replays all events prior to start_frame to reconstruct the active input state.
2. Prepends synthetic state_at_start rows so downstream consumers always know the full input state at frame 0 of the clip.
3. Slices in-window events within [start_frame, end_frame).

Supports both event-row format (frame_number, event_type, input_name, value) and a time-column variant where frame_number is derived from a millisecond timestamp and a configured FPS.

Stage 5 — VLM filter (pipeline/vlm_filter.py)

Scores and captions clips with a local vision-language model (currently Qwen2-VL 7B; InternVL2 8B and LLaVA-1.6 are the MIT/Apache-licensed alternatives noted in the README). Runs entirely on-device — no API calls, no data egress.

Each clip is split into windows of window_s seconds. Per window, n_keyframes frames are sampled and scored 0–10. A window is also classified for first-person perspective: lobby, character-select, scoreboard, kill-cam, and spectator views are flagged first_person=false and treated as dropped regardless of score. A clip is kept only if min_keep_ratio of its windows pass both the score threshold and the first-person check.

When trim_bad_windows is true, a second ffmpeg pass cuts kept windows out as separate sub-clips. drop_margin_s shaves additional seconds from kept windows that border a dropped window — useful for menu transitions that bleed into otherwise-clean gameplay.

manifest_vlm.json stores the full window timeline (score, caption, reason, action, kept) per clip, so the keep/drop decision can be replayed at a different threshold without re-running the model.

Stage 6 — Acceptance evaluation (pipeline/eval.py, step_11_eval)

The final, thin validation layer. It runs no model and re-reads no video: it loads the finished manifest.json, compares its measured metrics to a bound quest (quests/<quest_id>.json) and catalog spec (quests/catalog_specs/<spec_id>.json), and writes the self-describing data product data_product.json — the dp_schema_version + base + eval + payload envelope defined in Pipeline Output Data Products.

• manifest.json is untouched — the review UI and existing readers keep working. data_product.json is a new sibling that wraps the manifest as its payload.
• The verdict (pass / fail / pending) is a pure function of metrics already in the manifest, so it is re-derivable without re-running any GPU stage. scripts/backtest_eval.py re-judges a whole tree of recordings in seconds — point it at the batch-run output to get the accept/reject distribution and reject reasons.
• A criterion whose metric is null (e.g. a VLM/KPI signal a numeric-only run did not produce) is left unevaluated, so the numeric tier can certify on fps / validated minutes / party size alone while the VLM tier fills in later.

Configured under the eval section of config.json (quest_id, catalog_spec_id, quests_dir, catalog_specs_dir); eval.enabled=false or an empty quest_id skips the step and leaves the product as raw L0.

Hardware and dependencies

• NVIDIA GPU with 10 GB+ VRAM (tested on A10G 23 GB); CUDA 12.x or newer.
• Conda environment in environment.yml plus transformers >= 4.45, accelerate, qwen-vl-utils, torchvision.
• ffmpeg from conda-forge (the default bundle is missing required codecs).

CLI entry point

test_pipeline.py is the primary entry point and takes positional arguments so no source edits are needed between runs:

python test_pipeline.py <video.mp4> <events.csv> <out_dir>/ [--vlm] [--drop-margin S] [--edge-trim S]

Without --vlm the pipeline stops after Stage 4 (motion → segmentation → cutting → CSV join). With --vlm it runs the full five-stage pipeline including windowed VLM scoring and optional trimming.

Relationship to other repos

• playroll-cpp produces the MP4 + CSV + JSON metadata bundle that data-filtering consumes. The CSV schema (frame_number, event_type, input_name, value) matches the pure-v2 capture event log.
• playroll-docs (this site) describes the contracts on the upstream side; this page documents the offline-training-data pipeline that builds on top of them.