3D Data Representations — Concepts

This page is the vocabulary and mental map for 3D reconstruction. Before you can reason about an alignment bug, a meshing artifact, or which file format to ship, you need to know what kind of data you are holding and where it sits in the pipeline. That is what this page gives you.

It is written to be read two ways:

• As a newcomer — read top to bottom. Each representation has a plain-language card: what the data is, where it comes from, and its trade-offs.
• As an engineer working in 3d-data-reconstruction — jump to The reconstruction pipeline and Transformation algorithms; those are the arrows this repo implements (point-cloud alignment/fusion, voxel/TSDF, meshing).

Scope

This is conceptual / domain reference, not a description of a specific Playroll service. The 3D reconstruction work lives in its own repo (3d-data-reconstruction). This page documents the field so the rest of the team shares one vocabulary; repo-specific pipeline docs can link here for the "what is a TSDF?" background instead of re-explaining it each time.

The one-sentence mental model

3D data is geometry (where surfaces are) + appearance (what they look like) + capture metadata (where the sensor was) — and reconstruction is a chain of lossy transforms that turns raw sensor data into a clean geometry + appearance asset, leaving a pile of throwaway intermediate structures along the way.

Everything below is a refinement of that sentence.

---

The two core representations

Point cloud

• Data structure — an unordered set of 3D points. Minimum: (x, y, z) per point.
• Optional per-point attributes — (r,g,b) color · (nx,ny,nz) normal · intensity (LiDAR) · confidence · semantic class · timestamp.
• Defining property — samples of a surface with no connectivity. No point knows its neighbors. There is no topology and no explicit surface — just points floating in space.
• Comes from — LiDAR · photogrammetry / Structure-from-Motion · depth cameras (RealSense, Kinect) · ToF sensors. It is almost always the first raw product of reconstruction.
• Pros — simple · sensor-native · easy to fuse and align.
• Cons — gaps between points · no continuous surface · heavy when dense · noisy.
• File formats — .ply, .pcd, .las/.laz, .xyz, .e57.

Mesh

• Data structure — an explicit surface = vertices + edges + faces (usually triangles, sometimes quads).
• Defining property — connectivity / topology. The faces say how points join into a surface. In one line: mesh = point cloud + how the points connect.
• Extra attributes — UV coordinates (to map textures) · per-vertex / per-face normals · material assignments.
• Comes from — almost always derived from a point cloud via surface reconstruction. Rarely a primary sensor product.
• Pros — continuous, renderable surface · lightweight vs a dense cloud (a few faces cover a flat wall) · the standard for game engines and graphics.
• Cons — hard to generate cleanly (holes, noise, "manifoldness") · loses detail when simplified (decimated).
• File formats — .obj, .glb/.gltf, .fbx, .stl, .ply.

---

The rest of the family (geometry byproducts)

Representation	Data structure	Key idea	Pipeline role
Voxel grid	Regular 3D grid of cells (3D pixels); each cell occupied/empty or a value	"Minecraft" — discretize space into cubes	Intermediate volume; memory grows as resolution³
TSDF (Truncated Signed Distance Function)	Voxel grid where each cell stores the signed distance to the nearest surface	The surface lives where distance = 0 (the "zero crossing")	Classic bridge between depth maps and mesh (KinectFusion-style)
Depth map / RGB-D	2D image; each pixel = distance from the camera (+ aligned color for RGB-D)	This is 2.5D — you only see the surface facing the camera	Often the input a point cloud is back-projected from
Neural implicit (SDF / occupancy)	A function (neural net): point in → inside/outside or signed distance out	The surface is computed, not stored	Modern learning-based reconstruction (DeepSDF, Occupancy Networks)
NeRF	Neural net: (position, view dir) → (color, density)	Represents the light field, not the surface	Photorealistic novel-view rendering; geometry extracted as a second step
3D Gaussian Splatting	Set of 3D Gaussians (position, ellipsoidal covariance, color, opacity)	The recent evolution of NeRF — real-time rendering	Between a point cloud and a continuous field
Parametric / CAD (B-Rep, NURBS)	Surfaces defined by equations	Exact, analytic geometry	Not from scanning — from authoring. .step, .iges

---

The organizing frame: two axes

Most of the confusion in this space disappears once you place each representation on two axes:

• Explicit ⟷ Implicit — is the surface stored directly (explicit) or computed by a function (implicit)?
• Discrete ⟷ Continuous — is space sampled / gridded (discrete) or defined everywhere (continuous)?

Gaussian Splatting is the fun edge case: explicit primitives that blend into a continuous-looking field.

---

The reconstruction pipeline

This is the spine. Each arrow is a transform where information is gained or lost.

Where this repo lives on the spine

3d-data-reconstruction owns the middle of the spine: it takes per-view point clouds, aligns and fuses them (the feature/pointcloud-alignment work), builds a voxel representation, and saves point-cloud / voxel / 3D outputs. The "ALIGN

• FUSE" arrow is the most interesting engineering surface — see the algorithms below.

---

The layers around the geometry spine

The spine above is only the geometry lane. A real 3D asset and a real capture rig involve three more lanes. Think of them as swim-lanes wrapping the spine.

Lane 1 — Raw capture data (upstream of point clouds)

Node	Structure	Why it matters
Posed images / image set	2D photos + per-image camera pose + intrinsics	Literal raw input for photogrammetry / NeRF / Gaussian Splatting
Camera intrinsics / extrinsics	Focal length, principal point, distortion (intrinsics); position + rotation (extrinsics)	The glue that lets you back-project 2D → 3D
Raw LiDAR scan	Per-beam range, azimuth, elevation, intensity, return number	Before it becomes XYZ; multi-return sees through foliage
IMU / odometry / GPS	Time-series of position / orientation / acceleration	Registers scans in motion (SLAM, drones, mobile mapping)
Structured-light / stereo pair	Two+ images or a projected pattern	How depth cameras compute depth before you see a depth map

Key concept — SLAM (Simultaneous Localization and Mapping): builds the map and tracks the sensor pose at the same time. The engine behind real-time scanning (robots, AR, mobile LiDAR).

Lane 3 — Appearance data (the "skin" on geometry)

Geometry is only half an asset. The other half is how it looks.

Map	What it encodes
Albedo / diffuse	Base color, no lighting baked in
Normal map	Fake high-frequency detail in a 2D texture — a low-poly mesh looks high-poly
Roughness / metallic / specular	PBR (Physically Based Rendering) material properties
Displacement / height	Real geometric offset, not faked bumps
Ambient Occlusion (AO)	Baked contact shadows
UV layout	The 2D unwrap mapping texture pixels → 3D surface

Key concept — PBR material set: albedo + normal + roughness + metallic shipped together. Moving detail from a high-poly to a low-poly mesh is called baking.

Lane 4 — Mid-pipeline / intermediate artifacts

These are the "mid-rendering" products you don't ship but you debug and cache.

Node	What it is
Sparse cloud + camera graph	SfM's first output: tie-points + camera poses — the skeleton before dense reconstruction
Dense depth maps (post-MVS)	Refined per-pixel depth after Multi-View Stereo, fused into the dense cloud
Pose graph / factor graph	Nodes = poses, edges = relative transforms with uncertainty. This is literally what multiway registration optimizes
Octree / KD-tree / BVH	Spatial acceleration structures — fast "nearest point" / ray queries (ICP, ray-casting)
Confidence / weight volume	Alongside a TSDF: how trustworthy each voxel is, for fusing many views
G-buffer	Render-side analog: per-pixel depth / normals / albedo / motion vectors written before final shading

Lane 5 — Scene / animation data (beyond one object)

Node	What it adds
Scene graph	Hierarchy of transforms — objects parented in a tree
Rig + skinning weights	Bones + how each vertex follows them — makes a static mesh animatable
Animation curves	Keyframed transforms over time
USD (Universal Scene Description)	Pixar's format; the emerging standard for whole-scene interchange
LOD chain	Same object at multiple resolutions, swapped by distance

---

Transformation algorithms

These are the names to attach to the arrows — the "interesting engineering choices" where quality is won or lost.

Transformation	Algorithm(s) to know
Image features → camera poses + sparse cloud	SfM (Structure-from-Motion), COLMAP
Align two point clouds	ICP (Iterative Closest Point)
Align many clouds at once	Multiway registration / pose-graph optimization
Point cloud → mesh	Poisson surface reconstruction · Ball Pivoting · Delaunay
Voxel / TSDF → mesh	Marching Cubes
Estimate per-point normals	PCA on a local neighborhood (needed before Poisson)
Reduce mesh size	Decimation / quadric edge collapse
Clean a cloud	Statistical / radius outlier removal, voxel downsampling

Relevant to feature/pointcloud-alignment

ICP aligns a pair of clouds by iterating "find closest points → solve the rigid transform that minimizes their distance → repeat." Multiway registration generalizes this to N clouds at once by building a pose graph (one node per scan, edges = pairwise ICP results) and globally optimizing all poses together, so errors don't accumulate around a loop. That pose graph is the Lane-4 artifact above.

---

File-format quick reference

Format	Carries	Typical use
.ply	Point cloud or mesh (+ per-vertex color/normal)	Reconstruction interchange, research
.pcd	Point cloud (PCL-native)	Point Cloud Library workflows
.las / .laz	Point cloud (+ LiDAR attributes)	Aerial / survey LiDAR (.laz = compressed)
.e57	Point cloud + imagery + metadata	Industrial / survey scans
.obj	Mesh + UVs (materials via .mtl)	Universal mesh interchange
.glb / .gltf	Mesh + PBR materials + scene + animation	Runtime / web / game delivery
.fbx	Mesh + rig + animation	DCC tools, game engines
.stl	Mesh (geometry only)	3D printing
.step / .iges	Parametric / CAD (B-Rep, NURBS)	Engineering / CAD
.usd / .usdz	Whole scene (geometry + materials + layers + variants)	Film / large-scene interchange

---

How this maps to our code (3d-data-reconstruction)

The sections above are the generic field. This section anchors them to what the repo actually produces today, so you can tell the textbook from our implementation. (Verified by reading the source on the feature/pointcloud-alignment branch.)

Concept on this page	Do we produce it?	Where / what is stored
Depth map / RGB-D	✅ as input	rgbd_render/data/loader.py → .npz with images (uint8), depth (float32), intrinsic (3×3), extrinsic (3×4 W2C), optional confidence
Camera intrinsics / extrinsics	✅	Loaded with the RGB-D .npz; c2w poses derived into the Scene. Intrinsics used only during unprojection, not re-saved
Point cloud	✅ fused, GPU-unprojected	geometry/unproject.py::unproject_depth_batch_gpu → pipeline/builder.py. Stores xyz + rgb + per-point frame index. No normals at this stage. Saved inside the voxel .npz
Voxel grid	✅ Morton-coded	geometry/voxel.py::VoxelGridCUDA. Per voxel: center xyz, mean rgb, earliest frame, voxel_size. Saved via np.savez_compressed (voxels.npz). Marked deprecated in favor of the octree
Octree / spatial structure	✅ primary structure	geometry/octree.py::OctreeSPC (Kaolin SPC). Multi-level centroids + colors + Morton cell IDs, with per-frame LOD selection (lod_select). In-memory only — rebuilt from the .npz, not persisted
TSDF	❌ not produced	The pipeline goes RGB-D → point cloud → voxel/octree directly. There is no signed-distance fusion step. (So the "TSDF" node in the spine diagram above is a general-pipeline stage we skip.)
Mesh	⚠️ point-cloud .glb only	interactive_viewer/npz_to_glb.py and lingbot_map/vis/glb_export.py export a trimesh.PointCloud (+ camera frustums) to .glb; voxel_vis.py builds cube billboards for display. No marching cubes / no real surface mesh
NeRF / Gaussian / neural SDF	❌ none	This is a classical unproject → octree-LOD pipeline, not a differentiable/neural one

The alignment step in our terms

align_pointclouds.py is a textbook multiway registration, not just pairwise ICP. The chain, with the vocabulary from Transformation algorithms:

• Input — per-view clouds as .npz (xyz + optional rgb), loaded into Open3D (load_npz_as_o3d).
• Preprocessing — voxel_down_sample (collapses ~80M points to a few million), estimate_normals (needed for point-to-plane), and FPFH features.
• Coarse → fine — FPFH RANSAC for the initial guess, then point-to-plane ICP (TransformationEstimationPointToPlane) to refine.
• Global — an Open3D PoseGraph: sequential edges trusted, all other pairs added as loop-closure edges marked uncertain so bad matches are dropped, then global_optimization with Levenberg-Marquardt. This is the pose-graph / factor-graph Lane-4 artifact, made concrete.
• Output — one 4×4 transform_<name>.npy per cloud (into node-0's frame), plus a downsampled merged_preview.ply for eyeballing. Full-resolution out-of-core merge is still TODO (see the handoff).

Known limitation (from the handoff)

Monocular reconstructions have arbitrary scale and internal drift, and repeated identical assets in game footage (crates, walls) can make FPFH/RANSAC false-match. Aligning duplicated clouds does not fix a data-association / loop-closure gap — the handoff recommends evaluating a loop-closing stack (MASt3R-SLAM / COLMAP+hloc) rather than extending the current alignment alone. Always inspect merged_preview.ply and the per-edge fitness scores.

---

See also

• Pure-v2 Capture Pipeline — for comparison, how the Playroll recorder pipelines a different kind of data (frames → MP4) with the same "stage → queue → artifact" shape.
• Repo: 3d-data-reconstruction — align_pointclouds.py, demo_render/rgbd_render/, and HANDOFF-pointcloud-alignment.md for the current status and next steps.