3D Reconstruction: From Pixels to Radiance Fields

Imagine walking around a wooden park bench, snapping a dozen photos on your phone.

Your brain effortlessly synthesizes these 2D snapshots into a continuous 3D mental model. You instinctively know where the bench ends, how far the bicycle rests behind it, and what the scene looks like from angles you never directly captured.

A computer gets none of this for free. To an algorithm, an image is just a flat 2D grid of RGB values. Reconstructing physical 3D reality from 2D images is the core challenge of 3D Reconstruction and Novel-View Synthesis. Today, this technology underpins robotics, spatial computing, VFX, and digital twins.

Let’s unpack how we solve this fundamental problem—and explore how representations evolved from rigid geometric meshes to modern neural radiance fields and 3D Gaussian splats.

• The Core Challenge: An Ill-Posed Inverse Problem
• Classical 3D Scene Representations
  • 1. Voxels: Space as Discrete Cubes
  • 2. Point Clouds: Sparse Coordinates
  • 3. Polygon Meshes: The Graphics Standard
• The Foundation: Structure-from-Motion (SfM)
• Implicit Neural Representations: NeRF
  • Differentiable Volume Rendering
  • The NeRF Bottleneck
• Explicit Neural Splatting: 3D Gaussian Splatting (3DGS)
  • Tile-Based Rasterization: Real-Time Performance
• Head-to-Head: NeRF vs. 3D Gaussian Splatting
• Why 3D Representations Matter for Robotics
• Key Takeaways
• Further Reading

---

The Core Challenge: An Ill-Posed Inverse Problem

Why is 3D vision hard? Because taking a photograph is a lossy projection:

\[\mathbb{R}^3 \xrightarrow{\text{Perspective Projection}} \mathbb{R}^2\]

When a 3D point projects onto a 2D pixel, depth is permanently discarded. A single red pixel could originate from an object 1 meter away or a massive structure 100 meters away.

Recovering 3D geometry from 2D images is an ill-posed inverse problem: infinitely many 3D configurations can project onto the exact same set of 2D images. Resolving this ambiguity requires choosing the right mathematical representation of space.

---

Classical 3D Scene Representations

Before neural representations existed, vision systems relied on explicit geometric structures—each bringing critical trade-offs.

1. Voxels: Space as Discrete Cubes

The simplest approach divides 3D space into a uniform volumetric grid (think Minecraft).

• The Drawback: Cubic complexity $\mathcal{O}(N^3)$. Doubling the resolution consumes $8\times$ more memory. Most voxel memory is wasted storing empty air.

---

2. Point Clouds: Sparse Coordinates

Instead of empty grids, store an unordered set of 3D spatial points:

\[\mathcal{} = \{p_i \in \mathbb{R}^3 \mid i = 1, \dots, N\}\]

Sensors like LiDAR and depth cameras naturally produce point clouds.

• The Drawback:
  • The “Screen-Door” Effect: Because points lack surface area, zooming in or viewing from grazing angles reveals gaps and background bleed-through.
  • No Surface Normals or Occlusion: Points do not encode topology, making realistic lighting and solid surface rendering difficult.

---

3. Polygon Meshes: The Graphics Standard

Meshes explicitly define continuous surfaces using collections of vertices, edges, and triangular faces.

Meshes are the backbone of real-time graphics engines, but extracting clean meshes via traditional multi-view photogrammetry breaks down on complex real-world materials:

• Fuzziness & Hair: Thin strands, fur, and tree leaves cannot be modeled cleanly by rigid polygons without millions of degenerate micro-triangles.
• Transparency & Volumetrics: Smoke, glass, water, and fog have no singular hard boundary.
• Rigidity & Non-Manifold Artifacts: Optimization algorithms struggle to handle sharp topological changes or self-occluding boundaries.

---

The Foundation: Structure-from-Motion (SfM)

Regardless of whether we use traditional or neural rendering, every pipeline begins with camera calibration. If we do not know the precise location of each camera in 3D space, inverse rendering is impossible.

Structure-from-Motion (SfM) recovers camera extrinsic parameters ($R, \mathbf{t}$), intrinsic parameters, and a sparse point cloud using a 4-step pipeline:

1. Feature Detection & Matching: Identify distinctive visual landmarks (e.g., SIFT, SuperPoint) across images.
2. Epipolar Geometry: Calculate essential and fundamental matrices to derive relative camera motions.
3. Triangulation: Cast intersecting rays from matched features to calculate initial 3D positions.
4. Bundle Adjustment: Jointly optimize camera poses $\mathbf{}_i$ and 3D points $\mathbf{}_j$ by minimizing reprojection error across all views:

\[\min_{\{\mathbf{}_i\}, \{\mathbf{}_j\}} \sum_{i,j} \left\| \mathbf{}_{i,j} - \pi(\mathbf{}_i, \mathbf{}_j) \right\|^2\]

---

Implicit Neural Representations: NeRF

What if we stop storing explicit geometry (points/triangles) altogether and instead store the entire scene inside the weights of a neural network?

Introduced in 2020, Neural Radiance Fields (NeRF) parameterize a scene as a continuous 5D function parameterized by a Multi-Layer Perceptron (MLP):

\[F_\Theta: (x, y, z, \theta, \phi) \longrightarrow (\mathbf{}, \sigma)\]

• $(x, y, z)$: 3D coordinates in space
• $(\theta, \phi)$: 2D viewing direction vector $\mathbf{d}$
• $\mathbf{} = (r, g, b)$: Emitted RGB radiance
• $\sigma$: Volumetric density (opacity/thickness)

Differentiable Volume Rendering

To render a pixel, NeRF casts a camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ through the scene, samples discrete points along the ray, queries the MLP for color and density, and composites them:

\[C(\mathbf{r}) = \int_{t_n}^{t_f} T(t) \sigma(\mathbf{r}(t)) \mathbf{}(\mathbf{r}(t), \mathbf{d}) \, dt, \quad \text{where } T(t) = \exp\left(-\int_{t_n}^{t} \sigma(\mathbf{r}(s)) \, ds\right)\]

Because every step of this numerical integration is differentiable, we optimize network weights directly via gradient descent on a photometric loss:

\[\mathcal{}_{\text{photo}} = \sum_{\mathbf{r} \in \mathcal{R}} \left\| C(\mathbf{r}) - C_{\text{gt}}(\mathbf{r}) \right\|_2^2\]

The NeRF Bottleneck

While NeRF handles fine hair, reflections, and smoke with ease, rendering is painfully slow. Generating a single $1920 \times 1080$ frame requires evaluating the MLP hundreds of millions of times, making real-time interactive rendering difficult without extensive acceleration tricks.

---

Explicit Neural Splatting: 3D Gaussian Splatting (3DGS)

In 2023, 3D Gaussian Splatting addressed NeRF’s computational bottleneck by abandoning MLPs in favor of explicit, differentiable primitives.

Instead of querying a neural network for every point along a ray, 3DGS models the world as millions of 3D Gaussians:

\[G(\mathbf{x}) = \exp\left( -\frac{1}{2} (\mathbf{x}-\boldsymbol{\mu})^\top \Sigma^{-1} (\mathbf{x}-\boldsymbol{\mu}) \right)\]

where:

• $\mathbf{x} \in \mathbb{R}^3$ is a point in 3D space.
• $\boldsymbol{\mu} \in \mathbb{R}^3$ is the Gaussian’s center.
• $\Sigma \in \mathbb{R}^{3\times3}$ is the covariance matrix, which determines its scale and orientation.

Each primitive maintains:

1. Position ($\boldsymbol{\mu}$): 3D center $(x, y, z)$
2. Covariance ($\Sigma$): Decomposed into rotation quaternion $q$ and 3D scale vector $s$ ($\Sigma = R S S^\top R^\top$) to guarantee positive semi-definiteness
3. Opacity ($\alpha$): Volumetric density
4. Color: Modeled via Spherical Harmonics (SH) coefficients to capture view-dependent specular highlights

Tile-Based Rasterization: Real-Time Performance

3DGS replaces slow ray-marching with GPU-accelerated rasterization:

1. Projection: Project 3D Gaussians into 2D screen-space ellipses using a local affine approximation.
2. Tile Sorting: Partition the screen into $16 \times 16$ pixel tiles and fast-sort the Gaussians by depth (using radix sort).
3. $\alpha$-Blending: Blend the overlapping Gaussians in front-to-back order within each tile in parallel.

---

Head-to-Head: NeRF vs. 3D Gaussian Splatting

Dimension	Neural Radiance Fields (NeRF)	3D Gaussian Splatting (3DGS)
Representation	🧠 Implicit — Neural Network / MLP	🟢 Explicit — 3D Gaussians
Rendering Engine	🐢 Ray Marching	⚡ Tile-Based Splatting & Rasterization
Rendering Speed	❌ 0.1–5 FPS	✅ 100–200+ FPS
Training Time	❌ Hours to Days	✅ 15–45 Minutes
Memory Footprint	✅ Compact ~5–50 MB	❌ Higher ~200 MB–1.5 GB
Fluffy / Thin Structures	✅ Excellent	✅ Excellent
Real-Time Rendering	❌ Challenging	✅ Yes
Photorealistic Quality	✅ Excellent	✅ Excellent

---

Why 3D Representations Matter for Robotics

A physical robot cannot plan trajectories in flat pixel space. It needs persistent spatial representations to handle contact dynamics, obstacle avoidance, and manipulation.

• Dense Spatial Reasoning: Knowing unoccupied free space vs. solid collision boundaries.
• Novel View Synthesis for Simulation: Generating synthetic training rollouts for visuomotor policy learning without running thousands of real-world trials.
• Differentiable Physics Interfaces: Coupling explicit representations like 3DGS with physical simulators to perform real-time robot grasp planning and visual tracking.

---

Key Takeaways

1. 3D reconstruction is an ill-posed inverse problem because 2D projections discard absolute depth and introduce occlusions.
2. Meshes struggle with complex materials like smoke, fur, and thin structures; point clouds suffer from the screen-door effect.
3. NeRF introduced continuous neural volumetric rendering, solving fuzziness and specularities at the cost of high rendering latency.
4. 3D Gaussian Splatting delivers the best of both worlds: explicit volumetric primitives optimized end-to-end and rendered at real-time frame rates via GPU rasterization.

---

Further Reading

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (Mildenhall et al., ECCV 2020)
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering (Kerbl et al., SIGGRAPH 2023)
• Structure-from-Motion Revisited (Schönberger & Frahm, CVPR 2016)
• COLMAP Documentation — The standard open-source library for SfM and multi-view reconstruction.