Updates

09-2025

08-2025

06-2025

Differentiable Gaussians-to-Mesh Extraction

MILo provides a differentiable pipeline for extracting a mesh only from Gaussian parameters, including both vertex locations and connectivity. At each training iteration, MILo proceeds as follows:

1. Each Gaussians spawns a set of pivots. These pivots are likely to be located near the surface of the scene, and their coordinates are differentiable with respect to the Gaussian parameters.
2. Delaunay triangulation is applied to the pivots. The triangulation can be cached to avoid recomputing it at every iteration.
3. SDF values are assigned to the pivots.
4. Differentiable Marching Tetrahedra is used to extract a mesh from the pivots and SDF values.

MILo enables gradient flow from the extracted mesh to Gaussians. Supervision on this mesh (through differentiable depth and normal renderings) allows us to impose a soft prior on the Gaussians which results in better reconstructed surfaces. Conceptually, Gaussians can be seen as proxies for parameterizing a surface mesh extracted at every iteration.

Higher-Quality Meshes with Fewer Vertices

(a) Stump (320 MB)

(b) Garden (301 MB)

(c) Barn (313 MB)

MILo produces significantly lighter meshes, weighing between 60MB and 400MB depending on the scene and configurations used. Compared to previous approaches such as Gaussian Opacity Fields or RaDe-GS, MILo increases performance while generating an order of magnitude fewer mesh vertices.

Part of this efficiency is achieved by selecting only a subset of Gaussians to spawn pivots, specifically targeting those most likely to be located on the surface. To this end, we repurpose the importance sampling strategy introduced in Mini-Splatting2.

Despite their lightweight nature, our meshes capture the complete scene, including all background elements. This makes our method significantly more scalable than many concurrent approaches that rely on TSDF applied to regular grids—a technique that scales poorly and is typically limited to foreground-only reconstruction.

Depth-Order Regularization

(a) Colored mesh

(b) Mesh without depth-order regul.

(c) Mesh with depth-order regul.

MILo scales to full scenes, and allows for extracting meshes including all background elements. However, without any additional learned prior, the background can contain inaccurate geometry and messy structures.

In our codebase, we propose an optional depth-order regularization loss to drastically improve the quality of the background. This loss relies on DepthAnythingV2, and enforces the order of the depth values in the rendered depth maps to match the order of the depth values in predicted depth maps: If one pixel is predicted to be closer than another, it should also be closer in the rendered depth map.

This loss does not require the predicted depth maps to be multi-view consistent. Please note that this loss is not used in the paper.

Animation

(a) Initial scenes

(b) Editing and animating in Blender

(c) Resulting animated Gaussians

While MILo provides a differentiable solution for extracting meshes from 3DGS representations, it also implicitly encourages Gaussians to align with the surface of the mesh. As a result, any modification made to the mesh can be easily propagated to the Gaussians, making the reconstructed mesh an excellent proxy for editing and animating the 3DGS representation.

Similarly to previous works SuGaR and Gaussian Frosting, we provide a Blender addon allowing to combine, edit and animate 3DGS representations just by manipulating meshes reconstructed with MILo in Blender. No code is needed to use the addon.

Integrating MILo to your Own Project

MILo can be easily integrated into your existing Gaussian Splatting pipeline. In milo.functional, we provide straightforward functions to leverage our differentiable Gaussians-to-Mesh pipeline in your own 3DGS projects.

These functions only require Gaussian parameters as inputs (means, scales, rotations, opacities) and can extract a mesh from these parameters in a differentiable manner, allowing for performing differentiable operations on the surface mesh and backpropating gradients directly to the Gaussians.

An example is provided below. Please refer to our codebase for more details.

Show/Hide Code Example

from functional import (
    sample_gaussians_on_surface,
    extract_gaussian_pivots,
    compute_initial_sdf_values,
    compute_delaunay_triangulation,
    extract_mesh,
    frustum_cull_mesh,
)

# Load or initialize a 3DGS-like model and training cameras
gaussians = ...
training_cameras = ...

# Define a simple wrapper for your Gaussian Splatting rendering function, 
# following this template. It will be used only for initializing SDF values.
# The wrapper should accept just a camera as input, and return a dictionary 
# with "render" and "depth" keys. Here is an example:
from gaussian_renderer.radegs import render_radegs
pipe = ...
background = torch.tensor([0., 0., 0.], device="cuda")
def render_func(view):
    render_pkg = render_radegs(
        viewpoint_camera=view, 
        pc=gaussians, 
        pipe=pipe, 
        bg_color=background, 
        kernel_size=0.0, 
        scaling_modifier = 1.0, 
        require_coord=False, 
        require_depth=True
    )
    return {
        "render": render_pkg["render"],
        "depth": render_pkg["median_depth"],
    }

# Only the parameters of the Gaussians are needed for extracting the mesh.
means = gaussians.get_xyz
scales = gaussians.get_scaling
rotations = gaussians.get_rotation
opacities = gaussians.get_opacity

# Sample Gaussians on the surface.
# Should be performed only once, or just once in a while.
# In this example, we sample at most 600_000 Gaussians.
surface_gaussians_idx = sample_gaussians_on_surface(
    views=train_cameras,
    means=means,
    scales=scales,
    rotations=rotations,
    opacities=opacities,
    n_max_samples=600_000,
    scene_type='indoor',
)

# Compute initial SDF values for pivots. Should be performed only once.
# In the paper, we propose to learn optimal SDF values by maximizing the 
# consistency between volumetric renderings and surface mesh renderings.
initial_pivots_sdf = compute_initial_sdf_values(
    views=train_cameras,
    render_func=render_func,
    means=means,
    scales=scales,
    rotations=rotations,
    gaussian_idx=surface_gaussians_idx,
)

# Compute Delaunay Triangulation.
# Can be performed once in a while.
delaunay_tets = compute_delaunay_triangulation(
    means=means,
    scales=scales,
    rotations=rotations,
    gaussian_idx=surface_gaussians_idx,
)

# Differentiably extract a mesh from Gaussian parameters, including initial 
# or updated SDF values for the Gaussian pivots.
# This function is differentiable with respect to the parameters of the Gaussians, 
# as well as the SDF values. Can be performed at every training iteration.
mesh = extract_mesh(
    delaunay_tets=delaunay_tets,
    pivots_sdf=initial_pivots_sdf,
    means=means,
    scales=scales,
    rotations=rotations,
    gaussian_idx=surface_gaussians_idx,
)

# You can now apply any differentiable operation on the extracted mesh, 
# and backpropagate gradients back to the Gaussians!
# In the paper, we propose to use differentiable mesh rendering.
from scene.mesh import MeshRasterizer, MeshRenderer
renderer = MeshRenderer(MeshRasterizer(cameras=train_cameras))

# We cull the mesh based on the view frustum for more efficiency
i_view = np.random.randint(0, len(train_cameras))
mesh_render_pkg = renderer(
    frustum_cull_mesh(mesh, train_cameras[i_view]), 
    cam_idx=i_view, 
    return_depth=True, return_normals=True
)
mesh_depth = mesh_render_pkg["depth"]
mesh_normals = mesh_render_pkg["normals"]
      

Gallery

Click on any scene below to open the interactive 3D viewer with both mesh and Gaussian Splatting representations. For a smooth 60 FPS experience, please make sure WebGL uses your graphics card and not your CPU.

For resource constraints, we only display in this gallery results obtained with the low-resolution, base version of MILo: As a result, Gaussian Splatting representations contain much fewer Gaussians than usual, and the meshes have much fewer vertices than concurrent approaches.

Still, these lighter meshes reach higher quality on T&T and DTU benchmarks than concurrent approaches. The mesh sizes reported below include vertex colors.

Please note that these meshes were not optimized with depth-order regularization; As a result, the background can contain inaccurate geometry and messy structures.

Stump

Mesh size: 349 MB

Nb Gaussians: 582K

Ignatius

Mesh size: 256 MB

Nb Gaussians: 417K

Red Robot

Mesh size: 150 MB

Nb Gaussians: 261K

Garden

Mesh size: 354 MB

Nb Gaussians: 684K

Buzz

Mesh size: 68 MB

Nb Gaussians: 120K

Bicycle

Mesh size: 298 MB

Nb Gaussians: 563K

Knight

Mesh size: 58 MB

Nb Gaussians: 103K

Blue Robot

Mesh size: 61 MB

Nb Gaussians: 105K

Kitchen

Mesh size: 217 MB

Nb Gaussians: 409K

More Scenes

Coming Soon

 

BibTex

      @article{guedon2025milo,
        author       = {Gu{\'e}don, Antoine and Gomez, Diego and Maruani, Nissim and Gong, Bingchen and Drettakis, George and Ovsjanikov, Maks},
        title        = {MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction},
        journal      = {ACM Transactions on Graphics},
        number       = {},
        volume       = {},
        month        = {},
        year         = {2025},
        url          = {https://anttwo.github.io/milo/}
      }
    

Acknowledgements

Parts of this work were supported by:

• The ERC Consolidator Grant "VEGA" (No. 101087347)
• The ERC Advanced Grant "explorer" (No. 101097259)
• The ERC Advanced Grant "NERPHYS" (No. 101141721)
• The ANR AI Chair AIGRETTE

• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes
• RaDe-GS: Rasterizing Depth in Gaussian Splatting
• Mini-Splatting2: Building 360 Scenes within Minutes via Aggressive Gaussian Densification
• Modular Primitives for High-Performance Differentiable Rendering

Further information

If you like this project, check out our previous works related to 3D reconstruction, radiance fields or Gaussian splatting:

• Guédon et al. - MAtCha Gaussians: Atlas of Charts for High-Quality Geometry and Photorealism From Sparse Views (CVPR 2025)
• Gomez et al. - FourieRF: Few-Shot NeRFs via Progressive Fourier Frequency Control (3DV 2025)
• Guédon and Lepetit. - Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering (ECCV 2024)
• Maruani et al. - PoNQ: a Neural QEM-based Mesh Representation (CVPR 2024)
• Guédon and Lepetit. - SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (CVPR 2024)
• Maruani et al. - VoroMesh: Learning Watertight Surface Meshes with Voronoi Diagrams (ICCV 2023)

© You are welcome to copy the code of the webpage, please attribute the source with a link back to this page and remove the analytics.