SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering

CVPR 2024

Antoine GuédonVincent Lepetit

LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS

full_teaser.png


We introduce a method that extracts accurate and editable meshes from 3D Gaussian Splatting representations within minutes on a single GPU. The meshes can be edited, animated, composited, etc. with very realistic Gaussian Splatting rendering, offering new possibilities for Computer Graphics. Note for example that we changed the posture of the robot between the captured scene on the bottom left and the composited scene on the right, all rendered using Gaussian Splatting rasterizer.

Updates


Abstract


We propose a method to allow precise and extremely fast mesh extraction from 3D Gaussian Splatting (SIGGRAPH 2023).
Gaussian Splatting has recently become very popular as it yields realistic rendering while being significantly faster to train than NeRFs. It is however challenging to extract a mesh from the millions of tiny 3D Gaussians as these Gaussians tend to be unorganized after optimization and no method has been proposed so far.

Our first key contribution is a regularization term that encourages the 3D Gaussians to align well with the surface of the scene.

We then introduce a method that exploits this alignment to sample points on the real surface of the scene and extract a mesh from the Gaussians using Poisson reconstruction, which is fast, scalable, and preserves details, in contrast to the Marching Cubes algorithm usually applied to extract meshes from Neural SDFs.

Finally, we introduce an optional refinement strategy that binds Gaussians to the surface of the mesh, and jointly optimizes these Gaussians and the mesh through Gaussian splatting rendering. This enables easy editing, sculpting, rigging, animating, or relighting of the Gaussians using traditional softwares (Blender, Unity, Unreal Engine, etc.) by manipulating the mesh instead of the Gaussians themselves.

Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on neural SDFs, while providing a better rendering quality in terms of PSNR, SSIM and LPIPS.


SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction


1. Aligning the Gaussians with the Surface

gaussian_arrangement.png

To facilitate the creation of a mesh from the Gaussians, we first propose a regularization term that encourages the Gaussians to be well distributed over the scene surface so that they capture much better the scene geometry.
Our approach is to derive a volume density from the Gaussians under the assumption that they are flat and well distributed over the scene surface. By minimizing the difference between this density and the actual one computed from the Gaussians during optimization, we encourage the 3D Gaussians to represent well the surface geometry.

2. Efficient Mesh Extraction

steps.jpg

Gaussian Splatting representations of Real scenes typically end up with one or several millions of 3D Gaussians with different scales and rotations, the majority of them being extremely small in order to reproduce texture and details in the scene. This results in a density function that is close to zero almost everywhere, and the Marching Cubes algorithm fails to extract proper level sets of such a sparse density function even with a fine voxel grid.
Instead, we introduce a method that very efficiently samples points on the visible part of a level set of the density function, allowing us to run the Poisson reconstruction algorithm on these points to obtain a triangle mesh. This approach is scalable, by contrast with the Marching Cubes algorithm for example, and reconstructs a highly detailed surface mesh within minutes on a single GPU, compared to other state of the art methods relying on Neural SDFs for extracting meshes from radiance fields.

3. Binding New 3D Gaussians to the Mesh

bound_gaussians.jpg

After extracting this mesh, we propose an optional refinement strategy: We bind new Gaussians to the mesh triangles and optimize the Gaussians and the mesh jointly using the Gaussian Splatting rasterizer. This optimization enables high-quality rendering of the mesh using Gaussian splatting rendering rather than traditional textured mesh rendering. We provide several examples below.
This results in higher performance in terms of rendering quality than other radiance field models relying on an underlying mesh at inference.

This binding strategy also makes possible the use of traditional mesh-editing tools for editing a Gaussian Splatting representation of a scene, offering endless possibilities for Computer Graphics.

Example of scene composition


In the following video, we show how to compose scenes in Blender using the meshes extracted with SuGaR. As we show at the end of the video, this composition can be rendered using the Gaussian Splatting rasterizer and the gaussians bound to the surface of the mesh.

Example of scene edition and character animation


In the following video, we show how we are able to animate a character in Blender using the meshes extracted by SuGaR. As we show at the end of the video, this animation can be rendered using the Gaussian Splatting rasterizer and the gaussians bound to the surface of the mesh.

In the following video, we show how we segment and rig a character using the meshes extracted with SuGaR.

Meshing and rendering scenes from Mip-NeRF360, DeepBlending and casual captures


Kitchen

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Garden

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Knight (casual capture with a smartphone)

kitchen_render.png Mesh with bound Gaussians
kitchen_mesh.png Mesh without texture

Robot (casual capture with a smartphone)

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Bonsai

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Playroom

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Counter

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Dr Johnson

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Room

kitchen_render.png Mesh rendered with bound Gaussians
kitchen_mesh.png Mesh without texture

Resources


Paper

Code

BibTeX

If you find this work useful for your research, please cite:
      @article{guedon2023sugar,
        title={SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering},
        author={Gu{\'e}don, Antoine and Lepetit, Vincent},
        journal={CVPR},
        year={2024}
      }

Further information


If you like this project, check out our previous works related to 3D reconstruction, where we introduce models that learn simultaneously to explore and reconstruct 3D environments from RGB images:

Acknowledgements


This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011013387R1 made by GENCI. We thank George Drettakis and Elliot Vincent for inspiring discussions and valuable feedback.

© You are welcome to copy the code of the webpage, please attribute the source with a link back to this page and remove the analytics.