Updates

• 03/22/2024: Added examples of animation to the presentation video and the web page.
• 03/21/2024: Initial release of the paper.

Abstract

We propose Gaussian Frosting–or Frosting, for short–, a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time.

Our approach builds on the recent 3D Gaussian Splatting (SIGGRAPH 2023) framework, which optimizes a set of 3D Gaussians to approximate a radiance field from images. We propose first extracting a base mesh from Gaussians during optimization, then building and refining an adaptive layer of Gaussians with a variable thickness around the mesh to better capture the fine details and volumetric effects near the surface, such as hair or grass. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. The fuzzier the material, the thicker the frosting.

We also introduce a parameterization of the Gaussians to enforce them to stay inside the frosting layer and automatically adjust their parameters when deforming, rescaling, editing or animating the mesh. Our representation allows for efficient rendering using Gaussian splatting, as well as editing and animation by modifying the base mesh.

We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches.

Scene composition

Example 1: Buzz riding a giant kitten

(d) Posing meshes

(e) Rendering composition

In the example above, we were able to animate both Buzz and the kitten, changing their original pose (d) while preserving high-quality rendering (e). Contrary to SuGaR (g), very fine and fuzzy details such as the kitten's hair can be seen covering Buzz's legs in a realistic way (f):

Example 2: Knight resting in the forest

(d) Posing meshes

(e) Rendering composition

Animation

Reconstructing complex scenes with fuzzy materials

(a) Rendering

(b) Normals

Frosting reaches better rendering performance than other editable radiance field methods, and obtains competitive results compared to vanilla 3D Gaussian Splatting. Frosting is even able to outperform vanilla 3DGS when reconstructing scenes with many fuzzy materials, such as the scenes from the Shelly dataset.

(c) Ground Truth

(d) 3D Gaussian Splatting

(e) Frosting (Ours)

Gaussian Frosting: Overview

1. Forward Process: From Volume to Surface

(a) Using the predefined, large parameter D as in SuGaR

(b) Using our automatically computed D that adapts to the complexity of the 3DGS

We start by optimizing a 3D Gaussian Splatting reconstruction for a short period of time and we extract an editable surface mesh with optimal resolution. We improve the surface reconstruction from SuGaR by automatically estimating a good value for a critical hyperparameter used by Poisson reconstruction, namely the octree depth D. Selecting the right value for D can drastically improve both the quality of the mesh and the rendering performance of our model.

2. Backward Process: From Surface to Volume

(a) Rendering

(b) Thickness of the Frosting layer

After extracting a base mesh, we build a Frosting layer with a variable thickness and containing Gaussians around this mesh. We want this layer to be thicker in areas where more volumetric rendering is necessary near the surface, such as fuzzy material like hair or grass for example. On the contrary, this layer should be very thin near the parts of the scene that corresponds to well-defined flat surfaces, such as wood or plastic for example.

3. Frosting optimization and edition

(a) Original pose

(b) Edited pose

Once we constructed the Frosting layer, we initialize a densified set of Gaussians inside this layer and optimize them using 3DGS rendering loss. To make sure the Gaussians stay inside the frosting layer during optimization, we introduce a new parameterization of the Gaussians. This parameterization also allows for easily adjusting the Gaussians' parameters when editing the scene and animating characters.

Resources

BibTeX

      @article{guedon2024frosting,
        title={Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering},
        author={Gu{\'e}don, Antoine and Lepetit, Vincent},
        journal={arXiv preprint arXiv:2403.14554},
        year={2024}
      }

Further information

• Guédon et Lepetit. - SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (CVPR 2024)
• Guédon et al. - MACARONS: Mapping And Coverage Anticipation with RGB ONline Self-supervision (CVPR 2023)
• Guédon et al. - SCONE: Surface Coverage Optimization in Unknown Environments by Volumetric Integration (NeurIPS 2022, Spotlight)

Acknowledgements

This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011013387R1 made by GENCI.

© You are welcome to copy the code of the webpage, please attribute the source with a link back to this page and remove the analytics.