We propose to represent surfaces by a mesh covered with a "Frosting" layer of varying thickness and made of 3D Gaussians.
This representation captures both complex volumetric effects created by fuzzy materials such as hair or grass as well as flat surfaces.
Built from RGB images only, it can be rendered in real-time and animated using traditional animation tools.
We propose Gaussian Frosting–or Frosting, for short–, a novel mesh-based representation for high-quality rendering
and editing of complex 3D effects in real-time.
Our approach builds on the recent
3D Gaussian Splatting (SIGGRAPH 2023) framework,
which optimizes a set of 3D Gaussians to approximate a radiance field from images.
We propose first extracting a base mesh from Gaussians during optimization, then building and refining
an adaptive layer of Gaussians with a variable thickness around the mesh to better capture the fine
details and volumetric effects near the surface, such as hair or grass. We call this layer Gaussian Frosting,
as it resembles a coating of frosting on a cake. The fuzzier the material, the thicker the frosting.
We also introduce a parameterization of the Gaussians to enforce them to stay inside the frosting layer and
automatically adjust their parameters when deforming, rescaling, editing or animating the mesh.
Our representation allows for efficient rendering using Gaussian splatting, as well as editing
and animation by modifying the base mesh.
We demonstrate the effectiveness of our method on various synthetic and real scenes,
and show that it outperforms existing surface-based approaches.
(a) Scene 1: Bicycle
(b) Scene 2: Buzz
(c) Scene 3: Kitten
(d) Posing meshes
(e) Rendering composition
In the example above, we were able to animate both Buzz and the kitten, changing their original pose (d) while preserving high-quality rendering (e). Contrary to SuGaR (g), very fine and fuzzy details such as the kitten's hair can be seen covering Buzz's legs in a realistic way (f):
(f) Fuzzy details with Frosting - occlusions are correctly rendered
(g) Rendering with SuGaR - the fur does not occlude the legs correctly
(a) Scene 1: Stump
(b) Scene 2: Knight
(c) Scene 3: Horse
(d) Posing meshes
(e) Rendering composition
(a) A knight practicing fencing
(b) Buzz dancing on a bench
(a) Rendering
(b) Normals
Frosting reaches better rendering performance than other editable radiance field methods, and obtains competitive results compared to vanilla 3D Gaussian Splatting. Frosting is even able to outperform vanilla 3DGS when reconstructing scenes with many fuzzy materials, such as the scenes from the Shelly dataset.
(c) Ground Truth
(d) 3D Gaussian Splatting
(e) Frosting (Ours)
We start by optimizing a 3D Gaussian Splatting reconstruction for a short period of time and we extract an editable surface mesh with optimal resolution. We improve the surface reconstruction from SuGaR by automatically estimating a good value for a critical hyperparameter used by Poisson reconstruction, namely the octree depth D. Selecting the right value for D can drastically improve both the quality of the mesh and the rendering performance of our model.
(a) Rendering
(b) Thickness of the Frosting layer
After extracting a base mesh, we build a Frosting layer with a variable thickness and containing Gaussians around this mesh. We want this layer to be thicker in areas where more volumetric rendering is necessary near the surface, such as fuzzy material like hair or grass for example. On the contrary, this layer should be very thin near the parts of the scene that corresponds to well-defined flat surfaces, such as wood or plastic for example.
(a) Original pose
(b) Edited pose
Once we constructed the Frosting layer, we initialize a densified set of Gaussians inside this layer and optimize them using 3DGS rendering loss. To make sure the Gaussians stay inside the frosting layer during optimization, we introduce a new parameterization of the Gaussians. This parameterization also allows for easily adjusting the Gaussians' parameters when editing the scene and animating characters.
@article{guedon2024frosting, title={Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering}, author={Gu{\'e}don, Antoine and Lepetit, Vincent}, journal={ECCV}, year={2024} }
This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011013387R1 made by GENCI.
© You are welcome to copy the code of the webpage, please attribute the source with a link
back to this page and remove the analytics.