Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering

arXiv 2024

Antoine GuédonVincent Lepetit

LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS

We propose to represent surfaces by a mesh covered with a ``Frosting'' layer of varying thickness and made of 3D Gaussians. This representation captures both complex volumetric effects created by fuzzy materials such as hair or grass as well as flat surfaces. Built from RGB images only, it can be rendered in real-time and animated using traditional animation tools.



We propose Gaussian Frosting–or Frosting, for short–, a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time.

Our approach builds on the recent 3D Gaussian Splatting (SIGGRAPH 2023) framework, which optimizes a set of 3D Gaussians to approximate a radiance field from images. We propose first extracting a base mesh from Gaussians during optimization, then building and refining an adaptive layer of Gaussians with a variable thickness around the mesh to better capture the fine details and volumetric effects near the surface, such as hair or grass. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. The fuzzier the material, the thicker the frosting.

We also introduce a parameterization of the Gaussians to enforce them to stay inside the frosting layer and automatically adjust their parameters when deforming, rescaling, editing or animating the mesh. Our representation allows for efficient rendering using Gaussian splatting, as well as editing and animation by modifying the base mesh.

We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches.

Scene composition

Example 1: Buzz riding a giant kitten

(a) Scene 1: Bicycle

(b) Scene 2: Buzz

(c) Scene 3: Kitten


(d) Posing meshes

(e) Rendering composition

In the example above, we were able to animate both Buzz and the kitten, changing their original pose (d) while preserving high-quality rendering (e). Contrary to SuGaR (g), very fine and fuzzy details such as the kitten's hair can be seen covering Buzz's legs in a realistic way (f):


(f) Fuzzy details with Frosting - occlusions are correctly rendered

(g) Rendering with SuGaR - the fur does not occlude the legs correctly

Example 2: Knight resting in the forest

(a) Scene 1: Stump

(b) Scene 2: Knight

(c) Scene 3: Horse


(d) Posing meshes

(e) Rendering composition


(a) A knight practicing fencing

(b) Buzz dancing on a bench

Reconstructing complex scenes with fuzzy materials

(a) Rendering


(b) Normals

Frosting reaches better rendering performance than other editable radiance field methods, and obtains competitive results compared to vanilla 3D Gaussian Splatting. Frosting is even able to outperform vanilla 3DGS when reconstructing scenes with many fuzzy materials, such as the scenes from the Shelly dataset.


(c) Ground Truth


(d) 3D Gaussian Splatting


(e) Frosting (Ours)

Gaussian Frosting: Overview

1. Forward Process: From Volume to Surface

(a) Using the predefined, large parameter D as in SuGaR
(b) Using our automatically computed D that adapts to the complexity of the 3DGS

We start by optimizing a 3D Gaussian Splatting reconstruction for a short period of time and we extract an editable surface mesh with optimal resolution. We improve the surface reconstruction from SuGaR by automatically estimating a good value for a critical hyperparameter used by Poisson reconstruction, namely the octree depth D. Selecting the right value for D can drastically improve both the quality of the mesh and the rendering performance of our model.

2. Backward Process: From Surface to Volume


(a) Rendering


(b) Thickness of the Frosting layer

After extracting a base mesh, we build a Frosting layer with a variable thickness and containing Gaussians around this mesh. We want this layer to be thicker in areas where more volumetric rendering is necessary near the surface, such as fuzzy material like hair or grass for example. On the contrary, this layer should be very thin near the parts of the scene that corresponds to well-defined flat surfaces, such as wood or plastic for example.

3. Frosting optimization and edition


(a) Original pose


(b) Edited pose

Once we constructed the Frosting layer, we initialize a densified set of Gaussians inside this layer and optimize them using 3DGS rendering loss. To make sure the Gaussians stay inside the frosting layer during optimization, we introduce a new parameterization of the Gaussians. This parameterization also allows for easily adjusting the Gaussians' parameters when editing the scene and animating characters.





If you find this work useful for your research, please cite:
        title={Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering},
        author={Gu{\'e}don, Antoine and Lepetit, Vincent},
        journal={arXiv preprint arXiv:2403.14554},

Further information

If you like this project, check out our previous works related to 3D reconstruction:


This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011013387R1 made by GENCI.

© You are welcome to copy the code of the webpage, please attribute the source with a link back to this page and remove the analytics.