Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering

ECCV 2024 (Oral)

Antoine GuédonVincent Lepetit

LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS


We propose to represent surfaces by a mesh covered with a "Frosting" layer of varying thickness and made of 3D Gaussians. This representation captures both complex volumetric effects created by fuzzy materials such as hair or grass as well as flat surfaces. Built from RGB images only, it can be rendered in real-time and animated using traditional animation tools.

Updates



Abstract


We propose Gaussian Frosting–or Frosting, for short–, a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time.

Our approach builds on the recent 3D Gaussian Splatting (SIGGRAPH 2023) framework, which optimizes a set of 3D Gaussians to approximate a radiance field from images. We propose first extracting a base mesh from Gaussians during optimization, then building and refining an adaptive layer of Gaussians with a variable thickness around the mesh to better capture the fine details and volumetric effects near the surface, such as hair or grass. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. The fuzzier the material, the thicker the frosting.

We also introduce a parameterization of the Gaussians to enforce them to stay inside the frosting layer and automatically adjust their parameters when deforming, rescaling, editing or animating the mesh. Our representation allows for efficient rendering using Gaussian splatting, as well as editing and animation by modifying the base mesh.

We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches.


Scene composition


Example 1: Buzz riding a giant kitten

(a) Scene 1: Bicycle

(b) Scene 2: Buzz

(c) Scene 3: Kitten

buzz_kitten_composition.png

(d) Posing meshes

(e) Rendering composition


In the example above, we were able to animate both Buzz and the kitten, changing their original pose (d) while preserving high-quality rendering (e). Contrary to SuGaR (g), very fine and fuzzy details such as the kitten's hair can be seen covering Buzz's legs in a realistic way (f):

gt_detail.png
3dgs_detail.png
gt_detail.png
3dgs_detail.png

(f) Fuzzy details with Frosting - occlusions are correctly rendered

(g) Rendering with SuGaR - the fur does not occlude the legs correctly


Example 2: Knight resting in the forest

(a) Scene 1: Stump

(b) Scene 2: Knight

(c) Scene 3: Horse

knight_horse_composition.png

(d) Posing meshes

(e) Rendering composition


Animation


(a) A knight practicing fencing

(b) Buzz dancing on a bench


Reconstructing complex scenes with fuzzy materials


(a) Rendering

sleepycat_normals.png

(b) Normals


Frosting reaches better rendering performance than other editable radiance field methods, and obtains competitive results compared to vanilla 3D Gaussian Splatting. Frosting is even able to outperform vanilla 3DGS when reconstructing scenes with many fuzzy materials, such as the scenes from the Shelly dataset.

gt_detail.png

(c) Ground Truth

3dgs_detail.png

(d) 3D Gaussian Splatting

frosting_detail.png

(e) Frosting (Ours)


Gaussian Frosting: Overview


1. Forward Process: From Volume to Surface

khady_mesh.png
pug_mesh.png
horse_mesh.png
kitten_mesh.png
(a) Using the predefined, large parameter D as in SuGaR
khady_mesh.png
pug_mesh.png
horse_mesh.png
kitten_mesh.png
(b) Using our automatically computed D that adapts to the complexity of the 3DGS


We start by optimizing a 3D Gaussian Splatting reconstruction for a short period of time and we extract an editable surface mesh with optimal resolution. We improve the surface reconstruction from SuGaR by automatically estimating a good value for a critical hyperparameter used by Poisson reconstruction, namely the octree depth D. Selecting the right value for D can drastically improve both the quality of the mesh and the rendering performance of our model.

2. Backward Process: From Surface to Volume

panda_rgb.png

(a) Rendering

panda_thickness.png

(b) Thickness of the Frosting layer

After extracting a base mesh, we build a Frosting layer with a variable thickness and containing Gaussians around this mesh. We want this layer to be thicker in areas where more volumetric rendering is necessary near the surface, such as fuzzy material like hair or grass for example. On the contrary, this layer should be very thin near the parts of the scene that corresponds to well-defined flat surfaces, such as wood or plastic for example.

3. Frosting optimization and edition

original_pose.png

(a) Original pose

edited_pose.png

(b) Edited pose

Once we constructed the Frosting layer, we initialize a densified set of Gaussians inside this layer and optimize them using 3DGS rendering loss. To make sure the Gaussians stay inside the frosting layer during optimization, we introduce a new parameterization of the Gaussians. This parameterization also allows for easily adjusting the Gaussians' parameters when editing the scene and animating characters.


BibTex


If you find this work useful for your research, please cite:

      @article{guedon2024frosting,
        title={Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering},
        author={Gu{\'e}don, Antoine and Lepetit, Vincent},
        journal={ECCV},
        year={2024}
      }


Further information


If you like this project, check out our previous works related to 3D reconstruction:


Acknowledgements


This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011013387R1 made by GENCI.

© You are welcome to copy the code of the webpage, please attribute the source with a link back to this page and remove the analytics.