As a part of the curriculum I’m taking through Studio Arts, I’m taking a survey course in virtual production and the use of Unreal Engine for previsualization. The class is taught by Damian Allen, a vfx supervisor and previs artist.
Key Software
The primary tool we used was Reality Capture, which was acquired by Epic Games and is now free for use with Unreal. Reality Capture is primarily for photogrammetry, and excels at generating 3D models based on reference images.
By comparison, another popular technique is gaussian splatting, which generates a point cloud much like photogrammetry, but then each point is used to render the final image. Gaussian splatting is also unique in that it’s capable of parsing transparency, and can render fine detail far beyond what photogrammetry can typically capture. Chaos Group has an excellent blog post about this technique. NeRF Studio is one of the primary tools used to train neural networks based on point cloud data to create a gaussian splat. NeRF in this case stands for Neural Radiance Fields. Gaussian splatting does not explicitly require a neural network to be used, but can benefit from it. If you’ve used a real estate website like Zillow and used their virtual tour features, you’ve already encountered a simple version of a gaussian splat, where they use a low detail gaussian splat to create an interactive tour of a room.
In addition to Reality Capture, it’s worth mentioning Meshroom, a popular open source tool for photogrammetry that relies on the AliceVision network for generating meshes based on input photographs.