Reality Capture Basics

As a part of the curriculum I’m taking through Studio Arts, I’m taking a survey course in virtual production and the use of Unreal Engine for previsualization. The class is taught by Damian Allen, a vfx supervisor and previs artist.

Key Software

The primary tool we used was Reality Capture, which was acquired by Epic Games and is now free for use with Unreal. Reality Capture is primarily for photogrammetry, and excels at generating 3D models based on reference images.

By comparison, another popular technique is gaussian splatting, which generates a point cloud much like photogrammetry, but then each point is used to render the final image. Gaussian splatting is also unique in that it’s capable of parsing transparency, and can render fine detail far beyond what photogrammetry can typically capture. Chaos Group has an excellent blog post about this technique. NeRF Studio is one of the primary tools used to train neural networks based on point cloud data to create a gaussian splat. NeRF in this case stands for Neural Radiance Fields. Gaussian splatting does not explicitly require a neural network to be used, but can benefit from it. If you’ve used a real estate website like Zillow and used their virtual tour features, you’ve already encountered a simple version of a gaussian splat, where they use a low detail gaussian splat to create an interactive tour of a room.

In addition to Reality Capture, it’s worth mentioning Meshroom, a popular open source tool for photogrammetry that relies on the AliceVision network for generating meshes based on input photographs.

Building a Desert Landscape in UE5

As a part of a class I’m taking through Studio Arts here in LA, taught by Milton Mariscal, I’m working on a cinematic in Unreal. The first key step to this is to setup a landscape environment to use as the backdrop for the animation and short film.

I ended up doing several of these, in part through trial and error and also because I had several ideas for environments I wanted to experiment with, each with their own biome and geology.

I have a few of these which I’ll document, the first of which is a desert landscape that I’ve been working on. I took inspiration in part from a trip I took to Uluru/Kata Tjuta National Park in Australia. Much like the American southwest, Northern Territory and Western Australia have some of the most remarkable ‘desertscapes’ and geology in the world.

To start I put together a mood board, shown below:

The center photos show Kata Tjuta, which is my primary reference for rock texture. I also included several photos of Monument Valley in Utah, which I used primarily for structural reference.

Painting the Landscape

Using only UE’s Landscape mode I was able to paint a complex organic landscape, including ground transitions and buttes much like the sandstone formations in Utah. I primarily relied on the Sculpt, Smooth, and Flatten tools to achieve the shape, as well as the Erosion and Noise brushes to introduce some indication of water erosion and general weathering. The landscape system isn’t a substitute for something like ZBrush or Houdini for procedural landscapes, but is great for quickly prototyping a base mesh.

Setting the Mood Through Lighting

My goal was to create a classic western sunset feel, drawing on reference from my time in Australia, as well as Los Angeles. Both Sydney and Los Angeles have very similar landscapes and sunset colors, though LA’s are much more influenced by its pollution levels. Air temperature, position relative to the equator, and climate all play a role in the appearance of the sky. In researching this I found that in fact Los Angeles and Sydney are roughly similar distance from the equator, at around 2,300 miles. While the seasons are opposite, because of their similar relative distance from the equator they may experience a similar angle of sunlight in the same season. In the image above you can see where I was experimenting more with the key light angle and intensity, as well as the atmospheric scattering.

A key control for determining the look of the sky in Unreal is the Rayleigh and Mie Scattering color. The Mie scattering color will determine the absorption for short wavelengths of light, while the Rayleigh scattering color determines the absorption of longer wavelengths of light. Put simply, when the sun is low in the sky it’s emitted light is scattered through a thicker layer of atmosphere. Therefore, the primary light that reaches the viewers eye is red, since that’s the longest wavelength of light in the visible spectrum. The result is a warmer look at the horizon at sunset. We can further adjust this by tinting the Rayleigh scattering color as well as the Mie scattering color on the Sky Atmosphere. Scratchapixel has a good overview of the history and science behind these algorithms.

Setting the rayeigh and mie scattering to blue and a slightly purple tone respectively results in a consistent warm tone on the ground and a warmer horizon.

The default clouds feel somewhat pixelated. I wanted a lighter, more abstract sky with softer cirrus type cloud. These were the settings I ended up with, including the scatter color settings.

Changing the sky fill level and reducing it as well as adjusting the sun intensity can yield a more dramatic higher contrast look.

Layered Materials for a Landscape

I pulled in a number of Quixel assets at raw quality to start generating a tileable, paintable material. I wanted to use a single shader for the landscape and blend between multiple materials to a convincing desert biome ground. One minor error I ran into is common in UE5, where it warns you about running out of texture streaming memory. To fix this I entered the following console command:

r.Streaming.PoolSize 3000

Prior to setting up my materials I also made a few quality changes. To work compromise a bit on render quality so I could work quicker I set my render settings to Epic, so that I could still preview the volumetric clouds. I also turned off auto exposure compensation so I could get a better sense of the landscape color and have a more consistent view while exploring the environment in play mode.

I clicked Settings > Project Settings, then searched for Auto Exposure to find this setting.  Leaving Auto Exposure Bias at 0.1 allows for some exposure compensation when looking at very bright highlights, but prevents the camera from constantly adjusting as I roam in and out of shadowed areas.

The next step is to put together a custom material network. The process is pretty simple - create a new material, right click, and edit. From there we can drag in our textures from the content drawer. In this case I’m starting with a rough desert rock texture as the basis. One important map to take note of is the ORM map - Occlusion, Roughness, and Displacement. To enable displacement we need to enable Nanite in UE5. To enable Nanite, we need to enable it in the Project Settings, as well as enabling the plugin.`

Enabling the Nanite plugin requires an engine restart.

We then need to close the engine, and edit some parameters in the .ini file under the config folder in our project director. Once we’ve done that we can then restart UE5 and work with Nanite displacement. There are 2 lines we need to add, shown below:

Allow Tessellation needs to be set to 1, or on, for Nanite tessellation to work.

UE5: Hotkeys

Navigation:

• RMB + WASD - navigate the camera, pan up/down/left/right

• F - frame on the selected object

• LMB - rotate view

• Alt + LMB - rotates view around the current view point

• Alt + LMB after F - rotates about the object once its framed

• Alt + RMB - dolly camera

• RMB + WASD - Game style movement

• LMB/RMB - Pan and rotate from current viewpoint

• Scroll wheel - zoom in and out

• CTRL + ` - toggles between world space and object/local space for transforms and rotation

Selection and Viewing:

• CTRL + LMB Click - select multiple objects in the outliner

• Q - selection key

• Alt + LMB drag - duplicate selection and move

• Double click on object in the Outliner - selects, frames on the object and locates it in the scene

• Alt + G - Perspective view

• Alt + H - Front view

• Alt + K - Side view

• Alt + J - Top view

• CTRL + 0-9 - Book mark camera view. Press the corresponding number to jump to that view

Level Editing:

• CTRL + Space - Opens the drawer

• Shift + 1-8 - switch between editing modes, ie. Selection, Landscape

• H - Hide selected

• CTRL + H - Unhide all hidden objects

• F11 - Immersive

• G - Game view

• CTRL + K - search in blue print

• End, or fn + End - Snap to floor, depending on your keyboard

• Alt + End - Snap pivot to floor

• CTRL + End - Snap origin to grid

• Shift + End - Snap bounds to floor

• ESC - clear selection

• ~ - Open command console

Viewport Shortcuts:

• F - focus on selection

• Alt + 2 - Wireframe view

• Alt + 3 - Unlit view

• Alt + 4 - Lit view

• Grid size decrease - [

• Grid size increase - ]

• Rotation size decrease - Shift + [

• Rotation size increase - Shift + ]

• Show mouse cursor in play mode - Shift + F1

Modes

• Shift + 1 - Select/Placement

• Shift + 2 - Landscape painting mode

• Shift + 3 - Foliage, paint foliage placement interactively using source assets

• Shift + 4 - Mesh paint

• Shift + 5 - Modelling toolset

• Shift + 6 - Fracture

• Shift + 7 - Brush editing, for editing box brushes, etc.

• Shift + 8 - Animation toolset

UI

• CTRL + Shift + W - opens the widget reflector, allowing you to globally change the application scale including icons and font