Stylized Shading in UE5: Part Three

In the previous Part Two installment I documented each component of the stylized material I made. By combining and layering together a lot of simple concepts you can easily make a more complex network that’s visually interesting.

In this section I want to expand upon that further and demonstrate the parameters, document the post process materials I developed to enhance the look, and finally bring everything together in a simple test level using the Third Person template, where we’ll run around a stylized woodland.

The models I used are from Quaternius • Free Game Assets. I could make some simple assets of course, but my feeling was it’s even more valuable to test a shader on assets that you didn’t model yourself, to ensure that it performs well regardless of the geometry you give it. I also like the style of the models he makes. They’re CC0 Creative Commons license, but I want to give credit as he was nice to give his permission for me to use them in this piece.

Post Process Shaders

In addition to the base stylized material, I wanted to experiment with some post process materials. In Part Two I showed a turn table of the shader where the outline width can vary slightly depending on the surface topology and the viewing angle using a fresnel effect. In addition to this I wanted to experiment with the following:

• General outline shader

• Desaturation based on player distance from an object

• Desaturation of the ground base on player location, where the ground is desaturated after a certain falloff distance from the player

• Alternative texturing of the shadows, where the shadows can have a hatched effect

For the alternative texturing of shadows, the goal was to not only have the object self shadowing be hatched, but also the ground shadows, or allow the user to feed in a custom texture. I got the idea for this from a post by Tom Looman: Textured Shadows Trick in Unreal Engine - Tom Looman. In this post he demonstrates how to access the Light Attenuation buffer and query an objects UVs to project a texture into the shadow regions. The Light Attenuation buffer is technically no longer accessible in UE5 to my knowledge, at least not easily. Render Bucket, an excellent YouTube channel details an alternative workflow for creating an equivalent ‘shadow catcher’ pass: Unreal Engine 5+ Shadow Catcher / Shadow Pass (youtube.com). Effectively

Desaturation of distant object and areas on the ground is easy to capture by sampling the depth buffer and applying an Overlay material to relevant objects. Overlay materials can be prohibitive in terms of performance, but in this case the operation is simple, where I’m just desaturating the primary color.

An outline material relies on basic edge detection. There are a number of algorithms that allow for this, generally all relying on kernel convolution. A notable operator is the Sobel filter: Finding the Edges (Sobel Operator) - Computerphile (youtube.com). Laplachian style filtering generates a more complex edge with more interior details. Edge Detection Using Laplacian | Edge Detection (youtube.com) In this case I really only wanted the external edge, so relied on a Sobel style operator.

A key source for ideas on how to implement these effects in UE5 came from a GDC 2024 talk that actually released while I was in the midst of working on this. Chris Murphy of Epic Games covered a huge amount of information in just 30 minutes: Simple Stylization Techniques in Unreal Engine | GDC 2024 - YouTube. This talk shed light on how to access the correct frame buffers to use in my post process shaders.

Outline Shader

As mentioned the main additional post process effect I wanted was an outline material, to compliment the edge detection in the base material. By creating a more uniform black outline my feeling was this would enhance the read of objects and improve visual clarity.

Creating an outline material is fairly simple. We can change the material assigned to the Post Process Volume to overlay an outline over the entire scene by querying the incoming color and implementing an edge detection algorithm. It sounds complicated, but it’s actually very easy to implement in UE’s shading network.

Stylized Shading in UE5: Part Two

Translating the Gooch shading equation to HLSL

In Part One of this short series I’m putting together I talked about the basis for stylized shader, which is essentially a Gooch shading model. Part One

Gooch style shading, developed primarily by Amy Gooch and Bruce Gooch, has origins in technical illustration, where visual clarity is critical. Unlike Lambertian shading, there is no falloff to black in the shadow areas, but rather a predetermined color value. This creates a feeling of indirect bounce lighting, and is inherently similar to a cel shaded or two tone color look. Many stylized games seem to make use of this style of shading, where colors in the midtones and shadow regions are carefully controlled to create a softer look that mimics global illumination, without the use of expensive GI calculations or even baked solutions - it can all be achieved in shader.

Acerola has an excellent breakdown of this shading model on his tech art focused YouTube channel. I took some inspiration from this video, along with the book Non Photorealistic Rendering that I mentioned in Part One.

Below I show 2 samples of the Gooch component of the shader, which creates the soft gradient color transition on the ground plane. This serves 2 purposes - as mentioned, it’s aesthetically somewhat similar to global illumination, and it also adds visual clarity, distinguishing the curvature of the ground plane. Normals pointing upward get a lighter color green, while normals pointing away from the light vector direction of the keylight get a darker teal/green tone rather than black in Lambertian shading. To me this is a hallmark of stylized games, where the player has visual clarity and shadows are generally defined by darker monochromatic colors.

Preview image showing the Gooch shader applied to a ground plane, showing a transition from cool to warm colors in this case.  In the background a ground plane shows a transition from a teal to light green in a more traditional monochromatic color scheme.

To create this effect, I needed to translate the original equation into format that would work with Unreal Engine’s node based shading workflow. Effectively, with the node based implementation of HLSL in UE5, you need to do operations in reverse. So our goal is get color final as the output.

The ‘cool’ color or top color equation is: 1 + the dot product of the incoming light vector, dotted with the surface normal, all divided by 2, multiplied by the kcool RGB color value. This results in a gradient, where as the normal direction is further away from the sun vector, the influence of the cool color decays to zero, and we see more of the ‘warm’ color.

The ‘warm’ color or shadow side color equation is effectively: 1 - (1 + the dot product of the incoming light vector dotted with the surface normal, divided by 2) multiplied by kwarm. The result is then where the surface normal is pointing away from the sun, we transition to the warm color. The result of this and the corresponding HLSL nodes are shown below.

Preview of the Gooch shading algorithm on a sphere, showing the soft transition from the cool color to warm color in the shadows.  In this case I made the warm color quite bright to illustrate the creative control it allows over the shadow appearance.

There are many components to this shader. These are the key features I wanted to implement:

• Cross hatching effect to emphasize shadowed areas and form of the object

• A stylized outline similar to cel shading that would vary in thickness based on the object controu

• A secondary post process outline, uniform in thickness to help increase visual clarity

• A stylized specular response, similar to a hand drawn highlight, with a clean circular look

• Posterization, where the user can introduce a posterize or banding effect in the shader, to make it more similar to a graphic cel shaded look

• An overall specular component that will allow the user to differentiate material types while still looking stylized. Similar to the effect seen in Breath of the Wild, where many object have a broader specular layer.

The complete shader graph.  By combining many simple components, I was able to come up with something that had all the features and artist control I wanted to implement to start.

Cross hatching in HLSL using a simple sine wave

Above I show the complete shader network. Highlighted in red is the Gooch component I just explained in the previous section. Below to the left is the next key component, the cross hatch effect.

The goal of this as mentioned was to allow the user to emphasize the shadow regions of an object with a hatching effect that actually follows the geometry contours. There are many implementations of a hatch shader as a post process effect that look excellent. But temporal stability is poor generally, with obvious swimming and other artifacts as the player moves. Because of this, I chose to implement this effect in shader, using the surface normals and luminance.

I got the basics of the idea from this thread on the Nuke forums, where users were discussing how to achieve a Spiderverse style hatching effect: Halftone Gizmo.

I’ve implemented this in Nuke in the past. The concept is simple, but can be expanded upon to create some interesting artistic effects. Effectively all I’m doing is feeding a Linear Gradient node into a Sine wave, and multiplying it by a Constant in HLSL to control the frequence of the wave, as shown below:

Here I’m previewing the Linear Sine node applied to a sphere, which generates a hatched texture.  Hatch Up Angle controls the direction of the hatching.  I later use the surface normal and pixel normal to isolate the darker midtone areas where I want the hatch effect to appear.

The next step is to restrict the hatching effect. In this case I didn’t want excessively dark, dense shadowing, but mainly wanted to use the hatching to highlight the contours of an object and add definition to the mid tones. I use the nodes labeled Black Point and White Point to effectively control the contrast. Below I show how I used the dot product of the Vertex Normal and Sun Direction and multiply that against the Pixel Normal green component(the shadow direction) to isolate the midtones. The result is a black and white mask with a falloff that I multiply against the incoming hatch texture to create what’s shown below.

A mask isolating the mid tone shadow regions.

Previewing the hatch color.  This will be multiplied against the original to allow the user to tint the color of hatching.  In this case I’m keeping it monochromatic and just choosing a darker, less saturated version of the input. 

Cel shading banding effects, edge detection, and stylized specular

The final portion of the shader applies a banding effect to the incoming Gooch shading color, an edge outline, false stylized spec highlight, and a general spec layer. Adding these on top of the incoming result enhances the look, and gets a bit closer to

Combining the hatched effect with banding and the incoming color yields a more complex toon shaded result.

Creating the banding effect is fairly simple. Again I take the dot product of the incoming Sun Vector compared with the Pixel Normal in World Space, rather than the Vertex Normal. I then multiply that by a Step Number, in this case 8, for 8 bands and divide by the Step Number and Clamp. The end result is a stair stepped color value. Rather than a smooth gradient, it clamps to 8 different color value, yielding a stylized ‘lofi’ shaded result. This contributes to more of a ‘low poly’ feel that simplifies the look of objects in the scene.

Above is the result of my edge detection setup, which is really just a slight modification of a Fresnel node.

To create an edge detection effect, we can use a Fresnel node. However like Lambertian and Gooch shading, the result is relatively smooth initially. To create a hard edge outline, I again implement the same Clamp, Divide, and Round to create a banding effect to isolate the Fresnel to a restricted area. The Exponent Edge Thickness is just a float constant that I use to control the thickness of the outline. On a basic sphere shape, the outline is consistent. But on an object with more surface variation, the Pixel Normal will influence the apparent edge thickness. The banding effect in this case really only has 1 step, because we only want to return a black outline and a white interior, rather than multiple steps of gradation in value. Color Tint Outline is a color parameter I’m using to tint the outline so that’s its not fully black. Much like in Breath of the Wild, I chose the outline color based on the base color, in this case a brown tone to compliment the original orange value.

Adding a round toon style specular highlight and overall specular.

To create this spec hit I again add the Sun Vector to the Camera Vector, then Normalize it and take the Dot product compared with the Pixel Normal. A Saturate node ensures that the value range stays within 0 to 1. I then use a Power node and feed it a float constant I labeled Specular Power, then Round and Clamp the output. Taking the power to 12 restricts the highlight to a small area.

The stylized spec result.

Combining all these elements together yields an interesting stylized result. The final component I add on top is a stylized fresnel rim. Again I use the Fresnel node, clamp it and restrict to the top edge in the direction of the sun, and multiply it against a preset color.

Adding a separate edge fresnel rim to define the edges of the sphere.

Putting all together, in the video below I preview the result of the material applied to a 3d headscan to test the parameters.
In Part Three I’ll demonstrate using this material to look dev a stylized environment for a game, and also go into some post process materials to complete the effect.