Translating the Gooch shading equation to HLSL
In Part One of this short series I’m putting together I talked about the basis for stylized shader, which is essentially a Gooch shading model. Part One
Gooch style shading, developed primarily by Amy Gooch and Bruce Gooch, has origins in technical illustration, where visual clarity is critical. Unlike Lambertian shading, there is no falloff to black in the shadow areas, but rather a predetermined color value. This creates a feeling of indirect bounce lighting, and is inherently similar to a cel shaded or two tone color look. Many stylized games seem to make use of this style of shading, where colors in the midtones and shadow regions are carefully controlled to create a softer look that mimics global illumination, without the use of expensive GI calculations or even baked solutions - it can all be achieved in shader.
Acerola has an excellent breakdown of this shading model on his tech art focused YouTube channel. I took some inspiration from this video, along with the book Non Photorealistic Rendering that I mentioned in Part One.
Below I show 2 samples of the Gooch component of the shader, which creates the soft gradient color transition on the ground plane. This serves 2 purposes - as mentioned, it’s aesthetically somewhat similar to global illumination, and it also adds visual clarity, distinguishing the curvature of the ground plane. Normals pointing upward get a lighter color green, while normals pointing away from the light vector direction of the keylight get a darker teal/green tone rather than black in Lambertian shading. To me this is a hallmark of stylized games, where the player has visual clarity and shadows are generally defined by darker monochromatic colors.
Preview image showing the Gooch shader applied to a ground plane, showing a transition from cool to warm colors in this case. In the background a ground plane shows a transition from a teal to light green in a more traditional monochromatic color scheme.
To create this effect, I needed to translate the original equation into format that would work with Unreal Engine’s node based shading workflow. Effectively, with the node based implementation of HLSL in UE5, you need to do operations in reverse. So our goal is get color final as the output.
The ‘cool’ color or top color equation is: 1 + the dot product of the incoming light vector, dotted with the surface normal, all divided by 2, multiplied by the kcool RGB color value. This results in a gradient, where as the normal direction is further away from the sun vector, the influence of the cool color decays to zero, and we see more of the ‘warm’ color.
The ‘warm’ color or shadow side color equation is effectively: 1 - (1 + the dot product of the incoming light vector dotted with the surface normal, divided by 2) multiplied by kwarm. The result is then where the surface normal is pointing away from the sun, we transition to the warm color. The result of this and the corresponding HLSL nodes are shown below.
Preview of the Gooch shading algorithm on a sphere, showing the soft transition from the cool color to warm color in the shadows. In this case I made the warm color quite bright to illustrate the creative control it allows over the shadow appearance.
There are many components to this shader. These are the key features I wanted to implement:
Cross hatching effect to emphasize shadowed areas and form of the object
A stylized outline similar to cel shading that would vary in thickness based on the object controu
A secondary post process outline, uniform in thickness to help increase visual clarity
A stylized specular response, similar to a hand drawn highlight, with a clean circular look
Posterization, where the user can introduce a posterize or banding effect in the shader, to make it more similar to a graphic cel shaded look
An overall specular component that will allow the user to differentiate material types while still looking stylized. Similar to the effect seen in Breath of the Wild, where many object have a broader specular layer.
The complete shader graph. By combining many simple components, I was able to come up with something that had all the features and artist control I wanted to implement to start.
Cross hatching in HLSL using a simple sine wave
Above I show the complete shader network. Highlighted in red is the Gooch component I just explained in the previous section. Below to the left is the next key component, the cross hatch effect.
The goal of this as mentioned was to allow the user to emphasize the shadow regions of an object with a hatching effect that actually follows the geometry contours. There are many implementations of a hatch shader as a post process effect that look excellent. But temporal stability is poor generally, with obvious swimming and other artifacts as the player moves. Because of this, I chose to implement this effect in shader, using the surface normals and luminance.
I got the basics of the idea from this thread on the Nuke forums, where users were discussing how to achieve a Spiderverse style hatching effect: Halftone Gizmo.
I’ve implemented this in Nuke in the past. The concept is simple, but can be expanded upon to create some interesting artistic effects. Effectively all I’m doing is feeding a Linear Gradient node into a Sine wave, and multiplying it by a Constant in HLSL to control the frequence of the wave, as shown below:
Here I’m previewing the Linear Sine node applied to a sphere, which generates a hatched texture. Hatch Up Angle controls the direction of the hatching. I later use the surface normal and pixel normal to isolate the darker midtone areas where I want the hatch effect to appear.
The next step is to restrict the hatching effect. In this case I didn’t want excessively dark, dense shadowing, but mainly wanted to use the hatching to highlight the contours of an object and add definition to the mid tones. I use the nodes labeled Black Point and White Point to effectively control the contrast. Below I show how I used the dot product of the Vertex Normal and Sun Direction and multiply that against the Pixel Normal green component(the shadow direction) to isolate the midtones. The result is a black and white mask with a falloff that I multiply against the incoming hatch texture to create what’s shown below.
A mask isolating the mid tone shadow regions.
Previewing the hatch color. This will be multiplied against the original to allow the user to tint the color of hatching. In this case I’m keeping it monochromatic and just choosing a darker, less saturated version of the input.
Cel shading banding effects, edge detection, and stylized specular
The final portion of the shader applies a banding effect to the incoming Gooch shading color, an edge outline, false stylized spec highlight, and a general spec layer. Adding these on top of the incoming result enhances the look, and gets a bit closer to
Combining the hatched effect with banding and the incoming color yields a more complex toon shaded result.
Creating the banding effect is fairly simple. Again I take the dot product of the incoming Sun Vector compared with the Pixel Normal in World Space, rather than the Vertex Normal. I then multiply that by a Step Number, in this case 8, for 8 bands and divide by the Step Number and Clamp. The end result is a stair stepped color value. Rather than a smooth gradient, it clamps to 8 different color value, yielding a stylized ‘lofi’ shaded result. This contributes to more of a ‘low poly’ feel that simplifies the look of objects in the scene.
Above is the result of my edge detection setup, which is really just a slight modification of a Fresnel node.
To create an edge detection effect, we can use a Fresnel node. However like Lambertian and Gooch shading, the result is relatively smooth initially. To create a hard edge outline, I again implement the same Clamp, Divide, and Round to create a banding effect to isolate the Fresnel to a restricted area. The Exponent Edge Thickness is just a float constant that I use to control the thickness of the outline. On a basic sphere shape, the outline is consistent. But on an object with more surface variation, the Pixel Normal will influence the apparent edge thickness. The banding effect in this case really only has 1 step, because we only want to return a black outline and a white interior, rather than multiple steps of gradation in value. Color Tint Outline is a color parameter I’m using to tint the outline so that’s its not fully black. Much like in Breath of the Wild, I chose the outline color based on the base color, in this case a brown tone to compliment the original orange value.
Adding a round toon style specular highlight and overall specular.
To create this spec hit I again add the Sun Vector to the Camera Vector, then Normalize it and take the Dot product compared with the Pixel Normal. A Saturate node ensures that the value range stays within 0 to 1. I then use a Power node and feed it a float constant I labeled Specular Power, then Round and Clamp the output. Taking the power to 12 restricts the highlight to a small area.
The stylized spec result.
Combining all these elements together yields an interesting stylized result. The final component I add on top is a stylized fresnel rim. Again I use the Fresnel node, clamp it and restrict to the top edge in the direction of the sun, and multiply it against a preset color.
Adding a separate edge fresnel rim to define the edges of the sphere.
Putting all together, in the video below I preview the result of the material applied to a 3d headscan to test the parameters.
In Part Three I’ll demonstrate using this material to look dev a stylized environment for a game, and also go into some post process materials to complete the effect.