Hello again! I wrote my last technobabble on a paint effect shader for our LD39 jam entry, SPACEJAMMED, a few days ago, so do have a read to stay up to date on where I left off! c:
Quick update before we start: I set up an itch.io yesterday—and have officially joined the ranks of cool indie kids™—so now SPACEJAMMED along with eight earlier LD games can be found there. I'll dig up some more on my old PC later. I've fixed up old games to work on web too, and most work in fullscreen, so you can play in the comfort of your own browser! 🐱
Today I'll be talking about how I got the foreground elements, or gameplay elements, like the character and the puzzle pieces, to remain clear and unaffected by the painted and smudgy shader effect I'd applied to the image as a post-effect but really only wanted to have on the background. c:
And please do give SPACEJAMMED a play 🎮 to acquaint yourself with it first. ε: It's not as ambitious as our last entry, Blomst 🌻, but hopefully you'll like it anyway!
Painted background
So last time we gave the whole image a bit of a funky, smudgy effect like so:
But smudging out the character and the interactable puzzle is not that nice. It might be a good idea to clearly distinguish the immutable background from the gameplay elements, not just from a gameplay perspective, but an æsthetic one as well.
Clear foreground
The solution is more or less complex depending on what we need, and I did initially go for the more complex one before I realised this particular game didn't need it, and ended up simplifying it not to waste unnecessary render time and framerate. Let's consider the simple version first, and then explore why there can be issues with it and how to solve those as well.
Simple solution
The absolute easiest way to do something like this in Unity, which was used for this game, is to slap on another camera linked to the transform of the main camera and set up some render layers. This way we can have one camera render only the background elements and apply the painted post-effect to those, and then have a second camera render only the gameplay elements on top, without applying an effect, or at least not the same one.
Camera setup
The first camera should work as usual, clearing to a skybox or a colour or what have you, and draw everything considered part of the background, but not the render layer for the foreground elements (in my project called Clearer):
Note the setup for the culling mask, excluding this layer:
The second camera, for the foreground, renders only this layer, and has been set to clear depth only, so that it renders right on top of whatever was below:
Finally, the foreground camera has a higher depth value than the background camera so that it draws last, and of course only the background camera has the MainCamera tag and an audio listener.
Object/mesh setup
Then the objects with the 3D meshes themselves need to be marked accordingly. I simply left the background elements on whatever layer they were supposed to be anyway, while all the foreground elements, such as the character model, were set to the Clearer render layer, like so:
Result
Et voilà! Clearer indeed:
Problems
However, this only works because our game happened to have a fixed camera where all the foreground elements are in front of the background elements! Look what happens when I move the character behind one of the parts of the spaceship interior sticking out from the wall, compared to how it's supposed to look, when looking at the editor where the character is rendered properly:
in-game | editor
- | -
| 
Since everything on the foreground layer is drawn after everything on the background layer, it doesn't matter what's actually in front of what; foreground elements always end up on top! Again, not a problem for this particular game since this situation would never happen, but let's look at how to solve it anyway, since I actually did solve it initially before I realised it wasn't necessary.
Complex solution
To fix this, we need to do what is actually done to solve problems like this for each individual camera anyway, as the objects rendered are not necessarily drawn in order of their distance from the camera either, and even if one object has its centre farther away than another, there may still be parts sticking out of it that should occlude objects whose centres are nearer the camera, so that solution doesn't work either.
Depth buffer
Instead, a depth buffer is used to encode the depth of any pixel, or fragment, rendered to the screen, and whenever something new is to be rendered in the same position, it is first checked whether the depth of the new pixel is less than the old value for that pixel, and only then will it be considered in front and replace the old value.
Here's an example of the depth buffer of the scene, with both the foreground and the background elements:
White pixels, which have a value of 1.0, represent pixels far away from the camera and black, 0.0, ones very close. Then there's a greyscale in between. We can use these values to compare depths and figure out whether to override an old pixel with a new one whenever we draw something, based on whether it's actually in front, i.e. has a smaller value.
Getting depths
Luckily for us, Unity has made it quite easy to get to depth buffers, tho we need to do it two ways. First, we need to change up the pipeline a bit. Now we want the foreground camera to draw before the background camera instead so that it's all done when we get to the background camera, because we're going to manually be drawing what the foreground camera sees on top of what the background camera sees in the same post-effect shader that applies the painted effect.
To get the background camera's depth buffer in the post-effect shader attached to it is easy as pie! 🍰 Unity already has a built-in sampler, _CameraDepthTexture, which we can get like any other sampler by doing this somewhere in the pass block:
sampler2D _CameraDepthTexture;
We can then use this in the fragment function. But hold on for a moment. We need the foreground camera's depth as well. This is slightly more complicated, but not terribly so. First we need to add two texture properties to the shader that are going to be filled in by a script later.
_ForegroundTex ("Base (RGB)", 2D) = "white" {}
_ForegroundDepth ("Base (RGB)", 2D) = "white" {}
We'll have to add samplers for those as well:
sampler2D _ForegroundTex;
sampler2D _ForegroundDepth;
The first, _ForegroundTex, will represent what's rendered by the foreground camera. _ForegroundDepth will hold its depth buffer. Now let's attach a script to the foreground camera, so that we can fill these values in.
```
public class ClearCam : MonoBehaviour
{
[SerializeField] private Camera m_maincam;
private RenderTexture m_tex, m_depth;
private int m_widthLast = 0, m_heightLast = 0;
void Update()
{
int w = Screen.width;
int h = Screen.height;
if (m_tex == null || m_widthLast != w || m_heightLast != h)
{
var cam = GetComponent<Camera>();
cam.depthTextureMode = DepthTextureMode.Depth;
m_widthLast = w;
m_heightLast = h;
if (m_tex != null)
{
cam.targetTexture = null;
m_tex.Release();
m_depth.Release();
}
m_tex = new RenderTexture(w, h, 8);
m_depth = new RenderTexture(w, h, 8, RenderTextureFormat.Depth);
cam.SetTargetBuffers(m_tex.colorBuffer, m_depth.depthBuffer);
var mat = m_maincam.GetComponent<CamPost>().material;
mat.SetTexture("_ClearTex", m_tex);
mat.SetTexture("_ClearDepth", m_depth);
}
}
}
```
This is quite a bit of code, but fear not! I shall explain. 🙀 First, we need an inspector value where we can pop in the other camera, the background camera. I've called this variable m_maincam, since the background camera has the MainCamera tag. So in the inspector, drag that there:
Then there are two private render texture variables to hold the colour buffer (simply what the camera sees) and the depth buffer, which correspond to the two texture properties and samplers we just added to the shader. Finally two variables to keep track of the last width and height of the screen so that we can recreate these textures in case the resolution of the game changes, so that they're always the same size as that.
In the update function we get the foreground camera's own camera component, and we check if we need to update the buffers, or create them for the first time. If we update them, it's important to manually release the old buffers, or we'll be leaking memory and the computer is going to have a bad day eventually. Then comes the important part: use SetTargetBuffers() to be able to specify colour and depth buffers separately as render targets for our camera, and then finally get the post-effect shader material from the background camera and pass the buffers into it.
Compositing and comparing
Now we can finally use those samplers in the shader with the correct data in them, and do the final composite. After applying all the paint effects to the fragment from the background camera's colour buffer, at the very end of the fragment function, we'll do a few more things.
First, let's get the depth values for the background and the foreground respectively:
fixed depthBG = tex2D(_CameraDepthTexture, UV).r;
fixed depthFG = tex2D(_ForegroundDepth, UV).r;
We can use the same UV coördinates we used to sample the background's colour buffer, as in the post-effect shader, all the values simply correspond to the entire screen. These are only single scalars/values since depths are greyscale and we don't need multiple colour channels, so we're only grabbing the first value, i.e. the "red" value. Then we need the colour buffer of the foreground:
fixed colFG = tex2D(_ForegroundTex, UV);
Assuming our background colour buffer variable has been correspondingly named colBG, we can now alpha blend the foreground texture onto the background texture, but only if the depth is greater. One way to do it would be like so:
if (depthFG < depthBG) colBG.rgb = colFG.a * colFG.rgb + (1.0 - colFG.a) * colBG.rgb;
However, branching is discouraged in shaders, and it might be faster with a convoluted hack like this, tho I haven't done any benchmarking:
float alpha = colFG.a * ceil(clamp(depthBG - depthFG, 0.0, 1.0));
colBG.rgb = alpha * colFG.rgb + (1.0 - alpha) * colBG.rgb;
And there we go!
⚠️ Note that to get the complex method to work, at least for me, it's necessary to set the cameras to forward rendering instead of deferred or no foreground will show up! ⚠️
Next time
That is it! Like I said at the end of last post, there is still a bit of a lie in this image: at this point the shadows from the gameplay elements were not actually cast right below them as there are light sources coming in from the sides in the scene. This was confusing from a gameplay perspective, so a little trickery was taken to in order to force those shadows to be cast right below instead. That's for next time!
Until then, stay safe! 💓
| 
