Multi-color Outline Post Process in Unreal Engine 4

With some of the recent changes to Unreal Engine 4, rendering multi-color outlines is now possible! This is done through the use of Custom Stencil, a new buffer similar to Custom Depth – but allowing meshes to render as integer values. This provides us with a 1-255 range of indices that can be used to apply different outline colors to meshes and even combine multiple features such as the decal x-ray shown below by using a different stencil index.

ue4_coloredoutlines01_small

The Original Custom Depth

The original outline material I made last year was based on Custom Depth, before Custom Stencil was available. This meant there was no way to determine the mesh type after it was drawn into the buffer as single channel depth value. More info on the effect and other uses for Custom Depth are available in my of my earlier posts. The original effect uses fewer instructions (97 vs. 144 of the new material) so if you don’t require multiple colors in your game you can consider sticking to the old effect.

The new effect is still using Custom Depth to determine the (optional) occlusion which adds the faint overlayed color adjusted by tweaking the FillAlpha parameter in the post process. This occlusion can be turned off in the material by unchecking FillOcclusion in the material instance.

ue4_coloredoutlines02

Above: the Custom Depth visualizer.

Enabling Custom Stencil

Custom Stencil is disabled by default, to enable go to Window > Project Settings > Rendering > Post Process > Custom Depth-stencil Pass and set it to Enabled with Stencil.

Some of the meshes are not visible in the Custom Stencil visualizer in this example, their Stencil value is set to 0 (default), excluding them from this buffer.

ue4_coloredoutlines03

To enable this visualizer go to your viewport, look for Lit > Buffer Visualizer > Custom Stencil.

You can enable Custom Depth and change the Stencil index through the editor menu of a mesh under the Rendering category.

ue4_coloredoutlines05

If you’re using C++ you can define the stencil indices in your game header (ex. SurvivalGame.h) for your convenience and bookkeeping.

/* Stencil index mapping to PP_OutlineColored */
#define STENCIL_FRIENDLY_OUTLINE 252;
#define STENCIL_NEUTRAL_OUTLINE 253;
#define STENCIL_ENEMY_OUTLINE 254;
#define STENCIL_ITEMHIGHLIGHT 255;

Enabling custom depth and setting the index in C++ is pretty straightforward.

GetMesh()->SetRenderCustomDepth(true);
GetMesh()->CustomDepthStencilValue = STENCIL_ENEMY_OUTLINE; // or simply assign an int32 within 1-255 range.

Setting up the Post Process

To enable the outline you need to place a Post Process Volume. Make sure you set it to Unbound so it’s available regardless whether camera is inside the volume or not. With the post process volume selected, go to Settings > Blendables and add the PPI_OutlineColored as the first entry.

Download

Multi-Color Outline Post Process

Outline post process for Unreal Engine 4.9 and above. Allows (4) different color outlines (up to 255 colors theoritically supported)

Previous posts on subject

122 Comments

  1. Hi Tom , I’m wondering for performance wise, if it’s better to use this or 2 layers of blendables in PP to create different colour ( using non stencil ).

    • That wouldn’t work. You can’t tell the post process the object type without Stencils. So you end up drawing the outlines twice with only the last result visible for all outlined objects.

      • right .. that made sense , however I was actually previously thinking of turning one off before enabling another. allowing me to use two separate color but not at once.

  2. Thanks for the update, Tom. I used your previous guide to help me with a project a while back and one of my main gripes was that I couldn’t define different coloured outlines. Now to put this to good use…

  3. Hi Tom,
    Great addition!
    One question, how do you set up the “x-ray” decal to only show particular stencil buffers?

    Great work. Also just tried your fresnel material – pure gold.

    • It’s much like the outline effect that filters based on stencil index. You can for example use an if-node to check for corrct index or normlize your x-ray index so it becomes a 0-1 range so you may use it as a multiplication when figuring out if we should show x-ray or not.

      – Tom

  4. Hi Tom,

    first of all, thanks for the tutorials on Custom Depth and Custom Stencil buffers – very informative! I have a query that you might be able to help with.

    I’m using UE4 for cinematics rather than games, and I’d really like to be able to tag assets as belonging to separate layers (foreground, characters etc), and render them as separate images with an alpha. I did hack the ‘use custom depth as mask’ feature to do this, but found that the results were horribly aliased. Presumably the masking operation happens after antialiasing. Do you have any advice on how to produce an antialiased image containing just the tagged assets with an alpha channel? I know I could apply a custom material, but that’s a bit of a labour-intensive way of working…

    cheers!

  5. Hi Tom!

    I’ve been following your tutorials before and they taught me plenty. This time however, I’m not getting the desired results.

    I made a test level (4.9.2), Project Settings set to “Enbled with Stencil” and with an Unbound PostProcess-Volume in the level. I added PPI_OutlineColored as a blendable material (even picked PP_OutlineColored as parent for the instance). I placed a static mesh actor in the level, set Render CustomDepth Pass to true and picked a stencil value higher than 0.

    I can see the custom depth being visualized when I choose “Buffer Visualization – Custom Stencil”, and the color changes when I set the stencil value, but there’s no outline being rendered for the mesh, in the viewport or at runtime.

    I’ve tried solving this myself but the steps seems to be few and easy so I’m not sure what I’m doing wrong.

    I appreciate your help!

    / Victor

    • The index needs to be 252 or higher when using the post effect from the download, that’s what the first index is set to. Doing so means we can still easily use other features with stencil indices without interferring with our outlines.

      – Tom

  6. Hello Tom – Great tutorials and many thanks! Is there a way to make the outline more prominent / obvious to the user? It’s a little too subtle for my taste. I’d like the make the edge of the outline thicker.

    • Yes I’m on 4.10 and it works just fine. I hf to do the turorial step by step and read through the comments here about any mistakes you might have done

  7. can we set an RGB value specifically? Or are we limited to set colors from 1 to 255? Or am I misunderstanding the purpose of the index?

    • Hi George,

      Yes I believe you misunderstand the Index. The 1-255 are the “layers” and not a color itself, it’s a single channel in the render code so we need to translate this index into any color ourselves.

      To apply colors we assign our own color parameters to specific index values. The sample material checks for 3 different indices and assigns the Red, Green, Blue colors.

      Hope that helps!

      – Tom

  8. Let me get this right; with Custom Stencil we are therefore now able to have up to 255 custom depth channels, 256 if you count the original Custom Depth as well?

    That would be so awesome.

    • Stencil works a little bit different, it doesn’t store depth values and instead it just “tags” pixels with values between 1-255.

      That means you still only have one depth channel, but you can now do a lot more neat tricks for filtering pixels in post-process etc.

      – Tom

    • Yes there is, you can compare the CustomDepth with the SceneDepth and ignore all pixels where SceneDepth (which contains all pixels from any opaque object in the scene) is closer to the screen than CustomDepth (which contains only pixels of objects that are part of the outline)

      As you can see in the first image, I am already doing this comparison to get a fill-color on anything that is blocked by a wall.

      – Tom

      • Getting a little frustrated figuring out how to “compare the CustomDepth & SceneDepth”. Everything works fine if I want to have a fill, but all I wanted was to display the outline on objects visible. Every time I compare the described elements, I think it masks out the outline.

        How am I suppose to do that?

        • Have a look at the “Alpha” parameter, it controls the fill alpha and should give you a hint as to where to look for fill vs. outline-only implementation.

  9. Hi Tom, Would your method work on mobile devices since Post Processing Volumes are disabled on mobile?

    • It wouldn’t work on Mobile, it only works with post processing. You can use simple tricks like Fresnel to add similar highlighting effects to objects on mobile.

      – Tom

      • Can you show us an example of highlighting or silhouette effect on occluded actors on android? I know that pixel depth and scene depth work on android. Is that enough to get this kind of effect or a post process is absolutely needed.

    • Try multiplying the outline color with something above 1 to make it glow. It depends on the chain of post processors if it’ll work. The bloom post processing needs to come after this outline post process (I believe it is by default, but not 100% sure)

      Hope that works.

      – Tom

  10. I’ve been trying to merge this with another one that does not show the occluded portion, but I’m having a hard time getting it to work. How can I tweak this one to be able to block the outline from showng through other objects like walls and the player?

  11. I tried to change the depth stencil value at runtime using blueprint, but it just doesn’t work. The value stays the same. Is there something else that needs to be done to be able to change it?

    • What I have done, and it seems to work great, is set the color (Set Custom Depth StencilValue) at the same time I Set Render Custom Depth. That way it changes only when activated. When focus/mouseover/etc is no longer activated, I set it back to whatever I wanted the default color to be. Hope that helps!

    • That will only work if you use different Stencil Indices for both those meshes. Otherwise there are rendered as one big blob with your current result.

      – Tom

  12. You mentioned it briefly in an earlier post, but I’m having a hard time getting it to work: how can we hide the effect behind other objects? I love how well it works, but i don’t necessarily need to see the outline if the object is hiding behind another object or wall. Thoughts?

    • The material already checks for occlusion (this intensity can be modified by the Alpha parameter, so you might want to look around that node) that is the only value you need at that point and multiply that on the outline value. That should only give you a colored output on the occluded parts of the mesh.

      This sounds like a good thing to have in the original material, I might add it at some point, but for now I hope this gets you going in the right direction.

  13. Tom excellent job. Your tutorials simply don’t exist.
    I’m pretty noob with all this technical post-processing. But i need to obtain 4 colors in the outilines. Blue, Red, Gray and White. How can i achieve this?
    If you could explain a little bit more how this all works or point me into a direction that explains this subject to a complete newcomer. Custom Depth, Stencils… i dont get it =/

    • Another issue is that this doesn’t work on IOS Shader Model 4 (DX10, OpenGl 3.3) how can i enable it?

    • Make sure you have a postprocess volume in the scene with PPI_OutlineColored as a blendable material.

      Open up PPI_OutlineColored. This is a material instance where you can change the colors on 5 different “channels”. Each channel has a Stencil Index number, for example 255. Say we set channel 255 to gray, and we would like all weapon meshes to have a gray outline to them. On each weapon mesh you will set “Render Custom Depth Buffer” to true, and below change the Stencil Index to 255.

      You can then set 254 to red within the material instance, and set the stencil index on all button meshes to outline them in red. And so forth.

      I usually set “Render Custom Depth Buffer” to select the stencil index (it’s grayed out otherwise), and then unset so that I can toggle it on and off when I want to.

    • Tom
      March 12, 2016 at 9:00 PM Reply

      The material already checks for occlusion (this intensity can be modified by the Alpha parameter, so you might want to look around that node) that is the only value you need at that point and multiply that on the outline value. That should only give you a colored output on the occluded parts of the mesh.

      • Hey Victor thanks for the explanation i got it working, however it only works on my Windows machine. When i try to run this project in a MAC OS X it doesn’t even work =/ Does opengl have support for this?

  14. I think im misunderstanding how this works (or I just can’t figure it out)
    I can’t see a way of making the stencil only visible if its behind an object. The outline is working fine, but I only want it visible when behind something.

    Really appreciate the guide Mr Looman. And your survival series has been an invaluable resource. Thank you very much!

  15. I’m a tad stuck here, pretty new to post processes.

    I’m attempting to get the line to appear all the way around an object instead of being occluded by another object using a different Stencil Value. The current effect is this: http://i.imgur.com/f51oSuy.png

    The effect I’m hoping to achieve is this: http://i.imgur.com/ne5KUsT.png
    Is this effect possible since they’re using different Stencil Values? If it is, could I get an indication of where to look?

    • Hi Dominic,

      That issue is most likely because I use Custom Depth to determine whether or not an outline should be drawn, and the stencil values only to check which color to draw it as. I feel like this is something I should update in the original effect at some point. For now though, the hint I can give you is that a big chunk of the effect code is about checking Custom Depth values (I grab a bunch of samples around the current pixel we calculating to see if this pixel should be drawn as an outline or regular scene pixel) you *could* update this to use stencil values instead, I’d have to experiment with this myself too (on the surface it sounds like this could be more expensive, but I’m not sure)

      I hope that helps you a least a little bit. I doubt I’ll have time on the short term to fix this in the sample, but I’d like to get around to this eventually.

      – Tom

      • Hi Tom, did you happen to ever get back around to messing with this? I am having the same problem with every example I follow, the outline won’t follow the entire edge around. The editor does it perfectly when you select multiple items (orange outline) just can’t get the same effect using post.

    • That’s entirely possible. You can add more colors as long as there is still Stencil Indices available (and you have 255)

  16. hi tom, great effect! i used your previous version of this as well, prior to stenciling, and was able to occlude things without issue. however, with this effect, since the outline is drawn outside of the object, i’m having issues with the math on how to remove occluded pixels. since the outline isn’t part of the object being considered for occlusion, what would an alternate approach be that could still occlude outline fx? (per previous comments, occluding the fill effect isn’t a problem because it is calculated based on the original object – or just using 0 for alpha)

    • Hi Hadar,

      Hmm you’re right that does add some complexity. I’d have to dive back into this for a proper solution, I didn’t expect this to be as big of a problem as it’s now turning out to be!

      – Tom

  17. Hey Tom, I’ve been using your PP material in my project for the last 12 months and it’s awesome! Thank you so much. I’m using 4 different stencil colors and, currently, the only enemy target that gets highlighted is the one that the player is ‘focused’ on. What I’d like to do is change it so that all enemies on screen are highlighted with the outline effect but only the focused enemy has the Overlay Effect enabled. Do you have any idea how I could make the overlay effect apply to only some of the on screen enemies?

    Thanks so much!
    Jonathan

    • Well, I don’t really know how it works, but I’ve got a solution. I simply created two of each color stencil value – one for overlay and one for non-overlay and then a simple IF statement to set the overlay amount.

      I was thinking that setting that value would change the overlay effect of the entire blendable, but somehow it’s limited to only the meshes I want. I don’t know how it works but I’ll take an easy win. Thanks again Tom!

  18. You should probably add a note for Mac Users that this won’t work unless they have 10.11 installed with the metal renderer. The stencil buffer isn’t available in OpenGL on the mac. Took me a little bit discover a post on the answer hub that pointed that out.

  19. It’s not working, no matter what i do

    – Enabled Custom Depth-Stecil Pass with Stencil
    – Create post-processing and add PPI_OutlineColored to first index of blendable
    – Create a sphere, enable render custom depth, adjust CustomDepth Stencil value

    And nothing happended. Where did i go wrong?

  20. Hey buddy, great tutorial! I simply love your projects! Could you help me with something? It’s not working on my project lol I do everything right and it simply doesn’t work. Maybe some update just crashed it?

  21. Sweet, thanks! this saves me a lot of work! What do I actually get after masking stencil with R, the index? I want to be able to draw different thickness outlines on various objects (character vs environment), thought I could do this with just some math with width to index but apparently that’s not working..

    • Uh also, the outline doesn’t draw if one stencil object is partially occluded by another. Any ideas how to make sure outline always draws?

      • Hmm I think that’ll be a more costly material as you must check not only against CustomDepth but also compare StencilIndex values to check whether a line must be drawn. I haven’t tried this, it might be tricky..

        – Tom

    • You should get the index value yes, you can see how I did that in the Color picking part of the material.

  22. Hi, so this may be a silly question to ask but I was wondering with this post effect if it would at all be possible edit the material so that the outline drawn around the mesh would be occluded by other geometry. Or if not do you have any ideas on how I could approach this problem and draw an outline around a mesh that can be partially occluded by geometry. Thanks in advance, Luke.

  23. Thanks for this Tom, it’s exactly what I was looking for for a project. I do have one question. Is there a way to add more Vector Parameters? I tried setting some in the parent material after going over what you did, but afterwards the vector parameters I set didn’t work. I could set them, and compile, they just wouldn’t outline. I’ve lowered the Stencil buffer values a bit to give me room for new colors as well, starting at 240.

    • Hi Morgrhim,

      It should be possible to add additional color (Vectors) parameters. If you altered the material it will be a bit difficult for me to guess what’s going wrong. Just double-check the indices and if all the interpolation between the Vectors is setup OK. I can probably help a bit better if you can include your material (as either images or the material itself)

      – Tom

  24. Sorry Tom but I’m new to Post Processing in Unreal Engine 4. How can I make this pp effect to work on my project? Thanks in advance!

    • I managed to know how to use it but unfortunately it doesn’t work I do not know what the problem is I just placed it in the blendables of a post process volume is that the way to use it?

  25. Well after several experimentation I did make it work. Cool content please continue sharing your knowledge on Unreal Engine 4. Thanks!

  26. Hi Tom. I wanted to say thanks for this awesome material. I have been using it in my project and it is awesome!

    I am currently trying to figure out a way to only draw the outline on parts of the object that are NOT occluded by other objects. This would result in the outline only showing up on parts of the object you can see, and not showing up through other objects. I noticed a few other people also asking this question, but didn’t see any responses. Is there a simple change to be made in the material, or does it require more than that?

    Thanks!

  27. Hi Tom. Thanks for this awesome material, I got it work but my question is about AA
    When the object that has outline moving(camera move in the editor has the same problem), the outline become unstable and sandy, only when hold still about 1s will it become normal. I searched the answerhub and found TemporalAA is to blame and change it to FXAA will avoid this problem
    But if I want to keep using TemporalAA, is there anything I can do to make the outline stable and nice?

      • After my test, Set material to “Before Tonemapper” indeed solves the Flickering problem, but still sandy…I know when deal with translucent surface material, I can turn “responsive AA” on. Then can solve the edge sandy problem for meshes, but it doesn’t work on post process material…the checkbox is uncheckable

    • Should be possible Gabriel. You can use the Screen UVs node in the material editor and plug it into a TextureSample node (with your desired pattern) and then multiply this with the nodes used for the fill color (may need to dig around a little to find the correct place, look for near the “Alpha” parameter node)

      – Tom

      • I was experimenting with this and got a gradient effect on the outline, so beautiful!

        And another thing about Deus Ex HR likeness, how can i turn off the custom depth if the object that has it eneabled is behind a wall or something?

  28. Hi Tom, in your original outline blog you had talked about the next steps being to do a Fade-Out like the Left 4 Dead outline, or like the Deus Ex outline Gabriel posted above has; can you think of a way to incorporate a nice gradient falloff to soften the outline?

  29. Hi Tom! Really like your work and the way you explain everything. I just have a small question, I am trying to use this material in a html5 project and when I am trying to launch the game I get this : “LogMaterial:Warning: Failed to compile Material /Game/Materials/PP_OutlineColored.PP_OutlineColored for platform GLSL_ES2_WEBGL, Default Material will be used in game.” I was wondering if there is any way I could use this in my html or if there is another way to gain a similar effect.

    Kind regards!
    Mani

    • My knowledge of what HTML5 supports is very limited to be honest. Perhaps there is a way to emulate/force compile of GLSL_ES2_WEBGL while still in the editor so you can see what (material node) exactly it’s failing on or if cutom post process is perhaps not supported yet?

      – Tom

  30. Hi Tom,

    I have a question on the Custom Depth Stencil. How do I set it up so that the stencil output is just a white alpha mask? Is there a way to disable the actual stencil number? I’m using it to render out frames for comping externally.

  31. Hi Hi Tom,
    Colorful outline is not as long as a “Post” can be achieved?
    And how to determine the different depth of the mesh to render different colors?
    Ash often thanks.

  32. Hi! Please correct your code. SetCustomDepthStencilValue(index) must be used instead CustomDepthStencilValue = index. It is very important because second variant has an effect only first time after you run program.

  33. Hi, Tom! I’m like many people here struggling with trying to make outline only be seein if object is not hidden behind other objects. I was trying to compare “CustomDepth” and “SceneDepth” as you suggested in a manner shown on related screenshot, but it seems, like Custom is alwas greater than Scene, so no matter what I plug, I always get “A>B” output.
    https://gyazo.com/7db3d926106b206f379adcb1c897cc18
    Here’s screenshot example
    https://gyazo.com/ac7ec3ea68a6a95330b93c4ff5505b65
    And the other question I had, is there a way to make outline appear instantly and not in a fading manner (I use mouseover to change color and it is always fades when color changes) and if there is, how should I do it?
    Aaaand another one, how should I control stroke size depending on camera distance to the object?

    Thanks 🙂

  34. Would you know how to make lines appear only when approaching and looking at an object? Thank you

  35. there might be a more performant way of doing this with occlusion, but here’s the solution that i’ve been using and works with occlusion. It nulls the outline and nulls the fill (hence the two conditionals) if it’s occluded. But the problem is that it uses two conditionals, so when doing optimization checks, this shader is usually first on the list. Open to any alternatives…

    Overall nodes: http://imgur.com/87yXAX6
    depth testing: http://imgur.com/SoK5j4p
    occluding: http://imgur.com/lEfhowK

  36. Hello. I`m a Korean developer. I will try to use this BP for our program. Are there any license restrictions?

  37. Hello,
    is there any way to use two instances of this material but with different presets? It seems like one instance overrides the second one.
    I’d be very grateful for help

    • Hi mate, If you need to “hot-swap” different presets for whatever reason you will need to remove the original instance in the post process chain to avoid conflicts.

      In Blueprint there is a AddOrUpdateBlendable node available in your camera component. You can use this to remove the original post process (set Weight variable in the node to 0.0) and then add the other instance with a weight of 1.0.

      The “Blendable Object” parameter of that node is the material instance (eg. your post process)

      Hope that helps!

      Tom

      • Hello,
        thanks for the answer, I had no idea about that, I’m sure it will be useful in the future 🙂
        Unfortunately, I’m looking for a solution which allows to have outline on some objects, and only xray on others.
        I tried to modify your material to show outlines on objects that have Custom Stencil set to 255 (providing the StencilStartIndex is set to 252), but no luck so far.

      • PPS. Sorry for not editing my previous comments but it seems like there’s no such feature 😀
        By “Custom Stencil”, I meant “CustomDepth Stencil”

  38. Hi thanks for your work!
    I was not able to import your material in my content…So I create the material following this tutorial https://www.youtube.com/watch?v=IEHxjw6VEMQ, I’m seeing the line trough walls…Jon Farrell linked your website, but I can’t find the solution here, sorry, it’s surely not complicated but I’m new with all that!
    Bye
    Gui

  39. Not sure what I’m doing wrong. Using UE4.15 and i cant seem to get this to work. I figured that since the Blendables does not seem to exist anymore that “Post Process Materials” must be the new name? Still no luck though. Any pointers on getting it to work in UE4.15?

  40. well i got it working, forgot to set the “Render CustomDepth Pass”. but now if i don’t use 255 as the value, it wont show up, which means i cant change the color… anyone else have this issue?

  41. and!!! i didn’t realize i accidentally set the StencilStartIndex to 255…. hence the issue i was having haha…. Thanks for your awesome Shader Tom.
    PS.. wish i could just edit my above posts so it doesn’t look like i spammed this wall…

  42. hi
    thanks your tots,
    Is there any way to make it use the translucent material for outlines?

    • You can enable Custom Depth rendering on translucent materials, for this you need to enable “Allow Custom Depth Writes” in the translucent material

    • I also have this problem with the default version.

      I don’t know anything about shaders but from trying different things, I found disconnecting the SceneDepth seemed to make occlusion work normally, though I have no idea why.
      With your custom setup, maybe try skipping over the first ‘if’ node and just directly hooking up the “Mask (R)” node coming off the CustomDepth directly into ‘A’ for the ‘Multiply’ node.

      Might be a temporary fix until Tom replies with whatever would be the more correct approach.

      • I’m not sure if what I said actually makes sense for changing your current graph, here is the part for occlusion if you still have that part setup, where the blue outline shows how I resolved it and the red outline shows what was disconnected:
        http://i.imgur.com/XVOWAJn.png

  43. Hello. Just curious if there is any possibility of updating this article to one that works with version 4.15 of the engine? When I came here for help in 4.14 all of this worked perfectly, but now many things have been moved around (of course without matching doc updates) and people all over the unreal forums keep referencing this article – which is great – but all it does is confuse people more (me included) at this point and it is very frustrating to say the least. I’m sure others that will be pointed here in the future would GREATLY appreciate the effort (me included!) since this is the only article that explains it at all to begin with in an easy-to-understand way.

    And thanks for all your help!

    • Hi Russ!

      Haven’t used this effect in 4.15 yet, what kind of issues did you run into? I am surprised they changed so much that could even affect this material.

  44. The issues are with the changes to the Details panel and the placement of selections like Blendables inside the Post Process Volume, which no longer exists in 4.15 and since I saw mention of this change (in this post or your previous one) I had hoped maybe an update was here somewhere. I believe “Blendables” is now “Post Process Materials” so I added the material there.

    None of the code I’m using was changed between what I implemented in 4.14 (where it works perfectly) and 4.15 (where it does not) and it compiled without a hitch so I’m assuming I’m missing something else that changed in Unreal but I have no idea what that is.

Comments are closed.