Mesh Outlines Without Post-Processing in Unreal Engine
Some time ago I saw a neat solution by Cory Spooner on outlining meshes using particle sprites. The concept has been done before – but it’s interesting enough to cover it regardless for Unreal Engine specifically. We render the outline on a translucent cube or sphere that is tightly fitted around the desired mesh. We apply the outline material to this cube instead of a post-process chain. This let’s us affect only small portions of the screen rather then pay the full-screen cost.
Post Processing especially affects notebook and mobile performance which commonly have much lower fill-rate throughput than desktop graphics cards. Optimizing this type of post-effect is useful for any platform of course. Outlined meshes in games usually only cover a small portion of the screen. And in the worst case you don’t even outline anything on-screen, but still pay the post processing cost.
In this post I dive into the possibilities of using the ‘local’ outlines to reduce render cost. By drawing translucent cubes around the outlined meshes so we heavily reduce the number of pixels evaluated. We effectively add a draw-call (the cube mesh) and reduce the fill-rate/pixelshader cost. Additionally, this gives us more per-object control as each cube can apply it’s own material effects and colors.
Certain games like League of Legends already do this as it can greatly reduce cost on potato-quality machines. Riot Games actually has a fantastic Tech Blog that I highly recommend checking out…
Download Assets
You can get access to the material files here.
Fill-rate Reduction
A quick check using an Image Color Extract Tool tells me that roughly 66% of the above image contains the brightest green. (Screenshot taken from the Shader Complexity visualizer in the editor) This means it performed zero evaluations of our outline material! In a normal post-processing scenario, 100% of our pixels would need to be evaluated.
Only a tiny fraction is colored red which means that pixels has been evaluated twice in this specific scenario. (and therefor ‘more expensive’ than the regular PP setup) The scenario is even a particularly bad-case scenario as we have 6 tightly packed outlined objects on which we zoomed in. With more real-world conditions the percentage of saved pixels can be far greater.
Color Bleeding
Using simple outline implementations you might run into issues where tightly fitted outlines from different objects overlap. In some earlier experiments I ran into problems where I would shade pixels behind the cubes bounds. Eventually I fixed this by relying on stencil indices for each unique material. (eg. 1 index for the blue outlines and another for red)
Depending on your specific outline implementation you may need additional filtering. Mainly when dealing with multiple semi-translucent outlines that may negatively affect overal Alpha. I will be updating the material files as more of these edge cases get solved.
Performance
There is a trade-off in performance by using this implementation over the more traditional post-processing based outlines. You add more draw-calls (for the translucent cubes) but save a large amount of fill-rate. Where the post-processing has a cost based on screen resolution. Instead, the per-object approach is a variable cost based on the amount of outlines objects currently on-screen and how much screen-space they take.
Since the translucent cubes are depth-tested, we get a small win with occluded pixels. And when the object is fully occluded, the whole outline cube will be culled. This benefit is lost of course if you need outlines through walls, where depth-testing needs to be disabled.
Bonus: Local-Space Fall-off
Additionally I wanted to give the ‘World of Warcraft’-style outline a try. They use depth-fade to blend outline intensity based on the difference in depth, a common technique using in particle effects. This doesn’t add an outline around the entire object keeping the object more grounded with the world. A downside is that objects that don’t stick out a lot, will fail to render a clear outline. (see below)
I’ve included a sample fade-out in the downloadable material files. Now that we have more data available to us, such as the object’s bounds we can easily use that data to fade towards the Z-axis. You can even rotate the object and use the local rotation to fade towards the wall instead of down.
Further Experiments
To show the flexibility, I did a quick mash-up of adding colors and a panning line in world-space. Since we know the object’s bounds it’s easy to interpolate between colors based on the pixel’s position. Here is what came out of that:
Closing
Whether or not this technique is practical for your project depends entirely on your own constraints in performance and workflow. I originally started this as an experiment to see if the problems presented (color bleeding among others) could be solved. The results so far are looking good and are being used in our internal game projects. There are still cases where a full-screen post process are more desirable as it has that single-pass benefit.
Download link at the top of the Page or on here!
Ben Peck let his domain expire ; _ ;
Anyone know of what he did different for VR and why?
Just wanted to put a note here for anyone struggling with stencil outlines around translucent materials: Lower the “Opacity Mask Clip Value” in the Translucent material you want to outline and the stencil will draw around it just fine!
The stencil uses the Opacity Mask Clip Value as the threshold for what to stencil, even if you’re using a Translucent material (not Opacity Masked).
e.g. I set mine to 0.001 (instead of the default 0.3333)
Works for me in Unreal 5.3.2 anyway.
Just as a Note. As of 5.0 and 5.1 this will not work on Nanite meshes. Not a big deal honestly, but save yourself some time use non-nanite meshes for objects that need to be stencil’d 😛
I can get this going in UE5 in the editor viewport but targeting the Mobile ES3.1 viewport (I’m developing for the Quest 2) this doesn’t work. I’m researching custom depth and custom stencil support for ES3.1 and Vulkan but can’t seem to find much confirmation on whether or not these rendering features are supported on mobile.
Hey, Is it possible to allow overlap in the material in the material like this https://imgur.com/XS1YOh7
where overlapping character parts have individual outlines or would that not be supported in this.
Thanks You!
I struggled getting this to work for so long, I wanted to write up some more explicit instructions for anyone who might be similarly struggling.
1. Project Settings -> Render -> *Enabled with Stencil*
For some ActorClass, ensure you have two at least two components
* MeshComponent – Your standard mesh
* OutlineComponent – The primitive (Cube/Sphere) used for the outline bounds
On OutlineComponent
* Apply a MI of the material provided in the download (You might need to create one. For me, the instances provided in the zip did not work.)
* Note the Stencil Index in the MI
On MeshComponent
* Check RenderCustomDepth
* Set the Stencil Index to an index matching the MI
Hi, will this support UE5? Would love to try using this!
It should be available to use in 5.0, I don’t think anything changed that would prevent that.
Got it working well! Thanks!
No idea why I can’t change the colour of the outline anymore, it was working fine before. My steps as followed:
1) Place cube and drag the M_LocalOutlines so it’s invisible
2) Have all my meshes in the cube which are to be highlighted
3) Duplicated M_LocalOutlines as Material Instance
4) Changed the Material Instance StencilIndex to 2
5) Change the mesh CustomDepth Stencil Value to 2
Render CustomDepth Pass enabled for meshes, and Custom Depth-Stencil Pass: Enabled with Stencil in project settings.
It only shows the outline blue that’s displayed inside the M_LocalOutlines nodes.
If I change the colour within the nodes it works, but it’s not working within the Material Instance. Within the nodes, the default StencilIndex is 2, but if I want to switch the value to 4, I’d have to change it to 4 within the nodes whiles also changing the mesh CustomDepth Stencil Value to 4 so that it works.
Did I miss something to get the Material Instance to work?
Nevermind, turns out instead of slapping the M_LocalOutlines, I just slapped the duplicated Material Instance onto the cube and it works with the latest changes.
Hi Tom, any suggestions on how to use this to outline a single target in a group of actors with a shared base class that has the target cube?
What I’m seeing is that the cube from the target actor will cause neighbor actors to be pulled into the stencil. Imgur example: https://imgur.com/gxrNnjp
You can see the target apple on the right and the neighbor apple on the left is picking up some adjacent outlining.
I can fit the cube better, but this only takes things so far. Since you mentioned League of Legends I will use that example. Image the case of minion targeting. There will be a group of actors with the same class and need to outline exactly one. In this case, cube fitting can not be sufficient in itself.
One idea I had would be to programmatically iterate through the stencil indexes at Actor spawn time. But I wonder if this has been solved in a simpler way.
In your particular case you might be better off disabling custom depth for the object you don’t want to highlight in the moment. It’s kind of rendering redundant info into the Customdepth buffer at that time, so it’s better to do anyway performance wise as well. If I understood the problem correctly, that’ll solve your overlapping issue too.
I’m having trouble with flat and wide meshes and the vertical gradient not being as pronounced, looking a bit off, what could be the issue? I made sure the box containing the mesh is the same size as my mesh bounds so it’s not that …
FYI Tom and anyone who finds this, I was able to get this working relatively easily in the UE5 early access release. I couldn’t import the uasset but I opened it in 4.26 and just copied the material nodes over, copied the material parameters and it worked
To anyone looking at this as a solution for multiple outlines to be generated concurrently, being able to overlap between one another without artifacting, look elsewhere. The setup available for purchase doesn’t work if you have an object behind your outlined object with a lower depth stencil value (so if you assign the second bit to your object, anything behind it with the first bit assigned to it will also generate an outline).
If that’s your desired output, by all means, buy this and be happy. If not, you’ll need to customize the shader network to account for that and move a few things around. Word of warning as it’s not mentioned anywhere in the article, gumroad page or any available links (AFAIK at the time of writing)
Hi Tom, great tutorial, thank you – I just bought the materials because I really appreciate these writeups you do.
As someone else mentioned, in a VR project, the outline shows up only in the left eye. Any idea on what it would take to get it working for both eyes? A fix to the materials would be great but, short of that, any pointers or suggestions of where to dig would be greatly appreciated too. Thanks!
Update: in 4.25 at least, it works in both eyes if instanced stereo is disabled. Not sure if that’s a helpful clue or not.
I have no idea how stereo will influence this. It might be something Epic needs to solve if it only doesn’t work for stereo vr rendering. There is no option I know of that would ‘fix’ this.
Hi Tom,
Works great, but I’m trying to repeat the process to create additional custom stencil index depths and not seeing the expected results, thought the explanation might be useful to others, who may have made the same assumption I did
Step 1: Duplicate material
Step 2: Set Stencil Index parameter default on the new material to 3 (from 2)
Step 3: Apply new material to mesh
Step 4: Set custom depth stencil index to mesh to 3
This doesn’t work.
You can’t just update the *default* parameter for StencilIndex on the material, you have to edit the StencilIndex nodes in the material directly. Feels like that value should be falling through and updating the parameter nodes but for some reason, isn’t.
If you want to change the highlight colour of the actor in question then you just update the material applied to it.
Hope this helps someone
Hi Tom,
I appreciate all your tutorials and work in general. I am experimenting with your outline material with UE 4.25.
I have an split-screen setup where I set a post-process material on each character camera( by using PostProcessSettings). I assgin a unique stencilIndex to each local player, however only the Player0 renders the outline. The outline for Player0 is visible in all the viewports of the other splitscreen players. But the other players outline is never visible.
Any pointer about what I need to change to make it work with splitscreen?
Thank you
Hello. I dont know where is the problem, Iam using your outlines materials in Unreal 4.25. I am not sure if its work, i will try to instal older version of unreal engine. Thank you
Hello!
I have a question. When the camera get too close to the Mesh outlined (in my case the manequin), the mesh will get 100% colored. Any way to disable or fix it?
A link to the EULA? It will speed our purchase request thru legal.
It is working for me (4.25), but when I set the thickness much beyond 5 and move backwards, the object’s outline splits into 3 separate outlines and looks a bit off. The material is still usable, but its a tiny bit disappointing to have to keep the border so thin.
edit: that came across as a little negative. Its actually quite a good shader on larger objects, which incidentally I will be using. It was only smaller objects I noticed the issue above. I will probably be using this in production in a few months
It’s not working on 4.25
It should work fine in 4.25, I suspect your project doesn’t have Stencils enabled (this is one of the options under Custom Depth settings in the Project Settings)
Hey,
I am trying to have two different objects with different stencil values have effect from two different material instances derived from this same material, ie outline with two different colors for two objects in the scene at the same time. But it is not working in this case, I could see only one object is affected at one time, even if I am adding two material instances in the Post process volume
Looks like you are using the effect incorrectly. This is NOT a post process material as mentioned in the post. You need to apply this material to a MESH instead. This mesh should surround the object you are trying to outline, eg. you use a cube with this outline material on it to surround your character.
For anyone potentially struggling, my issue was that the ‘Custom Depth-Stencil Pass’ option in the ‘Project Settings’ needed to be set to ‘Enabled with Stencil’, not simply ‘Enabled’. I’m using 4.24.3 so not sure if this was changed or added since 4.22.
My default was set to ‘Enabled’ so I just assumed it was correct without actually checking the drop-down for other options.
Doesn’t work on Oculus Quest (Using mobile ES3.1)
I can’t figure out how to make it work
Hi Tom,
Do you have any ideas on how to make the line thickness consistent across different resolutions? Right now, the outlines appear thicker at lower resolutions.
Thanks for your time.
Hi Tom, great stuff, love your blogs.
I was trying this out in my VR project but instead of just highlighting the outside it highlights the whole object.
I believe this is because of forward rendering coupled with MSAA.
In an empty PC project, If I have either off these off then it works, If I have both on then I get this whole mesh highlighted situation.
Is there anything to be done about this?
Forward rendering is pretty much required for my project so I’ll probably have to find another way to outline without PP I expect? (Inversed normals solution looks trash)
Sorry forgot to link my picture.
https://imgur.com/a/mF4xDxe
Hmm I haven’t tried this with Forward/MSAA – you might be better of with regular post process outlines in your case (I’m not sure if this is a forward render bug or a limitation actually)
Thanks for your reply!
Trouble is I’m running VR on Oculus Quest so post processing is basically impossible.
I tried out the Inversed normals solution prior to this but it leaves open edges and looks quite ugly in practise.
Hello.. I already did the instruction. But the outline not shown.. Can you make simple tutorial video how to do it, pls?
Hi, i tried the outline in UE4.23 and it worked in PC exports but not in Android. When I enable ES3.1 Preview it disappears. What can i do? Thanks.
Helo. I downloaded the files, but they refuse to show up in the editor – it shows like an emply folder in there.
They were made using UE 4.22, so if you’re on an older version they will not show up. The material nodes can be copy-pasted into a empty material asset in case you really need it for the older project.
Hello,
I am having some issues setting this up. Using 4.22
I have set up custom depth with stencil.
https://i.imgur.com/IPagu9n.png
Stencil index enable custom render.
https://i.imgur.com/5IodasJ.png
No outline
https://i.imgur.com/g2VJ9xQ.png
I sample project would be super helpful.
Thanks in advance.
Hi Tom,
Thanks for the material. I’m seeing jaggedness in the outline on 90° corners. I think it might be because my game camera is very far away from the object. Is there a way to remedy this?
https://imgur.com/a/qEEDIIa
Thanks for your time.
That’s odd, you usually only see this if the pixels don’t check the corners (eg. only up/down left/right instead of top-left, top-right etc. for sampling the stencil and depth. I’d have to look deeper into the material again to see why that would be (as afaik I do sample the corner pixels too)
Hi Tom,
Any word on how to remove the stepping from corners?
Cheers.
Hi, just tried the material but i have some problems when to make working it , can you share a sample project please? So we can understand what may be wrong.
I don’t have a stand-alone project for it to share but there may be a few steps to help:
Make sure you custom depth and stencil values are working, since this may not be enabled by default. Custom Depth should be in your project settings to enable (Stencil Index is in the same dropdown)
The basics is pretty simple to apply: you need a primitive shape to apply the material too, place it around the object to outline and make sure your material instance has the stencil index that matches that which you have set in the mesh you want to outline (eg. both on 1 or above) the mesh to outline needs to have “Render Custom Depth” enabled AND have a stencil index set to match the outline-material.
Quick question: does this work with Forward Rendering? Thanks.
It should, as Custom Depth etc. is still available in the forward rendering, although since Forward rendering is a second-class citizen in Unreal I can’t guarantee that there could be some issues or things missing (for example SceneDepth sampling SHOULD be available in forward rendering, but I haven’t checked)
https://imgur.com/ualyyVJ
Seems like it’s not working properly right out of the box! I’m not too familiar with shaders so I wouldn’t know how to modify it to potentially make it work. The shader had no errors after opening it up so nothing is technically broken, but something isn’t right.
Wanted to update after playing around with the material some more. The outlines are working, it’s just the same thing that Turk experienced: the inside of the stencil is not removing.
However, I’d like to point out that this totally works to ‘fake’ shadows unto unlit, emissive material wielding objects. Just don’t overlap primitives or the effect is multiplied. 😉
I can simply use a fresnel to give a sort of highlight when interacting with something as opposed to an outline, which, in my opinion, looks slightly cooler.
Whoops, spoke too soon! In VR, only the left eye gets rendered with the overlap, but not the right eye, creating an almost 3D effect. Anyways, still playing with it and project settings.
Whoops, spoke too soon. In VR, looking at the object creates an almost 3D effect since it’s being rendered in the left eye but not the right. Anyways, I’m continuing to play with it. Thanks for the material!
Sorry about the double post. Tried deferred, forward, instanced stereo rendering on/off, and unlit/default lit material modes on 4.22.3 to see if something different would happen but it didn’t.
It would amazing if this ended up working as a way to fake shadows in VR (for unlit/emissive materials) but since it only renders in the left eye and not the right, it’s not feasible. Is there a chance that you could look it over for a quick fix? My google-fu says it might be something to do with ResolvedView.StereoPassIndex but I’m not sure how to set it up with your material.
Thanks
This is what I’m seeing on iOS,
The inside of the stencil seems to not get removed.
https://i.imgur.com/IpXTP5X.jpg
I’ve heard someone else reply with issues on iOS while Android works fine (I don’t have an iOS device to repro). I think at that point it’s likely to be an engine bug with rendering translucent surfaces (perhaps only then those try to sample the scene buffers) You may want to report this to Epic’s rendering team.
Thanks for your work! This works perfectly in Android devices too, but in iOS, outline looks ugly. Outline displays at not only in edges, but also in meshes, like stripe pattern. I want to attach my screenshot to show you, but I can’t find a way to attach screenshot in comment.
Do you already know about this kind of problem and know the solution? If not, can you suggest where to look first in your material nodes to solve this problem?
You can post it on imgur.com
I don’t have an iOS device to check, but seeing as it works work on Android it may be an mobile renderer bug but I’m quite unfamiliar with the entire mobile side of the engine to be honest.
Managed to make the outlines soft https://i.imgur.com/joajxgP.png , but some bugs are still present. Mainly the fact that all stencils are affected with the blur. I guess the HLSL code needs some adjustments.
Here’s hoping for an update addressing obscured outlines.
Cheers!
How can i make it to show colors like after tonemapping?
Hmm that might be unavailable due to when transparency is rendered in the chain. With post-processing you can render after or before tonemapping with a simple checkbox. I’m unaware of this being available for transparency pass.
Dear Tom. Love your work!
One question. It works well in 4.22, would it be possible to make it backwards compatible with earlier versions?
Thank you, Ivan
An easy way to do that (w/ 4.22 installed) is to select all the nodes, and copy/paste them into the new material. While files can’t be loaded into older versions, the nodes can still be copied into older versions of the engine easily.
Thank you Tom for the fast reply!
Bought it. Is my understanding correct that there’s no way of making it visible through other objects? (like normal PP outline) — perhaps its biggest downside if that is the case…
In theory that is still possible. With “depth testing” disabled in the material. Although I did realize that I added in a filter to remove any pixels ‘behind’ SceneDepth in the material. You can see that by looking for the SceneDepth node on the graph (it’s somewhere on the right where I do the multiplications)
I did have a test case already where it was possible to see the outline through the walls, so its not a technique limit, although the material currently may need some adjusting to account for that (I’ll see if I can update this later on to support it out of the box or w/ a variant file)
Thanks for the reply. I hope you can add this in a future update. If you could also throw in the option to make the outlines soft, like in the linked tweet, that would literally make it the best publicly available outline material, better than any marketplace outline pack too. Do that and feel absolutely free to raise the minimum asking price to 25-30 bucks. I’ll buy it again.
I have assigned the material to the cube and turned on the Render Custom Depth and matched the StencilIndex with the material, but i am still not able to see the the outline effect. I don’t know what i am doing wrong here please help.
Oh my bad sorry.. had to change custom depth-stencil pass- Enable to Enable with stencil from project settings.
Good catch. I’ll add this to the Gumroad description for users to check their Project Settings.