Add stencil support to spatial materials

[apples]

Adds stencil buffer support to spatial materials.

Linked Issues

• Supersedes [WIP] Added stencil function and depth function #78542
• Resolves Expose an intuitive subset of stencil operations godot-proposals#7174
• Bugsquad edit: Resolves Add support for depth function in spatial materials and/or shaders (GL_GREATER, …) godot-proposals#1298
• Currently integrates Add support for depth function in spatial materials #73527 The depth function design included here has deviated from its original PR.

Testing with the sample project https://github.com/apples/godot-stencil-demo

Screenshots

Standard outline, standard x-ray, and a custom outline effect.
Made entirely with the Material settings.

A more interesting x-ray example with a custom material.

ww_light.webm

A cheap light effect that mimics the lighting effects in Wind Waker.

campfire_480.webm

A simple stylized fire effect implemented entirely with StandardMaterial3D.

Example shader

shader_type spatial;
render_mode depth_draw_never, cull_back, unshaded;
stencil_mode write_depth_fail, 2;

void fragment() {
	ALPHA = 0.0; // Currently needed because we can't control color mask.
}

Differences from the design

• Added compare operator ALWAYS. The ALWAYS operator is needed for some effects.
• Added outline and xray presets ahead of schedule because it was easy and made testing easier.
  • The way that next_pass materials are handled is a bit strange right now, but I tried to make it as non-destructive as possible.

Currently implemented

• Added stencil_mode to shaders, which works very similarly to render_mode.
  • read - enables stencil comparisons.
  • write - enables stencil writes on depth pass.
  • write_depth_fail - enables stencil writes on depth fail.
  • compare_(never|less|equal|less_or_equal|greater|not_equal|greater_or_equal|always) - sets comparison operator.
  • (integer) - sets the reference value.
• Modified the depth_test_disabled render mode to be split into depth_test_{default,disabled,inverted} modes.
  • depth_test_default - Depth test enabled, standard sorting.
  • depth_test_disabled - Depth test disabled, same behavior as currently implemented.
  • depth_test_inverted - Depth test enabled, inverted sorting.
  • VisualShader now has special handling for depth_test_ modes: The disabled mode is kept as-is and presented as a bool flag, while the other two modes are presented as a enum mode dropdown which excludes the disabled mode.
• BaseMaterial3D stencil properties.
  • depth_test - Determines whether the depth test is inverted or not. Hidden when no_depth_test is true.
  • stencil_mode - choose between disabled, custom, or presets.
  • stencil_flags - set read/write/write_depth_fail flags.
  • stencil_compare - set stencil comparison operator.
  • stencil_reference - set stencil reference value.
  • stencil_effect_color - used by outline and xray presets.
  • stencil_outline_thickness - used by outline preset.
• BaseMaterial3D stencil presets.
  • STENCIL_MODE_OUTLINE - adds a next pass which uses the grow property to create an outline.
  • STENCIL_MODE_XRAY - adds a next pass which uses depth_test_disabled to draw a silhouette of the object behind other geometry.

Supported Renderers

• Forward+
• Mobile
• Compatibility

Depth Prepass

Stencil effects work well when rendered with a second opaque pass immediately following the current opaque pass. However, with depth prepass enabled, the depth information from any stencil materials is not available for screen-space effects. For instance, with SSAO enabled, opaque stencil materials have strong visual artifacts. Due to this, there might not be a good way to support opaque stencil effects when depth prepass is enabled.

This has been resolved by simply not supporting opaque-pass stencil-read materials.

GLES3

More work is needed for GLES3 support. The depth buffer attachment for the scene FBO must be changed to a depth-stencil attachment. Making this change without breaking existing projects will be difficult.

• See f2ec38d for an experimental implementation, and find concerns in this comment: Add stencil support to spatial materials #80710 (comment).

Update: GLES3 and therefore the Compatibility renderer are now included in this PR. It may potentially cause problems for XR, but requires further testing.

Sorting Problems

Currently the biggest annoyance with sorting, is that next_pass materials aren't necessarily rendered immediately after their parent. Users have to rely entirely on render_priority to get correct sorting.

Individual stencil effects which have material passes in both the opaque and the alpha passes will have more difficulty being correctly sorted. This might make effects such as portals and impossible geometry more challenging to implement.

[QbieShay]

Woahh awesome @apples ! Thank you so much! I am busy busy until next week but I'll review it then!

[Calinou]

Tested locally (rebased on top of master ff5c884), it mostly works as expected. MSAA, TAA and FXAA correctly smooth out stencil rendering as expected too.

In the demo project, the Wind Waker Light appears much darker when using the Mobile rendering method though. I assume this is due to the use of a RGB10A2 color buffer instead of RGBA16, but maybe this is due to the 2× color multiplier that has to be applied? It doesn't affect the x-ray outline on the second screenshot further below.

Forward+	Mobile

Forward+	Mobile

I also noticed the stencil modes don't feature syntax highlighting and autocompletion in the shader editor, unlike the render modes:

Shader syntax validation also doesn't work if the stencil type has anything written after write, even if it makes no sense:

Here, I don't get a shader compilation error but I logically should.

---

I also suggest changing the default outline color in the material preset to be opaque black (or opaque white), so that you can see it immediately after setting it up. Right now, the default color is fully transparent.

PS: Using a Material Overlay property, it's possible to have an object that has both an outline and x-ray at the same time 🙂

Also, if you're curious about how a stencil outline compares with an inverted hull outline. Inverted hull on the left, stencil outline on the right:

[Calinou]

Is it possible to create stencil shadows that don't appear through walls with this PR? Logically, it seems feasible, but I've tried to modify the Wind Waker light shader to no avail.

[apples]

Is it possible to create stencil shadows that don't appear through walls with this PR?

@Calinou My intuition says that stencil shadows would require the write operations INCREMENT/DECREMENT to be exposed, but there might be some clever way to implement them without those operators.

Right now, since the write operator isn't exposed, it's just set to REPLACE internally.

the Wind Waker Light appears much darker when using the Mobile rendering method

I noticed this too and haven't had a chance to debug it. It's something to do with how the raw color values are being mixed (multiply), it seems to be clamping the values to 1.0 at some point. The xray/outline effects in my demo just use standard alpha blending with regular colors, so the problem doesn't affect them.

EDIT: If we want to expose those write operations for stencil shadows, we'll likely need to expose the compare/write masks as well. This can't really be done in the material properties for bloat reasons, but we could expose them in the shader stencil_mode without any trouble. (edit again: this is assuming it's necessary at all, which it might not be)

[QbieShay]

Tested with my project, I didn't attempt windwaker lights because I have completely forgot how the setup for that is supposed to work.

I tested my "particles passing through hole in the ground" scene and also the uotline and xray mode.

My own particles worked great! I am glad that this syntax seems indeed to be an update from the full stencil one:

Outlines&Xray: works great! The default configuration however seems to use stencil reference 0 and not work, it started working as soon as i changed it to something that isn't 0 (maybe that has to do with the mask like you were saying @apples ?)

[QbieShay]

RE: stencil shadows

I have not seen a huge demand amongst users for this. Since it seems to add some complexity to the API, i'd leave it for a future PR in case there's need for it. I'd specifically wait to see if anyone needs those features also for other reasons.

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines?
(based on this, https://www.reddit.com/r/opengl/comments/4mjkv1/comment/d3w8m28/?utm_source=share&utm_medium=web2x&context=3 )

[QbieShay]

Tested on my test project for stencil, looks good!

I've noticed this PR also adds depth test operations, which I'm assuming they're needed for the stencil write if depth fail.

While exposing the whole thing just for this may be overkill, I see no harm in adding it for the users, I think they'll find ways to use it (plus i think the requested windwaker thing isn't possible without it).

Thank you Apples! My comments on default configurations can be tweaked in a later PR (and doesn't need to be you since changing defaults is something good for a first contributor as well ^^ )

[Calinou]

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines?

This is true, but sometimes art direction prevails above all else 🙂

[QbieShay]

Oh. Didn't know they looked different! I really know nothing of them

[Lielay9]

RE: stencil shadows

I have not seen a huge demand amongst users for this. Since it seems to add some complexity to the API, i'd leave it for a future PR in case there's need for it. I'd specifically wait to see if anyone needs those features also for other reasons.

Wasn't the original proposal #3373 made explicitly because of stencil shadows? I can attest to that at the very least one of the 40 thumbs ups was for those. If they won't be supported the proposal should probably be reopened. 👀

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines? (based on this, https://www.reddit.com/r/opengl/comments/4mjkv1/comment/d3w8m28/?utm_source=share&utm_medium=web2x&context=3 )

I wouldn't put much weight to a 7 year old comment with 5 likes. As Calinou alluded to, stencil shadows have a distinct look and properties that are better fit for stylized games. I've commonly encountered them in platforming games, where the environment uses shadow maps and characters have down pointing stencil shadows. I could've swore some of the mario games worked like this but I can't recall which.

[QbieShay]

@Lielay9 could you test this PR, and see if it supports them? I don't understand how they work, so I can't quite test them myself.

You can go on the "checks" tab and at the right there is an "artifacts" menu drop down - you'll find a build for this PR there.

Wasn't the original proposal godotengine/godot-proposals#3373 made explicitly because of stencil shadows? I can attest to that at the very least one of the 40 thumbs ups was for those. If they won't be supported the proposal should probably be reopened. 👀

The proposal was indeed for stencil shadows specifically, but they cannot be done without stencil at all. It should be reassessed with this API.

[apples]

@QbieShay

The default configuration however seems to use stencil reference 0 and not work, it started working as soon as i changed it to something that isn't 0 (maybe that has to do with the mask like you were saying @apples ?)

Nothing to do with masks, it's just that the buffer is initialized to all zeros every frame, so choosing 0 as a default reference value is kind of useless. It could easily be changed to 1 as the default.

I've noticed this PR also adds depth test operations, which I'm assuming they're needed for the stencil write if depth fail.

Some stencil effects (Wind Waker lights in particular) will need the depth test operator exposed. Since #73527 seems like it's on track to be merged, I included it in this PR for testing purposes. I'm hoping that PR gets merged first so I can just rebase this one on top of it.

[apples]

Regarding stencil shadows:

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines?

Modern shadows are typically based on shadow maps, which can be a bit fuzzy/pixelated since they are stored as textures. Stencil shadows, in comparison, use geometry-based stencil volumes instead of textures, so they have infinitely crisp edges. Some users will desire this for artistic reasons, typically because they're trying to follow the style of retro games.

If they [stencil shadows] won't be supported the proposal should probably be reopened.

I don't think that'll be necessary. This PR is a good foundation, and will be very easy to expand upon in future PRs.

Additionally, stencil shadows might require some other changes not directly related to stencils, such as exposing depth clamping, which is needed for infinite shadow volumes. Also, finding ways to generate the infinite shadow volumes in the first place and avoid culling them is outside the scope of this PR.

Rest assured: I am one of those people who wants stencil shadows. It is not something forgotten with this PR. It just might take multiple PRs to fully support them.

[QbieShay]

Some stencil effects (Wind Waker lights in particular) will need the depth test operator exposed. Since #73527 seems like it's on track to be merged, I included it in this PR for testing purposes. I'm hoping that PR gets merged first so I can just rebase this one on top of it.

@clayjohn ?

[Lielay9]

2023-08-24.01-26-39.mp4

Well, I guess this is best you can do now. Works if volumes don't overlap; a simple torus, for example, will break it.

Calinou My intuition says that stencil shadows would require the write operations INCREMENT/DECREMENT to be exposed, but there might be some clever way to implement them without those operators.

These are indeed necessary for shadows. Besides that, the only thing missing would be culling the shadows correctly which is outside the scope of this PR. Thereafter, stencil shadows are viable, if not exactly painless to use.

Extra note: Some primitive shapes seem to have small holes/gaps and therefore didn't play too nicely with the code generating the shadow meshes.

[Calinou]

Extra note: Some primitive shapes seem to have small holes/gaps and therefore didn't play too nicely with the code generating the shadow meshes.

Could you post your code somewhere? I think the current primitive meshes are fine as they are, although some of them have hard seams as you'd typically expect (e.g. boxes aren't smooth-shaded).

[akien-mga]

Thanks for the amazing feature work @apples!

We managed to finally integrate this before its 2-year anniversary mark, and your patience and flexibility during that long review process has been really great. Thanks to everyone involved reviewing, designing and testing this feature!

[stuartcarnie]

Great work, @apples

May I suggest you include some screen shots of your demo in your PR description, as it will help user's coming to this PR to understand the visuals.

[AThousandShips]

Amazing work and excited to try this out!

[akien-mga]

May I suggest you include some screen shots of your demo in your PR description, as it will help user's coming to this PR to understand the visuals.

Might also be worth including https://github.com/apples/godot-stencil-demo in https://github.com/godotengine/godot-demo-projects if you're up for it.

[catboomer1105]

great work! we can do this now >_<

[Glorax]

@apples

I'm hoping some clever person will figure out how to use this to do Antichamber-style optical illusions or seamless portals without using additional viewports.

Rendering portals this way requires the stencil buffer technique used by Valve when they made the Portal series. I'm not aware of any engine apart from Valve's Source engine, and possibly Unity, that is capable of this. This is part of the reason why I abandoned Unreal for Godot, because Unreal is too difficult to mod.

Does this PR support the stencil buffer technique, or will that require additional engine modifications?

https://www.reddit.com/r/unrealengine/comments/15evwzu/comment/kctu5vp/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[tmilker]

@Glorax

Does this PR support the stencil buffer technique, or will that require additional engine modifications?

Godot doesn't need stencils to do portals. They're possible in 4.4 right now, the only thing missing is a custom camera projection to setup an oblique clipping plane for the portal camera. I think even stencil based portals would need this. See:

godotengine/godot-proposals#11436
#85529

@BastiaanOlij has it on the list to implement, which I hope is soon.

[apples]

Rendering portals this way requires the stencil buffer technique used by Valve when they made the Portal series.

Godot doesn't need stencils to do portals. They're possible in 4.4 right now, the only thing missing is a custom camera projection to setup an oblique clipping plane for the portal camera. I think even stencil based portals would need this.

A limited form of stencil portals is possible with this PR, but it requires everything to be put into the alpha pass and manually sorted using render_priority. Not very feasible for complex scenes, but it works for small tricks.

The key feature Godot would need to support Portal-style stencil portals is a multi-pass/programmable render pipeline (and maybe software stencil buffers).

Viewport-based portals are much more flexible and possible now, although the math involved is quite tricky.

[Glorax]

@tmilker

Godot doesn't need stencils to do portals. They're possible in 4.4 right now, the only thing missing is a custom camera projection to setup an oblique clipping plane for the portal camera. I think even stencil based portals would need this. See:

I realize, but my impression is that the stencil buffer technique is the only way to implement them that is reasonably performant. Otherwise, you're re-rendering the scene for every portal surface, which is horrible for performance.

Unreal had this exact same problem, and I left because it was impossible to use the buffer technique without heavy engine modifications that nobody knew how to do. The only way to do portals in Unreal was the horrifically slow scene capture method, which I presume is analogous to Godot's viewport method.

I've tested the viewport method in Godot, and from what I've observed, it effectively multiples the performance load of SDFGI, ambient occlusion, and the like.

[Glorax]

@apples

Viewport-based portals are much more flexible and possible now, although the math involved is quite tricky.

I'd love to use the viewport portals, but it's just not practical in a game with anything above the most basic graphics. For something like Antichamber, it'd work. For a game with realistic graphics, absolutely not. Every additional viewport multiplies the performance load of the lighting passes.

The performance impact of Valve's method isn't negligible, and it gets quite high with recursive rendering in the mix, but it's a vast improvement over the roughly 50% performance tax of the scene capture / viewport method.

Also, in the past, I've had serious issues with getting the portals to look seamless, as some post-processing effects seem to apply themselves to not just the main viewport but also the nested portal viewports, causing the portal image to look overly bright and saturated. Perhaps stencils will help mitigate this problem, even with the viewport method.

Not very feasible for complex scenes, but it works for small tricks.

What kinds of complex scenes, and what sorts of small tricks?

[akien-mga]

I'd suggest moving this discussion to another platform (e.g. the Godot forum), as it's no longer relevant to the context of this merged PR.

If there are actionable improvements to make to Godot to support such use cases, you can open a new proposal on godot-proposals to discuss them.

[JeeeJeee]

Does this PR also expose the stencils to compute shaders in any way? Is there some buffer texture somewhere that can be sampled similar to how Depth is sampled using RenderSceneBuffersRd.GetDepthLayer()

[shafnaz]

Any solutions to the face separation for sharp corners? I havn't found any non-shader appraoch to this yet.

[Delsin-Yu]

Any solutions to the face separation for sharp corners? I havn't found any non-shader appraoch to this yet.

Because these faces are being extruded away from their vertex normals, and since these vertices have sharp edges, their normals are not rounded. There are several existing approaches to address this problem.

See Pixel-Perfect Outline Shaders for Unity/Handling sharp edges for more info.

Sharp edges in 3D meshes are achieved by duplicating vertices along the edge. Each side of this cube has four unique vertices with normals oriented perpendicular to those of adjacent sides. When we translate these vertices’ positions along their normals, it’s equivalent to exploding the sides of the cube.

1. One way to address this problem is to make the edges of the cube smooth, and bevel them. This will impact the look of the cube itself.
2. Another option is to use a separate mesh that has smooth normals exclusively for drawing the outline. This requires us to author or generate in a script a separate mesh for each mesh that has sharp edges that we’d like to outline.
3. Finally, we can store smooth normal data in another channel of the mesh we aren’t using, e.g. in the vertex colors.

[SomeRanDev]

Does this PR also expose the stencils to compute shaders in any way? Is there some buffer texture somewhere that can be sampled similar to how Depth is sampled using RenderSceneBuffersRd.GetDepthLayer()

At least for compatibility, it appears the stencil values are contained on the depth buffer?? But I'm far from an expert on this, so I'm not sure how it works. If there is a way to sample from the stencil buffer through a CompositorEffect (or even better, on gdshader), could someone document it?

[yoyotam3]

It was mentioned earlier in the PR that increment/decrement operations were not exposed, meaning it is not possible to implement stencil shadows. Is this still the case?

[QbieShay]

@shafnaz that's an issue with any mesh with sharp edges unfortunately.

@yoyotam3 increment and decrement are indeed not exposed. please open a proposal to detail your usecase, in particular what are the benefits of stencil shadows (linking other articles it's fine too). I suspect we'll run into the current limitation of reading being possible only in the alpha pass and where stencil happens in the pipeline, but I may be wrong.

We need to understand if exposing those feature works with the current limitations, otherwise we will expose them later down the line.

[QbieShay]

@JeeeJeee there's been no explicit work to expose to compositor, not sure if it works with compute.

[tmilker]

@QbieShay

in particular what are the benefits of stencil shadows (linking other articles it's fine too)

The reason increment is needed is almost as old as stencil shadows themselves: Without it, you get double(or more) darkening from overlapping parts of the model. I guess you could maybe get around it with an if in your shader but those aren't good for performance in shaders, especially fragment ones.

This is an image from a slide show by Mark Kilgard in 1998. Here's the full talk, still archived 27 years later.

This is just the simple version of stencil shadows too. For rendering shadow volumes properly, you increment and decrement the stencil buffer. See https://ogldev.org/www/tutorial40/tutorial40.html

[QbieShay]

Please put that in a proposal so it doesn't get lost

[yoyotam3]

@QbieShay, the benefits of stencil shadows themselves, as mentioned by others, are mostly stylistic. They do also allow for volumetric shadowing though which I don't think is as easily achievable with shadowmaps. I don't think only being available in the alpha pass should be an issue, given the current system can be used to implement wind waker-style lighting which uses the stencil buffer in a similar manner. But I'm not entirely sure, I haven't implemented stencil shadows before.

[yoyotam3]

I have created a proposal for the required features here, please tell me if there are any issues with it.

[QbieShay]

Thank you! No worries, style benefits are in fact very very important - we make games and games are art :)

[eimfach]

Will this be in 4.5 still ? Or is it there already ? Can't see to read smth about it..

[AThousandShips]

It is already in 4.5, it's mentioned in the beta1 release