Skip to content

Improvements to multiple layer support #108156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

BastiaanOlij
Copy link
Contributor

@BastiaanOlij BastiaanOlij commented Jul 1, 2025

Introduction

For advanced XR headsets we're required to render more than a single stereoscopic result. This approach has now been added as a standard to OpenXR for use by multiple headset vendors and support for it is growing. The Khronos Group is funding work for us to implement this.

This primary use case is well described in Varjo's documentation on the subject.
Here we are rendering two stereoscopic buffers of different resolutions and with different camera positions and projection settings.

This PR makes changes to the rendering engine in support of this.

Alternatives

Before settling on this approach we trailed two alternative solutions for this:

  1. Increasing the number of layers we support for multiview from 2 to 4. This is the quickest and least intrusive option but does not cater for the requirement of different resolutions and introduces an increased memory footprint even when the feature is not used.
  2. Using a separate viewport for the extra rendering, while promising as a solution and possibly worth exploring further, this approach ultimately added complexity to the viewport system that is unwieldy for the end user.

Bonus objectives

The approach in this PR solves more than the ability to render multiple sets of views in a viewport. It also:

  1. Moves some of the XR logic out of the rendering core and into the viewport allow us to setup multiview viewports outside of an XR use case. Once we also perform needed camera improvements this will enable a number features closed off to us such as stereo mirrors and stereo portals.
  2. Introduces the ability to increase the number of views beyond the two we support for multiview and loop through them in pairs. This both results in finally having a fallback (which has been requested many times for webXR) when multiview is not supported, and allows us to render for multiscopic displays such as for looking glass.

TODO

This is still a work in progress:

  • complete OpenXRInterface changes for quad view support.
  • complete fallback logic including adding a check for asymmetry in projection matrices and correct use of view count.
  • cleanup _check_xr_size logic in viewport.
  • test if turning use_xr on/off behaves correctly
  • add in compatibility functions so the API remains backwards compatible!!!! <-- must have, very important.
  • ...

Implements: godotengine/godot-proposals#12572

Contributed by Khronos Group through the Godot Integration Project

@BastiaanOlij BastiaanOlij added this to the 4.x milestone Jul 1, 2025
@BastiaanOlij BastiaanOlij self-assigned this Jul 1, 2025
@BastiaanOlij BastiaanOlij force-pushed the improve_multilayered_rendering branch 2 times, most recently from bf479d9 to 9fb28dc Compare July 2, 2025 07:37
@@ -2704,6 +2706,9 @@ void RendererSceneCull::render_camera(const Ref<RenderSceneBuffers> &p_render_bu
// For now just cull on the first camera
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dsnopek I haven't had time to test it yet but I believe this is where the issues with occlusion come from. We're initialising the occlusion system for the left eye instead of the combined frustum.

} else if (view_count == 2) {
camera_data.set_multiview_camera(view_count, transforms, projections, false, false, camera->vaspect);
} else if (p_view_count == 2) {
camera_data.set_multiview_camera(p_view_count, transforms, projections, false, false, camera->vaspect);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dsnopek @m4gr3d This is the other bit not directly related to this PR that I want to dive into more.

As you guys know, the logic that combines the view frustums of the left and right eye creates an encompassing frustum that we use for culling but in order for this to fit, we move the center position of the camera backwards.

As the player moves their head, this position will thus start rotating around the head position.

I think this may be what is causing the shadow flickering we've experienced in XR as this may trigger unnecessary movement in the directional light cascades.

Not as part of this PR but I might see if I can adjust this logic to keep the resulting transform centered on the eyes and encode the offset in the projection matrix. Just wanted to note it here so we don't forget about it.

@BastiaanOlij BastiaanOlij force-pushed the improve_multilayered_rendering branch from 9fb28dc to abc5748 Compare July 2, 2025 07:53
@BastiaanOlij BastiaanOlij force-pushed the improve_multilayered_rendering branch from abc5748 to 439bed7 Compare July 9, 2025 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant