Add support for rasterizing mesh to texture with custom shader

[beicause]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Related:

• Implement Drawable Textures #7379

One feature that is widely requested in Godot is the ability to easily have a texture and just draw to it (or even run a custom shader).
Examples of things you want to do:
Animated textures
Smoke and fire using ping-pong transfers
Water plane effects
Effects such as painting
etc.

However, the drawable texture proposal doesn't address the support for mesh/vertex shader yet.

Current MeshTexture is just a wrapper for canvas_item_add_mesh() and doesn't store texture data itself, which means it can't be used in shader uniform or 3D scene. And it can't do regional drawing.

ViewportTexture is heavy and not convenient for just rendering a simple mesh. In addition, ViewportTexture can't render 3D mesh if compling engine with 3d disabled because Node3Ds is not available.

Moreover, this is also a way to perform GPGPU using the rasterization pipeline.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Implement mesh rasterizer api and RasterizedMeshTexture. This texture accepts a mesh and a ShaderMaterial to do custom drawing.

Main differences from DrawableTexture2D:

• It supports vertex shader and vertex attributes (UV and vertex color)
• It clears the texture before each drawing, instead of blit the texture mult-times
• It only writes one color output to itself. DrawableTexture2D can write up to 4 color outputs simultaneously.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

My proposal PR is here: godotengine/godot#106331

Current api desgin:

	enum RasterizedTextureFormat {
		RASTERIZED_TEXTURE_FORMAT_RGBA8,
		RASTERIZED_TEXTURE_FORMAT_RGBA8_SRGB,
		RASTERIZED_TEXTURE_FORMAT_RGBAH,
		RASTERIZED_TEXTURE_FORMAT_RGBAF
	};
	virtual RID mesh_rasterizer_create(int p_width, int p_height, RasterizedTextureFormat p_texture_format, bool p_generate_mipmaps = false) = 0;
	virtual void mesh_rasterizer_set_bg_color(RID p_mesh_rasterizer, const Color &p_bg_color) = 0;
	virtual void mesh_rasterizer_set_mesh(RID p_mesh_rasterizer, RID p_mesh, int p_surface_index) = 0;
	virtual void mesh_rasterizer_set_material(RID p_mesh_rasterizer, RID p_material) = 0;
	virtual void mesh_rasterizer_draw(RID p_mesh_rasterizer) = 0;
	virtual RID mesh_rasterizer_get_texture(RID p_mesh_rasterizer) = 0;

And the core shader:

#[vertex]
#version 450

#VERSION_DEFINES

layout(set = 0, binding = 0, std430) restrict readonly buffer GlobalShaderUniformData {
	vec4 data[];
}
global_shader_uniforms;

#include "samplers_inc.glsl"

layout(location = 0) in vec3 vertex_attrib;
layout(location = 1) in vec2 uv_attrib;
layout(location = 2) in vec4 color_attrib;

layout(location = 0) out vec2 uv_interp;
layout(location = 1) out vec4 color_interp;

#ifdef MATERIAL_UNIFORMS_USED
/* clang-format off */
layout(set = 1, binding = 0, std140) uniform MaterialUniforms {
#MATERIAL_UNIFORMS
}
material;
/* clang-format on */
#endif

#GLOBALS

void main() {
	// Flip y-axis and convert z-axis to [0,1].
	mat4 m = mat4(
			1.0, 0.0, 0.0, 0.0,
			0.0, -1.0, 0.0, 0.0,
			0.0, 0.0, 0.5, 0.0,
			0.0, 0.0, 0.5, 1.0);
	vec4 vertex = m * vec4(vertex_attrib, 1);
	vec2 uv = uv_attrib;
	vec4 color = color_attrib;

	{
#CODE : VERTEX
	}

	uv_interp = uv;
	color_interp = color;
	gl_Position = vertex;
}

#[fragment]
#version 450

#VERSION_DEFINES

layout(set = 0, binding = 0, std430) restrict readonly buffer GlobalShaderUniformData {
	vec4 data[];
}
global_shader_uniforms;

#include "samplers_inc.glsl"

layout(location = 0) in vec2 uv_interp;
layout(location = 1) in vec4 color_interp;

layout(location = 0) out vec4 frag_color;

#ifdef MATERIAL_UNIFORMS_USED
/* clang-format off */
layout(set = 1, binding = 0, std140) uniform MaterialUniforms {
#MATERIAL_UNIFORMS
}
material;
/* clang-format on */
#endif

#GLOBALS

void main() {
	vec2 uv = uv_interp;
	vec4 color = color_interp;

	{
#CODE : FRAGMENT
	}

	frag_color = color;
}

And a new resource RasterizedMeshTexture, which is a wrapper of RenderingServer api.

If this enhancement will not be used often, can it be worked around with a few lines of script?

This can be worked around using RenderingDevice ( See also https://github.com/beicause/GodotMeshTextureRd ), but not convenient

Is there a reason why this should be core and not an add-on in the asset library?

It's core

[ydeltastar]

Can this support supersampling like Viewport.scaling_3d_scale to improve rendering quality like in godotengine/godot#97468?

[beicause]

I think it's easier to add MSAA support.

Alternatively, you can render a larger texture and then render it into a smaller quad mesh, but it doesn't make sense and would be better to just use a larger texture. Or you can use RenderingServer.texture_set_size_override() to override the size of texture, though it's currently broken godotengine/godot#104500

[clayjohn]

I think this should be a part of the drawable texture API instead of being its own resource type (Proposal here: #7379).

But it would help to understand what the game you are working on is and why this is required first.

[beicause]

I don't think the current drawable texture is better than this proposal for drawing mesh. It is still hard to use in terms of UX.

I think this offers an additional way of GPU-accelerated procedural texture generation, rather than using compute shaders. And it can nest itself for multi-pass processing.

I think this should be a part of the drawable texture API instead of being its own resource type (Proposal here: #7379).

This can be a part of the drawable texture or not. DrawableTexture performs multiple blits without clearing the texture. I think its use cases are different. You can also use RasterizedMeshTexture alongside DrawableTexture

[ydeltastar]

DrawableTexture performs multiple blits without clearing the texture.

This API should do the same; otherwise, we won't be able to iterate mesh rasterization for things like overlays and brush effects.

I agree with clayjohn that this could be merged with DrawableTexture. If we want to generate a texture like the one below, we would first blit a 2D shadow texture and then rasterize a mesh over it.

The process would be easy if DrawableTexture had one more function, which combines the API described here.

void rasterize_mesh(RID p_mesh, int p_surface_index, RID p_material, const Color &p_bg_color);

As for implementation, isn't a texture blit just a call to rasterize_mesh() with a quad and a basic texture material?

I don't think the current drawable texture is better than this proposal for drawing mesh. It is still hard to use in terms of UX.

Once the RenderingServer API is done, we can create any resource type based on it. We can have both the DrawableTexture for manual advanced use and the MeshTexture-like resource where we just set a mesh and it handles everything. In fact, we could just update MeshTexture to utilize the new API.

[beicause]

This API should do the same; otherwise, we won't be able to iterate mesh rasterization for things like overlays and brush effects.

You can nest rasterized mesh texture itself to iterate. Or blit rasterized mesh texture in drawable texture.

IMO it's more common to redraw the entire texture after updating material parameters to draw animation frame by frame.

rasterize_mesh(RID p_mesh, int >_surface_index, RID p_material, const Color &p_bg_color);

Uploading a mesh is expensive.

This makes the API more complex and difficult to optimize, as well as makes users harder to understand.

[beicause]

It's easy to add clear color and color blend state arguments to my PR.
For example ( This is not user-friendly, but can achieve custom blend ):

	virtual void mesh_rasterizer_draw(RID p_mesh_rasterizer, RID p_material, const Color &p_bg_color, bool p_clear = true, const Ref<RDPipelineColorBlendState> &p_color_blend_state = Ref<RDPipelineColorBlendState>()) = 0;

I just feel drawable texture is somewhat over-designed.

[beicause]

Here is another API proposal, which separates drawable texture from mesh rasterizer, so you can use multiple mesh rasterizer to draw in the same texture:

	/* TEXTURE DRAWABLE API*/

	enum TextureDrawableFormat {
		TEXTURE_DRAWABLE_FORMAT_RGBA8,
		TEXTURE_DRAWABLE_FORMAT_RGBA8_SRGB,
		TEXTURE_DRAWABLE_FORMAT_RGBAH,
		TEXTURE_DRAWABLE_FORMAT_RGBAF
	};

	virtual RID texture_drawable_create(int p_width, int p_height, TextureDrawableFormat p_texture_format, bool p_use_mipmaps = false) = 0;
	virtual void texture_drawable_generate_mipmaps(RID p_texture_drawable) = 0;

	/* MESH RASTERIZER API */

	enum RasterizerBlendMode {
		RASTERIZER_BLEND_MODE_CLEAR,
		RASTERIZER_BLEND_MODE_MIX,
		RASTERIZER_BLEND_MODE_ADD,
		RASTERIZER_BLEND_MODE_SUB,
		RASTERIZER_BLEND_MODE_MUL,
		RASTERIZER_BLEND_MODE_PREMULT_ALPHA
	};

	virtual RID mesh_rasterizer_create(RID p_mesh, int p_surface_index) = 0;
	virtual void mesh_rasterizer_draw(RID p_mesh_rasterizer, RID p_material, RID p_texture_drawable, RasterizerBlendMode p_blend_mode, const Color &p_clear_color, RD::TextureSamples p_multisample = RD::TEXTURE_SAMPLES_1) = 0;

cc @ColinSORourke Any thoughts of the API?
See also my implemention of the above drawable texture in RD renderer: https://github.com/godotengine/godot/pull/106331/files#diff-8015f93172c06761438fa715a3829e9cee65c23cc96496bc758ad7623b980a2e

Edit: More thoughts: it's possible for drawable texture to support all rd texture format that godot supports, just needs to pass RD::DataFormat as argument. And also possible to pass custom color blend state to blend.

[ydeltastar]

That works great. I think the major flaw of the first design is that the rasterizer hides the drawable texture. Being able to pass it as an argument makes the system much less limited.