Fixing Fragment Shader Silent Optimization Issue (Issue #284)

[zmr-233]

This PR addresses Issue #284, where the fragment shader successfully compiles but is silently optimized to have no output operations, causing silent rendering without errors or warnings, making debugging very difficult.

⚠️ ERROR or WARN ?

After adding this feature, it was discovered that many of the previous test cases no longer pass in the Github CI, as they are completely optimized out by DCE (Dead Code Elimination). Upon analysis, directly marking fragment shaders with no output as errors does indeed lead to unnecessary test failures.
In the current PR, I have still set this as an ERROR, as whether to use WARN or ERROR is yet to be decided by the maintainers. I hope to discuss this issue further to make the most reasonable decision.

Root Cause Analysis

After a deep analysis of DCE (Dead Code Elimination), we discovered that Issue #284 is actually a "false scenario," based on the following two core facts:

1. The Entry-point function is never deleted: In the current DCE implementation, all functions in the module’s OpEntryPoint list are unconditionally marked as "root," becoming reachable rooted objects. In other words, even if all instructions in the function body are identified as pure calculations and removed, the function definition and OpEntryPoint declaration are still retained. DCE will not remove the entire entry-point function.

2. The phenomenon of "silently compiling out" is actually the simplification of the function body, not the disappearance of the entire entry-point: When no "obvious output" is written, DCE will remove internal calculations (such as OpCompositeConstruct, temporary OpLoad, etc.) and leave only a direct OpReturn. However, it will not remove %main_fs = OpFunction... or OpEntryPoint Fragment %main_fs "main_fs" .... In the SPIR-V module generated by the compiler, this fragment shader still exists—it's just that it no longer writes to the Output variable.

Therefore:

• The phenomenon in Issue Fragment shader can be silently compiled out #284 where "the entire fragment shader is silently compiled out" is actually a misunderstanding of the behavior "function body is simplified to not write outputs" ⇢ "no color is written."

• The initially proposed "silent compile-out" didn’t actually happen: The SPIR-V still contains empty functions and entry-point declarations; it’s just that nothing is done during the rendering phase.

Solution

Thus, the solution is to add a check/error for fragment shaders with "empty outputs" so that users can immediately know: "Hey, this fragment shader doesn't write anything to output, it’s definitely a bug," instead of generating an invalid SPIR-V that "renders nothing."

Therefore, I added a validation step after DCE to check for fragment shaders with no output operations:

• validate_fragment_shader_outputs(): The main validation function called after DCE.
• fragment_shader_writes_output(): Recursively checks for output operations.
• has_partial_output_writes(): Detects evidence of partial writes that were optimized away.
• is_output_storage_variable(): Identifies output variables in different scopes.

Example Error Message

A test case that always fails was added to verify that the silent failure would trigger an error message:

• tests/compiletests/ui/dis/issue-284-dead-fragment.rs: A fragment shader with no output (correctly fails and provides a clear error).

Before the fix (silent failure): The shader compiled successfully but rendered nothing.

After the fix (clear error and guidance):

error: fragment shader `main_fs` produces no output
  |
  = help: fragment shaders must write to output parameters to produce visible results
  = note: use complete assignment like `*out_frag_color = vec4(r, g, b, a)` instead of partial component assignments
  = note: partial component assignments may be optimized away if not all components are written

Since the validation only runs after DCE, doesn't affect the hot compilation path, and only checks fragment shaders (not vertex/compute shaders), it has a minimal performance impact.

[Firestar99]

This addition is failing CI, it's probably something that should remain a local change on your machine.

Thanks for your contribution, will review the rest later :D

[zmr-233]

Thanks for the feedback! Understood, I'll revert this change and keep it local on my machine. I'll wait for the review of the rest of the changes.

Thanks for your help! :D

[Firestar99]

@zmr-233 You don't need to close the PR, just update this one and keep pushing to the same branch

[zmr-233]

I’ve removed the local [profile.dev] codegen-backend = "llvm" change, so that’s no longer affecting CI. The CI failures you see are expected, as they’re caused by the new validation I added validate_fragment_shader_outputs.

Some existing tests are now failing because of the “no output in fragment shader” error.
I set it as an error, but we can switch it to a warning if you prefer. Let me know how you want to handle it :D