Policy: Empower reviewers to reject burdensome PRs

[jieyouxu]

Note

Wording draft: https://hackmd.io/@rust-compiler-team/S1iwe5GBxx

Proposal: Empower reviewers to reject burdensome PRs

Reviewer time and effort are very precious and limited resources. However, there are Pull Requests (PRs) with certain qualities that place undue burden on reviewers. We are proposing a policy that reviewers can use as justification for rejecting a PR without them having to excessively defend their position or relitigate the reasons for closing.

Motivation

In recent times, there are rather frequent contributions that are fully or partially produced by generative AI (e.g. LLMs and friends) which exhibit burdensome characteristics (see below) that cause a lot of burden on reviewers. While we've seen PRs with burdensome characteristics produced entirely by humans, generate AI tools have significantly lowered the level of effort required to produce "plausibly-looking" contributions that are entirely inadmissible, and so have become a major source of burdensome PRs. This policy is thus drafted as a response targeted at the problem of an increasing frequency of burdensome PRs, without requiring reviewers to figure out how they are produced.

PR characteristics that are burdensome to reviewers

• Needlessly or overly verbose descriptions or responses.
• Not internally coherent or even self-contradictory.
• Demonstrates misunderstanding of important aspects of what the code is doing or the purpose of the change.

It is okay to not be fully confident in a proposed change, but this should be disclaimed on the PR itself.

Policy

To mitigate the burden on reviewers, when encountering PRs with one or more of the characteristics, reviewers may choose to exercise one or more of the following options:

• Ask the PR author to make their PR descriptions or responses concise.
• Edit out verbose PR descriptions.
• Reject (close) the PR.

Mentors or Reviewers

None specifically; compiler team.

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Caution

Concerns (1 active)

• wait-for-some-discussions-to-settle resolved in this comment
• make-sure-messaging-does-not-imply-discouragement-of-ai-tools-as-aid resolved in this comment
• drop-disclosure-requirement resolved in this comment
• focus-on-the-problem resolved in this comment
• ai-should-still-be-mentioned-in-motivation resolved in this comment
• ai-should-still-be-mentioned-in-motivation resolved in this comment
• wording-nits

Managed by @rustbot—see help for details.

[rustbot]

Important

This issue is not meant to be used for technical discussion. There is a Zulip stream for that.
Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections can formally be registered here by adding a comment.

@rustbot concern reason-for-concern
<description of the concern>

Concerns can be lifted with:

@rustbot resolve reason-for-concern

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler

[workingjubilee]

@rustbot second

[jieyouxu]

This deserves some more baking to discuss how we should frame this, most importantly, what this does not try to reject/discourage.

@rustbot concern wait-for-some-discussions-to-settle

[jieyouxu]

it's less about the intent of the MCP, and more about whether there is messaging in there that people using AI as a dev aide (i.e. not as a spam tool), would be pushed away from contributing or made feel inadequate as devs

@rustbot concern make-sure-messaging-does-not-imply-discouragement-of-ai-tools-as-aid

[jieyouxu]

I don't personally feel very strongly about having a declaration or not -- that could be a separate proposal. My core intention for this proposal was to have a

backing policy like this [that] allows the reviewer to close the low-effort PR quickly, without needing to spend a lot of energy or effort arguing.

Whether requiring disclosure or not IMO doesn't matter as much for this purpose.

Probably consider dropping the disclosure requirement, as it's not clear what should require a disclosure -- does simply using copilot for auto-completion for a very small subset of a PR count? That is likely better as a separate proposal.

@rustbot concern drop-disclosure-requirement

[jieyouxu]

AI tools are simply a method to produce overly verbose descriptions/responses and invalid changes, but is not the only way to arrive at them. We should focus on the problems, since there are many methods to arrive at them.

@rustbot concern focus-on-the-problem

[RalfJung]

@Kivooeo please read this comment.

[jieyouxu]

I adjusted this policy proposal following feedback on the thread. This policy is now focused only on burdensome PRs, and intentionally does not mention how they are produced (AI or not). Any responsible AI usage or AI usage disclaimer policies are intentionally beyond the scope of this MCP (but may be follow-ups).

EDIT: Generate AI is mentioned but only in the motivation capacity as "lowering the barrier to produce plausibly-looking contributions" without being focused on how burdensome PRs are produced.

@rustbot resolve make-sure-messaging-does-not-imply-discouragement-of-ai-tools-as-aid
@rustbot resolve drop-disclosure-requirement
@rustbot resolve focus-on-the-problem

[jieyouxu]

We seem to have a convergence on focusing on the characteristics of burdensome PRs (which seems uncontroversial) instead of the methodology of producing those, so:

@rustbot resolve wait-for-some-discussions-to-settle