Investigate options for automated PR reviews

[aneta-petrova]

We currently use certain Vale style rules to help automate PR reviews. The goal of this issue is to investigate other automation options for PR reviews to further enhance efficiency and reduce the documentation team's workload when reviewing PRs.

[aneta-petrova]

The following is a summary of the tools I'm investigating and the project guidelines I'm reviewing for automation:

Sourcery: AI-based review automation

https://sourcery.ai/ provides AI-generated code reviews. theforeman/foreman_rh_cloud#1124 (review) is an example of how it's used in foreman_rh_cloud.

https://docs.sourcery.ai/Code-Review/Configuration/#review-rules describes how to add review rules to provide context for the AI reviews. We could use this to define requirements such as quality standards.

Project-specific Vale rules: AI-free review automation

Not all conventions are suitable for AI reviews. For example, ensuring compliance with the one-sentence-per-line rule or checking how many words are used in a heading seem better suited for Vale. We could introduce a project-specific Vale style for some of these (and add others to https://github.com/redhat-documentation/vale-at-red-hat/ which we already use).

Automating existing contribution guidelines

A good starting point for investigating whether further PR automation is useful would be to review https://github.com/theforeman/foreman-documentation/blob/master/CONTRIBUTING.md. The questions to ask are: Which of these conventions could be implemented as a Vale rule? Which could be implemented as a review rule for Sourcery?

[aneta-petrova]

I'm currently working on an MVP version of both Sourcery and a project-specific Vale style.

[aneta-petrova]

I'll share random notes that I've taken and call it a progress update.

For the Vale part, #4465 is now acked. It adds a custom Vale style for our project. Right now it only contains a rule to check for one-sentence-per-line violations but in the future could be extended with more project-specific rules. Looking at https://github.com/theforeman/foreman-documentation/blob/master/CONTRIBUTING.md, another thing that we could move from the document to the Vale style are capitalization rules and branded terminology (Foreman, orcharhino, Satellite, etc.). However, that's for the future, not part of this issue.

For the Sourcery (AI review) part, I've tried a few basic things. These are some of the AI-generated review comments I like best:

After these initial tests, I want to approach the Sourcery part more systematically now. https://redhat-documentation.github.io/peer-review/ is a resource that has recently been removed from CONTRIBUTING.md because it's kind of impractical to use it when reviewing PRs. However, it could be a source of material for AI review rules.

One thing I dislike very much is that I don't see an option to save the AI review rules in our repo and keep them under version control. Sourcery docs mention .sourcery.yaml but only as a legacy configuration option. So far I've only been using the UI page for adding review rules.

[aneta-petrova]

Hi all, per the sync meeting earlier today, I enabled Sourcery AI in our repo.

In the Sourcery web UI, I configured these review instructions:

Path patterns: guides/common/modules/proc*.adoc
Instructions: - Troubleshooting and error recognition steps are included where appropriate.
- Verification steps are included where appropriate.

Path patterns: guides/common/*.adoc,guides/common/modules/*.adoc
Instructions: - Documentation describes a user story rather than a product feature.
- Documentation uses user-centric language rather than product-centric language.
- Avoid passive voice in verbs.

I chose these because:

• AI proved quite good at suggesting active voice alternatives to passive voice.
• Per an earlier (downstream only) meeting, we have room for improvement for user-story-based and user-centric documentation.
• The troubleshooting suggestions seem to be well targeted.

I'd rather not define too much, at least not in the beginning, to avoid too much noise.

The Sourcery bot will reply to all PRs upon their creation. This is defined on the level of the GitHub organization so shared between Foreman repos. We cannot disable this just for foreman-documentation to get only on-demand reviews.

Feedback on this feature is welcome.

[aneta-petrova]

In an effort to reduce the noise the AI bot produces, I dropped this line from the instructions:

- Documentation uses user-centric language rather than product-centric language.

Let's see if it will result in an improvement with regards to duplicated comments and the number of comments in general.