Skip to content

[Backend Test] Backend test reporting skeleton #12296

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

GregoryComer
Copy link
Member

@GregoryComer GregoryComer commented Jul 9, 2025

Summary

Add the initial skeleton of reporting code for the backend tester. This PR is primarily focused on putting the hooks and runner structure in place. Follow-up work will expand the scope of collection and reporting outputs.

This PR adds the following:

  • CLI runner for the test suite.
  • Basic test result breakdown by success / fail and cause (failing in lowering vs output mismatch, for example).
  • Refactoring of test suite logic to clean things up.

Next steps:

  • Aggregate results by flow (backend).
  • Add additional CLI flags to allow filtering backends and dtypes.
  • Land more of the operator test suite.
  • Wire up flows for quantized operators.

Note that this PR is stacked on (and thus includes) #11960. I accidently broke my ghstack, so I'm converting this to a normal PR.

Sample output (XNNPACK):

Test Session Summary:

   84 Passed / 95
   11 Failed / 95
    0 Skipped / 95

[Success]
   66 Delegated
   18 Undelegated

[Failure]
    4 Lowering Fail
    0 PTE Load Fail
    0 PTE Run Fail
    6 Output Mismatch Fail

Reproduce with ET_TEST_ENABLED_BACKENDS=xnnpack python -m executorch.backends.test.suite.runner.executorch.backends.test.suite. I've temporarily commented out non-f32 dtypes to work around some crashes in XNNPACK, which are non-recoverable from Python.

Copy link

pytorch-bot bot commented Jul 9, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12296

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit 05f933d with merge base f9a3ca8 (image):

NEW FAILURE - The following job has failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2025
@GregoryComer GregoryComer force-pushed the backend-test-reporting branch from ac4a118 to 05f933d Compare July 9, 2025 03:54
@GregoryComer GregoryComer marked this pull request as ready for review July 9, 2025 03:55
@GregoryComer GregoryComer added the release notes: none Do not include this in the release notes label Jul 9, 2025
@GregoryComer GregoryComer requested a review from digantdesai July 9, 2025 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: none Do not include this in the release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants