Skip to content

[benchmarks] overhaul benchmarks #11565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 91 commits into from
Jul 4, 2025
Merged

[benchmarks] overhaul benchmarks #11565

merged 91 commits into from
Jul 4, 2025

Conversation

sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented May 16, 2025

What does this PR do?

This PR considerably simplifies how we do benchmarks. Instead of using entire pipeline-level benchmarks across different tasks, we will now ONLY benchmark the diffusion network that is the most compute-intensive part in a standard diffusion workflow.

To make the estimates more realistic, we will make use of pre-trained checkpoints and dummy inputs with reasonable dimensionalities.

I ran benchmarking_flux.py on an 80GB A100 on a batch size of 1 and got the following results:

image

Analyze the results in this Space: https://huggingface.co/spaces/diffusers/benchmark-analyzer

By default, all benchmarks will use a batch size of 1, eliminating CFG.

How to add your benchmark?

Adding benchmarks for a new model class (SanaTransformer2DModel, for example) boils down to the following:

  1. Define the dummy inputs of the model.
  2. Define the benchmarking scenarios we should run the benchmark on.

This is what benchmarking_flux.py does. More modularization can be shipped afterward.

Idea would be to merge this PR with pre-configured benchmarks for a few popular models and open others to the community.

TODOs

Utilities:

  • To fire the execution of the individual model-level benchmarks sequentially.
  • To combine CSVs from multiple different model classes.
  • Central dataset update and Slack notification.

@DN6 could you give the approach a quick look? I can then work on resolving the TODOs.



if __name__ == "__main__":
scenarios = [
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered the following scenarios:

  • Regular BF16 with compilation
  • NF4
  • Layerwise upcasting
  • Group offloading

@sayakpaul
Copy link
Member Author

sayakpaul commented May 20, 2025

Added SDXL, Wan (14B), and LTX (13B) on top of Flux:

Results
scenario model_cls num_params_M flops_M time_plain_s mem_plain_GB time_compile_s mem_compile_GB fullgraph mode
0 Wan-AI/Wan2.1-T2V-14B-Diffusers-bf16 WanTransformer3DModel 14288.5 7.85612e+08 10.797 31.17 8.974 31.77 1 default
1 Wan-AI/Wan2.1-T2V-14B-Diffusers-layerwise-upcasting WanTransformer3DModel 14288.5 7.85612e+08 10.702 26.78 nan nan nan nan
2 Wan-AI/Wan2.1-T2V-14B-Diffusers-group-offload-leaf WanTransformer3DModel 14288.5 7.85612e+08 10.83 4.48 nan nan nan nan
3 stabilityai/stable-diffusion-xl-base-1.0-bf16 UNet2DConditionModel 2567.46 5.9791e+06 0.085 5.05 0.058 5.39 1 default
4 stabilityai/stable-diffusion-xl-base-1.0-layerwise-upcasting UNet2DConditionModel 2567.46 5.9791e+06 0.175 4.89 nan nan nan nan
5 stabilityai/stable-diffusion-xl-base-1.0-group-offload-leaf UNet2DConditionModel 2567.46 5.9791e+06 0.383 0.2 nan nan nan nan
6 black-forest-labs/FLUX.1-dev-bf16 FluxTransformer2DModel 11901.4 5.95295e+07 0.535 22.61 0.388 22.85 1 default
7 black-forest-labs/FLUX.1-dev-bnb-nf4 FluxTransformer2DModel 5952.25 17263.8 0.574 6.7 nan nan nan nan
8 black-forest-labs/FLUX.1-dev-layerwise-upcasting FluxTransformer2DModel 11901.4 5.95295e+07 0.621 22.18 nan nan nan nan
9 black-forest-labs/FLUX.1-dev-group-offload-leaf FluxTransformer2DModel 11901.4 5.95295e+07 1.536 0.53 nan nan nan nan
10 Lightricks/LTX-Video-0.9.7-dev-bf16 LTXVideoTransformer3DModel 13042.6 1.67583e+08 1.446 25.21 1.137 25.63 1 default
11 Lightricks/LTX-Video-0.9.7-dev-layerwise-upcasting LTXVideoTransformer3DModel 13042.6 1.67583e+08 1.529 24.38 nan nan nan nan
12 Lightricks/LTX-Video-0.9.7-dev-group-offload-leaf LTXVideoTransformer3DModel 13042.6 1.67583e+08 1.917 1.04 nan nan nan nan

@sayakpaul sayakpaul marked this pull request as ready for review May 20, 2025 11:08
@sayakpaul sayakpaul changed the title [WIP][benchmarks] overhaul benchmarks [benchmarks] overhaul benchmarks May 20, 2025
@sayakpaul
Copy link
Member Author

Cc: @a-r-r-o-w if you want to add some caching benchmarks (in a later PR), I think that would be really great!

@sayakpaul sayakpaul requested a review from DN6 May 20, 2025 12:09
@sayakpaul
Copy link
Member Author

sayakpaul commented May 20, 2025

@DN6 this is ready for a review.

This is how the final CSV for this stage looks like:
https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/collated_results.csv

I have confirmed in this run that it works as expected:
https://github.com/huggingface/diffusers/actions/runs/15138495257/job/42570011907

@sayakpaul
Copy link
Member Author

@DN6 a gentle ping.

@@ -3,25 +3,26 @@ name: Benchmarking tests
on:
workflow_dispatch:
schedule:
- cron: "30 1 1,15 * *" # every 2 weeks on the 1st and the 15th of every month at 1:30 AM
- cron: "0 17 * * 1" # every monday at 5 PM.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker. But why run every week? Is a monthly benchmark not sufficient?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. Changing to bi-weekly.

@sayakpaul
Copy link
Member Author

@anijain2305 just a ping to let you know that we're merging this PR which will run the benchmarking suite bi-weekly and report the results here: https://huggingface.co/datasets/diffusers/benchmarks/blob/main/collated_results.csv

@anijain2305
Copy link
Contributor

Thanks for setting this up. This will be really helpful for tracking progress and identifying regression

@sayakpaul
Copy link
Member Author

Since everything is passing now, will merge this PR :)
https://github.com/huggingface/diffusers/actions/runs/16065987231/job/45340516693

@sayakpaul sayakpaul merged commit e6639fe into main Jul 4, 2025
12 checks passed
@sayakpaul sayakpaul deleted the benchmarking-overhaul branch July 4, 2025 05:34
@sayakpaul
Copy link
Member Author

Also cc @a-r-r-o-w for #11565 (comment) (not urgent, when you get time).

tolgacangoz pushed a commit to tolgacangoz/diffusers that referenced this pull request Jul 5, 2025
* start overhauling the benchmarking suite.

* fixes

* fixes

* checking.

* checking

* fixes.

* error handling and logging.

* add flops and params.

* add more models.

* utility to fire execution of all benchmarking scripts.

* utility to push to the hub.

* push utility improvement

* seems to be working.

* okay

* add torchprofile dep.

* remove total gpu memory

* fixes

* fix

* need a big gpu

* better

* what's happening.

* okay

* separate requirements and make it nightly.

* add db population script.

* update secret name

* update secret.

* population db update

* disable db population for now.

* change to every monday

* Update .github/workflows/benchmark.yml

Co-authored-by: Dhruv Nair <[email protected]>

* quality improvements.

* reparate hub upload step.

* repository

* remove csv

* check

* update

* update

* threading.

* update

* update

* updaye

* update

* update

* update

* remove peft dep

* upgrade runner.

* fix

* fixes

* fix merging csvs.

* push dataset to the Space repo for analysis.

* warm up.

* add a readme

* Apply suggestions from code review

Co-authored-by: Luc Georges <[email protected]>

* address feedback

* Apply suggestions from code review

* disable db workflow.

* update to bi weekly.

* enable population

* enable

* updaye

* update

* metadata

* fix

---------

Co-authored-by: Dhruv Nair <[email protected]>
Co-authored-by: Luc Georges <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants