Add Stable Diffusion 3 #8483

DN6 · 2024-06-12T14:18:34Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

sayakpaul · 2024-06-12T14:20:42Z

docs/source/en/api/pipelines/stable_diffusion/stable_diffusion_3.md

+The abstract from the paper is:
+
+*Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations.*
+


Suggested change

Original model weights can be found here: [https://huggingface.co/stabilityai/](https://huggingface.co/stabilityai/).

Was this meant to be [here](https://huggingface.co/stabilityai/)?

Yeah feel free to shorten it. I wanted to unfurl it.

docs/source/en/api/pipelines/stable_diffusion/stable_diffusion_3.md

src/diffusers/models/transformers/transformer_sd3.py

sayakpaul

Todo:

You are already adding tests. I think the we should add the SD3Transformer2DModel tests too: https://github.com/huggingface/new-model-addition-diffusers/pull/30.
Add docs for SD3Transformer2DModel and the scheduler.

src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py

yiyixuxu · 2024-06-12T15:49:14Z

src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py

+        width: Optional[int] = None,
+        num_inference_steps: int = 50,
+        timesteps: List[int] = None,
+        guidance_scale: float = 5.0,


should we set it to be 7.0 (I think that's the default they want)

vladmandic · 2024-06-12T16:07:55Z

no support for from_single_file?
like i said many times, majority of users prefer to be able to download a model first - especially since stabilityai provided pre-packaged single-file versions that they actually promote.
also, using from_pretrained to automatically download a model is of limited use here since user needs to agree to terms&conditions first.

sayakpaul · 2024-06-12T16:14:15Z

It will come with single-file support. The PR isn't fully ready yet.

…fusion_3.py Co-authored-by: YiYi Xu <[email protected]>

src/diffusers/image_processor.py

src/diffusers/models/autoencoders/autoencoder_kl.py

kadirnar · 2024-06-12T18:08:08Z

Will you write the inpaint pipeline code?

GavChap · 2024-06-12T18:16:06Z

requirements_sd3.txt is missing

Co-authored-by: YiYi Xu <[email protected]>

…_3.md Co-authored-by: Sayak Paul <[email protected]>

sayakpaul · 2024-06-12T18:33:41Z

@vladmandic single-file support is there ;)

src/diffusers/loaders/single_file_utils.py

src/diffusers/models/autoencoders/autoencoder_kl.py

apolinario

wanted to make a comment and started a review by accident

examples/dreambooth/README_sd3.md

GavChap · 2024-06-12T18:56:28Z

I get this error as well when attempting to run the training (after figuring out the requirements)

An error occurred while trying to fetch stabilityai/stable-diffusion-3-medium-diffusers: stabilityai/stable-diffusion-3-medium-diffusers does not appear to have a file named diffusion_pytorch_model.safetensors.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Traceback (most recent call last):
  File "/home/me/diffusers/venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/home/me/diffusers/venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers/resolve/48ef03a6b806e781a368fe71b3855f515e789bb0/transformer/diffusion_pytorch_model.bin

sayakpaul · 2024-06-12T19:01:13Z

@GavChap I would appreciate if you don't post it in between an ongoing PR as the issue is not descriptive enough. So, I would suggest you wait for a little while and try it out. If you face any issue, open a separate thread.

* up * add sd3 * update * update * add tests * fix copies * fix docs * update * add dreambooth lora * add LoRA * update * update * update * update * import fix * update * Update src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py Co-authored-by: YiYi Xu <[email protected]> * import fix 2 * update * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/models/autoencoders/autoencoder_kl.py Co-authored-by: YiYi Xu <[email protected]> * update * update * update * fix ckpt id * fix more ids * update * missing doc * Update src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py Co-authored-by: YiYi Xu <[email protected]> * Update src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py Co-authored-by: YiYi Xu <[email protected]> * Update docs/source/en/api/pipelines/stable_diffusion/stable_diffusion_3.md Co-authored-by: Sayak Paul <[email protected]> * Update docs/source/en/api/pipelines/stable_diffusion/stable_diffusion_3.md Co-authored-by: Sayak Paul <[email protected]> * update' * fix * update * Update src/diffusers/models/autoencoders/autoencoder_kl.py * Update src/diffusers/models/autoencoders/autoencoder_kl.py * note on gated access. * requirements * licensing --------- Co-authored-by: sayakpaul <[email protected]> Co-authored-by: YiYi Xu <[email protected]>

DN6 added 4 commits June 12, 2024 12:38

up

e979394

add sd3

15fb675

update

e82f8c4

update

761e677

sayakpaul reviewed Jun 12, 2024

View reviewed changes

add tests

02881e2

sayakpaul reviewed Jun 12, 2024

View reviewed changes

docs/source/en/api/pipelines/stable_diffusion/stable_diffusion_3.md Show resolved Hide resolved

sayakpaul reviewed Jun 12, 2024

View reviewed changes

docs/source/en/api/pipelines/stable_diffusion/stable_diffusion_3.md Show resolved Hide resolved

sayakpaul reviewed Jun 12, 2024

View reviewed changes

src/diffusers/models/transformers/transformer_sd3.py Outdated Show resolved Hide resolved

fix copies

db90e8d

sayakpaul reviewed Jun 12, 2024

View reviewed changes

DN6 added 7 commits June 12, 2024 14:30

fix docs

215c2e0

update

cea25b6

add dreambooth lora

6b49a13

add LoRA

8eb8e21

update

617997d

update

706f1b9

update

167fa3b

painebenjamin mentioned this pull request Jun 12, 2024

Support for stable-diffusion-3-medium model #8482

Closed

update

ed45a09

yiyixuxu reviewed Jun 12, 2024

View reviewed changes

boraykasap mentioned this pull request Jun 12, 2024

Support for SD3 LoRA #8485

Closed

sayakpaul and others added 3 commits June 12, 2024 17:19

import fix

f7794f9

update

3ea4669

Update src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_dif…

07dafaf

…fusion_3.py Co-authored-by: YiYi Xu <[email protected]>

yiyixuxu reviewed Jun 12, 2024

View reviewed changes

src/diffusers/image_processor.py Show resolved Hide resolved

yiyixuxu reviewed Jun 12, 2024

View reviewed changes

src/diffusers/models/autoencoders/autoencoder_kl.py Outdated Show resolved Hide resolved

yiyixuxu reviewed Jun 12, 2024

View reviewed changes

src/diffusers/models/autoencoders/autoencoder_kl.py Outdated Show resolved Hide resolved

Merge branch 'sd3' of https://github.com/huggingface/diffusers into sd3

da68e58

missing doc

ba76d29

DN6 and others added 5 commits June 12, 2024 23:48

Update src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py

cbd29d2

Co-authored-by: YiYi Xu <[email protected]>

Update src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py

f22c199

Co-authored-by: YiYi Xu <[email protected]>

Update docs/source/en/api/pipelines/stable_diffusion/stable_diffusion…

61e60db

…_3.md Co-authored-by: Sayak Paul <[email protected]>

Update docs/source/en/api/pipelines/stable_diffusion/stable_diffusion…

c921fd2

…_3.md Co-authored-by: Sayak Paul <[email protected]>

update'

0297d9e

Merge branch 'main' into sd3

877e765

yiyixuxu reviewed Jun 12, 2024

View reviewed changes

src/diffusers/loaders/single_file_utils.py Outdated Show resolved Hide resolved

sayakpaul and others added 3 commits June 12, 2024 19:38

fix

1a69918

update

7f611fb

Merge branch 'sd3' of https://github.com/huggingface/diffusers into sd3

e2389fe

yiyixuxu reviewed Jun 12, 2024

View reviewed changes

src/diffusers/models/autoencoders/autoencoder_kl.py Outdated Show resolved Hide resolved

Update src/diffusers/models/autoencoders/autoencoder_kl.py

752febe

yiyixuxu reviewed Jun 12, 2024

View reviewed changes

src/diffusers/models/autoencoders/autoencoder_kl.py Outdated Show resolved Hide resolved

yiyixuxu and others added 2 commits June 12, 2024 08:40

Update src/diffusers/models/autoencoders/autoencoder_kl.py

e78462e

note on gated access.

b2dba96

apolinario approved these changes Jun 12, 2024

View reviewed changes

examples/dreambooth/README_sd3.md Show resolved Hide resolved

requirements

04e38b3

licensing

ba17c16

sayakpaul merged commit 04717fd into main Jun 12, 2024

sayakpaul deleted the sd3 branch June 12, 2024 19:44

Beinsezii mentioned this pull request Jun 12, 2024

StableDiffusion3Pipeline from_pretrained Exception #8488

Closed

vladmandic mentioned this pull request Jun 15, 2024

SD3 lora support non functional #8579

Closed

		The abstract from the paper is:

		Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations.



	Original model weights can be found here: [https://huggingface.co/stabilityai/](https://huggingface.co/stabilityai/).

Add Stable Diffusion 3 #8483

Add Stable Diffusion 3 #8483

Uh oh!

Conversation

DN6 commented Jun 12, 2024

What does this PR do?

Before submitting

Who can review?

Uh oh!

sayakpaul Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

DN6 Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaul Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiyixuxu Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

vladmandic commented Jun 12, 2024

Uh oh!

sayakpaul commented Jun 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kadirnar commented Jun 12, 2024

Uh oh!

GavChap commented Jun 12, 2024

Uh oh!

sayakpaul commented Jun 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

apolinario left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GavChap commented Jun 12, 2024

Uh oh!

sayakpaul commented Jun 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants