You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+95-9Lines changed: 95 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,29 +20,30 @@ as a modular toolbox for inference and training of diffusion models.
20
20
21
21
More precisely, 🤗 Diffusers offers:
22
22
23
-
- State-of-the-art diffusion pipelines that can be run in inference with just a couple of lines of code (see [src/diffusers/pipelines](https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines)).
23
+
- State-of-the-art diffusion pipelines that can be run in inference with just a couple of lines of code (see [src/diffusers/pipelines](https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines)). Check [this overview](https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/README.md#pipelines-summary) to see all supported pipelines and their corresponding official papers.
24
24
- Various noise schedulers that can be used interchangeably for the prefered speed vs. quality trade-off in inference (see [src/diffusers/schedulers](https://github.com/huggingface/diffusers/tree/main/src/diffusers/schedulers)).
25
25
- Multiple types of models, such as UNet, can be used as building blocks in an end-to-end diffusion system (see [src/diffusers/models](https://github.com/huggingface/diffusers/tree/main/src/diffusers/models)).
26
26
- Training examples to show how to train the most popular diffusion models (see [examples/training](https://github.com/huggingface/diffusers/tree/main/examples/training)).
27
27
- Inference examples to show how to create custom pipelines for advanced tasks such as image2image, in-painting (see [examples/inference](https://github.com/huggingface/diffusers/tree/main/examples/inference))
28
28
29
+
29
30
## Quickstart
30
31
31
32
In order to get started, we recommend taking a look at two notebooks:
32
33
33
34
- The [Getting started with Diffusers](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/diffusers_intro.ipynb)[](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/diffusers_intro.ipynb) notebook, which showcases an end-to-end example of usage for diffusion models, schedulers and pipelines.
34
35
Take a look at this notebook to learn how to use the pipeline abstraction, which takes care of everything (model, scheduler, noise handling) for you, and also to understand each independent building block in the library.
35
-
- The [Training a diffusers model](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/training_example.ipynb)[](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/training_example.ipynb) notebook summarizes diffuser model training methods. This notebook takes a step-by-step approach to training your
36
-
diffuser model on an image dataset, with explanatory graphics.
36
+
- The [Training a diffusers model](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/training_example.ipynb)[](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/training_example.ipynb) notebook summarizes diffusion models training methods. This notebook takes a step-by-step approach to training your
37
+
diffusion models on an image dataset, with explanatory graphics.
37
38
38
-
## **New 🎨🎨🎨** Stable Diffusion is now fully compatible with `diffusers`!
39
-
40
-
Stable Diffusion is a text-to-image latent diffusion model created by the researchers and engineers from [CompVis](https://github.com/CompVis), [Stability AI](https://stability.ai/) and [LAION](https://laion.ai/). It's trained on 512x512 images from a subset of the [LAION-5B](https://laion.ai/blog/laion-5b/) database. This model uses a frozen CLIP ViT-L/14 text encoder to condition the model on text prompts. With its 860M UNet and 123M text encoder, the model is relatively lightweight and runs on a GPU with at least 10GB VRAM.
39
+
## **New ** Stable Diffusion is now fully compatible with `diffusers`! Stable Diffusion is a text-to-image latent diffusion model created by the researchers and engineers from [CompVis](https://github.com/CompVis), [Stability AI](https://stability.ai/) and [LAION](https://laion.ai/). It's trained on 512x512 images from a subset of the [LAION-5B](https://laion.ai/blog/laion-5b/) database. This model uses a frozen CLIP ViT-L/14 text encoder to condition the model on text prompts. With its 860M UNet and 123M text encoder, the model is relatively lightweight and runs on a GPU with at least 10GB VRAM.
41
40
See the [model card](https://huggingface.co/CompVis/stable-diffusion) for more information.
42
41
43
42
You need to accept the model license before downloading or using the Stable Diffusion weights. Please, visit the [model card](https://huggingface.co/CompVis/stable-diffusion-v1-3), read the license and tick the checkbox if you agree. You have to be a registered user in 🤗 Hugging Face Hub, and you'll also need to use an access token for the code to work. For more information on access tokens, please refer to [this section](https://huggingface.co/docs/hub/security-tokens) of the documentation.
44
43
45
-
```py
44
+
### Text-to-Image generation with Stable Diffusion
45
+
46
+
```python
46
47
# make sure you're logged in with `huggingface-cli login`
47
48
from torch import autocast
48
49
from diffusers import StableDiffusionPipeline, LMSDiscreteScheduler
@@ -54,10 +55,13 @@ lms = LMSDiscreteScheduler(
54
55
)
55
56
56
57
pipe = StableDiffusionPipeline.from_pretrained(
57
-
"CompVis/stable-diffusion-v1-3",
58
+
"CompVis/stable-diffusion-v1-4",
59
+
revision="fp16",
60
+
torch_dtype=torch.float16,
58
61
scheduler=lms,
59
62
use_auth_token=True
60
-
).to("cuda")
63
+
)
64
+
pipe = pipe.to("cuda")
61
65
62
66
prompt ="a photo of an astronaut riding a horse on mars"
63
67
with autocast("cuda"):
@@ -66,6 +70,88 @@ with autocast("cuda"):
66
70
image.save("astronaut_rides_horse.png")
67
71
```
68
72
73
+
### Image-to-Image text-guided generation with Stable Diffusion
74
+
75
+
The `StableDiffusionImg2ImgPipeline` lets you pass a text prompt and an initial image to condition the generation of new images.
76
+
77
+
```python
78
+
from torch import autocast
79
+
import requests
80
+
fromPILimport Image
81
+
from io import BytesIO
82
+
83
+
from diffusers import StableDiffusionImg2ImgPipeline
You can also run this example on colab [](https://colab.research.google.com/github/patil-suraj/Notebooks/blob/master/image_2_image_using_diffusers.ipynb)
110
+
111
+
### In-painting using Stable Diffusion
112
+
113
+
The `StableDiffusionInpaintPipeline` lets you edit specific parts of an image by providing a mask and text prompt.
114
+
115
+
```python
116
+
from io import BytesIO
117
+
118
+
from torch import autocast
119
+
import requests
120
+
importPIL
121
+
122
+
from diffusers import StableDiffusionInpaintPipeline
You can generate your own latents to reproduce results, or tweak your prompt on a specific result you liked. [This notebook](stable-diffusion-seeds.ipynb) shows how to do it step by step. You can also run it in Google Colab [](https://colab.research.google.com/github/pcuenca/diffusers-examples/blob/main/notebooks/stable-diffusion-seeds.ipynb).
153
+
154
+
69
155
For more details, check out [the Stable Diffusion notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_diffusion.ipynb)[](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_diffusion.ipynb)
70
156
and have a look into the [release notes](https://github.com/huggingface/diffusers/releases/tag/v0.2.0).
Copyright 2022 The HuggingFace Team. All rights reserved.
3
+
Licensed under the Apache License, Version 2.0 (the "License");
4
+
you may not use this file except in compliance with the License.
5
+
You may obtain a copy of the License at
6
+
7
+
http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+
Unless required by applicable law or agreed to in writing, software
10
+
distributed under the License is distributed on an "AS IS" BASIS,
11
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+
See the License for the specific language governing permissions and
13
+
limitations under the License.
14
+
-->
15
+
16
+
# 🧨 Diffusers Examples
17
+
18
+
Diffusers examples are a collection of scripts to demonstrate how to effectively use the `diffusers` library
19
+
for a variety of use cases.
20
+
21
+
**Note**: If you are looking for **official** examples on how to use `diffusers` for inference,
22
+
please have a look at [src/diffusers/pipelines](https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines)
23
+
24
+
Our examples aspire to be **self-contained**, **easy-to-tweak**, **beginner-friendly** and for **one-purpose-only**.
25
+
More specifically, this means:
26
+
27
+
-**Self-contained**: An example script shall only depend on "pip-install-able" Python packages that can be found in a `requirements.txt` file. Example scripts shall **not** depend on any local files. This means that one can simply download an example script, *e.g.*[train_unconditional.py](https://github.com/huggingface/diffusers/blob/main/examples/unconditional_image_generation/train_unconditional.py), install the required dependencies, *e.g.*[requirements.txt](https://github.com/huggingface/diffusers/blob/main/examples/unconditional_image_generation/requirements.txt) and execute the example script.
28
+
-**Easy-to-tweak**: While we strive to present as many use cases as possible, the example scripts are just that - examples. It is expected that they won't work out-of-the box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs. To help you with that, most of the examples fully expose the preprocessing of the data and the training loop to allow you to tweak and edit them as required.
29
+
-**Beginner-friendly**: We do not aim for providing state-of-the-art training scripts for the newest models, but rather examples that can be used as a way to better understand diffusion models and how to use them with the `diffusers` library. We often purposefully leave out certain state-of-the-art methods if we consider them too complex for beginners.
30
+
-**One-purpose-only**: Examples should show one task and one task only. Even if a task is from a modeling
31
+
point of view very similar, *e.g.* image super-resolution and image modification tend to use the same model and training method, we want examples to showcase only one task to keep them as readable and easy-to-understand as possible.
32
+
33
+
We provide **official** examples that cover the most popular tasks of diffusion models.
34
+
*Official* examples are **actively** maintained by the `diffusers` maintainers and we try to rigorously follow our example philosophy as defined above.
35
+
If you feel like another important example should exist, we are more than happy to welcome a [Feature Request](https://github.com/huggingface/diffusers/issues/new?assignees=&labels=&template=feature_request.md&title=) or directly a [Pull Request](https://github.com/huggingface/diffusers/compare) from you!
36
+
37
+
Training examples show how to pretrain or fine-tune diffusion models for a variety of tasks. Currently we support:
In addition, we provide **community** examples, which are examples added and maintained by our community.
46
+
Community examples can consist of both *training* examples or *inference* pipelines.
47
+
For such examples, we are more lenient regarding the philosophy defined above and also cannot guarantee to provide maintenance for every issue.
48
+
Examples that are useful for the community, but are either not yet deemed popular or not yet following our above philosophy should go into the [community examples](https://github.com/huggingface/diffusers/tree/main/examples/community) folder. The community folder therefore includes training examples and inference pipelines.
49
+
**Note**: Community examples can be a [great first contribution](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) to show to the community how you like to use `diffusers` 🪄.
50
+
51
+
## Important note
52
+
53
+
To make sure you can successfully run the latest versions of the example scripts, you have to **install the library from source** and install some example-specific requirements. To do this, execute the following steps in a new virtual environment:
**The inference examples folder is deprecated and will be removed in a future version**.
4
+
**Officially supported inference examples can be found in the [Pipelines folder](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines)**.
5
+
6
+
- For `Image-to-Image text-guided generation with Stable Diffusion`, please have a look at the official [Pipeline examples](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines#examples)
7
+
- For `In-painting using Stable Diffusion`, please have a look at the official [Pipeline examples](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines#examples)
8
+
- For `Tweak prompts reusing seeds and latents`, please have a look at the official [Pipeline examples](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines#examples)
0 commit comments