Skip to content

Commit b91d5dd

Browse files
[Docs] Fix typos, improve, update at Using Diffusers' Loading & Hub page (huggingface#5584)
* Fix typos, improve, update * Change to trending and apply some Grammarly fixes * Grammarly fixes * Update loading_adapters.md * Update loading_adapters.md * Update other-formats.md * Update push_to_hub.md * Update loading_adapters.md * Update loading.md * Update docs/source/en/using-diffusers/push_to_hub.md Co-authored-by: Steven Liu <[email protected]> * Update schedulers.md * Update docs/source/en/using-diffusers/loading.md Co-authored-by: Steven Liu <[email protected]> * Update docs/source/en/using-diffusers/loading_adapters.md Co-authored-by: Steven Liu <[email protected]> * Update A1111 LoRA files part * Update other-formats.md --------- Co-authored-by: Steven Liu <[email protected]>
1 parent 2a8cf8e commit b91d5dd

File tree

8 files changed

+156
-103
lines changed

8 files changed

+156
-103
lines changed

β€Ždocs/source/en/using-diffusers/custom_pipeline_overview.mdβ€Ž

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,4 +163,4 @@ video_frames = pipeline(
163163
).frames
164164
```
165165

166-
Here, notice the use of the `trust_remote_code` argument while initializing the pipeline. It is responsible for handling all the "magic" behind the scenes.
166+
Here, notice the use of the `trust_remote_code` argument while initializing the pipeline. It is responsible for handling all the "magic" behind the scenes.

β€Ždocs/source/en/using-diffusers/loading.mdβ€Ž

Lines changed: 44 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,11 @@ This guide will show you how to load:
2929

3030
<Tip>
3131

32-
πŸ’‘ Skip to the [DiffusionPipeline explained](#diffusionpipeline-explained) section if you interested in learning in more detail about how the [`DiffusionPipeline`] class works.
32+
πŸ’‘ Skip to the [DiffusionPipeline explained](#diffusionpipeline-explained) section if you are interested in learning in more detail about how the [`DiffusionPipeline`] class works.
3333

3434
</Tip>
3535

36-
The [`DiffusionPipeline`] class is the simplest and most generic way to load any diffusion model from the [Hub](https://huggingface.co/models?library=diffusers). The [`DiffusionPipeline.from_pretrained`] method automatically detects the correct pipeline class from the checkpoint, downloads and caches all the required configuration and weight files, and returns a pipeline instance ready for inference.
36+
The [`DiffusionPipeline`] class is the simplest and most generic way to load the latest trending diffusion model from the [Hub](https://huggingface.co/models?library=diffusers&sort=trending). The [`DiffusionPipeline.from_pretrained`] method automatically detects the correct pipeline class from the checkpoint, downloads, and caches all the required configuration and weight files, and returns a pipeline instance ready for inference.
3737

3838
```python
3939
from diffusers import DiffusionPipeline
@@ -42,7 +42,7 @@ repo_id = "runwayml/stable-diffusion-v1-5"
4242
pipe = DiffusionPipeline.from_pretrained(repo_id, use_safetensors=True)
4343
```
4444

45-
You can also load a checkpoint with it's specific pipeline class. The example above loaded a Stable Diffusion model; to get the same result, use the [`StableDiffusionPipeline`] class:
45+
You can also load a checkpoint with its specific pipeline class. The example above loaded a Stable Diffusion model; to get the same result, use the [`StableDiffusionPipeline`] class:
4646

4747
```python
4848
from diffusers import StableDiffusionPipeline
@@ -51,7 +51,7 @@ repo_id = "runwayml/stable-diffusion-v1-5"
5151
pipe = StableDiffusionPipeline.from_pretrained(repo_id, use_safetensors=True)
5252
```
5353

54-
A checkpoint (such as [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) or [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5)) may also be used for more than one task, like text-to-image or image-to-image. To differentiate what task you want to use the checkpoint for, you have to load it directly with it's corresponding task-specific pipeline class:
54+
A checkpoint (such as [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) or [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5)) may also be used for more than one task, like text-to-image or image-to-image. To differentiate what task you want to use the checkpoint for, you have to load it directly with its corresponding task-specific pipeline class:
5555

5656
```python
5757
from diffusers import StableDiffusionImg2ImgPipeline
@@ -103,12 +103,10 @@ Let's use the [`SchedulerMixin.from_pretrained`] method to replace the default [
103103
Then you can pass the new [`EulerDiscreteScheduler`] instance to the `scheduler` argument in [`DiffusionPipeline`]:
104104

105105
```python
106-
from diffusers import DiffusionPipeline, EulerDiscreteScheduler, DPMSolverMultistepScheduler
106+
from diffusers import DiffusionPipeline, EulerDiscreteScheduler
107107

108108
repo_id = "runwayml/stable-diffusion-v1-5"
109-
110109
scheduler = EulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
111-
112110
stable_diffusion = DiffusionPipeline.from_pretrained(repo_id, scheduler=scheduler, use_safetensors=True)
113111
```
114112

@@ -121,6 +119,9 @@ from diffusers import DiffusionPipeline
121119

122120
repo_id = "runwayml/stable-diffusion-v1-5"
123121
stable_diffusion = DiffusionPipeline.from_pretrained(repo_id, safety_checker=None, use_safetensors=True)
122+
"""
123+
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide by the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend keeping the safety filter enabled in all public-facing circumstances, disabling it only for use cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
124+
"""
124125
```
125126

126127
### Reuse components across pipelines
@@ -163,18 +164,18 @@ stable_diffusion_img2img = StableDiffusionImg2ImgPipeline(
163164

164165
## Checkpoint variants
165166

166-
A checkpoint variant is usually a checkpoint where it's weights are:
167+
A checkpoint variant is usually a checkpoint whose weights are:
167168

168169
- Stored in a different floating point type for lower precision and lower storage, such as [`torch.float16`](https://pytorch.org/docs/stable/tensors.html#data-types), because it only requires half the bandwidth and storage to download. You can't use this variant if you're continuing training or using a CPU.
169-
- Non-exponential mean averaged (EMA) weights which shouldn't be used for inference. You should use these to continue finetuning a model.
170+
- Non-exponential mean averaged (EMA) weights, which shouldn't be used for inference. You should use these to continue fine-tuning a model.
170171

171172
<Tip>
172173

173174
πŸ’‘ When the checkpoints have identical model structures, but they were trained on different datasets and with a different training setup, they should be stored in separate repositories instead of variations (for example, [`stable-diffusion-v1-4`] and [`stable-diffusion-v1-5`]).
174175

175176
</Tip>
176177

177-
Otherwise, a variant is **identical** to the original checkpoint. They have exactly the same serialization format (like [Safetensors](./using_safetensors)), model structure, and weights have identical tensor shapes.
178+
Otherwise, a variant is **identical** to the original checkpoint. They have exactly the same serialization format (like [Safetensors](./using_safetensors)), model structure, and weights that have identical tensor shapes.
178179

179180
| **checkpoint type** | **weight name** | **argument for loading weights** |
180181
|---------------------|-------------------------------------|----------------------------------|
@@ -202,7 +203,7 @@ stable_diffusion = DiffusionPipeline.from_pretrained(
202203
)
203204
```
204205

205-
To save a checkpoint stored in a different floating point type or as a non-EMA variant, use the [`DiffusionPipeline.save_pretrained`] method and specify the `variant` argument. You should try and save a variant to the same folder as the original checkpoint, so you can load both from the same folder:
206+
To save a checkpoint stored in a different floating-point type or as a non-EMA variant, use the [`DiffusionPipeline.save_pretrained`] method and specify the `variant` argument. You should try and save a variant to the same folder as the original checkpoint, so you can load both from the same folder:
206207

207208
```python
208209
from diffusers import DiffusionPipeline
@@ -247,15 +248,15 @@ The above example is therefore deprecated and won't be supported anymore for `di
247248
<Tip warning={true}>
248249
249250
If you load diffusers pipelines or models with `revision="fp16"` or `revision="non_ema"`,
250-
please make sure to update to code and use `variant="fp16"` or `variation="non_ema"` respectively
251+
please make sure to update the code and use `variant="fp16"` or `variation="non_ema"` respectively
251252
instead.
252253
253254
</Tip>
254255
-->
255256

256257
## Models
257258

258-
Models are loaded from the [`ModelMixin.from_pretrained`] method, which downloads and caches the latest version of the model weights and configurations. If the latest files are available in the local cache, [`~ModelMixin.from_pretrained`] reuses files in the cache instead of redownloading them.
259+
Models are loaded from the [`ModelMixin.from_pretrained`] method, which downloads and caches the latest version of the model weights and configurations. If the latest files are available in the local cache, [`~ModelMixin.from_pretrained`] reuses files in the cache instead of re-downloading them.
259260

260261
Models can be loaded from a subfolder with the `subfolder` argument. For example, the model weights for `runwayml/stable-diffusion-v1-5` are stored in the [`unet`](https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main/unet) subfolder:
261262

@@ -281,17 +282,17 @@ You can also load and save model variants by specifying the `variant` argument i
281282
from diffusers import UNet2DConditionModel
282283

283284
model = UNet2DConditionModel.from_pretrained(
284-
"runwayml/stable-diffusion-v1-5", subfolder="unet", variant="non-ema", use_safetensors=True
285+
"runwayml/stable-diffusion-v1-5", subfolder="unet", variant="non_ema", use_safetensors=True
285286
)
286-
model.save_pretrained("./local-unet", variant="non-ema")
287+
model.save_pretrained("./local-unet", variant="non_ema")
287288
```
288289

289290
## Schedulers
290291

291292
Schedulers are loaded from the [`SchedulerMixin.from_pretrained`] method, and unlike models, schedulers are **not parameterized** or **trained**; they are defined by a configuration file.
292293

293294
Loading schedulers does not consume any significant amount of memory and the same configuration file can be used for a variety of different schedulers.
294-
For example, the following schedulers are compatible with [`StableDiffusionPipeline`] which means you can load the same scheduler configuration file in any of these classes:
295+
For example, the following schedulers are compatible with [`StableDiffusionPipeline`], which means you can load the same scheduler configuration file in any of these classes:
295296

296297
```python
297298
from diffusers import StableDiffusionPipeline
@@ -300,8 +301,8 @@ from diffusers import (
300301
DDIMScheduler,
301302
PNDMScheduler,
302303
LMSDiscreteScheduler,
303-
EulerDiscreteScheduler,
304304
EulerAncestralDiscreteScheduler,
305+
EulerDiscreteScheduler,
305306
DPMSolverMultistepScheduler,
306307
)
307308

@@ -324,9 +325,9 @@ pipeline = StableDiffusionPipeline.from_pretrained(repo_id, scheduler=dpm, use_s
324325
As a class method, [`DiffusionPipeline.from_pretrained`] is responsible for two things:
325326

326327
- Download the latest version of the folder structure required for inference and cache it. If the latest folder structure is available in the local cache, [`DiffusionPipeline.from_pretrained`] reuses the cache and won't redownload the files.
327-
- Load the cached weights into the correct pipeline [class](./api/pipelines/overview#diffusers-summary) - retrieved from the `model_index.json` file - and return an instance of it.
328+
- Load the cached weights into the correct pipeline [class](../api/pipelines/overview#diffusers-summary) - retrieved from the `model_index.json` file - and return an instance of it.
328329

329-
The pipelines underlying folder structure corresponds directly with their class instances. For example, the [`StableDiffusionPipeline`] corresponds to the folder structure in [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5).
330+
The pipelines' underlying folder structure corresponds directly with their class instances. For example, the [`StableDiffusionPipeline`] corresponds to the folder structure in [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5).
330331

331332
```python
332333
from diffusers import DiffusionPipeline
@@ -338,13 +339,13 @@ print(pipeline)
338339

339340
You'll see pipeline is an instance of [`StableDiffusionPipeline`], which consists of seven components:
340341

341-
- `"feature_extractor"`: a [`~transformers.CLIPFeatureExtractor`] from πŸ€— Transformers.
342+
- `"feature_extractor"`: a [`~transformers.CLIPImageProcessor`] from πŸ€— Transformers.
342343
- `"safety_checker"`: a [component](https://github.com/huggingface/diffusers/blob/e55687e1e15407f60f32242027b7bb8170e58266/src/diffusers/pipelines/stable_diffusion/safety_checker.py#L32) for screening against harmful content.
343344
- `"scheduler"`: an instance of [`PNDMScheduler`].
344345
- `"text_encoder"`: a [`~transformers.CLIPTextModel`] from πŸ€— Transformers.
345346
- `"tokenizer"`: a [`~transformers.CLIPTokenizer`] from πŸ€— Transformers.
346347
- `"unet"`: an instance of [`UNet2DConditionModel`].
347-
- `"vae"` an instance of [`AutoencoderKL`].
348+
- `"vae"`: an instance of [`AutoencoderKL`].
348349

349350
```json
350351
StableDiffusionPipeline {
@@ -379,7 +380,7 @@ StableDiffusionPipeline {
379380
}
380381
```
381382

382-
Compare the components of the pipeline instance to the [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) folder structure, and you'll see there is a separate folder for each of the components in the repository:
383+
Compare the components of the pipeline instance to the [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main) folder structure, and you'll see there is a separate folder for each of the components in the repository:
383384

384385
```
385386
.
@@ -388,12 +389,18 @@ Compare the components of the pipeline instance to the [`runwayml/stable-diffusi
388389
β”œβ”€β”€ model_index.json
389390
β”œβ”€β”€ safety_checker
390391
β”‚Β Β  β”œβ”€β”€ config.json
391-
β”‚Β Β  └── pytorch_model.bin
392+
| β”œβ”€β”€ model.fp16.safetensors
393+
β”‚ β”œβ”€β”€ model.safetensors
394+
β”‚ β”œβ”€β”€ pytorch_model.bin
395+
| └── pytorch_model.fp16.bin
392396
β”œβ”€β”€ scheduler
393397
β”‚Β Β  └── scheduler_config.json
394398
β”œβ”€β”€ text_encoder
395399
β”‚Β Β  β”œβ”€β”€ config.json
396-
β”‚Β Β  └── pytorch_model.bin
400+
| β”œβ”€β”€ model.fp16.safetensors
401+
β”‚ β”œβ”€β”€ model.safetensors
402+
β”‚ |── pytorch_model.bin
403+
| └── pytorch_model.fp16.bin
397404
β”œβ”€β”€ tokenizer
398405
β”‚Β Β  β”œβ”€β”€ merges.txt
399406
β”‚Β Β  β”œβ”€β”€ special_tokens_map.json
@@ -402,9 +409,17 @@ Compare the components of the pipeline instance to the [`runwayml/stable-diffusi
402409
β”œβ”€β”€ unet
403410
β”‚Β Β  β”œβ”€β”€ config.json
404411
β”‚Β Β  β”œβ”€β”€ diffusion_pytorch_model.bin
405-
└── vae
406-
β”œβ”€β”€ config.json
407-
β”œβ”€β”€ diffusion_pytorch_model.bin
412+
| |── diffusion_pytorch_model.fp16.bin
413+
β”‚ |── diffusion_pytorch_model.f16.safetensors
414+
β”‚ |── diffusion_pytorch_model.non_ema.bin
415+
β”‚ |── diffusion_pytorch_model.non_ema.safetensors
416+
β”‚ └── diffusion_pytorch_model.safetensors
417+
|── vae
418+
. β”œβ”€β”€ config.json
419+
. β”œβ”€β”€ diffusion_pytorch_model.bin
420+
β”œβ”€β”€ diffusion_pytorch_model.fp16.bin
421+
β”œβ”€β”€ diffusion_pytorch_model.fp16.safetensors
422+
└── diffusion_pytorch_model.safetensors
408423
```
409424

410425
You can access each of the components of the pipeline as an attribute to view its configuration:
@@ -424,10 +439,11 @@ CLIPTokenizer(
424439
"unk_token": AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=True),
425440
"pad_token": "<|endoftext|>",
426441
},
442+
clean_up_tokenization_spaces=True
427443
)
428444
```
429445

430-
Every pipeline expects a `model_index.json` file that tells the [`DiffusionPipeline`]:
446+
Every pipeline expects a [`model_index.json`](https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/model_index.json) file that tells the [`DiffusionPipeline`]:
431447

432448
- which pipeline class to load from `_class_name`
433449
- which version of 🧨 Diffusers was used to create the model in `_diffusers_version`

0 commit comments

Comments
Β (0)