You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/en/using-diffusers/custom_pipeline_overview.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -163,4 +163,4 @@ video_frames = pipeline(
163
163
).frames
164
164
```
165
165
166
-
Here, notice the use of the `trust_remote_code` argument while initializing the pipeline. It is responsible for handling all the "magic" behind the scenes.
166
+
Here, notice the use of the `trust_remote_code` argument while initializing the pipeline. It is responsible for handling all the "magic" behind the scenes.
Copy file name to clipboardExpand all lines: docs/source/en/using-diffusers/loading.md
+44-28Lines changed: 44 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,11 +29,11 @@ This guide will show you how to load:
29
29
30
30
<Tip>
31
31
32
-
π‘ Skip to the [DiffusionPipeline explained](#diffusionpipeline-explained) section if you interested in learning in more detail about how the [`DiffusionPipeline`] class works.
32
+
π‘ Skip to the [DiffusionPipeline explained](#diffusionpipeline-explained) section if you are interested in learning in more detail about how the [`DiffusionPipeline`] class works.
33
33
34
34
</Tip>
35
35
36
-
The [`DiffusionPipeline`] class is the simplest and most generic way to load any diffusion model from the [Hub](https://huggingface.co/models?library=diffusers). The [`DiffusionPipeline.from_pretrained`] method automatically detects the correct pipeline class from the checkpoint, downloads and caches all the required configuration and weight files, and returns a pipeline instance ready for inference.
36
+
The [`DiffusionPipeline`] class is the simplest and most generic way to load the latest trending diffusion model from the [Hub](https://huggingface.co/models?library=diffusers&sort=trending). The [`DiffusionPipeline.from_pretrained`] method automatically detects the correct pipeline class from the checkpoint, downloads, and caches all the required configuration and weight files, and returns a pipeline instance ready for inference.
You can also load a checkpoint with it's specific pipeline class. The example above loaded a Stable Diffusion model; to get the same result, use the [`StableDiffusionPipeline`] class:
45
+
You can also load a checkpoint with its specific pipeline class. The example above loaded a Stable Diffusion model; to get the same result, use the [`StableDiffusionPipeline`] class:
A checkpoint (such as [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) or [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5)) may also be used for more than one task, like text-to-image or image-to-image. To differentiate what task you want to use the checkpoint for, you have to load it directly with it's corresponding task-specific pipeline class:
54
+
A checkpoint (such as [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) or [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5)) may also be used for more than one task, like text-to-image or image-to-image. To differentiate what task you want to use the checkpoint for, you have to load it directly with its corresponding task-specific pipeline class:
55
55
56
56
```python
57
57
from diffusers import StableDiffusionImg2ImgPipeline
@@ -103,12 +103,10 @@ Let's use the [`SchedulerMixin.from_pretrained`] method to replace the default [
103
103
Then you can pass the new [`EulerDiscreteScheduler`] instance to the `scheduler` argument in [`DiffusionPipeline`]:
104
104
105
105
```python
106
-
from diffusers import DiffusionPipeline, EulerDiscreteScheduler, DPMSolverMultistepScheduler
106
+
from diffusers import DiffusionPipeline, EulerDiscreteScheduler
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide by the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend keeping the safety filter enabled in all public-facing circumstances, disabling it only for use cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
A checkpoint variant is usually a checkpoint where it's weights are:
167
+
A checkpoint variant is usually a checkpoint whose weights are:
167
168
168
169
- Stored in a different floating point type for lower precision and lower storage, such as [`torch.float16`](https://pytorch.org/docs/stable/tensors.html#data-types), because it only requires half the bandwidth and storage to download. You can't use this variant if you're continuing training or using a CPU.
169
-
- Non-exponential mean averaged (EMA) weights which shouldn't be used for inference. You should use these to continue finetuning a model.
170
+
- Non-exponential mean averaged (EMA) weights, which shouldn't be used for inference. You should use these to continue fine-tuning a model.
170
171
171
172
<Tip>
172
173
173
174
π‘ When the checkpoints have identical model structures, but they were trained on different datasets and with a different training setup, they should be stored in separate repositories instead of variations (for example, [`stable-diffusion-v1-4`] and [`stable-diffusion-v1-5`]).
174
175
175
176
</Tip>
176
177
177
-
Otherwise, a variant is **identical** to the original checkpoint. They have exactly the same serialization format (like [Safetensors](./using_safetensors)), model structure, and weights have identical tensor shapes.
178
+
Otherwise, a variant is **identical** to the original checkpoint. They have exactly the same serialization format (like [Safetensors](./using_safetensors)), model structure, and weights that have identical tensor shapes.
178
179
179
180
|**checkpoint type**|**weight name**|**argument for loading weights**|
To save a checkpoint stored in a different floatingpoint type or as a non-EMA variant, use the [`DiffusionPipeline.save_pretrained`] method and specify the `variant` argument. You should try and save a variant to the same folder as the original checkpoint, so you can load both from the same folder:
206
+
To save a checkpoint stored in a different floating-point type or as a non-EMA variant, use the [`DiffusionPipeline.save_pretrained`] method and specify the `variant` argument. You should try and save a variant to the same folder as the original checkpoint, so you can load both from the same folder:
206
207
207
208
```python
208
209
from diffusers import DiffusionPipeline
@@ -247,15 +248,15 @@ The above example is therefore deprecated and won't be supported anymore for `di
247
248
<Tip warning={true}>
248
249
249
250
If you load diffusers pipelines or models with `revision="fp16"` or `revision="non_ema"`,
250
-
please make sure to update to code and use `variant="fp16"` or `variation="non_ema"` respectively
251
+
please make sure to update the code and use `variant="fp16"` or `variation="non_ema"` respectively
251
252
instead.
252
253
253
254
</Tip>
254
255
-->
255
256
256
257
## Models
257
258
258
-
Models are loaded from the [`ModelMixin.from_pretrained`] method, which downloads and caches the latest version of the model weights and configurations. If the latest files are available in the local cache, [`~ModelMixin.from_pretrained`] reuses files in the cache instead of redownloading them.
259
+
Models are loaded from the [`ModelMixin.from_pretrained`] method, which downloads and caches the latest version of the model weights and configurations. If the latest files are available in the local cache, [`~ModelMixin.from_pretrained`] reuses files in the cache instead of re-downloading them.
259
260
260
261
Models can be loaded from a subfolder with the `subfolder` argument. For example, the model weights for `runwayml/stable-diffusion-v1-5` are stored in the [`unet`](https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main/unet) subfolder:
261
262
@@ -281,17 +282,17 @@ You can also load and save model variants by specifying the `variant` argument i
Schedulers are loaded from the [`SchedulerMixin.from_pretrained`] method, and unlike models, schedulers are **not parameterized** or **trained**; they are defined by a configuration file.
292
293
293
294
Loading schedulers does not consume any significant amount of memory and the same configuration file can be used for a variety of different schedulers.
294
-
For example, the following schedulers are compatible with [`StableDiffusionPipeline`] which means you can load the same scheduler configuration file in any of these classes:
295
+
For example, the following schedulers are compatible with [`StableDiffusionPipeline`], which means you can load the same scheduler configuration file in any of these classes:
As a class method, [`DiffusionPipeline.from_pretrained`] is responsible for two things:
325
326
326
327
- Download the latest version of the folder structure required for inference and cache it. If the latest folder structure is available in the local cache, [`DiffusionPipeline.from_pretrained`] reuses the cache and won't redownload the files.
327
-
- Load the cached weights into the correct pipeline [class](./api/pipelines/overview#diffusers-summary) - retrieved from the `model_index.json` file - and return an instance of it.
328
+
- Load the cached weights into the correct pipeline [class](../api/pipelines/overview#diffusers-summary) - retrieved from the `model_index.json` file - and return an instance of it.
328
329
329
-
The pipelines underlying folder structure corresponds directly with their class instances. For example, the [`StableDiffusionPipeline`] corresponds to the folder structure in [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5).
330
+
The pipelines' underlying folder structure corresponds directly with their class instances. For example, the [`StableDiffusionPipeline`] corresponds to the folder structure in [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5).
330
331
331
332
```python
332
333
from diffusers import DiffusionPipeline
@@ -338,13 +339,13 @@ print(pipeline)
338
339
339
340
You'll see pipeline is an instance of [`StableDiffusionPipeline`], which consists of seven components:
340
341
341
-
-`"feature_extractor"`: a [`~transformers.CLIPFeatureExtractor`] from π€ Transformers.
342
+
-`"feature_extractor"`: a [`~transformers.CLIPImageProcessor`] from π€ Transformers.
342
343
-`"safety_checker"`: a [component](https://github.com/huggingface/diffusers/blob/e55687e1e15407f60f32242027b7bb8170e58266/src/diffusers/pipelines/stable_diffusion/safety_checker.py#L32) for screening against harmful content.
343
344
-`"scheduler"`: an instance of [`PNDMScheduler`].
344
345
-`"text_encoder"`: a [`~transformers.CLIPTextModel`] from π€ Transformers.
345
346
-`"tokenizer"`: a [`~transformers.CLIPTokenizer`] from π€ Transformers.
346
347
-`"unet"`: an instance of [`UNet2DConditionModel`].
347
-
-`"vae"` an instance of [`AutoencoderKL`].
348
+
-`"vae"`: an instance of [`AutoencoderKL`].
348
349
349
350
```json
350
351
StableDiffusionPipeline {
@@ -379,7 +380,7 @@ StableDiffusionPipeline {
379
380
}
380
381
```
381
382
382
-
Compare the components of the pipeline instance to the [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) folder structure, and you'll see there is a separate folder for each of the components in the repository:
383
+
Compare the components of the pipeline instance to the [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main) folder structure, and you'll see there is a separate folder for each of the components in the repository:
383
384
384
385
```
385
386
.
@@ -388,12 +389,18 @@ Compare the components of the pipeline instance to the [`runwayml/stable-diffusi
388
389
βββ model_index.json
389
390
βββ safety_checker
390
391
βΒ Β βββ config.json
391
-
βΒ Β βββ pytorch_model.bin
392
+
| βββ model.fp16.safetensors
393
+
β βββ model.safetensors
394
+
β βββ pytorch_model.bin
395
+
| βββ pytorch_model.fp16.bin
392
396
βββ scheduler
393
397
βΒ Β βββ scheduler_config.json
394
398
βββ text_encoder
395
399
βΒ Β βββ config.json
396
-
βΒ Β βββ pytorch_model.bin
400
+
| βββ model.fp16.safetensors
401
+
β βββ model.safetensors
402
+
β |ββ pytorch_model.bin
403
+
| βββ pytorch_model.fp16.bin
397
404
βββ tokenizer
398
405
βΒ Β βββ merges.txt
399
406
βΒ Β βββ special_tokens_map.json
@@ -402,9 +409,17 @@ Compare the components of the pipeline instance to the [`runwayml/stable-diffusi
Every pipeline expects a `model_index.json` file that tells the [`DiffusionPipeline`]:
446
+
Every pipeline expects a [`model_index.json`](https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/model_index.json) file that tells the [`DiffusionPipeline`]:
431
447
432
448
- which pipeline class to load from `_class_name`
433
449
- which version of 𧨠Diffusers was used to create the model in `_diffusers_version`
0 commit comments