Skip to content

Commit 3ab9211

Browse files
authored
[Community] [WIP] LCM Interpolation Pipeline (huggingface#5767)
* wip: add interpolate pipeline for lcm * update documentation * update documentation
1 parent 6b04d61 commit 3ab9211

File tree

2 files changed

+1098
-0
lines changed

2 files changed

+1098
-0
lines changed

examples/community/README.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ sketch inpaint - Inpainting with non-inpaint Stable Diffusion | sketch inpaint m
4747
prompt-to-prompt | change parts of a prompt and retain image structure (see [paper page](https://prompt-to-prompt.github.io/)) | [Prompt2Prompt Pipeline](#prompt2prompt-pipeline) | - | [Umer H. Adil](https://twitter.com/UmerHAdil) |
4848
| Latent Consistency Pipeline | Implementation of [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378) | [Latent Consistency Pipeline](#latent-consistency-pipeline) | - | [Simian Luo](https://github.com/luosiallen) |
4949
| Latent Consistency Img2img Pipeline | Img2img pipeline for Latent Consistency Models | [Latent Consistency Img2Img Pipeline](#latent-consistency-img2img-pipeline) | - | [Logan Zoellner](https://github.com/nagolinc) |
50+
| Latent Consistency Interpolation Pipeline | Interpolate the latent space of Latent Consistency Models with multiple prompts | [Latent Consistency Interpolation Pipeline](#latent-consistency-interpolation-pipeline) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1pK3NrLWJSiJsBynLns1K1-IDTW9zbPvl?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) |
5051

5152

5253
To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.
@@ -2295,3 +2296,50 @@ num_inference_steps = 4
22952296

22962297
images = pipe(prompt=prompt, image=input_image, strength=strength, num_inference_steps=num_inference_steps, guidance_scale=8.0, lcm_origin_steps=50, output_type="pil").images
22972298
```
2299+
2300+
2301+
2302+
### Latent Consistency Interpolation Pipeline
2303+
2304+
This pipeline extends the Latent Consistency Pipeline to allow for interpolation of the latent space between multiple prompts. It is similar to the [Stable Diffusion Interpolate](https://github.com/huggingface/diffusers/blob/main/examples/community/interpolate_stable_diffusion.py) and [unCLIP Interpolate](https://github.com/huggingface/diffusers/blob/main/examples/community/unclip_text_interpolation.py) community pipelines.
2305+
2306+
```py
2307+
import torch
2308+
import numpy as np
2309+
2310+
from diffusers import DiffusionPipeline
2311+
2312+
pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_interpolate")
2313+
2314+
# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
2315+
pipe.to(torch_device="cuda", torch_dtype=torch.float32)
2316+
2317+
prompts = [
2318+
"Self-portrait oil painting, a beautiful cyborg with golden hair, Margot Robbie, 8k",
2319+
"Self-portrait oil painting, an extremely strong man, body builder, Huge Jackman, 8k",
2320+
"An astronaut floating in space, renaissance art, realistic, high quality, 8k",
2321+
"Oil painting of a cat, cute, dream-like",
2322+
"Hugging face emoji, cute, realistic"
2323+
]
2324+
num_inference_steps = 4
2325+
num_interpolation_steps = 60
2326+
seed = 1337
2327+
2328+
torch.manual_seed(seed)
2329+
np.random.seed(seed)
2330+
2331+
images = pipe(
2332+
prompt=prompts,
2333+
height=512,
2334+
width=512,
2335+
num_inference_steps=num_inference_steps,
2336+
num_interpolation_steps=num_interpolation_steps,
2337+
guidance_scale=8.0,
2338+
embedding_interpolation_type="lerp",
2339+
latent_interpolation_type="slerp",
2340+
process_batch_size=4, # Make it higher or lower based on your GPU memory
2341+
generator=torch.Generator(seed),
2342+
)
2343+
2344+
assert len(images) == (len(prompts) - 1) * num_interpolation_steps
2345+
```

0 commit comments

Comments
 (0)