Add guidance start/stop

[holwech]

Adds guidance start/stop to the img2img and text2img Stable Diffusion pipeline classe. This is a setting that is available in automatic1111. This PR is heavily inspired by this community pipeline that implements the same functionality.

I've used and tested the PR on my own project and it seems to work fine.

[patrickvonplaten]

Looks like a nice addition to me ,could we add some tests and also add it to the controlnet inpainting pipeline?

Tests should be added here:
https://github.com/huggingface/diffusers/tree/main/tests/pipelines/controlnet
Inpainting pipeline is here:
https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py

[sayakpaul]

Seems to be a nice addition. Would be cool to also see some results with and without it.

And as mentioned by @patrickvonplaten let's add some test cases for it to ensure robustness and feature compatibility.

[julien-blanchon]

controlnet_guidance_start and controlnet_guidance_end don't work for MultiControlnet. It should accept list/tuple for each controlnet

[holwech]

Thanks for the feedback. Seems like implementing this is slightly more complex than I anticipated, but I will try to look at this in the coming weeks. If anyone wants this implemented asap, feel free to take it over.

Todos

• Write tests
• Add guidance start/end to inpainting
• Add guidance start/end to MultiControlNet

[holwech]

Here is a before and after example:

modern apartment, grass, rocks, mountains, norway, summer, vray, hyper-realistic, high resolution, highly detailed
 Negative prompt: ugly, bad, jpeg artifacts, vignette
 Steps: 40, Sampler: DPM++ 2M, CFG scale: 7, Seed: 3313150245, Size: 896x512, Model: v1_realisticVisionV20_v20, ControlNet 0: "preprocessor: lineart_standard (from white bg & black line), model: control_v11p_sd15_lineart [43d4be0d], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: False, control mode: Balanced, preprocessor params: (512, 64, 64)", ControlNet 1: "preprocessor: depth_midas, model: control_v11f1p_sd15_depth [cfd03158], weight: 0.1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: False, control mode: Balanced, preprocessor params: (512, 64, 64)", Version: v1.3.2

modern apartment, grass, rocks, mountains, norway, summer, vray, hyper-realistic, high resolution, highly detailed
 Negative prompt: ugly, bad, jpeg artifacts, vignette
 Steps: 40, Sampler: DPM++ 2M, CFG scale: 7, Seed: 1996722551, Size: 896x512, Model: v1_realisticVisionV20_v20, ControlNet 0: "preprocessor: lineart_standard (from white bg & black line), model: control_v11p_sd15_lineart [43d4be0d], weight: 1, starting/ending: (0.05, 0.7), resize mode: Crop and Resize, pixel perfect: False, control mode: Balanced, preprocessor params: (512, 64, 64)", ControlNet 1: "preprocessor: depth_midas, model: control_v11f1p_sd15_depth [cfd03158], weight: 0.1, starting/ending: (0.05, 0.7), resize mode: Crop and Resize, pixel perfect: False, control mode: Balanced, preprocessor params: (512, 64, 64)", Version: v1.3.2

I find it's quite useful to set guidance end to a number lower than 1. This allows the model to clean up any artifacts caused by controlnet in the last steps of the process. By setting guidance start larger than 0, details that don't exist in the input images are added.

[holwech]

I've updated the PR to support multicontrolnet, can someone have a look at the changes and let me know if you are fine with the changes? I will add some tests afterwards if we agree to proceed.

[cylwin]

We need this for our qrcode controlnet demo.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[PhilSad]

Thanks for your PR! Here is a demo colab that shows how to use it to generate qrcodes.

I also made the lib sdqrcode to generate ai qrcodes with this method with a simple one-liner (colab)

[patrickvonplaten]

Hey @holwech,

Sorry for fiddling soo much with your PR. Some tests were missing and I think it's better if we try to not change the controlnet files. Here I just take advantage of the fact that zero'ing out the controlnet result is the same as skipping it. The gain we might get from skipping on controlnet is negligible IMO so design-wise I think it's cleaner to just zero it out.