AutoPipeline

[yiyixuxu]

new draft based on #4131 (comment)

to-do:

• add all the pipelines to the map
• add doc + tests

from diffusers import AutoPipelineForImageToImage, AutoPipelineForTextToImage, AutoPipelineForInpainting

pipe_img2img = AutoPipelineForImageToImage.from_pretrained("runwayml/stable-diffusion-v1-5")
pipe_txt2img = AutoPipelineForTextToImage.from_pretrained("runwayml/stable-diffusion-v1-5")
pipe_inpaint = AutoPipelineForInpainting.from_pretrained("runwayml/stable-diffusion-v1-5")

print(type(pipe_img2img), type(pipe_txt2img), type(pipe_inpaint))

<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_img2img.StableDiffusionImg2ImgPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_inpaint.StableDiffusionInpaintPipeline'>

pipe_txt2img = AutoPipelineForTextToImage.from_pipe(pipe_img2img)
pipe_inpaint = AutoPipelineForInpainting.from_pipe(pipe_txt2img)
pipe_img2img = AutoPipelineForImageToImage.from_pipe(pipe_inpaint)
print(type(pipe_img2img), type(pipe_txt2img), type(pipe_inpaint))

<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_img2img.StableDiffusionImg2ImgPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_inpaint.StableDiffusionInpaintPipeline'>

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Great! Now we just need to add a couple of tests and fix some nits

[patrickvonplaten]

Here we actually need to do a bit more because .components() doesn't have all the necessary inputs necessarily. E.g. SD has a require_safety_checker pipeline optional attribute that we need to pass here as well. What you could /should do here therefore is the following:

1. Retrieve all optional kwargs that the pipeline can accept. You can have a look here at how this can be done:
   

   diffusers/src/diffusers/pipelines/pipeline_utils.py

   Line 562 in c6e56e9

   expected_modules, optional_kwargs = self._get_signature_keys(self)

2. Now you check if these kwargs are in pipeline.config. If they are you should retrieve them and then pass them

[yiyixuxu]

actually, I don't quite understand this

we are not calling the __init__ method, we are calling the from_pretrained() - I think these steps are done inside from_pretrained() method no?

[patrickvonplaten]

@yiyixuxu this test currently fails, but it shouldn't. The reason is that we don't correctly forward requires_safety_checker.

Note that there might even be harder examples in case we go from SD-XL Image-2-Image over Text-2-Image back to SD-XL Image-2-Image.

Could you think about how we can make this work? We should probably just make sure we correctly save all the optional kwargs in from_pipe with register_to_config

[yiyixuxu]

ohhhhhh I got it now -- we need to just forward all the arguments passed to the original pipeline's from_pretrained(), instead of expecting users to pass them again

Thanks for demonstrating it!

[yiyixuxu]

@patrickvonplaten updated from_pipe here

[patrickvonplaten]

Can we add controlnet here as well?

[patrickvonplaten]

Can we add one more test for https://huggingface.co/hf-internal-testing/tiny-stable-diffusion-xl-pipe

where we go from img2img to text2img to img2img & make sure that the config stays correct?

[yiyixuxu]

@patrickvonplaten made it work with controlnet and added the tests

[patrickvonplaten]

Very cool only some final comments and then it would be great if we could add a "AutoPipeline" to the docs here: https://github.com/huggingface/diffusers/tree/main/docs/source/en/api/pipelines :-)

[stevhliu]

Super nice, I've actually been low-key following this! 😂

What do you think about also writing a tutorial for this new API abstraction so it gets more visibility and is potentially an even easier entry point for new users coming to the library (happy to help write or review this with you 🤗 )?

[sayakpaul]

Let's include all the major ones:

• SDXL
• UniDiffuser

I mention UniDiffuser because it's pre-trained on a fairly large dataset.

[sayakpaul]

Neat 👌

[sayakpaul]

Love the API design!

[patrickvonplaten]

Merging now already so that it's easier to work on: #4207