Fix SDXL conversion from original to diffusers

[duongna21]

Hi, long time no see. Great progress here.
I noticed a small bug that prevented the successful conversion of SDXL.

Traceback (most recent call last):
  File "scripts/convert_original_stable_diffusion_to_diffusers.py", line 138, in <module>
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1545, in download_from_original_stable_diffusion_ckpt
    pipe = pipeline_class(
TypeError: __init__() got an unexpected keyword argument 'text_encoder_2'

cc: @patrickvonplaten @sayakpaul

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Great catch!

[Mystfit]

This regresses #4108 . @patrickvonplaten, it looks as though we might have to add those safety checks to confirm if the pipeline supports text_encoder_2 after all.

[Mystfit]

The problem seems to be when pipeline_class is not used then the checkpoint loader will always use StableDiffusionXLPipeline even if you've specified another pipeline (see #4108). The convert_original_stable_diffusion_to_diffusers.py script does not set pipeline_class as it expects it to be inferred in download_from_original_stable_diffusion_ckpt. I proposed changing StableDiffusionXLPipeline to pipeline_class as when you attempt to load SDXL models from a single file you can't use StableDiffusionXLImg2ImgPipeline as pipeline_class is ignored in favour of the explicitly set StableDiffusionXLPipeline

To fix this, I'd like to propose that we populate the pipeline_class variable earlier in the function depending on the value of model_type and not explicitly use StableDiffusionXLPipeline when instantiating the pipeline. I'd suggest something like this - Mystfit@a6b1884

@patrickvonplaten Is it worth me opening up another PR for this?

[patrickvonplaten]

The problem seems to be when pipeline_class is not used then the checkpoint loader will always use StableDiffusionXLPipeline even if you've specified another pipeline (see #4108). The convert_original_stable_diffusion_to_diffusers.py script does not set pipeline_class as it expects it to be inferred in download_from_original_stable_diffusion_ckpt. I proposed changing StableDiffusionXLPipeline to pipeline_class as when you attempt to load SDXL models from a single file you can't use StableDiffusionXLImg2ImgPipeline as pipeline_class is ignored in favour of the explicitly set StableDiffusionXLPipeline

To fix this, I'd like to propose that we populate the pipeline_class variable earlier in the function depending on the value of model_type and not explicitly use StableDiffusionXLPipeline when instantiating the pipeline. I'd suggest something like this - Mystfit@a6b1884

@patrickvonplaten Is it worth me opening up another PR for this?

It'd be great if you could open a PR for it - @yiyixuxu did we solve a similar problem recently?

[yiyixuxu]

@patrickvonplaten

not similar but a related problem! actually I realize all I did here #4408 was to revert what this PR did 😲😲😲

I don't understand what was the issue PR trying to fixed though? I tested StableDiffusionXLPipeline.from_single_file() with #4408 and it definitely worked fine

[yiyixuxu]

oh I got it now - if you run the convert_original_stable_diffusion_to_diffusers.py you will get an error

I think @Mystfit is right - we just need to add a pipeline_class argument or something to the conversion script