Modular Guider ConfigMixin

[a-r-r-o-w]

import torch
from diffusers import ClassifierFreeGuidance
from diffusers.modular_pipelines import SequentialPipelineBlocks, ComponentSpec
from diffusers.modular_pipelines.stable_diffusion_xl import TEXT2IMAGE_BLOCKS

# create pipeline from official blocks preset
blocks = SequentialPipelineBlocks.from_blocks_dict(TEXT2IMAGE_BLOCKS)

modular_repo_id = "YiYiXu/modular-loader-t2i"
pipeline = blocks.init_pipeline(modular_repo_id)
pipeline.load_default_components(torch_dtype=torch.float16)
pipeline.to("cuda")

pipeline.loader.update(
    guider=ComponentSpec(
        name="cfg",
        type_hint=ClassifierFreeGuidance,
        config={"guidance_scale": 5.0},
        default_creation_method="from_config",
    )
)

# run pipeline, need to pass a "output=images" argument
image = pipeline(prompt="Astronaut in a jungle, cold color palette, muted colors, detailed, 8k", output="images", generator=torch.Generator().manual_seed(42))[0]
image.save("modular_t2i_out.png")

(Pdb) pipeline.components["guider"]
ClassifierFreeGuidance {
  "_class_name": "ClassifierFreeGuidance",
  "_diffusers_version": "0.35.0.dev0",
  "guidance_rescale": 0.0,
  "guidance_scale": 5.0,
  "start": 0.0,
  "stop": 1.0,
  "use_original_formulation": false
}

(Pdb) pipeline.loader.guider
ClassifierFreeGuidance {
  "_class_name": "ClassifierFreeGuidance",
  "_diffusers_version": "0.35.0.dev0",
  "guidance_rescale": 0.0,
  "guidance_scale": 5.0,
  "start": 0.0,
  "stop": 1.0,
  "use_original_formulation": false
}

(Pdb) pipeline.loader.guider.save_config("dump_guider")
(Pdb) # manually change the guidance scale to 10.0 in the file
*** SyntaxError: invalid syntax

(Pdb) pipeline.loader.guider.from_config("dump_guider")
/home/aryan/work/diffusers/src/diffusers/configuration_utils.py:248: FutureWarning: It is deprecated to pass a pretrained model name or path to `from_config`.
  deprecate("config-passed-as-path", "1.0.0", deprecation_message, standard_warn=False)
ClassifierFreeGuidance {
  "_class_name": "ClassifierFreeGuidance",
  "_diffusers_version": "0.35.0.dev0",
  "guidance_rescale": 0.0,
  "guidance_scale": 10.0,
  "start": 0.0,
  "stop": 1.0,
  "use_original_formulation": false
}

(Pdb) pipeline.loader.guider
ClassifierFreeGuidance {
  "_class_name": "ClassifierFreeGuidance",
  "_diffusers_version": "0.35.0.dev0",
  "guidance_rescale": 0.0,
  "guidance_scale": 5.0,
  "start": 0.0,
  "stop": 1.0,
  "use_original_formulation": false
}

(Pdb) pipeline.loader.update(guider=ClassifierFreeGuidance.from_config("dump_guider"))
*** ValueError: `ModularLoader` only supports components created from `ComponentSpec`.

(Pdb) pipeline.loader.guider = ClassifierFreeGuidance.from_config("dump_guider")

(Pdb) pipeline.loader.guider
ClassifierFreeGuidance {
  "_class_name": "ClassifierFreeGuidance",
  "_diffusers_version": "0.35.0.dev0",
  "guidance_rescale": 0.0,
  "guidance_scale": 10.0,
  "start": 0.0,
  "stop": 1.0,
  "use_original_formulation": false
}

[yiyixuxu]

thanks @a-r-r-o-w
do we have to refactor out LayerSkipConfig (and similar config class for other guider types) - I think they don't work with ConfigMixin

[yiyixuxu]

can you test it out to save it to a folder like https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/tree/main/scheduler

and load it back

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

@yiyixuxu The output shared above shows saving and loading the config -- is this what you meant or something else? The folder gets created and initialized with a guider_config.json file containing:

{
  "_class_name": "ClassifierFreeGuidance",
  "_diffusers_version": "0.35.0.dev0",
  "guidance_rescale": 0.0,
  "guidance_scale": 5.0,
  "start": 0.0,
  "stop": 1.0,
  "use_original_formulation": false
}

[yiyixuxu]

@a-r-r-o-w last time I tried it didn't work with some of the guiders with its own config class, e.g I think LayerSkipConfig isn't serializable

 skip_layer_config: Union[LayerSkipConfig, List[LayerSkipConfig]] = None,

[a-r-r-o-w]

I see. I will give it a look tomorrow as I'm going off for the day now. Will try to make it work, and if there's no easy way, we can think of other way to set the config options

[yiyixuxu]

@a-r-r-o-w I merged it so that I can start to update doc using CFG example sonner, you can make sure it works with the rest of the guiders in a new PR :)

(I think we probably just don't need the special config class now that it is a ConfigMixin, but it is up to you)