Modular PAG Guider

[a-r-r-o-w]

import torch
from diffusers import LayerSkipConfig, PerturbedAttentionGuidance
from diffusers.modular_pipelines import SequentialPipelineBlocks, ComponentSpec
from diffusers.modular_pipelines.stable_diffusion_xl import TEXT2IMAGE_BLOCKS

# create pipeline from official blocks preset
blocks = SequentialPipelineBlocks.from_blocks_dict(TEXT2IMAGE_BLOCKS)

modular_repo_id = "YiYiXu/modular-loader-t2i"
pipeline = blocks.init_pipeline(modular_repo_id)
pipeline.load_default_components(torch_dtype=torch.float16)
pipeline.to("cuda")

config = LayerSkipConfig(indices=[2, 9], fqn="mid_block.attentions.0.transformer_blocks", skip_attention=False, skip_attention_scores=True, skip_ff=False)
guider = PerturbedAttentionGuidance(
    guidance_scale=5.0, perturbed_guidance_scale=2.5, skip_layer_config=config
)
pipeline.loader.update(
    guider=ComponentSpec(
        name="pag",
        type_hint=PerturbedAttentionGuidance,
        config={
            "guidance_scale": 5.0,
            "perturbed_guidance_scale": 20.0,
            "skip_layer_config": config,
        },
        default_creation_method="from_config",
    )
)

# run pipeline, need to pass a "output=images" argument
image = pipeline(prompt="Astronaut in a jungle, cold color palette, muted colors, detailed, 8k", output="images", generator=torch.Generator().manual_seed(42))[0]
image.save("modular_t2i_out.png")

Left is PAG scale 10.0, right is PAG scale 20.0. With the default parameters, PAG is only active for the first 20% of the steps.

cc @sunovivid Could you review the PR and suggest any corrections/improvements as you see fit? 🤗

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sunovivid]

Looks great to me!

Thank you for the note on the paper in the method descriptions sections, as well as the specialized implementation.

Huge thanks, @a-r-r-o-w! 🤗