The Modular Diffusers #9672

yiyixuxu · 2024-10-14T19:00:44Z

TO-DOs before merge

We overrode changes in this PR https://github.com/huggingface/diffusers/pull/11652/files, need to make sure the updated code works for this case
PAG needs to be a guider on its own
One more guide on how to write a PipelineBlock; and how to assemble SequentialPipelineBlocks and AutoPipelineBlocks
a simple test case: slow case for now

TO-DOs

guider needs to be a ConfigMixin
ModularPipelineBlocks.from_pretrained needs to work with official blocks
Merge the concept of ModularLoader into ModularPipeline
Flux/Wan/ITX
API docs

Documentations

Documentations are live here

HuggingFaceDocBuilderDev · 2024-10-14T19:07:16Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yoland68 · 2024-10-16T17:20:48Z

Very cool!

… completely yet (look into later)

…oisestep

oozzy77 · 2024-10-30T14:50:49Z

hi this is very interesting! I'm making a Python pipeline flow visual scripting tool, that can auto-convert functions to visual nodes for fast and modular UI blocks demo. Itself is a pip package: https://pypi.org/project/nozyio/

I wanted to integrate diffusers with my flow nodes UI project but found its not very modular. But this PR may change that! Looking forward to see how this evolves.

github: https://github.com/oozzy77/nozyio happy to connect!

yiyixuxu · 2024-10-30T20:58:40Z

@oozzy77 thanks!
do you want to join a slack channel with me? if you want to experiment building something with this PR I'm eager to hear your feedback and iterate base on that

oozzy77 · 2024-10-31T20:10:20Z

Hi super willing to join slack channel with you! What’s the workspace channel I should join？or you can invite me ***@***.***

…

On Thu, Oct 31, 2024 at 4:59 AM YiYi Xu ***@***.***> wrote: @oozzy77 <https://github.com/oozzy77> thanks! do you want to join a slack channel with me? if you want to experiment building something with this PR I'm eager to hear your feedback and iterate base on that — Reply to this email directly, view it on GitHub <#9672 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BMBK3ZHNSKN56N262LBH3WLZ6FCBNAVCNFSM6AAAAABP5SYMXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGM3DQMBYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

yiyixuxu · 2024-10-31T23:07:43Z

@oozzy77 I sent an invite!

…into modular-diffusers

yiyixuxu · 2025-07-01T01:46:33Z

src/diffusers/commands/custom_blocks.py

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+


@DN6 @sayakpaul
I merged in this part of code without reviewing it (I need the save_pretrained() code and it works fine)
can you let me know if you want to keep it in here when we merge this PR or remove &convert it into a new PR

src/diffusers/commands/custom_blocks.py

yiyixuxu · 2025-07-01T01:47:16Z

src/diffusers/guiders/__init__.py

+
+    GuiderType = Union[
+        AdaptiveProjectedGuidance,
+        AutoGuidance,


@a-r-r-o-w can we make sure PAG has its own class before we merge?

yiyixuxu · 2025-07-01T01:48:29Z

src/diffusers/utils/dynamic_modules_utils.py

@@ -154,33 +159,87 @@ def check_imports(filename):
    return get_relative_imports(filename)


-def get_class_in_module(class_name, module_path, pretrained_model_name_or_path=None):
+def _raise_timeout_error(signum, frame):


@DN6 is there a test we can run for remote code? want to make sure we don't break anything here

Oh no test yet. Can add one.

stevhliu · 2025-07-02T20:53:17Z

Super comprehensive and nice set of guides! 👏

Here is my feedback:

The getting_started.md doc is quite large and may be intimidating for new users, especially as a "getting started" guide. I think it may be better to take a more top-down approach here with a brief explanation of the block types and then follow it up with a practical example of using these blocks to create a workflow (maybe mention the 4-step process here). This way users can more concretely and immediately grasp the main focus of Modular Diffusers.
Create a separate doc to go over PipelineState and BlockState.
The write_your_own_pipeline_block.md doc can be split into a separate doc for each block type.
Create a new doc for ModularPipeline now that users have gone over the state and block types.
I think the end_to_end_guide.md is actually a very nice "quickstart" doc. It's seems pretty similar to this section here though? Maybe we can turn this into a nice blog post for announcing Modular Diffusers?

The docs would look something like this:

- local: getting_started
  title: Getting started
- local: modular_diffusers_states
  title: Pipeline and block states
- local: pipeline_block
  title: PipelineBlock
- local: sequential_pipeline_blocks
  title: SequentialPipelineBlocks
- local: loop_sequential_pipeline_blocks
  title: LoopSequentialPipelineBlocks
- local: auto_pipeline_blocks
  title: AutoPipelineBlocks
- local: modular_pipeline
  title: ModularPipeline

yiyixuxu · 2025-07-03T03:31:21Z

@stevhliu
Thanks for the feedback! I'll wait to consolidate other feedback before making any changes, but I totally agree - reorganizing the docs into smaller, more digestible pieces makes a lot of sense.
I think maybe we can skip the getting_started doc for now and revisit it later though. It's not very clear yet what the bare minimum knowledge we need to get across is, since (at least for now) we're targeting more advanced users. This is a system for building new pipelines/workflows, so naturally there's a higher entry barrier.
We're not announcing Modular Diffusers publicly yet - current goal is to get it merged so it's easier to iterate with the team and the community members who are enthusiastic about this. That gives us some time to figure out the right onboarding approach.
I can consolidate the quickstart and end-to-end guide into one for now, but they won't be very "quick" at all 😅 - just have a more practical focus. And yeah, we can definitely turn them into a blog post when we're announcing it, and we should have a real "quick start" ready by then.

So I think we can aim for something like this for the merge, what do you think?

- local: modular_diffusers_states
  title: Pipeline and block states 
- local: pipeline_block
  title: PipelineBlock
- local: sequential_pipeline_blocks
  title: SequentialPipelineBlocks
- local: loop_sequential_pipeline_blocks
  title: LoopSequentialPipelineBlocks
- local: auto_pipeline_blocks
  title: AutoPipelineBlocks
- local: modular_pipeline
  title: ModularPipeline
 - local: end_to_end_guide
   title: An End-to-End Example (from building custom pipeline to UI node deployment) # look for better names lol

stevhliu · 2025-07-03T16:40:19Z

Sounds good to me, let me know how I can help!

src/diffusers/pipelines/components_manager.py

src/diffusers/modular_pipelines/components_manager.py

src/diffusers/modular_pipelines/modular_pipeline.py

src/diffusers/modular_pipelines/components_manager.py

DN6 · 2025-07-03T10:41:56Z

docs/source/en/modular_diffusers/getting_started.md

+
+<Tip>
+
+💡 We recommend using `ModularPipeline` with Component Manager by passing a `components_manager`:


Would just have the description of the ComponentManager before the code example.

DN6 · 2025-07-03T10:50:23Z

docs/source/en/modular_diffusers/getting_started.md

+
+```py
+>>> components = ComponentsManager()
+>>> pipeline = blocks.init_pipeline(modular_repo_id, components_manager=components)


What happens if no component manager is set?

src/diffusers/modular_pipelines/modular_pipeline.py

docs/source/en/modular_diffusers/getting_started.md

DN6 · 2025-07-03T17:14:34Z

docs/source/en/modular_diffusers/getting_started.md

+from diffusers import ComponentsManager
+# Set up component manager and turn on the offloading
+components = ComponentsManager()
+components.enable_auto_cpu_offload(device="cuda")


Would auto offloading work here? No components have been loaded into it yet?

yeah it works, once the strategy is on it's on, you can add/remove models
(currently I just reset it everytime a model is added/removed, so basically all the models get moved to cpu and start over; but we can certainly do it more efficiently if there is a demand for that)

* update * fix * update

* update * update * register to config pag

sayakpaul

Thanks for leading the charge here, @yiyixuxu! 15k LoC changes might seem a lot but I do believe that it's for the greater good!

My comments are mostly in the docs because I think even for reviewers, that should be the entrypoint to understanding the different moving parts involed in the modular design.

Additionally, I also reviewed the changes made to the existing core modules and all of them are okay to me.

Apart from that, I am quite interested to grow the offloading part in components manager as I believe that will be crucial for the UI-based workflows!

docs/source/en/modular_diffusers/getting_started.md

sayakpaul · 2025-07-05T04:42:33Z

docs/source/en/modular_diffusers/getting_started.md

+
+**Important**: It is important to always check the docstring because arguments can be different from standard pipelines that you're familar with. For example, in Modular Diffusers we standardized controlnet image input as `control_image`, but regular pipelines have inconsistencies over the names, e.g. controlnet text-to-image uses `image` while SDXL controlnet img2img uses `control_image`.
+
+**Note**: The `output` list might be longer than you expected - it includes everything in the intermediate state that you can choose to return. Most of the time, you'll just want `output="images"` or `output="latents"`.


Maybe we could show an example output?

sayakpaul · 2025-07-05T04:43:08Z

docs/source/en/modular_diffusers/getting_started.md

+```py
+t2i_pipeline.doc
+```


Does it need to be here?

docs/source/en/modular_diffusers/getting_started.md

sayakpaul · 2025-07-05T04:48:22Z

docs/source/en/modular_diffusers/getting_started.md

+
+**Components Manager**: Create one manager and pass it to `init_pipeline` along with a collection name. All models loaded by that pipeline will be added to the manager under that collection.
+
+**Auto Offloading**: All components are placed on CPU and only moved to device right before their forward pass. The manager monitors device memory and may move components off-device to make space for new ones. Unlike `DiffusionPipeline.enable_model_cpu_offload()`, this works across all components in the manager and all your workflows.


I think the component manager deserves its own section / page where we go deep into offloading, etc., and remove it from there.

I am also following what is meant by

Unlike DiffusionPipeline.enable_model_cpu_offload(), this works across all components in the manager and all your workflows.

If we do enable_model_cpu_offload() it works on all the model-components of the underlying pipeline, no?

it works across pipelines, say like you have 10 pipelines share 15 models, it works across these 15 models

sayakpaul · 2025-07-05T04:50:47Z

src/diffusers/pipelines/auto_pipeline.py

-            for model_name, pipeline in task_mapping.items():
-                if pipeline.__name__ == pipeline_class_name:
-                    return model_name
+def _get_model(pipeline_class_name):


I am guessing this is just refactoring?

* update * make style

Co-authored-by: Sayak Paul <[email protected]>

add

33f85fa

add dataflow info for each block in builder _repr_

52a7f1c

yiyixuxu added 8 commits October 16, 2024 20:56

add img2img support - output does not match with non-modular pipeline…

e8d0980

… completely yet (look into later)

update img2img, result match

ad3f9a2

add from_pipe + run_blocks

ddea157

controlnet

af9572d

fix controlnet

2b6dcbf

combine controlnetstep into contronetdesnoisestep

70272b1

refactor guider, remove prepareguidance step to be combinedd into den…

46ec174

…oisestep

update pag guider - draft

f1b3036

a-r-r-o-w mentioned this pull request Oct 25, 2024

[docs] add docstrings in pipline_stable_diffusion.py #9590

Merged

6 tasks

yiyixuxu added 7 commits October 26, 2024 21:17

refactor guider

540d303

up

6742f16

add

005195c

fix so that run_blocks can work with inputs in the state

024a9f5

remove img2img blocksgit status consolidate text2img and img2img

37e8dc7

refactor, from_pretrained, from_pipe, remove_blocks, replace_blocks

8b811fe

style

c70a285

yiyixuxu mentioned this pull request Nov 6, 2024

Adaptive Projected Guidance #9626

Closed

add autostep (not complete)

ffc2992

asomoza mentioned this pull request Nov 30, 2024

Analysis of Classifier-Free Guidance Weight Schedulers #7756

Open

2 tasks

yiyixuxu added the roadmap Add to current release roadmap label Dec 4, 2024

hlky mentioned this pull request Dec 5, 2024

Detail-Daemon diffusers #10112

Open

update/refactor

ace53e2

yiyixuxu added 8 commits June 30, 2025 22:09

more attemp to fix circular import

f27fbce

Merge branch 'modular-diffusers' of github.com:huggingface/diffusers …

98ea5c9

…into modular-diffusers

developer_guide -> end-to-end guide

b5db8aa

rename quick start- it is really not quick

4543d21

update docstree

1987c07

up up

2e20241

add modularpipelineblocks to be pushtohub mixin

13fe248

up upup

8cb5b08

yiyixuxu commented Jul 1, 2025

View reviewed changes

src/diffusers/commands/custom_blocks.py Outdated Show resolved Hide resolved

yiyixuxu commented Jul 1, 2025

View reviewed changes

yiyixuxu requested review from stevhliu, a-r-r-o-w and asomoza and removed request for stevhliu July 1, 2025 02:39

fix links in the doc

3e46c86

DN6 reviewed Jul 3, 2025

View reviewed changes

DN6 approved these changes Jul 3, 2025

View reviewed changes

a-r-r-o-w added 2 commits July 4, 2025 12:19

Modular PAG Guider (#11860)

13c51bb

* update * fix * update

Modular Guider ConfigMixin (#11862)

b750c69

* update * update * register to config pag

sayakpaul approved these changes Jul 5, 2025

View reviewed changes

a-r-r-o-w and others added 5 commits July 5, 2025 07:49

Modular custom config object serialization (#11868)

284f827

* update * make style

Apply suggestions from code review

2c66fb3

Co-authored-by: Sayak Paul <[email protected]>

resolve conflicnt

63e94cb

style + copy

4f8b6f5

add sub_blocks for pipelineBlock

23de59e


		<Tip>

		💡 We recommend using `ModularPipeline` with Component Manager by passing a `components_manager`:


		Important: It is important to always check the docstring because arguments can be different from standard pipelines that you're familar with. For example, in Modular Diffusers we standardized controlnet image input as `control_image`, but regular pipelines have inconsistencies over the names, e.g. controlnet text-to-image uses `image` while SDXL controlnet img2img uses `control_image`.

		Note: The `output` list might be longer than you expected - it includes everything in the intermediate state that you can choose to return. Most of the time, you'll just want `output="images"` or `output="latents"`.


		Components Manager: Create one manager and pass it to `init_pipeline` along with a collection name. All models loaded by that pipeline will be added to the manager under that collection.

		Auto Offloading: All components are placed on CPU and only moved to device right before their forward pass. The manager monitors device memory and may move components off-device to make space for new ones. Unlike `DiffusionPipeline.enable_model_cpu_offload()`, this works across all components in the manager and all your workflows.

The Modular Diffusers #9672

Are you sure you want to change the base?

The Modular Diffusers #9672

Uh oh!

Conversation

yiyixuxu commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TO-DOs before merge

TO-DOs

Documentations

Uh oh!

HuggingFaceDocBuilderDev commented Oct 14, 2024

Uh oh!

yoland68 commented Oct 16, 2024

Uh oh!

oozzy77 commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxu commented Oct 30, 2024

Uh oh!

oozzy77 commented Oct 31, 2024 via email

Uh oh!

yiyixuxu commented Oct 31, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevhliu commented Jul 2, 2025

Uh oh!

yiyixuxu commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevhliu commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiyixuxu commented Oct 14, 2024 •

edited

Loading

oozzy77 commented Oct 30, 2024 •

edited

Loading

yiyixuxu commented Jul 3, 2025 •

edited

Loading