SD 1.5 Support For Advanced Lora Training (train_dreambooth_lora_sdxl_advanced.py) #6449

brandostrong · 2024-01-04T06:05:35Z

A quick adaptation to support people interested in using this method on 1.5 models.

What does this PR do?

Adds support for 1.5 models on the advanced dreambooth training method laid out by @linoytsaban

Who can review?

A quick adaptation to support people interested in using this method on 1.5 models.

HuggingFaceDocBuilderDev · 2024-01-04T06:12:44Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

linoytsaban

Thanks for the PR! looks great, excited to see what LoRAs will come out of the advanced script with SD!

I left a few comments regarding compatibility issues for the prompt text encoding, and unet added conditions. I think all should be quite quick to fix if you don't mind taking another look 😊
examples/dreambooth/train_dreambooth_lora.py - this might help with making those adjustments :)

examples/advanced_diffusion_training/train_dreambooth_lora_sd15_advanced.py

brandostrong · 2024-01-04T10:25:35Z

Thank you so much for the comments Linoy! I appreciate how much detail you just provided me. Admittedly, this was something I quickly put together since I saw a demand for 1.5. It's rather late right now, so when I find time tomorrow I'll make the necessary changes.

Also, did you happen to test your script at all on 1.5, and do you have any comments on the efficacy of it?

linoytsaban · 2024-01-04T10:44:39Z

Awesome thanks so much😁
admittedly I only tried the advanced features with sdxl, so I'm intrigued!

@linoytsaban

as per @linoytsaban 's recommendations. Testing would be appreciated,

brandostrong · 2024-01-04T23:57:31Z

I've made the suggested changes, and I've gotten it to train a style so I would appreciate some testing. There seems to be some fundamental differences(at least in naming, this is a bit over my head) between the base sdxl and 1.5 training scripts, but despite this it seems to work. Could you give it a test as well? Also, is there an example dataset and full list of parameters that can be shared and compared?

linoytsaban · 2024-01-05T11:46:21Z

Thanks a lot!

There seems to be some fundamental differences(at least in naming, this is a bit over my head) between the base sdxl and 1.5 training scripts, but despite this it seems to work. Could you give it a test as well?

Sure! I'll go over it and run some experiments :)

brandostrong · 2024-01-07T22:53:58Z

Hi, I've spent the last few days testing and it holds up against my similar training in kohya. Both styles and likenesses are successful, though it really excels at styles.

sayakpaul · 2024-01-08T01:21:04Z

Thanks so much for your work!

Could we see some results? Also, let's get the code quality fixed, so that we can merge :)

Removed some mentions of SDXL, and some arguments that don't apply to sd 1.5, and cleaned up some comments.

brandostrong · 2024-01-22T00:17:34Z

Here are some results training a 2d style off the base sd 1.5 checkpoint, I've attached the embedding and lora of a style(and somewhat character) train.
I would say it trains just fine :). @sayakpaul @linoytsaban , thanks for the help, hopefully this meets the quality requirements.

offscriptstyleadv.zip
The instance prompt is "in the style of offscriptstyle"

And attached are some example images from comfyui and wandb:

sayakpaul · 2024-01-22T01:19:19Z

Cc: @linoytsaban

linoytsaban · 2024-01-22T08:33:56Z

Looks good @brandostrong! if you don't mind taking a look and merge my 2 suggestions (bug fixes for checkpoint saving) I think we'd be good to go 😁

linoytsaban · 2024-01-22T08:36:01Z

Here are some results training a 2d style off the base sd 1.5 checkpoint, I've attached the embedding and lora of a style(and somewhat character) train. I would say it trains just fine :). @sayakpaul @linoytsaban , thanks for the help, hopefully this meets the quality requirements.

offscriptstyleadv.zip The instance prompt is "in the style of offscriptstyle"

And attached are some example images from comfyui and wandb:

btw @brandostrong - did you try prompting "in the style of <s0><s1> offscriptstyle"? Just curious because that way we also leverage the new optimized embeddings which should improve results even more :)

brandostrong · 2024-01-22T18:31:26Z

Ah oops, the entire essence of this training workflow, totally forgot. I'll send screenshots in an hour or so with the same seed/prompt. I'm running a long animatediff so my VRAM is currently parked at max

brandostrong · 2024-01-22T18:34:29Z

Looks good @brandostrong! if you don't mind taking a look and merge my 2 suggestions (bug fixes for checkpoint saving) I think we'd be good to go 😁

Was the merge I just made the suggestions? I'm not that experienced with much more than solo git.

linoytsaban · 2024-01-22T18:36:39Z

Ah oops, the entire essence of this training workflow, totally forgot. I'll send screenshots in an hour or so with the same seed/prompt. I'm running a long animatediff so my VRAM is currently parked at max

no worries! 😁

linoytsaban · 2024-01-22T18:44:58Z

Looks good @brandostrong! if you don't mind taking a look and merge my 2 suggestions (bug fixes for checkpoint saving) I think we'd be good to go 😁

Was the merge I just made the suggestions? I'm not that experienced with much more than solo git.

I believe they were not merged yet :) can you see my new comments? I think that would do it 😊

brandostrong · 2024-01-22T19:32:57Z

I'm super sorry, you may need to hold my hand on this one. They are not the original suggested changes, or are they(I see I can review changes and comment but not approve or request)? Can you link me to your new ones?

Here are the updated results with the proper tokens.

Very cool stuff.

examples/advanced_diffusion_training/train_dreambooth_lora_sd15_advanced.py

…5_advanced.py

linoytsaban · 2024-01-23T12:18:26Z

@brandostrong I just committed 2 minor changes to save_model_hook - makes sure we save checkpoints correctly, as per the bug fix in #6464. I think we can ship now 😁🛳️

brandostrong

Awesome, thank you :D

@linoytsaban

…_advanced.py) (huggingface#6449) * sd1.5 support in separate script A quick adaptation to support people interested in using this method on 1.5 models. * sd15 prompt text encoding and unet conversions as per @linoytsaban 's recommendations. Testing would be appreciated, * Readability and quality improvements Removed some mentions of SDXL, and some arguments that don't apply to sd 1.5, and cleaned up some comments. * make style/quality commands * tracker rename and run-it doc * Update examples/advanced_diffusion_training/train_dreambooth_lora_sd15_advanced.py * Update examples/advanced_diffusion_training/train_dreambooth_lora_sd15_advanced.py --------- Co-authored-by: Linoy Tsaban <[email protected]>

sd1.5 support in separate script

28fc94a

A quick adaptation to support people interested in using this method on 1.5 models.

linoytsaban requested changes Jan 4, 2024

View reviewed changes

sd15 prompt text encoding and unet conversions

015d8a1

as per @linoytsaban 's recommendations. Testing would be appreciated,

Merge branch 'main' into sd15-support

bd3480b

linoytsaban self-assigned this Jan 9, 2024

brandostrong added 4 commits January 21, 2024 15:11

Readability and quality improvements

9d20bb0

Removed some mentions of SDXL, and some arguments that don't apply to sd 1.5, and cleaned up some comments.

make style/quality commands

18beb38

Merge branch 'main' into sd15-support

3c94ed8

tracker rename and run-it doc

65f688e

Merge branch 'main' into sd15-support

3a3b32e

Merge branch 'main' into sd15-support

775b328

linoytsaban approved these changes Jan 23, 2024

View reviewed changes

examples/advanced_diffusion_training/train_dreambooth_lora_sd15_advanced.py Show resolved Hide resolved

examples/advanced_diffusion_training/train_dreambooth_lora_sd15_advanced.py Outdated Show resolved Hide resolved

linoytsaban added 2 commits January 23, 2024 14:10

Update examples/advanced_diffusion_training/train_dreambooth_lora_sd1…

00985c8

…5_advanced.py

Update examples/advanced_diffusion_training/train_dreambooth_lora_sd1…

0822b02

…5_advanced.py

Merge branch 'main' into sd15-support

470ef9a

brandostrong commented Jan 23, 2024

View reviewed changes

linoytsaban merged commit 16748d1 into huggingface:main Jan 24, 2024

SD 1.5 Support For Advanced Lora Training (train_dreambooth_lora_sdxl_advanced.py) #6449

SD 1.5 Support For Advanced Lora Training (train_dreambooth_lora_sdxl_advanced.py) #6449

Uh oh!

Conversation

brandostrong commented Jan 4, 2024

What does this PR do?

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Jan 4, 2024

Uh oh!

linoytsaban left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandostrong commented Jan 4, 2024

Uh oh!

linoytsaban commented Jan 4, 2024

Uh oh!

brandostrong commented Jan 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linoytsaban commented Jan 5, 2024

Uh oh!

brandostrong commented Jan 7, 2024

Uh oh!

sayakpaul commented Jan 8, 2024

Uh oh!

brandostrong commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Jan 22, 2024

Uh oh!

linoytsaban commented Jan 22, 2024

Uh oh!

linoytsaban commented Jan 22, 2024

Uh oh!

brandostrong commented Jan 22, 2024

Uh oh!

brandostrong commented Jan 22, 2024

Uh oh!

linoytsaban commented Jan 22, 2024

Uh oh!

linoytsaban commented Jan 22, 2024

Uh oh!

brandostrong commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

linoytsaban commented Jan 23, 2024

Uh oh!

brandostrong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

linoytsaban left a comment •

edited

Loading

brandostrong commented Jan 4, 2024 •

edited

Loading

brandostrong commented Jan 22, 2024 •

edited

Loading

brandostrong commented Jan 22, 2024 •

edited

Loading