Use CC12M for LCM WDS training example #5908

pcuenca · 2023-11-23T09:43:04Z

Had to make some adjustments to make it work with CC12M, as the metadata field names in the json are different. In particular, if no pwatermark field exists all images were rejected.

cc @stevhliu @sayakpaul @patil-suraj

Fixes #5868, #5770.
Possibly fixes #5743 for other datasets.

HuggingFaceDocBuilderDev · 2023-11-23T09:49:24Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

sayakpaul

Thanks for changes, they look good to me. Would it make sense to reflect the peft specific changes from #5778 to make it run more efficiently?

pcuenca · 2023-12-04T10:09:24Z

Would it make sense to reflect the peft specific changes from #5778 to make it run more efficiently?

Makes sense. I'd rather tackle that in a separate PR, if possible, so this one can be merged and helps the community with the issues linked in the first comment.

patrickvonplaten

Great idea! @patil-suraj @dg845 can you please check here?

dg845 · 2023-12-04T22:43:48Z

examples/consistency_distillation/train_lcm_distill_sd_wds.py

@@ -1097,7 +1097,7 @@ def compute_embeddings(prompt_batch, proportion_empty_prompts, text_encoder, tok
    for epoch in range(first_epoch, args.num_train_epochs):
        for step, batch in enumerate(train_dataloader):
            with accelerator.accumulate(unet):
-                image, text, _, _ = batch
+                image, text = batch


Just to confirm, is this change because the SD distillation script currently has a bug where it assumes that the Text2ImageDataset.train_dataloader batch consists of (image, text, orig_size, crop_coords) like the SDXL dataloader instead of just (image, text):

diffusers/examples/consistency_distillation/train_lcm_distill_sd_wds.py

Lines 176 to 178 in 110ac7f

wds.map(filter_keys({"image", "text"})),

wds.map(transform),

wds.to_tuple("image", "text"),

and is otherwise unrelated to CC12M support?

Looks like 711e468 mentions this. Feel free to close :).

Yes, exactly, it's actually a bug and not directly related to CC12M :)

examples/consistency_distillation/train_lcm_distill_lora_sd_wds.py

dg845 · 2023-12-04T23:08:22Z

examples/consistency_distillation/README.md

+export OUTPUT_DIR="path/to/saved/model"
+
+accelerate launch train_lcm_distill_sd_wds.py \
+    --pretrained_teacher_model=$MODEL_NAME \
    --output_dir=$OUTPUT_DIR \
    --mixed_precision=fp16 \
    --resolution=512 \
    --learning_rate=1e-6 --loss_type="huber" --ema_decay=0.95 --adam_weight_decay=0.0 \


Do the hyperparameters in the examples work well with the CC12M dataset or does it potentially make sense to revisit them?

Not sure to be honest. I haven't experimented much with it, considering that CC12M is very small for current standards.

Perhaps it would be easier to include a disclaimer stating that this dataset is used for illustrative purposes and users are encouraged to bring their own.

dg845 · 2023-12-04T23:28:03Z

I think it could be valuable to make sure WebdatasetFilter and Text2ImageDataset are consistent across the scripts (and maybe separating them into SD and SDXL versions, e.g. SDText2ImageDataset/SDXLText2ImageDataset), perhaps through the # Copied from mechanism. Not sure if it makes sense to do that in this PR though.

pcuenca · 2023-12-05T09:17:33Z

I think it could be valuable to make sure WebdatasetFilter and Text2ImageDataset are consistent across the scripts (and maybe separating them into SD and SDXL versions, e.g. SDText2ImageDataset/SDXLText2ImageDataset), perhaps through the # Copied from mechanism. Not sure if it makes sense to do that in this PR though.

I think that's a good idea! I'd rather work on that separately, if possible, so we can close those issues from the community.

I'll add a disclaimer about the illustrative nature of the dataset. Edit: done, @dg845 let me know if that'd be enough.

dg845

Looks good to me :).

pcuenca · 2023-12-06T09:35:33Z

Thanks for the great reviews!

* Fix SD scripts - there are only 2 items per batch * Adjustments to make the SDXL scripts work with other datasets * Use public webdataset dataset for examples * make style * Minor tweaks to the readmes. * Stress that the database is illustrative.

pcuenca added 3 commits November 23, 2023 10:37

Fix SD scripts - there are only 2 items per batch

711e468

Adjustments to make the SDXL scripts work with other datasets

a94cf3c

Use public webdataset dataset for examples

8c80814

This was referenced Nov 23, 2023

[Latent Consistency Distillation] training stuck at 0% #5743

Closed

[docs] LCM training #5796

Merged

pcuenca requested review from patil-suraj, sayakpaul and stevhliu November 26, 2023 09:50

sayakpaul mentioned this pull request Nov 27, 2023

[Training] Add datasets version of LCM LoRA SDXL #5778

Merged

3 tasks

sayakpaul approved these changes Nov 27, 2023

View reviewed changes

sayakpaul mentioned this pull request Nov 27, 2023

Dataset access of LCM training #5868

Closed

make style

110ac7f

patrickvonplaten reviewed Dec 4, 2023

View reviewed changes

dg845 reviewed Dec 4, 2023

View reviewed changes

examples/consistency_distillation/train_lcm_distill_lora_sd_wds.py Show resolved Hide resolved

dg845 reviewed Dec 4, 2023

View reviewed changes

pcuenca added 2 commits December 5, 2023 11:08

Minor tweaks to the readmes.

53abd8e

Stress that the database is illustrative.

309cfe1

pcuenca requested a review from dg845 December 5, 2023 10:11

dg845 approved these changes Dec 6, 2023

View reviewed changes

pcuenca merged commit ab6672f into main Dec 6, 2023

pcuenca deleted the lcm-wds-example branch December 6, 2023 09:35

pcuenca mentioned this pull request Dec 6, 2023

Can not run LCM distill pipeline, due to dataset access #5770

Closed

sayakpaul mentioned this pull request Dec 11, 2023

LCM Lora training #6084

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use CC12M for LCM WDS training example #5908

Use CC12M for LCM WDS training example #5908

Uh oh!

pcuenca commented Nov 23, 2023 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Nov 23, 2023

Uh oh!

sayakpaul left a comment

Uh oh!

pcuenca commented Dec 4, 2023

Uh oh!

patrickvonplaten left a comment

Uh oh!

dg845 Dec 4, 2023

Uh oh!

dg845 Dec 5, 2023

Uh oh!

pcuenca Dec 5, 2023

Uh oh!

Uh oh!

dg845 Dec 4, 2023

Uh oh!

pcuenca Dec 5, 2023

Uh oh!

dg845 commented Dec 4, 2023

Uh oh!

pcuenca commented Dec 5, 2023 •

edited

Loading

Uh oh!

dg845 left a comment

Uh oh!

pcuenca commented Dec 6, 2023

Uh oh!

Uh oh!

	wds.map(filter_keys({"image", "text"})),
	wds.map(transform),
	wds.to_tuple("image", "text"),

Use CC12M for LCM WDS training example #5908

Use CC12M for LCM WDS training example #5908

Uh oh!

Conversation

pcuenca commented Nov 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Nov 23, 2023

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

pcuenca commented Dec 4, 2023

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

dg845 Dec 4, 2023

Choose a reason for hiding this comment

Uh oh!

dg845 Dec 5, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca Dec 5, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dg845 Dec 4, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca Dec 5, 2023

Choose a reason for hiding this comment

Uh oh!

dg845 commented Dec 4, 2023

Uh oh!

pcuenca commented Dec 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dg845 left a comment

Choose a reason for hiding this comment

Uh oh!

pcuenca commented Dec 6, 2023

Uh oh!

Uh oh!

pcuenca commented Nov 23, 2023 •

edited

Loading

pcuenca commented Dec 5, 2023 •

edited

Loading