[SDXL and IP2P]: instruction pix2pix XL training and pipeline

[kfzyqin]

Support for Training Instruct Pix2Pix with Stable Diffusion XL

Description:

This pull request introduces support for training Instruct Pix2Pix with Stable Diffusion XL. The changes include two new files:

train_instruct_pix2pix_xl.py: This file contains the necessary code to train the Instruct Pix2Pix model using the Stable Diffusion XL method. It includes the training loop, loss function calculations, and backpropagation steps.

pipeline_stable_diffusion_xl_instruct_pix2pix.py: This file provides the pipeline for inferencing with the trained Instruct Pix2Pix model. It handles the pre-processing of input data, model inference, and post-processing of model outputs.

The addition of these files will allow users to train and inference with the Instruct Pix2Pix model of Stable Diffusion XL, which can potentially lead to improved performance and results.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[kfzyqin]

Bugs observed. Fixing now.

[andreemic]

Hey what bugs did you observe? I was working on the same thing, maybe I can help.

[kfzyqin]

Hey what bugs did you observe? I was working on the same thing, maybe I can help.

Training generates weird artifacts like below.

Looking into this now.

[kfzyqin]

Hey what bugs did you observe? I was working on the same thing, maybe I can help.

Training generates weird artifacts like below. Looking into this now.

Now bugs have been fixed. Reasonable images can be generated. Working on cleaning up code now.

[andreemic]

Did you take an SDXL training pipeline as base? Or what did you start with?

[kfzyqin]

Did you take an SDXL training pipeline as base? Or what did you start with?

I modified the original instruct pix2pix training pipeline to be compatible with SDXL. You may refer to the code.

[kfzyqin]

Hi @sayakpaul and @patrickvonplaten,

I have addressed the code review. Once I have run more experiments, I will give more detailed documentation on training IP2P with SDXL.

I will become very busy starting this week, since the semester has started and I need to teach and supervise students. If possible, can we merge this PR? If we have more interesting thoughts, we can address them in the future when I am not so busy.

[sayakpaul]

I have addressed the code review. Once I have run more experiments, I will give more detailed documentation on training IP2P with SDXL.

Having some good results documented (like the one you presented on interior design), preferably with the trained checkpoints, will be very beneficial to the community.

Maybe we could simply write something like WIP on the corresponding pages. @patrickvonplaten WDYT?

[kfzyqin]

Sure, we can do this in another PR, maybe this weekend. Let's please complete this one now.

[patrickvonplaten]

ew. Once I have run more experiments, I will give more detailed documentation on training IP2P with SDXL.

Ok for me!

[kfzyqin]

ew. Once I have run more experiments, I will give more detailed documentation on training IP2P with SDXL.

Ok for me!

@patrickvonplaten Can you merge the code? I believe this one is self-contained :-)

[patrickvonplaten]

If ok for @sayakpaul let's merge it :-)

[sayakpaul]

Let's merge. I think it's safe to revisit some stuff (documentation, examples, etc.) later. At least with the PR, we can enable the community.

[sayakpaul]

@harutatsuakiyama I took the liberty of adding mentions of SDXL InstructPix2Pix from the REAMDEs and also applied @patrickvonplaten's suggestions.

Will merge after the CI is green :)

[sayakpaul]

Thanks for your contribution!

[blldd]

@harutatsuakiyama oh wow! InstructPix2Pix is always very dear to my heart. This is why I worked on this: https://huggingface.co/blog/instruction-tuning-sd.
But anyway, could I get some results from your training runs before I do a deep review of your PR if it's not too much?

Sure. I trained on this dataset: https://www.zheyuanliu.me/CIRR/, since I have trained this dataset with IP2P before.

These are some examples:

Has dining room view, Change stools to long white benches, Add a chair on the right corner

Has similar layout of bed and pillows, Change bed to pink

The dataset is not very pix2pix, hence the change is big.

It's really great, but I follow the training script in reademe, the loss can't converge well, so I wonder if you can share ckpt?

[kfzyqin]

@harutatsuakiyama oh wow! InstructPix2Pix is always very dear to my heart. This is why I worked on this: https://huggingface.co/blog/instruction-tuning-sd.
But anyway, could I get some results from your training runs before I do a deep review of your PR if it's not too much?

Sure. I trained on this dataset: https://www.zheyuanliu.me/CIRR/, since I have trained this dataset with IP2P before.
These are some examples:
Has dining room view, Change stools to long white benches, Add a chair on the right corner
Has similar layout of bed and pillows, Change bed to pink
The dataset is not very pix2pix, hence the change is big.

It's really great, but I follow the training script in reademe, the loss can't converge well, so I wonder if you can share ckpt?

Will give more details on weekends. You will need to increase the batch size to be 4, which will improve performance. Also, be patient. I found loss may oscillate, but generated results are good.

If you have QQ, you can add me: 290956355. We can discuss more.

[blldd]

@harutatsuakiyama oh wow! InstructPix2Pix is always very dear to my heart. This is why I worked on this: https://huggingface.co/blog/instruction-tuning-sd.
But anyway, could I get some results from your training runs before I do a deep review of your PR if it's not too much?

Sure. I trained on this dataset: https://www.zheyuanliu.me/CIRR/, since I have trained this dataset with IP2P before.
These are some examples:
Has dining room view, Change stools to long white benches, Add a chair on the right corner
Has similar layout of bed and pillows, Change bed to pink
The dataset is not very pix2pix, hence the change is big.

It's really great, but I follow the training script in reademe, the loss can't converge well, so I wonder if you can share ckpt?

Will give more details on weekends. You will need to increase the batch size to be 4, which will improve performance. Also, be patient. I found loss may oscillate, but generated results are good.

If you have QQ, you can add me: 290956355. We can discuss more.

Nice

[bghira]

thanks to your great work @harutatsuakiyama i created a great extension for SimpleTuner, for general fine-tuning. you might find some improvements or changes there, that you might like. including min-SNR, which greatly assisted with convergence, and options to use D-adaptation for the same reason.