Skip to content

Commit 086c7f9

Browse files
anton-lpatrickvonplatenpcuenca
authored
Nightly integration tests (huggingface#1664)
* [WIP] Nightly integration tests * initial SD tests * update SD slow tests * style * repaint * ImageVariations * style * finish imgvar * img2img tests * debug * inpaint 1.5 * inpaint legacy * torch isn't happy about deterministic ops * allclose -> max diff for shorter logs * add SD2 * debug * Update tests/pipelines/stable_diffusion_2/test_stable_diffusion.py Co-authored-by: Patrick von Platen <[email protected]> * Update tests/pipelines/stable_diffusion/test_stable_diffusion.py Co-authored-by: Patrick von Platen <[email protected]> * fix refs * Update src/diffusers/utils/testing_utils.py Co-authored-by: Pedro Cuenca <[email protected]> * fix refs * remove debug Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Pedro Cuenca <[email protected]>
1 parent acd3178 commit 086c7f9

18 files changed

+1317
-925
lines changed

‎.github/workflows/nightly_tests.yml‎

Lines changed: 101 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Nightly integration tests
1+
name: Nightly tests on main
22

33
on:
44
schedule:
@@ -9,12 +9,108 @@ env:
99
HF_HOME: /mnt/cache
1010
OMP_NUM_THREADS: 8
1111
MKL_NUM_THREADS: 8
12-
PYTEST_TIMEOUT: 1000
12+
PYTEST_TIMEOUT: 600
1313
RUN_SLOW: yes
14+
RUN_NIGHTLY: yes
1415

1516
jobs:
16-
run_slow_tests_apple_m1:
17-
name: Slow PyTorch MPS tests on MacOS
17+
run_nightly_tests:
18+
strategy:
19+
fail-fast: false
20+
matrix:
21+
config:
22+
- name: Nightly PyTorch CUDA tests on Ubuntu
23+
framework: pytorch
24+
runner: docker-gpu
25+
image: diffusers/diffusers-pytorch-cuda
26+
report: torch_cuda
27+
- name: Nightly Flax TPU tests on Ubuntu
28+
framework: flax
29+
runner: docker-tpu
30+
image: diffusers/diffusers-flax-tpu
31+
report: flax_tpu
32+
- name: Nightly ONNXRuntime CUDA tests on Ubuntu
33+
framework: onnxruntime
34+
runner: docker-gpu
35+
image: diffusers/diffusers-onnxruntime-cuda
36+
report: onnx_cuda
37+
38+
name: ${{ matrix.config.name }}
39+
40+
runs-on: ${{ matrix.config.runner }}
41+
42+
container:
43+
image: ${{ matrix.config.image }}
44+
options: --shm-size "16gb" --ipc host -v /mnt/hf_cache:/mnt/cache/ ${{ matrix.config.runner == 'docker-tpu' && '--privileged' || '--gpus 0'}}
45+
46+
defaults:
47+
run:
48+
shell: bash
49+
50+
steps:
51+
- name: Checkout diffusers
52+
uses: actions/checkout@v3
53+
with:
54+
fetch-depth: 2
55+
56+
- name: NVIDIA-SMI
57+
if: ${{ matrix.config.runner == 'docker-gpu' }}
58+
run: |
59+
nvidia-smi
60+
61+
- name: Install dependencies
62+
run: |
63+
python -m pip install -e .[quality,test]
64+
python -m pip install git+https://github.com/huggingface/accelerate
65+
python -m pip install -U git+https://github.com/huggingface/transformers
66+
67+
- name: Environment
68+
run: |
69+
python utils/print_env.py
70+
71+
- name: Run nightly PyTorch CUDA tests
72+
if: ${{ matrix.config.framework == 'pytorch' }}
73+
env:
74+
HUGGING_FACE_HUB_TOKEN: ${{ secrets.HUGGING_FACE_HUB_TOKEN }}
75+
run: |
76+
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \
77+
-s -v -k "not Flax and not Onnx" \
78+
--make-reports=tests_${{ matrix.config.report }} \
79+
tests/
80+
81+
- name: Run nightly Flax TPU tests
82+
if: ${{ matrix.config.framework == 'flax' }}
83+
env:
84+
HUGGING_FACE_HUB_TOKEN: ${{ secrets.HUGGING_FACE_HUB_TOKEN }}
85+
run: |
86+
python -m pytest -n 0 \
87+
-s -v -k "Flax" \
88+
--make-reports=tests_${{ matrix.config.report }} \
89+
tests/
90+
91+
- name: Run nightly ONNXRuntime CUDA tests
92+
if: ${{ matrix.config.framework == 'onnxruntime' }}
93+
env:
94+
HUGGING_FACE_HUB_TOKEN: ${{ secrets.HUGGING_FACE_HUB_TOKEN }}
95+
run: |
96+
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \
97+
-s -v -k "Onnx" \
98+
--make-reports=tests_${{ matrix.config.report }} \
99+
tests/
100+
101+
- name: Failure short reports
102+
if: ${{ failure() }}
103+
run: cat reports/tests_${{ matrix.config.report }}_failures_short.txt
104+
105+
- name: Test suite reports artifacts
106+
if: ${{ always() }}
107+
uses: actions/upload-artifact@v2
108+
with:
109+
name: ${{ matrix.config.report }}_test_reports
110+
path: reports
111+
112+
run_nightly_tests_apple_m1:
113+
name: Nightly PyTorch MPS tests on MacOS
18114
runs-on: [ self-hosted, apple-m1 ]
19115

20116
steps:
@@ -46,7 +142,7 @@ jobs:
46142
run: |
47143
${CONDA_RUN} python utils/print_env.py
48144
49-
- name: Run slow PyTorch tests on M1 (MPS)
145+
- name: Run nightly PyTorch tests on M1 (MPS)
50146
shell: arch -arch arm64 bash {0}
51147
env:
52148
HF_HOME: /System/Volumes/Data/mnt/cache

‎.github/workflows/pr_tests.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Run fast tests
1+
name: Fast tests for PRs
22

33
on:
44
pull_request:

‎.github/workflows/push_tests.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Run all tests
1+
name: Slow tests on main
22

33
on:
44
push:
@@ -10,7 +10,7 @@ env:
1010
HF_HOME: /mnt/cache
1111
OMP_NUM_THREADS: 8
1212
MKL_NUM_THREADS: 8
13-
PYTEST_TIMEOUT: 1000
13+
PYTEST_TIMEOUT: 600
1414
RUN_SLOW: yes
1515

1616
jobs:

‎src/diffusers/utils/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@
5555
load_hf_numpy,
5656
load_image,
5757
load_numpy,
58+
nightly,
5859
parse_flag_from_env,
5960
require_torch_gpu,
6061
slow,

‎src/diffusers/utils/testing_utils.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ def parse_flag_from_env(key, default=False):
8383

8484

8585
_run_slow_tests = parse_flag_from_env("RUN_SLOW", default=False)
86+
_run_nightly_tests = parse_flag_from_env("RUN_NIGHTLY", default=False)
8687

8788

8889
def floats_tensor(shape, scale=1.0, rng=None, name=None):
@@ -111,6 +112,16 @@ def slow(test_case):
111112
return unittest.skipUnless(_run_slow_tests, "test is slow")(test_case)
112113

113114

115+
def nightly(test_case):
116+
"""
117+
Decorator marking a test that runs nightly in the diffusers CI.
118+
119+
Slow tests are skipped by default. Set the RUN_NIGHTLY environment variable to a truthy value to run them.
120+
121+
"""
122+
return unittest.skipUnless(_run_nightly_tests, "test is nightly")(test_case)
123+
124+
114125
def require_torch(test_case):
115126
"""
116127
Decorator marking a test that requires PyTorch. These tests are skipped when PyTorch isn't installed.

‎tests/pipelines/repaint/test_repaint.py‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,14 @@
1313
# See the License for the specific language governing permissions and
1414
# limitations under the License.
1515

16+
import gc
1617
import unittest
1718

1819
import numpy as np
1920
import torch
2021

2122
from diffusers import RePaintPipeline, RePaintScheduler, UNet2DModel
22-
from diffusers.utils.testing_utils import load_image, load_numpy, require_torch_gpu, slow, torch_device
23+
from diffusers.utils.testing_utils import load_image, load_numpy, nightly, require_torch_gpu, torch_device
2324

2425
from ...test_pipelines_common import PipelineTesterMixin
2526

@@ -83,9 +84,14 @@ def test_repaint(self):
8384
assert np.abs(image_slice.flatten() - expected_slice).max() < 1e-3
8485

8586

86-
@slow
87+
@nightly
8788
@require_torch_gpu
88-
class RepaintPipelineIntegrationTests(unittest.TestCase):
89+
class RepaintPipelineNightlyTests(unittest.TestCase):
90+
def tearDown(self):
91+
super().tearDown()
92+
gc.collect()
93+
torch.cuda.empty_cache()
94+
8995
def test_celebahq(self):
9096
original_image = load_image(
9197
"https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/"
@@ -104,6 +110,8 @@ def test_celebahq(self):
104110
scheduler = RePaintScheduler.from_pretrained(model_id)
105111

106112
repaint = RePaintPipeline(unet=unet, scheduler=scheduler).to(torch_device)
113+
repaint.set_progress_bar_config(disable=None)
114+
repaint.enable_attention_slicing()
107115

108116
generator = torch.Generator(device=torch_device).manual_seed(0)
109117
output = repaint(

‎tests/pipelines/stable_diffusion/test_onnx_stable_diffusion.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
OnnxStableDiffusionPipeline,
2828
PNDMScheduler,
2929
)
30-
from diffusers.utils.testing_utils import is_onnx_available, require_onnxruntime, require_torch_gpu, slow
30+
from diffusers.utils.testing_utils import is_onnx_available, nightly, require_onnxruntime, require_torch_gpu
3131

3232
from ...test_pipelines_onnx_common import OnnxPipelineTesterMixin
3333

@@ -128,7 +128,7 @@ def test_pipeline_dpm_multistep(self):
128128
assert np.abs(image_slice.flatten() - expected_slice).max() < 1e-2
129129

130130

131-
@slow
131+
@nightly
132132
@require_onnxruntime
133133
@require_torch_gpu
134134
class OnnxStableDiffusionPipelineIntegrationTests(unittest.TestCase):

‎tests/pipelines/stable_diffusion/test_onnx_stable_diffusion_img2img.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,13 @@
2727
PNDMScheduler,
2828
)
2929
from diffusers.utils import floats_tensor
30-
from diffusers.utils.testing_utils import is_onnx_available, load_image, require_onnxruntime, require_torch_gpu, slow
30+
from diffusers.utils.testing_utils import (
31+
is_onnx_available,
32+
load_image,
33+
nightly,
34+
require_onnxruntime,
35+
require_torch_gpu,
36+
)
3137

3238
from ...test_pipelines_onnx_common import OnnxPipelineTesterMixin
3339

@@ -134,7 +140,7 @@ def test_pipeline_dpm_multistep(self):
134140
assert np.abs(image_slice.flatten() - expected_slice).max() < 1e-1
135141

136142

137-
@slow
143+
@nightly
138144
@require_onnxruntime
139145
@require_torch_gpu
140146
class OnnxStableDiffusionImg2ImgPipelineIntegrationTests(unittest.TestCase):

‎tests/pipelines/stable_diffusion/test_onnx_stable_diffusion_inpaint.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,13 @@
1818
import numpy as np
1919

2020
from diffusers import LMSDiscreteScheduler, OnnxStableDiffusionInpaintPipeline
21-
from diffusers.utils.testing_utils import is_onnx_available, load_image, require_onnxruntime, require_torch_gpu, slow
21+
from diffusers.utils.testing_utils import (
22+
is_onnx_available,
23+
load_image,
24+
nightly,
25+
require_onnxruntime,
26+
require_torch_gpu,
27+
)
2228

2329
from ...test_pipelines_onnx_common import OnnxPipelineTesterMixin
2430

@@ -32,7 +38,7 @@ class OnnxStableDiffusionPipelineFastTests(OnnxPipelineTesterMixin, unittest.Tes
3238
pass
3339

3440

35-
@slow
41+
@nightly
3642
@require_onnxruntime
3743
@require_torch_gpu
3844
class OnnxStableDiffusionInpaintPipelineIntegrationTests(unittest.TestCase):

‎tests/pipelines/stable_diffusion/test_onnx_stable_diffusion_inpaint_legacy.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,17 +22,17 @@
2222
is_onnx_available,
2323
load_image,
2424
load_numpy,
25+
nightly,
2526
require_onnxruntime,
2627
require_torch_gpu,
27-
slow,
2828
)
2929

3030

3131
if is_onnx_available():
3232
import onnxruntime as ort
3333

3434

35-
@slow
35+
@nightly
3636
@require_onnxruntime
3737
@require_torch_gpu
3838
class StableDiffusionOnnxInpaintLegacyPipelineIntegrationTests(unittest.TestCase):

0 commit comments

Comments
 (0)