Skip to content

Commit 3daebe2

Browse files
sayakpaulsayakpaul
authored
use uv for installing stuff in the workflows. (huggingface#7116)
* use uv for installing stuff in the workflows. * fix: from source installation command when using uv. * fix uv venv issue * edit editable installation. * fix quality installation * checking * make editable. * more check * check * add: export step * venv handling. * checking. * fix: dependency workflows. * peft tests. * proper way to initialize env. * Empty-Commit * Empty-Commit
1 parent abd922b commit 3daebe2

11 files changed

+104
-57
lines changed

‎.github/workflows/benchmark.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,9 @@ jobs:
3131
- name: Install dependencies
3232
run: |
3333
apt-get update && apt-get install libsndfile1-dev libgl1 -y
34-
python -m pip install -e .[quality,test]
35-
python -m pip install pandas peft
34+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
35+
python -m uv pip install -e [quality,test]
36+
python -m uv pip install pandas peft
3637
- name: Environment
3738
run: |
3839
python utils/print_env.py

‎.github/workflows/nightly_tests.yml‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -60,9 +60,10 @@ jobs:
6060
6161
- name: Install dependencies
6262
run: |
63-
python -m pip install -e .[quality,test]
64-
python -m pip install -U git+https://github.com/huggingface/transformers
65-
python -m pip install git+https://github.com/huggingface/accelerate
63+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
64+
python -m uv pip install -e [quality,test]
65+
python -m uv pip install -U transformers@git+https://github.com/huggingface/transformers
66+
python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate
6667
6768
- name: Environment
6869
run: |
@@ -73,6 +74,7 @@ jobs:
7374
env:
7475
HUGGING_FACE_HUB_TOKEN: ${{ secrets.HUGGING_FACE_HUB_TOKEN }}
7576
run: |
77+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
7678
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \
7779
-s -v -k "not Flax and not Onnx" \
7880
--make-reports=tests_${{ matrix.config.report }} \
@@ -83,6 +85,7 @@ jobs:
8385
env:
8486
HUGGING_FACE_HUB_TOKEN: ${{ secrets.HUGGING_FACE_HUB_TOKEN }}
8587
run: |
88+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
8689
python -m pytest -n 0 \
8790
-s -v -k "Flax" \
8891
--make-reports=tests_${{ matrix.config.report }} \
@@ -93,6 +96,7 @@ jobs:
9396
env:
9497
HUGGING_FACE_HUB_TOKEN: ${{ secrets.HUGGING_FACE_HUB_TOKEN }}
9598
run: |
99+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
96100
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \
97101
-s -v -k "Onnx" \
98102
--make-reports=tests_${{ matrix.config.report }} \
@@ -132,10 +136,10 @@ jobs:
132136
- name: Install dependencies
133137
shell: arch -arch arm64 bash {0}
134138
run: |
135-
${CONDA_RUN} python -m pip install --upgrade pip
136-
${CONDA_RUN} python -m pip install -e .[quality,test]
137-
${CONDA_RUN} python -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu
138-
${CONDA_RUN} python -m pip install git+https://github.com/huggingface/accelerate
139+
${CONDA_RUN} python -m pip install --upgrade pip uv
140+
${CONDA_RUN} python -m uv pip install -e [quality,test]
141+
${CONDA_RUN} python -m uv pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu
142+
${CONDA_RUN} python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate
139143
140144
- name: Environment
141145
shell: arch -arch arm64 bash {0}

‎.github/workflows/pr_dependency_test.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,12 @@ jobs:
2525
python-version: "3.8"
2626
- name: Install dependencies
2727
run: |
28-
python -m pip install --upgrade pip
29-
pip install -e .
30-
pip install pytest
28+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
29+
python -m pip install --upgrade pip uv
30+
python -m uv pip install -e .
31+
python -m uv pip install pytest
3132
- name: Check for soft dependencies
3233
run: |
34+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
3335
pytest tests/others/test_dependencies.py
3436

‎.github/workflows/pr_flax_dependency_test.yml‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,14 @@ jobs:
2525
python-version: "3.8"
2626
- name: Install dependencies
2727
run: |
28-
python -m pip install --upgrade pip
29-
pip install -e .
30-
pip install "jax[cpu]>=0.2.16,!=0.3.2"
31-
pip install "flax>=0.4.1"
32-
pip install "jaxlib>=0.1.65"
33-
pip install pytest
28+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
29+
python -m pip install --upgrade pip uv
30+
python -m uv pip install -e .
31+
python -m uv pip install "jax[cpu]>=0.2.16,!=0.3.2"
32+
python -m uv pip install "flax>=0.4.1"
33+
python -m uv pip install "jaxlib>=0.1.65"
34+
python -m uv pip install pytest
3435
- name: Check for soft dependencies
3536
run: |
37+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
3638
pytest tests/others/test_dependencies.py

‎.github/workflows/pr_test_fetcher.yml‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@ jobs:
3333
- name: Install dependencies
3434
run: |
3535
apt-get update && apt-get install libsndfile1-dev libgl1 -y
36-
python -m pip install -e .[quality,test]
36+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
37+
python -m uv pip install -e [quality,test]
3738
- name: Environment
3839
run: |
3940
python utils/print_env.py
@@ -89,15 +90,18 @@ jobs:
8990
- name: Install dependencies
9091
run: |
9192
apt-get update && apt-get install libsndfile1-dev libgl1 -y
92-
python -m pip install -e .[quality,test]
93+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
94+
python -m pip install -e [quality,test]
9395
python -m pip install accelerate
9496
9597
- name: Environment
9698
run: |
99+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
97100
python utils/print_env.py
98101
99102
- name: Run all selected tests on CPU
100103
run: |
104+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
101105
python -m pytest -n 2 --dist=loadfile -v --make-reports=${{ matrix.modules }}_tests_cpu ${{ fromJson(needs.setup_pr_tests.outputs.test_map)[matrix.modules] }}
102106
103107
- name: Failure short reports
@@ -144,15 +148,18 @@ jobs:
144148
- name: Install dependencies
145149
run: |
146150
apt-get update && apt-get install libsndfile1-dev libgl1 -y
147-
python -m pip install -e .[quality,test]
151+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
152+
python -m pip install -e [quality,test]
148153
149154
- name: Environment
150155
run: |
156+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
151157
python utils/print_env.py
152158
153159
- name: Run Hub tests for models, schedulers, and pipelines on a staging env
154160
if: ${{ matrix.config.framework == 'hub_tests_pytorch' }}
155161
run: |
162+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
156163
HUGGINGFACE_CO_STAGING=true python -m pytest \
157164
-m "is_staging_test" \
158165
--make-reports=tests_${{ matrix.config.report }} \

‎.github/workflows/pr_test_peft_backend.yml‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -84,21 +84,24 @@ jobs:
8484
- name: Install dependencies
8585
run: |
8686
apt-get update && apt-get install libsndfile1-dev libgl1 -y
87-
python -m pip install -e .[quality,test]
87+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
88+
python -m uv pip install -e [quality,test]
8889
if [ "${{ matrix.lib-versions }}" == "main" ]; then
89-
python -m pip install -U git+https://github.com/huggingface/peft.git
90-
python -m pip install -U git+https://github.com/huggingface/transformers.git
91-
python -m pip install -U git+https://github.com/huggingface/accelerate.git
90+
python -m uv pip install -U peft@git+https://github.com/huggingface/peft.git
91+
python -m uv pip install -U transformers@git+https://github.com/huggingface/transformers.git
92+
python -m uv pip install -U accelerate@git+https://github.com/huggingface/accelerate.git
9293
else
93-
python -m pip install -U peft transformers accelerate
94+
python -m uv pip install -U peft transformers accelerate
9495
fi
9596
9697
- name: Environment
9798
run: |
99+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
98100
python utils/print_env.py
99101
100102
- name: Run fast PyTorch LoRA CPU tests with PEFT backend
101103
run: |
104+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
102105
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \
103106
-s -v \
104107
--make-reports=tests_${{ matrix.config.report }} \

‎.github/workflows/pr_tests.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -111,16 +111,19 @@ jobs:
111111
- name: Install dependencies
112112
run: |
113113
apt-get update && apt-get install libsndfile1-dev libgl1 -y
114-
python -m pip install -e .[quality,test]
115-
python -m pip install accelerate
114+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
115+
python -m uv pip install -e [quality,test]
116+
python -m uv pip install accelerate
116117
117118
- name: Environment
118119
run: |
120+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
119121
python utils/print_env.py
120122
121123
- name: Run fast PyTorch Pipeline CPU tests
122124
if: ${{ matrix.config.framework == 'pytorch_pipelines' }}
123125
run: |
126+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
124127
python -m pytest -n 2 --max-worker-restart=0 --dist=loadfile \
125128
-s -v -k "not Flax and not Onnx" \
126129
--make-reports=tests_${{ matrix.config.report }} \
@@ -129,6 +132,7 @@ jobs:
129132
- name: Run fast PyTorch Model Scheduler CPU tests
130133
if: ${{ matrix.config.framework == 'pytorch_models' }}
131134
run: |
135+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
132136
python -m pytest -n 2 --max-worker-restart=0 --dist=loadfile \
133137
-s -v -k "not Flax and not Onnx and not Dependency" \
134138
--make-reports=tests_${{ matrix.config.report }} \
@@ -137,6 +141,7 @@ jobs:
137141
- name: Run fast Flax TPU tests
138142
if: ${{ matrix.config.framework == 'flax' }}
139143
run: |
144+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
140145
python -m pytest -n 2 --max-worker-restart=0 --dist=loadfile \
141146
-s -v -k "Flax" \
142147
--make-reports=tests_${{ matrix.config.report }} \
@@ -145,7 +150,8 @@ jobs:
145150
- name: Run example PyTorch CPU tests
146151
if: ${{ matrix.config.framework == 'pytorch_examples' }}
147152
run: |
148-
python -m pip install peft
153+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
154+
python -m uv pip install peft
149155
python -m pytest -n 2 --max-worker-restart=0 --dist=loadfile \
150156
--make-reports=tests_${{ matrix.config.report }} \
151157
examples
@@ -194,15 +200,18 @@ jobs:
194200
- name: Install dependencies
195201
run: |
196202
apt-get update && apt-get install libsndfile1-dev libgl1 -y
197-
python -m pip install -e .[quality,test]
203+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
204+
python -m uv pip install -e [quality,test]
198205
199206
- name: Environment
200207
run: |
208+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
201209
python utils/print_env.py
202210
203211
- name: Run Hub tests for models, schedulers, and pipelines on a staging env
204212
if: ${{ matrix.config.framework == 'hub_tests_pytorch' }}
205213
run: |
214+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
206215
HUGGINGFACE_CO_STAGING=true python -m pytest \
207216
-m "is_staging_test" \
208217
--make-reports=tests_${{ matrix.config.report }} \

‎.github/workflows/pr_torch_dependency_test.yml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,12 @@ jobs:
2525
python-version: "3.8"
2626
- name: Install dependencies
2727
run: |
28-
python -m pip install --upgrade pip
29-
pip install -e .
30-
pip install torch torchvision torchaudio
31-
pip install pytest
28+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
29+
python -m pip install --upgrade pip uv
30+
python -m uv pip install -e .
31+
python -m uv pip install torch torchvision torchaudio
32+
python -m uv pip install pytest
3233
- name: Check for soft dependencies
3334
run: |
35+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
3436
pytest tests/others/test_dependencies.py

‎.github/workflows/push_tests.yml‎

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,9 @@ jobs:
3535
- name: Install dependencies
3636
run: |
3737
apt-get update && apt-get install libsndfile1-dev libgl1 -y
38-
python -m pip install -e .[quality,test]
39-
python -m pip install git+https://github.com/huggingface/accelerate.git
38+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
39+
python -m uv pip install -e [quality,test]
40+
python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate.git
4041
4142
- name: Environment
4243
run: |
@@ -79,8 +80,9 @@ jobs:
7980
- name: Install dependencies
8081
run: |
8182
apt-get update && apt-get install libsndfile1-dev libgl1 -y
82-
python -m pip install -e .[quality,test]
83-
python -m pip install git+https://github.com/huggingface/accelerate.git
83+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
84+
python -m uv pip install -e [quality,test]
85+
python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate.git
8486
- name: Environment
8587
run: |
8688
python utils/print_env.py
@@ -128,8 +130,9 @@ jobs:
128130
- name: Install dependencies
129131
run: |
130132
apt-get update && apt-get install libsndfile1-dev libgl1 -y
131-
python -m pip install -e .[quality,test]
132-
python -m pip install git+https://github.com/huggingface/accelerate.git
133+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
134+
python -m uv pip install -e [quality,test]
135+
python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate.git
133136
134137
- name: Environment
135138
run: |
@@ -177,9 +180,10 @@ jobs:
177180
- name: Install dependencies
178181
run: |
179182
apt-get update && apt-get install libsndfile1-dev libgl1 -y
180-
python -m pip install -e .[quality,test]
181-
python -m pip install git+https://github.com/huggingface/accelerate.git
182-
python -m pip install git+https://github.com/huggingface/peft.git
183+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
184+
python -m uv pip install -e [quality,test]
185+
python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate.git
186+
python -m uv pip install peft@git+https://github.com/huggingface/peft.git
183187
184188
- name: Environment
185189
run: |
@@ -227,8 +231,9 @@ jobs:
227231
- name: Install dependencies
228232
run: |
229233
apt-get update && apt-get install libsndfile1-dev libgl1 -y
230-
python -m pip install -e .[quality,test]
231-
python -m pip install git+https://github.com/huggingface/accelerate.git
234+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
235+
python -m uv pip install -e [quality,test]
236+
python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate.git
232237
233238
- name: Environment
234239
run: |
@@ -274,8 +279,9 @@ jobs:
274279
- name: Install dependencies
275280
run: |
276281
apt-get update && apt-get install libsndfile1-dev libgl1 -y
277-
python -m pip install -e .[quality,test]
278-
python -m pip install git+https://github.com/huggingface/accelerate.git
282+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
283+
python -m uv pip install -e [quality,test]
284+
python -m uv pip install accelerate@git+https://github.com/huggingface/accelerate.git
279285
280286
- name: Environment
281287
run: |
@@ -323,7 +329,8 @@ jobs:
323329
nvidia-smi
324330
- name: Install dependencies
325331
run: |
326-
python -m pip install -e .[quality,test,training]
332+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
333+
python -m uv pip install -e [quality,test,training]
327334
- name: Environment
328335
run: |
329336
python utils/print_env.py
@@ -363,7 +370,8 @@ jobs:
363370
nvidia-smi
364371
- name: Install dependencies
365372
run: |
366-
python -m pip install -e .[quality,test,training]
373+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
374+
python -m uv pip install -e [quality,test,training]
367375
- name: Environment
368376
run: |
369377
python utils/print_env.py
@@ -404,16 +412,19 @@ jobs:
404412
405413
- name: Install dependencies
406414
run: |
407-
python -m pip install -e .[quality,test,training]
415+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
416+
python -m uv pip install -e [quality,test,training]
408417
409418
- name: Environment
410419
run: |
420+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
411421
python utils/print_env.py
412422
413423
- name: Run example tests on GPU
414424
env:
415425
HUGGING_FACE_HUB_TOKEN: ${{ secrets.HUGGING_FACE_HUB_TOKEN }}
416426
run: |
427+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
417428
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile -s -v --make-reports=examples_torch_cuda examples/
418429
419430
- name: Failure short reports

0 commit comments

Comments
 (0)