Skip to content

chore: Install python dependencies with uv in workflows #4086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HaoXuAI merged 9 commits into from
Apr 23, 2024
Merged

chore: Install python dependencies with uv in workflows #4086

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6e165e0
install dependencies in unit-tests with uv
tokoko Apr 9, 2024
8ea1210
install dependencies in unit-tests with uv
tokoko Apr 9, 2024
2ba3708
enable caching, change linter job
tokoko Apr 9, 2024
e6400ac
change local integration tests to uv
tokoko Apr 9, 2024
8f3b362
change all installs to uv
tokoko Apr 10, 2024
55bb636
try adding uv cache
tokoko Apr 10, 2024
3e15126
fix lambda cache step name
tokoko Apr 10, 2024
4563ace
reenable caches for uv
tokoko Apr 11, 2024
36338c6
remove dangling cache step
tokoko Apr 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 4 additions & 18 deletions .github/workflows/linter.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,31 +11,17 @@ jobs:
- uses: actions/checkout@v3
- name: Setup Python
id: setup-python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: "3.9"
architecture: x64
- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"
- name: pip cache
uses: actions/cache@v2
with:
path: |
${{ steps.pip-cache.outputs.dir }}
/opt/hostedtoolcache/Python
/Users/runner/hostedtoolcache/Python
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install uv
run: pip install uv
- name: Install dependencies
run: |
make install-python-ci-dependencies
make install-python-ci-dependencies-uv
- name: Lint python
run: make lint-python
31 changes: 13 additions & 18 deletions .github/workflows/master_only.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v5
- name: Setup Python
id: setup-python
uses: actions/setup-python@v3
Expand All @@ -106,27 +106,22 @@ jobs:
aws-region: us-west-2
- name: Use AWS CLI
run: aws sts get-caller-identity
- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"
- name: pip cache
uses: actions/cache@v2
with:
path: |
${{ steps.pip-cache.outputs.dir }}
/opt/hostedtoolcache/Python
/Users/runner/hostedtoolcache/Python
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install uv
run: pip install uv
- name: Get uv cache dir
id: uv-cache
run: |
echo "::set-output name=dir::$(uv cache dir)"
- name: uv cache
uses: actions/cache@v4
with:
path: ${{ steps.uv-cache.outputs.dir }}
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-uv-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
- name: Install dependencies
run: make install-python-ci-dependencies
run: make install-python-ci-dependencies-uv
- name: Setup Redis Cluster
run: |
docker pull vishnunair/docker-redis-cluster:latest
Expand Down
37 changes: 16 additions & 21 deletions .github/workflows/nightly-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
with:
ref: master
- name: Setup Python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
id: setup-python
with:
python-version: "3.9"
Expand Down Expand Up @@ -90,18 +90,18 @@ jobs:
run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
- name: Cache Public ECR Image
id: lambda_python_3_9
uses: actions/cache@v2
uses: actions/cache@v4
with:
path: ~/cache
key: lambda_python_3_9
- name: Handle Cache Miss (pull public ECR image & save it to tar file)
if: steps.cache-primes.outputs.cache-hit != 'true'
if: steps.lambda_python_3_9.outputs.cache-hit != 'true'
run: |
mkdir -p ~/cache
docker pull public.ecr.aws/lambda/python:3.9
docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
- name: Handle Cache Hit (load docker image from tar file)
if: steps.cache-primes.outputs.cache-hit == 'true'
if: steps.lambda_python_3_9.outputs.cache-hit == 'true'
run: |
docker load -i ~/cache/lambda_python_3_9.tar
- name: Build and push
Expand Down Expand Up @@ -145,7 +145,7 @@ jobs:
ref: master
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
id: setup-python
with:
python-version: ${{ matrix.python-version }}
Expand Down Expand Up @@ -173,25 +173,20 @@ jobs:
aws-region: us-west-2
- name: Use AWS CLI
run: aws sts get-caller-identity
- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"
- name: pip cache
uses: actions/cache@v2
with:
path: |
${{ steps.pip-cache.outputs.dir }}
/opt/hostedtoolcache/Python
/Users/runner/hostedtoolcache/Python
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install uv
run: pip install uv
- name: Get uv cache dir
id: uv-cache
run: |
echo "::set-output name=dir::$(uv cache dir)"
- name: uv cache
uses: actions/cache@v4
with:
path: ${{ steps.uv-cache.outputs.dir }}
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-uv-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
- name: Install apache-arrow on ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
Expand Down
31 changes: 13 additions & 18 deletions .github/workflows/pr_integration_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ jobs:
ref: refs/pull/${{ github.event.pull_request.number }}/merge
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
id: setup-python
with:
python-version: ${{ matrix.python-version }}
Expand All @@ -133,27 +133,22 @@ jobs:
aws-region: us-west-2
- name: Use AWS CLI
run: aws sts get-caller-identity
- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"
- name: pip cache
uses: actions/cache@v2
with:
path: |
${{ steps.pip-cache.outputs.dir }}
/opt/hostedtoolcache/Python
/Users/runner/hostedtoolcache/Python
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install uv
run: pip install uv
- name: Get uv cache dir
id: uv-cache
run: |
echo "::set-output name=dir::$(uv cache dir)"
- name: uv cache
uses: actions/cache@v4
with:
path: ${{ steps.uv-cache.outputs.dir }}
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-uv-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
- name: Install dependencies
run: make install-python-ci-dependencies
run: make install-python-ci-dependencies-uv
- name: Setup Redis Cluster
run: |
docker pull vishnunair/docker-redis-cluster:latest
Expand Down
31 changes: 13 additions & 18 deletions .github/workflows/pr_local_integration_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,32 +33,27 @@ jobs:
ref: refs/pull/${{ github.event.pull_request.number }}/merge
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
id: setup-python
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"
- name: pip cache
uses: actions/cache@v2
with:
path: |
${{ steps.pip-cache.outputs.dir }}
/opt/hostedtoolcache/Python
/Users/runner/hostedtoolcache/Python
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install uv
run: pip install uv
- name: Get uv cache dir
id: uv-cache
run: |
echo "::set-output name=dir::$(uv cache dir)"
- name: uv cache
uses: actions/cache@v4
with:
path: ${{ steps.uv-cache.outputs.dir }}
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-uv-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
- name: Install dependencies
run: make install-python-ci-dependencies
run: make install-python-ci-dependencies-uv
- name: Test local integration tests
if: ${{ always() }} # this will guarantee that step won't be canceled and resources won't leak
run: make test-python-integration-local
32 changes: 14 additions & 18 deletions .github/workflows/unit_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,31 +19,27 @@ jobs:
- uses: actions/checkout@v3
- name: Setup Python
id: setup-python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"
- name: pip cache
uses: actions/cache@v2
with:
path: |
${{ steps.pip-cache.outputs.dir }}
/opt/hostedtoolcache/Python
/Users/runner/hostedtoolcache/Python
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install uv
run: |
pip install uv
- name: Get uv cache dir
id: uv-cache
run: |
echo "::set-output name=dir::$(uv cache dir)"
- name: uv cache
uses: actions/cache@v4
with:
path: ${{ steps.uv-cache.outputs.dir }}
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-uv-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
- name: Install dependencies
run: make install-python-ci-dependencies
run: make install-python-ci-dependencies-uv
- name: Test Python
run: make test-python-unit

Expand Down
5 changes: 5 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ install-python-ci-dependencies:
pip install --no-deps -e .
python setup.py build_python_protos --inplace

install-python-ci-dependencies-uv:
Copy link
Collaborator

@HaoXuAI HaoXuAI Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not familiar with uv. but using it to install in the CI, will it have a different result with direct pip install?

Copy link
Collaborator Author

@tokoko tokoko Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all intents and purposes, no, it's intended to be a drop-in replacement. There's of course some minor differences, pip tends to be more lenient towards packages with invalid metadata for example, version resolution (which we aren't changing in this pr, but I'll do in a follow-up) might also be negligibly different because they are using different resolvers with different heuristics. But at the end of the day, it shouldn't matter for us.

uv pip sync --system sdk/python/requirements/py$(PYTHON)-ci-requirements.txt
uv pip install --system --no-deps -e .
python setup.py build_python_protos --inplace

lock-python-ci-dependencies:
python -m piptools compile -U --extra ci --output-file sdk/python/requirements/py$(PYTHON)-ci-requirements.txt

Expand Down