Skip to content

Commit 547348a

Browse files
Cadenealiberts
Cadene
authored and
aliberts
committed
Improve dataset examples (#82)
Co-authored-by: Alexander Soare <[email protected]> Tests cleaning & simplification (#81) Fix tolerance for delta_timestamps (#84) Co-authored-by: Remi <[email protected]> Hotfix test_examples.py (#87) Quality of life patches for eval.py (#86) Add meta_data, revision v1.1 WIP add load functions + episode_data_index id -> index, finish moving compute_stats before hf_dataset push_to_hub Use v1.1, hf_transform_to_torch, Add 3 xarm datasets Remove Prod, Tests are passind Add tests/data small fix fix visualize_dataset fix online training fix online training fix online training Fixes for datasets 2.18 -> 2.19 update (#88) WIP Dockerfile Add Makefile Add cpu build Add libhdf5-dev in build-image Add gpu build Add libegl1-mesa-dev in cpu build WIP Add docker build ci Fix docker context Fix dockerfile path Fix dockerhub repo Free up more disk space on runner Include envs in cpu build remove rm -rf /usr/local/share/boost Test with pip only Add python version file Fix python version & extras Checkout pathed envs Add test artifacts Update PR template WIP Add nightly tests Push gpu & cpu images to gpu/cpu dockerhub repos WIP Test nightly WIP test cpu build WIP checkout pip install Use Makefile for end-to-end tests Fix cpu venv path Fix end-to-end Checkout envs Revert "Checkout envs" This reverts commit 7193f60. Add working-directory Fix builds (rebase hotfix #87 from main) Test fixed builds rebase from user/rcadene/2024_04_18_episode_data_index Test images Check nvidia-smi Fix opengl on gpu image Test gpu build Try nvidia/cudagl Test cudagl build Try different registry
1 parent 659c69a commit 547348a

File tree

11 files changed

+669
-97
lines changed

11 files changed

+669
-97
lines changed

‎.dockerignore‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Logging
2+
logs
3+
tmp
4+
wandb
5+
data
6+
outputs
7+
.vscode
8+
rl
9+
media
10+
11+
# HPC
12+
nautilus/*.yaml
13+
*.key
14+
15+
# Slurm
16+
sbatch*.sh
17+
18+
# Byte-compiled / optimized / DLL files
19+
__pycache__/
20+
*.py[cod]
21+
*$py.class
22+
23+
# C extensions
24+
*.so
25+
26+
# Distribution / packaging
27+
.Python
28+
build/
29+
develop-eggs/
30+
dist/
31+
downloads/
32+
eggs/
33+
.eggs/
34+
lib/
35+
lib64/
36+
parts/
37+
sdist/
38+
var/
39+
wheels/
40+
pip-wheel-metadata/
41+
share/python-wheels/
42+
*.egg-info/
43+
.installed.cfg
44+
*.egg
45+
MANIFEST
46+
47+
# PyInstaller
48+
# Usually these files are written by a python script from a template
49+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
50+
*.manifest
51+
*.spec
52+
53+
# Installer logs
54+
pip-log.txt
55+
pip-delete-this-directory.txt
56+
57+
# Unit test / coverage reports
58+
!tests/data
59+
htmlcov/
60+
.tox/
61+
.nox/
62+
.coverage
63+
.coverage.*
64+
.cache
65+
nosetests.xml
66+
coverage.xml
67+
*.cover
68+
*.py,cover
69+
.hypothesis/
70+
.pytest_cache/
71+
72+
# Translations
73+
*.mo
74+
*.pot
75+
76+
# Django stuff:
77+
*.log
78+
local_settings.py
79+
db.sqlite3
80+
db.sqlite3-journal
81+
82+
# Flask stuff:
83+
instance/
84+
.webassets-cache
85+
86+
# Scrapy stuff:
87+
.scrapy
88+
89+
# Sphinx documentation
90+
docs/_build/
91+
92+
# PyBuilder
93+
target/
94+
95+
# Jupyter Notebook
96+
.ipynb_checkpoints
97+
98+
# IPython
99+
profile_default/
100+
ipython_config.py
101+
102+
# pyenv
103+
.python-version
104+
105+
# pipenv
106+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
107+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
108+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
109+
# install all needed dependencies.
110+
#Pipfile.lock
111+
112+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
113+
__pypackages__/
114+
115+
# Celery stuff
116+
celerybeat-schedule
117+
celerybeat.pid
118+
119+
# SageMath parsed files
120+
*.sage.py
121+
122+
# Spyder project settings
123+
.spyderproject
124+
.spyproject
125+
126+
# Rope project settings
127+
.ropeproject
128+
129+
# mkdocs documentation
130+
/site
131+
132+
# mypy
133+
.mypy_cache/
134+
.dmypy.json
135+
dmypy.json
136+
137+
# Pyre type checker
138+
.pyre/

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,29 @@
11
# What does this PR do?
22

3-
Example: Fixes # (issue)
3+
Examples:
4+
- Fixes # (issue)
5+
- Adds new dataset
6+
- Optimizes something
47

8+
## How was it tested?
9+
10+
Examples:
11+
- Added `test_something` in `tests/test_stuff.py`.
12+
- Added `new_feature` and checked that training converges with policy X on dataset/environment Y.
13+
- Optimized `some_function`, it now runs X times faster than previously.
14+
15+
## How to checkout & try? (for the reviewer)
16+
17+
Examples:
18+
```bash
19+
DATA_DIR=tests/data pytest -sx tests/test_stuff.py::test_something
20+
```
21+
```bash
22+
python lerobot/scripts/train.py --some.option=true
23+
```
524

625
## Before submitting
7-
- Read the [contributor guideline](https://github.com/huggingface/lerobot/blob/main/CONTRIBUTING.md#submitting-a-pull-request-pr).
8-
- Provide a minimal code example for the reviewer to checkout & try.
9-
- Explain how you tested your changes.
26+
Please read the [contributor guideline](https://github.com/huggingface/lerobot/blob/main/CONTRIBUTING.md#submitting-a-pull-request-pr).
1027

1128

1229
## Who can review?

‎.github/workflows/build_docker_images.yml‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Inspired by
2+
# https://github.com/huggingface/peft/blob/main/.github/workflows/build_docker_images.yml
3+
name: Build Docker images (scheduled)
4+
5+
on:
6+
pull_request: # TODO(aliberts): for testing, delete before merge
7+
branches:
8+
- main
9+
workflow_dispatch:
10+
workflow_call:
11+
schedule:
12+
- cron: "0 1 * * *"
13+
14+
concurrency:
15+
group: docker-image-builds
16+
cancel-in-progress: false
17+
18+
# env:
19+
# CI_SLACK_CHANNEL: ${{ secrets.CI_DOCKER_CHANNEL }}
20+
21+
jobs:
22+
latest-cpu:
23+
name: "Latest LeRobot CPU [dev]"
24+
runs-on: ubuntu-latest
25+
steps:
26+
- name: Cleanup disk
27+
run: |
28+
sudo df -h
29+
# sudo ls -l /usr/local/lib/
30+
# sudo ls -l /usr/share/
31+
sudo du -sh /usr/local/lib/
32+
sudo du -sh /usr/share/
33+
sudo rm -rf /usr/local/lib/android
34+
sudo rm -rf /usr/share/dotnet
35+
sudo du -sh /usr/local/lib/
36+
sudo du -sh /usr/share/
37+
sudo df -h
38+
- name: Set up Docker Buildx
39+
uses: docker/setup-buildx-action@v3
40+
- name: Check out code
41+
uses: actions/checkout@v4
42+
- name: Checkout gym-aloha
43+
uses: actions/checkout@v4
44+
with:
45+
repository: huggingface/gym-aloha
46+
path: envs/gym-aloha
47+
ssh-key: ${{ secrets.SSH_PRIVATE_KEY }}
48+
- name: Checkout gym-xarm
49+
uses: actions/checkout@v4
50+
with:
51+
repository: huggingface/gym-xarm
52+
path: envs/gym-xarm
53+
ssh-key: ${{ secrets.SSH_PRIVATE_KEY }}
54+
- name: Checkout gym-pusht
55+
uses: actions/checkout@v4
56+
with:
57+
repository: huggingface/gym-pusht
58+
path: envs/gym-pusht
59+
ssh-key: ${{ secrets.SSH_PRIVATE_KEY }}
60+
- name: Login to DockerHub
61+
uses: docker/login-action@v3
62+
with:
63+
username: ${{ secrets.DOCKERHUB_USER_CUSTOM }}
64+
password: ${{ secrets.DOCKERHUB_PASSWORD_CUSTOM }}
65+
66+
- name: Build and Push CPU
67+
uses: docker/build-push-action@v5
68+
with:
69+
context: .
70+
file: ./docker/lerobot-cpu/Dockerfile
71+
push: true
72+
# tags: huggingface/lerobot-cpu
73+
tags: ${{ secrets.DOCKERHUB_REGISTRY_CUSTOM }}
74+
75+
# - name: Post to a Slack channel
76+
# id: slack
77+
# #uses: slackapi/[email protected]
78+
# uses: slackapi/slack-github-action@6c661ce58804a1a20f6dc5fbee7f0381b469e001
79+
# with:
80+
# # Slack channel id, channel name, or user id to post message.
81+
# # See also: https://api.slack.com/methods/chat.postMessage#channels
82+
# channel-id: ${{ env.CI_SLACK_CHANNEL }}
83+
# # For posting a rich message using Block Kit
84+
# payload: |
85+
# {
86+
# "text": "lerobot-cpu Docker Image build result: ${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head_commit.url }}",
87+
# "blocks": [
88+
# {
89+
# "type": "section",
90+
# "text": {
91+
# "type": "mrkdwn",
92+
# "text": "lerobot-cpu Docker Image build result: ${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head_commit.url }}"
93+
# }
94+
# }
95+
# ]
96+
# }
97+
# env:
98+
# SLACK_BOT_TOKEN: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}
99+
100+
latest-cuda:
101+
name: "Latest LeRobot GPU [dev]"
102+
runs-on: ubuntu-latest
103+
steps:
104+
- name: Cleanup disk
105+
run: |
106+
sudo df -h
107+
# sudo ls -l /usr/local/lib/
108+
# sudo ls -l /usr/share/
109+
sudo du -sh /usr/local/lib/
110+
sudo du -sh /usr/share/
111+
sudo rm -rf /usr/local/lib/android
112+
sudo rm -rf /usr/share/dotnet
113+
sudo du -sh /usr/local/lib/
114+
sudo du -sh /usr/share/
115+
sudo df -h
116+
- name: Set up Docker Buildx
117+
uses: docker/setup-buildx-action@v3
118+
- name: Check out code
119+
uses: actions/checkout@v4
120+
- name: Checkout gym-aloha
121+
uses: actions/checkout@v4
122+
with:
123+
repository: huggingface/gym-aloha
124+
path: envs/gym-aloha
125+
ssh-key: ${{ secrets.SSH_PRIVATE_KEY }}
126+
- name: Checkout gym-xarm
127+
uses: actions/checkout@v4
128+
with:
129+
repository: huggingface/gym-xarm
130+
path: envs/gym-xarm
131+
ssh-key: ${{ secrets.SSH_PRIVATE_KEY }}
132+
- name: Checkout gym-pusht
133+
uses: actions/checkout@v4
134+
with:
135+
repository: huggingface/gym-pusht
136+
path: envs/gym-pusht
137+
ssh-key: ${{ secrets.SSH_PRIVATE_KEY }}
138+
- name: Login to DockerHub
139+
uses: docker/login-action@v3
140+
with:
141+
username: ${{ secrets.DOCKERHUB_USERNAME }}
142+
password: ${{ secrets.DOCKERHUB_PASSWORD }}
143+
144+
- name: Build and Push GPU
145+
uses: docker/build-push-action@v5
146+
with:
147+
context: .
148+
file: ./docker/lerobot-gpu/Dockerfile
149+
push: true
150+
tags: huggingface/lerobot-gpu
151+
152+
# - name: Post to a Slack channel
153+
# id: slack
154+
# #uses: slackapi/[email protected]
155+
# uses: slackapi/slack-github-action@6c661ce58804a1a20f6dc5fbee7f0381b469e001
156+
# with:
157+
# # Slack channel id, channel name, or user id to post message.
158+
# # See also: https://api.slack.com/methods/chat.postMessage#channels
159+
# channel-id: ${{ env.CI_SLACK_CHANNEL }}
160+
# # For posting a rich message using Block Kit
161+
# payload: |
162+
# {
163+
# "text": "lerobot-gpu Docker Image build result: ${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head_commit.url }}",
164+
# "blocks": [
165+
# {
166+
# "type": "section",
167+
# "text": {
168+
# "type": "mrkdwn",
169+
# "text": "lerobot-gpu Docker Image build result: ${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head_commit.url }}"
170+
# }
171+
# }
172+
# ]
173+
# }
174+
# env:
175+
# SLACK_BOT_TOKEN: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}

0 commit comments

Comments
 (0)