Skip to content

Conversation

alexander-soare
Copy link
Contributor

Fixes an issue where the is_pad was being calculated incorrectly in some datasets due to the tolerance being too high.

Does so by setting the tolerance independently for each dataset depending on its fps.

Also makes the tolerance a required parameter to force the user to think about their choice.

@alexander-soare alexander-soare requested a review from Cadene April 18, 2024 17:26
Copy link
Collaborator

@Cadene Cadene left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small suggestion. LGTM thanks!!!!!

Comment on lines +41 to +42
- The same item with the queried frames for each modality specified in delta_timestamps, with an additional key for
each modality (e.g. "observation.image_is_pad").
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you chose to break the line in multiple lines? What's your vscode plugin ^^
ideally we should have this be part of a pre-commit
cc @aliberts

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gotten quite quick at doing it manually lol. And I have "editor.rulers": [110], in my settings.json to see where the end is.

@alexander-soare alexander-soare merged commit 8d98094 into main Apr 18, 2024
@alexander-soare alexander-soare deleted the alexander-soare/fix_tolerance_for_delta_timestamps branch April 18, 2024 17:48
AdilZouitine pushed a commit that referenced this pull request Apr 19, 2024
aliberts pushed a commit that referenced this pull request Apr 23, 2024
Co-authored-by: Alexander Soare <[email protected]>

Tests cleaning & simplification (#81)

Fix tolerance for delta_timestamps (#84)

Co-authored-by: Remi <[email protected]>

Hotfix test_examples.py (#87)

Quality of life patches for eval.py (#86)

Add meta_data, revision v1.1

WIP add load functions + episode_data_index

id -> index, finish moving compute_stats before hf_dataset push_to_hub

Use v1.1, hf_transform_to_torch, Add 3 xarm datasets

Remove Prod, Tests are passind

Add tests/data

small fix

fix visualize_dataset

fix online training

fix online training

fix online training

Fixes for datasets 2.18 -> 2.19 update (#88)

WIP Dockerfile

Add Makefile

Add cpu build

Add libhdf5-dev in build-image

Add gpu build

Add libegl1-mesa-dev in cpu build

WIP Add docker build ci

Fix docker context

Fix dockerfile path

Fix dockerhub repo

Free up more disk space on runner

Include envs in cpu build

remove rm -rf /usr/local/share/boost

Test with pip only

Add python version file

Fix python version & extras

Checkout pathed envs

Add test artifacts

Update PR template

WIP Add nightly tests

Push gpu & cpu images to gpu/cpu dockerhub repos

WIP Test nightly

WIP test cpu build

WIP checkout pip install

Use Makefile for end-to-end tests

Fix cpu venv path

Fix end-to-end

Checkout envs

Revert "Checkout envs"

This reverts commit 7193f60.

Add working-directory

Fix builds (rebase hotfix #87 from main)

Test fixed builds

rebase from user/rcadene/2024_04_18_episode_data_index

Test images

Check nvidia-smi

Fix opengl on gpu image

Test gpu build

Try nvidia/cudagl

Test cudagl build

Try different registry
aliberts pushed a commit that referenced this pull request Apr 24, 2024
Co-authored-by: Alexander Soare <[email protected]>

Tests cleaning & simplification (#81)

Fix tolerance for delta_timestamps (#84)

Co-authored-by: Remi <[email protected]>

Hotfix test_examples.py (#87)

Quality of life patches for eval.py (#86)

Add meta_data, revision v1.1

WIP add load functions + episode_data_index

id -> index, finish moving compute_stats before hf_dataset push_to_hub

Use v1.1, hf_transform_to_torch, Add 3 xarm datasets

Remove Prod, Tests are passind

Add tests/data

small fix

fix visualize_dataset

fix online training

fix online training

fix online training

Fixes for datasets 2.18 -> 2.19 update (#88)

WIP Dockerfile

Add Makefile

Add cpu build

Add libhdf5-dev in build-image

Add gpu build

Add libegl1-mesa-dev in cpu build

WIP Add docker build ci

Fix docker context

Fix dockerfile path

Fix dockerhub repo

Free up more disk space on runner

Include envs in cpu build

remove rm -rf /usr/local/share/boost

Test with pip only

Add python version file

Fix python version & extras

Checkout pathed envs

Add test artifacts

Update PR template

WIP Add nightly tests

Push gpu & cpu images to gpu/cpu dockerhub repos

WIP Test nightly

WIP test cpu build

WIP checkout pip install

Use Makefile for end-to-end tests

Fix cpu venv path

Fix end-to-end

Checkout envs

Revert "Checkout envs"

This reverts commit 7193f60.

Add working-directory

Fix builds (rebase hotfix #87 from main)

Test fixed builds

rebase from user/rcadene/2024_04_18_episode_data_index

Test images

Check nvidia-smi

Fix opengl on gpu image

Test gpu build

Try nvidia/cudagl

Test cudagl build

Try different registry
aliberts pushed a commit that referenced this pull request Apr 25, 2024
Co-authored-by: Alexander Soare <[email protected]>

Tests cleaning & simplification (#81)

Fix tolerance for delta_timestamps (#84)

Co-authored-by: Remi <[email protected]>

Hotfix test_examples.py (#87)

Quality of life patches for eval.py (#86)

Add meta_data, revision v1.1

WIP add load functions + episode_data_index

id -> index, finish moving compute_stats before hf_dataset push_to_hub

Use v1.1, hf_transform_to_torch, Add 3 xarm datasets

Remove Prod, Tests are passind

Add tests/data

small fix

fix visualize_dataset

fix online training

fix online training

fix online training

Fixes for datasets 2.18 -> 2.19 update (#88)

WIP Dockerfile

Add Makefile

Add cpu build

Add libhdf5-dev in build-image

Add gpu build

Add libegl1-mesa-dev in cpu build

WIP Add docker build ci

Fix docker context

Fix dockerfile path

Fix dockerhub repo

Free up more disk space on runner

Include envs in cpu build

remove rm -rf /usr/local/share/boost

Test with pip only

Add python version file

Fix python version & extras

Checkout pathed envs

Add test artifacts

Update PR template

WIP Add nightly tests

Push gpu & cpu images to gpu/cpu dockerhub repos

WIP Test nightly

WIP test cpu build

WIP checkout pip install

Use Makefile for end-to-end tests

Fix cpu venv path

Fix end-to-end

Checkout envs

Revert "Checkout envs"

This reverts commit 7193f60.

Add working-directory

Fix builds (rebase hotfix #87 from main)

Test fixed builds

rebase from user/rcadene/2024_04_18_episode_data_index

Test images

Check nvidia-smi

Fix opengl on gpu image

Test gpu build

Try nvidia/cudagl

Test cudagl build

Try different registry
aliberts pushed a commit that referenced this pull request Apr 25, 2024
Co-authored-by: Alexander Soare <[email protected]>

Tests cleaning & simplification (#81)

Fix tolerance for delta_timestamps (#84)

Co-authored-by: Remi <[email protected]>

Hotfix test_examples.py (#87)

Quality of life patches for eval.py (#86)

Add meta_data, revision v1.1

WIP add load functions + episode_data_index

id -> index, finish moving compute_stats before hf_dataset push_to_hub

Use v1.1, hf_transform_to_torch, Add 3 xarm datasets

Remove Prod, Tests are passind

Add tests/data

small fix

fix visualize_dataset

fix online training

fix online training

fix online training

Fixes for datasets 2.18 -> 2.19 update (#88)

WIP Dockerfile

Add Makefile

Add cpu build

Add libhdf5-dev in build-image

Add gpu build

Add libegl1-mesa-dev in cpu build

WIP Add docker build ci

Fix docker context

Fix dockerfile path

Fix dockerhub repo

Free up more disk space on runner

Include envs in cpu build

remove rm -rf /usr/local/share/boost

Test with pip only

Add python version file

Fix python version & extras

Checkout pathed envs

Add test artifacts

Update PR template

WIP Add nightly tests

Push gpu & cpu images to gpu/cpu dockerhub repos

WIP Test nightly

WIP test cpu build

WIP checkout pip install

Use Makefile for end-to-end tests

Fix cpu venv path

Fix end-to-end

Checkout envs

Revert "Checkout envs"

This reverts commit 7193f60.

Add working-directory

Fix builds (rebase hotfix #87 from main)

Test fixed builds

rebase from user/rcadene/2024_04_18_episode_data_index

Test images

Check nvidia-smi

Fix opengl on gpu image

Test gpu build

Try nvidia/cudagl

Test cudagl build

Try different registry
menhguin pushed a commit to menhguin/lerobot that referenced this pull request Feb 9, 2025
Kalcy-U referenced this pull request in Kalcy-U/lerobot May 13, 2025
ZoreAnuj pushed a commit to luckyrobots/lerobot that referenced this pull request Jul 29, 2025
Ricci084 pushed a commit to JeffWang987/lerobot that referenced this pull request Sep 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants