Optionally persist to a dataset #29
Conversation
Uses `huggingface_hub.CommitScheduler` to automatically persist the SQLite database to a dataset, if specified by the user. If dataset does not exist it is automatically created.
trackio/deploy.py
Outdated
| @@ -55,3 +59,16 @@ def deploy_as_space(title: str): | |||
| folder_path=trackio_path, | |||
| ignore_patterns=["README.md"], | |||
| ) | |||
|
|
|||
| # add HF_TOKEN so we have access to dataset to persist data | |||
| HF_TOKEN = os.environ.get("HF_TOKEN") | |||
There was a problem hiding this comment.
Suggest using huggingface_hub.utils.get_token so that it can read the token if they are logged in via the cli. Should we prompt users for an HF token if there is no saved token?
There was a problem hiding this comment.
Yes, when I ran examples/persist-dataset.py, my Space was in an invalid state, presumably because it could not fetch my hugging face token correctly: https://huggingface.co/spaces/abidlabs/metrics-test
|
It looks like right now, persisting to a dataset only works if we are streaming the metrics to a Space. However, I think these two things should be indepenent: you can run the dashboard locally and/or on Spaces, or you can persist in a Dataset or not. Right? |
|
Added to the UI as well: 
|
|
Thanks for the improvements @abidlabs! Merged. |
Uses
huggingface_hub.CommitSchedulerto automatically persist theSQLite database to a dataset, if specified by the user. If dataset does
not exist it is automatically created.