exp run: Use Cases

[casperdcl]

There are 2 main use cases I see for exp run, and it would be great to

a) confirm that they can be implemented, and
b) if needed, tidy up/implement a nice DVC API for them.

The use cases are:

1. Plot loss curves & model evolution (i.e. metrics & data evolving over epochs)
2. Hyperparameter search (tabulate metrics & corresponding input parameters)

The current concern is that a DVC project set up to do (1) cannot also do (2) without running out of disk space... i.e. when doing a hyperparameter search, DVC should not save e.g. model.h5 and predictions.zip per-epoch, but should continue to save parameters & metrics per-epoch.

The new checkpoint: true flag in dvc.yaml is apparently about not clearing outputs before execution. Perhaps it should be used instead to signify whether or not to put things into the run-cache.

[casperdcl]

CC @shcheklein

[dberenbaum]

Great point @casperdcl!

You have the scenarios right, but exp run is also intended to support reverting to previous checkpoints (like if your loss curve starts going the wrong direction), so I don't think we can assume that checkpoints need not save model weights or predictions or other generic outputs.

There is an open issue in #4428 to allow for users to disable the run-cache for outputs that are too big, so it seems like that might be the place to address this.

[casperdcl]

Yes #4428 looks about right. With this alpha feature I care about having theoretical configurability/options/implementations (a) first rather than the next step of finessing API defaults/assumptions (b).

May still have to discuss the naming though. checkpoint: true/cache: true/run-cache: true. DVC has a history of finding fault with established Git terminology so should be extra careful when introducing its own new terms.