ui: responsiveness

[skshetry]

DVC - with sizes and numbers of data that we aim to handle, DVC might never be fast. We could make it responsive with the progress bars and status prints to improve perceived performance and that it does not look broken/stuck.

We do this in a few places (push/pull/add), but not on others (list, metrics, etc.).
We have multiple progress bars in push, but not in add.

A few questions:

1. Is responsiveness important to us?
2. Responsiveness means communicating more, is that a good thing?
3. Can we achieve responsiveness with the current architecture?
4. What does being responsive mean in terms of parallel processing?

So, as I pointed out above, for example in push, we have heard complaints about jumping progress bars and too many progress bars in the CLI (we have one progress bar for each process). But, just showing one progress bar also does not seem to be that responsive, and in lots of cases where you have large files or large numbers of files, it is necessary to have multiple ones.

But again, when someone is dvc add-ing a million files, we only have one single progress bar at any time.

Regarding status, DVC provides very little hints of what it is doing or what it intends to do. We could use some status/spinners to share some information about what DVC is doing. But it raises a question about how responsive should we be? As an example, should we share that we are collecting the stages from the repo?

Please take a look at 2-4 points on Robustness guideline, and let me know what you think.

[dberenbaum]

I don't see any reason to deviate from the default guidelines. Applying them to individual scenarios may always raise questions about how to interpret them, but I think it's probably sufficient to adopt the guideline and raise questions (along with citing the guideline) as needed.

Regarding the concern of too much communication, I assume using --quiet will allow users to reduce responsiveness as needed, right? Looking at some other tools built to handle big data, they tend to be very verbose.

Dask:

Spark:

They tend to print out way too much info to be able to read in realtime. DVC doesn't necessarily need to emulate them, but I think there's good reason for being responsive especially for long, complex processes consuming lots of data, because it's important to have a sense of what's happening and to see at which point things went wrong.