Skip to content

Downloading batch files should check to see if any data is already downloaded #2600

Open
@gvnwst

Description

@gvnwst

Problem Description

After running a web.Batch() of simulations, it's time to download the results.
This is typically not a problem, but particularly for large batches I've run into:

  1. My computer decides to restart, killing the download task (it's windows...)
  2. There's some kind of connection issue and the download fails

I can easily use batch.load() or Batch.download() to get new files, but at the moment this re-downloads every file in the batch.

Desired Solution

I would love if there were an option to only download the "outstanding" files -- I'm already providing the download directory, so just check the set difference between the batch filenames and the filenames in the target directory and download the difference.

An even more advanced version of this would check that the downloaded files are complete, and finish downloading those which got cut off.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions