Downloading batch files should check to see if any data is already downloaded

[gvnwst]

Problem Description

After running a web.Batch() of simulations, it's time to download the results.
This is typically not a problem, but particularly for large batches I've run into:

1. My computer decides to restart, killing the download task (it's windows...)
2. There's some kind of connection issue and the download fails

I can easily use batch.load() or Batch.download() to get new files, but at the moment this re-downloads every file in the batch.

Desired Solution

I would love if there were an option to only download the "outstanding" files -- I'm already providing the download directory, so just check the set difference between the batch filenames and the filenames in the target directory and download the difference.

An even more advanced version of this would check that the downloaded files are complete, and finish downloading those which got cut off.

[tylerflex]

So at first glance, just inserting the following into Batch.download() should do the trick?

if os.path.exists(job_path):
    continue

into https://github.com/flexcompute/tidy3d/blob/develop/tidy3d/web/api/container.py#L926

Just need to think about whether there are any possible unintended consequences of this. Or more specifically whether there are times where we might actually want to overwrite the files. I can't think of any right now. If there are some, we could consider adding something like overwrite: bool = False as an optional field to Batch

@yaugenst-flex ?

[gvnwst]

I would definitely think this should be an optional flag, the default to be download all and overwrite (current behavior).

[yaugenst-flex]

Hey @gvnwst, agree this would be a great addition. As @tylerflex said, it should be pretty straightforward to add. I'm slightly worried about how this would be handled in web.run_async, but restricting it to direct interaction with Batch for now is definitely feasible.

[jewettaijfc]

So at first glance, just inserting the following into Batch.download() should do the trick?

if os.path.exists(job_path):
continue

Hi @tylerflex ! I worry about skipping over files that were partially downloaded. Here's a question for you or Yannick:

Some programs (eg. google chrome) will randomize the names of the files that are being downloaded. After the download is completed, google chrome will rename the file to the correct name. This makes the download atomic. (This guarantees that if a file with the correct name exists at the destination, then the download is complete.)

Does Tidy3D do this?

[tylerflex]

it does not. you'll have to do it manually

[jewettaijfc]

Thanks for the fast reply!
Making file downloads atomic is necessary to solve this problem safely. (And this gives users confidence that the file was successfully downloaded.)

May I attempt that?
Do you have any objections to using google-chrome's approach (renaming files during download)?

[tylerflex]

I'd say let's just use the os.path.exists() solution as written, get this feature out and then revisit this more generally in a followup. It would have to touch many areas of the codebase to do it properly. sound good? You can mention the potential risk in the description of the overwrite field

[jewettaijfc]

It could be a little risky for the user, but I can implement this. (In the interim period, I might add a warning message, telling the user to delete incompletely downloaded files beforehand. Risk is okay as long as the user is informed.)

[tylerflex]

since overwrite would be an optional field, we can just put a warning in the description of the field. I dont think we need to log a warning every time (in case that's what you meant?).

[gvnwst]

It would be a little more computationally intensive, but the data loading methods which exist for the various SimTypeData objects can tell whether a file is complete or not -- without needing to do renaming or anything, it should be possible to simply loop over the existing files and check if they are complete (otherwise re-download).

This would be a larger change than Tyler's original solution though, and less elegant than the proposed change from Andrew.

[tylerflex]

Turns out there's actually a replace_existing argument for web.load() that isn't in the Job actually. (note doesn't get passed to web.download().

I think a somewhat simple solution here could be

1. Add replace_existsing : bool = True to the Job class.
2. Make sure it's properly passed inside of job.load() https://github.com/flexcompute/tidy3d/blob/develop/tidy3d/web/api/container.py#L331
3. Add this field to Batch
4. Make sure Batch passes it to Job https://github.com/flexcompute/tidy3d/blob/develop/tidy3d/web/api/container.py#L612-L646
5. Not sure why Batch.run() does a download() before a load() but potentially we could remove that and let load() handle it.

this might solve the problem while still keeping everything consistent internally before we implement something more complex. just not sure about step 5

[jewettaijfc]

First of all, thanks for posting this issue to github. It's a great issue. It brings up some deeper questions:

1. I'm new to Tidy3D. I have never run a real Tidy3D simulation before, so I'm curious to know roughly how large a simulation file could be. What's a plausible upper bound for these kinds of files? 10GB?, 100GB? 1TB?, 10TB? Alternatively, how big are your files?

(I ask because files that are less than 10GB in size are comparatively easy to transfer. But files that are much larger than that means we should eventually really think about things like file-level interruption recovery and proper file-level chunking. Not something we have to do now. Just something to keep in mind in the long run. -For this issue, I'll assume file sizes are <=10GB, and I'll focus on making sure all the files in a batch are successfully downloaded.)

It would be a little more computationally intensive, but the data loading methods which exist for the various SimTypeData objects can tell whether a file is complete or not -- without needing to do renaming or anything

2. Cool! But I am unsure which loading methods you mean. If you know the name of one of these functions, feel free to post it here.

I confess I am not yet familiar with this part of the code. I also have never run a batch simulation, or used .load() or .download(). However in the next few days I am going through dozens of examples from the tidy3d-notebooks. Maybe I'll see it demonstrated there. Either way, I will probably wait a couple days before I tackle this task.

3. Meanwhile, feel free to post a few lines of code from your example that shows how you are running a simulations and downloading your data (if you feel comfortable doing that).

[tylerflex]

Hi @gvnwst feel free to share any info with me via email about the simulation size or other details if they are relevant to the problem at hand.