Support for listing all file assets associated with Zarr-encoded data

[paulmillar]

I've looked at the Zarr API and didn't see a way to list all the file assets (files, data objects, etc), associated with some Zarr-encoded data. The goal of such an API call would be to produce a concise and complete list of strings, where each string is a representation of the location (e.g., a URL or filesystem path) of some file-like object on some storage system. "Concise" in the sense that all listed files are needed (at least theoretically) by at least one Zarr program (the list should not include any extraneous files). "Complete" in the sense that the Zarr library will not access any data files other than those listed.

There are two use-cases that (I think) would benefit from such an API call.

1. copy all the data from one storage system to another.

In order to improve availability and reduce latency (assuming the data is read multiple times), a user might wish to make an additional replica of some Zarr-encoded data. The copy could be to some geographically distinct location (e.g., from Europe to USA), but could also be across multiple storage technologies (e.g., making some dataset temporarily available on space-constrained nVME storage).

2. validating that storage has all necessary files.

In some scenarios, it may be helpful to catch corruption problems ahead of data use. One simple test is to check whether all the expected files exist on the storage and that the client is authorised to open these files for reading.

From searching various forums, it seems (from related discussion) that use-cases like this are supported by the client simply listing the contents of the underlying storage (listing content of some AWS bucket, directory listing of some POSIX storage, etc.). One potential problem is a lack of guarantee that all the files (in the storage) belong to the Zarr-encoded data or that all the files (in the Zarr data) are present on the storage. For example, when copying data, if there are any extraneous/irrelevant files then they too would be copied; for validating data, its unclear what files should be present.

Is there already a way of producing a list of file assets of some Zarr data?

If not, would adding support for this make sense?

[FabricioArendTorres]

There is a Tree functionality in Zarr, which allows you to list all members of a group by iterating through zarr.Group.members , maybe that might be useful?

https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/core/_tree.py

I make use of an adjusted version of that for a tree representation, but I guess the relevant functionality would be zarr.Group.members for you?

Otherwise would any variation of rsync-like tools (e.g. rclone) not be a better fit for you, if it just about copying and synchronizing different storages?

[d-v-b]

@paulmillar we don't expose a single function or method that does this, but we do define the necessary routines. The most IO intensive part is taking a census of all the chunks associated with an array. the chunks_initialized function takes an AsyncArray and finds the keys for the chunks that have been persisted to storage. If you add the key for the metadata document(s) associated with the array to the list of the persisted chunk keys, you have defined the complete "support" for the array in terms of stored objects.

For a group we would probably use a recursive definition: the file assets for a group are its metadata document(s), and file assets of its members. As @FabricioArendTorres notes we expose a method for recursively iterative over the members of a group. So combining these elements should give you the result you want. Sorry there isn't a stand-alone function that does this yet.

[paulmillar]

Thanks for the feedback and helpful hints.

Just as further explanation, the use-cases comes from data management within the interTwin project. This project is looking into support ML training and exploitation, to build a Digital Twin Engine for various scientific use-cases. Within this context, part of the project's work involves investigating how to use existing high-volume scientific data management tools such as FTS and Rucio.

A rather limited subset of the training data uses Zarr, so we were exploring how to manage data locality for such Zarr data, using these data management tools. In essence, these tools work with lists of files. Therefore, one approach would be to query a source Zarr dataset and register the corresponding list of files. This would allow Rucio+FTS to manage that data's locality across multiple HPC facilities.

As it wasn't clear (at least, not to me!) how to use Zarr-python to obtaining a list of files/data objects from Zarr metadata, I wanted to open this issue to check I wasn't missing something, and to help identify this as additional functionality that might prove useful for others.

Unfortunately, there's very limited time remaining in the project, so I'm not sure to what extent we can build on the information provided, here.

[d-v-b]

cool, thanks for the background @paulmillar. In terms of effort I'd say this functionality should only take a few hours max to implement, but that assumes familiarity with the zarr python APIs / data model. Let me know if you want more pointers / help. In any case, I do think the functionality you are asking for should be part of our public API (e.g., as a method on arrays / groups). It just isn't yet.