keys() returning an empty list on a FsspecStore

[bevilacqc]

Zarr version

3.0.8

Numcodecs version

0.16.1

Python Version

3.11.7

Operating System

Mac

Installation

using pip into a virtual environment

Description

When calling the keys() function on the root group of a zarr file hosted in a S3 bucket I get an empty list, while accessing directly the key (of a group which I know exists in the file) works.

I am not sure whether it is a bug or there is something wrong in the way I stored the zarr file in the S3 bucket or opened it through the zarr library.

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr >= 3.0",
#   "zarr[remote]>=3"
#   "aiohttp"
# ]
# ///
#

import zarr
filename = r'https://s3.embl.de/bls-app-test/test.bls.zarr'

store = zarr.storage.FsspecStore.from_url(url=filename, read_only=True)
print(f"listing supported: {store.supports_listing}")

root = zarr.open_group(store=store, mode='r')

root["Brillouin_data"] # the key 'Brillouin_data' can e found
print(f"keys: {list(root.keys())}") # but when listing the keys in the root an empty list is returned

Additional output

listing supported: True
keys: []

[d-v-b]

It's possible that your bucket is not configured to allow listing. can you list the objects in the bucket using the s3 api (or fsspec)?

[bevilacqc]

@d-v-b thanks for your input. I checked and listing was already enabled.

I managed to make it work by creating a FsspecStore directly from a fsspec filesystem explicitly set to S3:

import fsspec                    

fs = fsspec.filesystem('s3', anon=True, asynchronous=True,
                    client_kwargs={'endpoint_url': f"https://s3.embl.de"})

store = zarr.storage.FsspecStore(fs, path = 'bls-app-test/test.bls.zarr',
                                read_only=True)


print(f"listing supported: {store.supports_listing}")

root = zarr.open_group(store=store, mode='r')

print(f"keys: {list(root.keys())}") # Now the keys are listed correctly

[d-v-b]

ok, I think I see what the problem was: https://s3.embl.de/bls-app-test/test.bls.zarr is not an s3 url (i.e, it doesn't start with s3://, so zarr-python can't infer that it should use the s3 backend, which results in the httpfilesystem backend to be used instead. I'm not sure there's anything zarr python can do here -- I think if you have an s3-compatible http service, you will need to explicitly request an s3 filesystem backend, as you did here.

[bevilacqc]

Maybe one could add a sentence in the documentation of the Remote store along the line of

The type of filesystem (e.g. S3, https, etc..) is inferred from the scheme of the url (e.g. s3://example-site.org). In case a specific filesystem is need one can explicitly create it:

fs = fsspec.filesystem('s3', anon=True, asynchronous=True,
                    client_kwargs={'endpoint_url': "https://s3.example-site.org"})
store = zarr.storage.FsspecStore(fs)

Something similar could also be added to the documentation of FsspecStore.from_url

[d-v-b]

@bevilacqc that's a great idea for a docs addition. would you like to open a PR with those changes? if so, let us know if you run into any issues getting the development environment set up