Issue Opening Files via SFTP with fsspec in Zarr v3

[dgegen]

I'm currently using Zarr and fsspec to access a Zarr file stored on a server via SFTP. However, I've encountered an issue while transitioning from Zarr v2 to v3.

Description

import zarr
import fsspec

# Specify the SFTP filesystem
storage_options = {
    "key_filename": "/path/to/key",
    "username": "username",
    "host": "host",
}
sftp_fs = fsspec.filesystem('sftp', **storage_options)
zarr_path = "/path/to/file/on/server.zarr"

fs_map = fsspec.FSMap(zarr_path, sftp_fs)

print(f"File exists: {sftp_fs.exists(zarr_path)}")  # True

In Zarr v2.18.3 I am able to open this file using

# Zarr v2.18.3
zarr_file = zarr.open(fs_map)

Now, attempting to achieve the same thing using Zarr v3.0.9 (including an additional import because of #3195), I use:

# Zarr v3.0.9
import fsspec.implementations.asyn_wrapper  # missing import in zarr.storage.FsspecStore
from zarr.storage import FsspecStore

store = FsspecStore.from_mapper(fs_map, read_only=True)
zarr_file = zarr.open(store)

However, the call zarr_file = zarr.open(store), never finishes, i.e. Python freezes.

Is this a bug? Or will SFPT no longer be supported in Zarr v3? Or am I using FsspecStore incorrectly?

Thanks a lot in advance!

[d-v-b]

However, the call zarr_file = zarr.open(store), never finishes, i.e. Python freezes.

that's a surprising failure mode. As our 3.0.9 release has been a bit bumpy, can you try running the same thing in 3.0.8? You will need to skip the fsmap and just give the filesystem to the fsspecstore I think.

[dgegen]

For me, Python also freezes using the following in v3.0.8

# Zarr v3.0.8
from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper
from zarr.storage import FsspecStore

store = FsspecStore(AsyncFileSystemWrapper(sftp_fs, asynchronous=True), path=remote_file_path)
zarr.open(store)

[d-v-b]

ok,that's "good" in the sense that it's not related to the 3.0.9 release, but it's bad insofar as it's a bug. Can you confirm that this isn't an fsspec issue?

[dgegen]

I'm not sure I understand what you mean by that. However, using the same environment, merely downgrading Numcodecs and Zarr using

pip install zarr==2.18.3
pip install numcodecs==0.13.1

and of course using the snippet for Zarr v2 resolves the issue. So in principal the file can be opened with fsspec v2025.5.1.

[d-v-b]

I'm not sure I understand what you mean by that.

Fsspecstore uses fsspec for all of its IO. So if fsspec's sftp implementation has a problem, then zarr's usage of fsspec's sftp implementation will reveal that problem, even if it comes from fsspec. It's worth noting that there is no sftp-specific logic in zarr's use of fsspec -- we are just using the fsspec filesystem object, no matter what storage backend it uses. So that's why I am wondering if the basic problem here comes from fsspec.

[dgegen]

However, considering that I can successfully open the file with the same version of fsspec using zarr v2.18.3, doesn't this suggest that the issue is related to how zarr v3.0.8/v3.0.9 is utilizing fsspec?

[d-v-b]

This would be much easier with a reproducible example that one of the devs could run. Could you create one with the dependencies in the header (like in our issue template)? You could try using the address of a public sftp server