Peformance regression: 3.0.9 issues twice as many GET requests when used with xarray

[TomNicholas]

Reporting something @dcherian noticed. Xarray has some neat tests which record the total number of http requests made by Zarr.

We are seeing that as of zarr 3.0.9 the number of GET requests required for our little tests has doubled. This likely translates to considerably slower interaction with high-latency stores (e.g. remote object stores).

Should be reproducible just by running that xarray test with the latest zarr, but @dcherian might have a standalone reproducer.

[rabernat]

Do you know what is triggering this and which calls are being duplicated?

[TomNicholas]

No-one has looked into this yet. I'm raising it here as @dcherian and I discussed temporarily downgrading xarray's expectations until it's hopefully fixed in zarr, and I needed a tracking issue to refer to.

[rabernat]

This is a major performance regression in Zarr which needs to be fixed asap.

Copying the details here

FAILED xarray/tests/test_backends.py::TestInstrumentedZarrStore::test_append - AssertionError: ('get', {'get': 8, 'list_dir': 2, 'list_prefix': 1, 'set': 16})
assert 8 <= 4
FAILED xarray/tests/test_backends.py::TestInstrumentedZarrStore::test_region_write - AssertionError: ('get', {'get': 8, 'list_dir': 2, 'list_prefix': 4, 'set': 16})
assert 8 <= 2

Relevant test code

https://github.com/pydata/xarray/blob/main/xarray/tests/test_backends.py#L3534-L3713

[TomNicholas]

This reproduces on main in the CI here. I'm still going to (temporarily) adjust xarray's expectations to make things green in our CI because I don't think this change of baseline is an xarray issue (we didn't change our code), it's a zarr issue.

[maxrjones]

Testing against expected get metrics is a great idea and would be great to implement in Zarr-Python as well.

I've been trying to produce a MVCE for this regression through the unsophisticated approach of printing the arguments every-time MemoryStore.get() is called but see that the following simplified code from xarray's tests makes the same 6 calls to MemoryStore.get() for both 3.0.8 and 3.0.9.

# /// script
# dependencies = [
#   "xarray @ git+https://github.com/pydata/xarray@main",
#   "zarr @ git+https://github.com/maxrjones/zarr-python@3.0.8-with-print",
#   "numpy==2.3.1",
#   "pooch",
#   "netcdf4",
# ]
# ///

import zarr
from zarr.storage import MemoryStore
import xarray as xr

print(zarr.__version__)
original = xr.Dataset({"foo": ("x", [1])}, coords={"x": [0]})

with MemoryStore() as store:
    original.to_zarr(store)

get(key=zarr.json, byte_range=None)
get(key=foo/zarr.json, byte_range=None)
get(key=x/zarr.json, byte_range=None)
get(key=zarr.json, byte_range=None)
get(key=x/zarr.json, byte_range=None)
get(key=foo/zarr.json, byte_range=None)

# /// script
# dependencies = [
#   "xarray",
#   "zarr @ git+https://github.com/maxrjones/zarr-python@3.0.9-with-print",
#   "numpy==2.3.1",
#   "pooch",
#   "netcdf4",
# ]
# ///

import zarr
from zarr.storage import MemoryStore
import xarray as xr

print(zarr.__version__)
original = xr.Dataset({"foo": ("x", [1])}, coords={"x": [0]})

with MemoryStore() as store:
    original.to_zarr(store)

get(key=zarr.json, byte_range=None)
get(key=foo/zarr.json, byte_range=None)
get(key=x/zarr.json, byte_range=None)
get(key=zarr.json, byte_range=None)
get(key=x/zarr.json, byte_range=None)
get(key=foo/zarr.json, byte_range=None)

separately, since MemoryStore implements set() independently from stores that operate over the network (i.e., FsspecStore or ObjectStore), I'm not 100% convinced by the statement that "This likely translates to considerably slower interaction with high-latency stores (e.g. remote object stores)." without independent testing using minio. I will look into that.

[maxrjones]

Even using the smarter way of leveraging LoggingStore I am seeing the same number of calls between the two versions:

• 3.0.8.log from https://github.com/maxrjones/test-scripts/blob/main/test-zarr/print-get-3.0.8.py
• 3.0.9.log from https://github.com/maxrjones/test-scripts/blob/main/test-zarr/print-get-3.0.9.py

[TomNicholas]

How weird. Something must have changed, even if it is in xarray...

[rabernat]

The regressions are only on Xarray append and region_write workflows.

[TomNicholas]

The regressions are only on Xarray append and region_write workflows.

Aren't those just the only ones tested in this way? (See test definition)

[rabernat]

Yes but @maxrjones's reproducer above did not test those workflows, that's why I'm pointing it out.

[maxrjones]

The first failure (see changes in pydata/xarray#10491) occurred before any appending operations, which is why my MVCE reduced to the first ds.to_zarr(store) operation.

[maxrjones]

I am becoming increasingly convinced that this relates to a change with xarray rather than the zarr release. Please note that this CI run for xarray's main branch used Zarr 3.0.9 and had all passing tests - https://github.com/pydata/xarray/actions/runs/16016164611/job/45183114206.

[maxrjones]

xarray's failures started with this commit - https://github.com/pydata/xarray/pull/10469/files#diff-e1c5f34f91e5b8a5242d9499ca9ff2765f3d5b7956c28196f80e4c399b29b1a6.