Clean-up indexing adapter classes

[benbovy]

• User visible changes (including notable bug fixes) are documented in whats-new.rst

Mostly cherry-picked from #10296 (cleaner in a separate PR).

This PR also prevents converting a whole pd.RangeIndex or a CoordinateTransform object into a numpy array when showing the array repr. The object is sliced / indexed into a smaller one before doing the conversion. For large indexes this can make a big difference:

>>> import pandas as pd
>>> import xarray as xr
>>> from xarray.indexes import PandasIndex

>>> xidx = PandasIndex(pd.RangeIndex(2_000_000_000), "x")
>>> ds = xr.Dataset(coords=xr.Coordinates.from_xindex(xidx))
>>> ds   # before this PR, was loading 16GB into memory!

However, I'm now wondering how we should format the array (inline) repr for those two cases. It is useful to show some concrete values but at the same time it is misleading to show them like plain numpy arrays.

>>> print(ds.x.variable)
>>> <xarray.IndexVariable 'x' (x: 2000000000)> Size: 16GB
array([[         0,          1,          2, ..., 1999999997, 1999999998,
        1999999999]])

>>> ds.x.variable._in_memory
True       # (not really, actually!)

It would certainly help to have some visual mark for lazy variables (perhaps next to their size), but that's another topic.

[dcherian]

Can't we get away with adding_repr_inline to CoordinateTransfromIndexingAdapter? I think it's preferable to avoid this kind of special casing.

[benbovy]

This function does not format the inline repr but the data of a DataArray or variable.

We could add a short_data_repr method to those adapter classes and check if internal_data has such a method here, although it is not much different from this special casing.

[dcherian]

I guess a better request is to see whether we can just reuse format_array_flat which already does indexing and should just work for these classes.

[benbovy]

Hmm again format_array_flat is for formatting the inline repr, whereas the special case introduced here is for formatting the data repr.

Without this special case, short_data_repr() will convert the indexing adapters into numpy arrays via PandasIndexingAdpater.get_duck_array() and CoordinateTransformIndexingAdapter.get_duck_array() over their full shape / size. For both the inline and the data reprs, we want to select only the first and last relevant items before doing this conversion.

first_n_items() and last_n_items() in xarray.core.formatting do similar things than PandasIndexingAdapter._get_array_subset() and CoordinateTransformIndexingAdapter._get_array_subset(). We could perhaps reuse the two former instead, although for the data repr (at least for CoordinateTransform, and possibly later for pd.IntervalIndex) we don't want a flattened result. So this would involve more refactoring. Also this wouldn't remove the special case here.

Alternatively we could tweak Variable._in_memory such that it returns False when ._data is a PandasIndexingAdapter (only when it wraps a pd.RangeIndex) or a CoordinateTransformIndexingAdapter, which will turn their data repr from, e.g.,

>>> xidx = PandasIndex(pd.RangeIndex(2_000_000_000), "x")
>>> ds = xr.Dataset(coords=xr.Coordinates.from_xindex(xidx))
>>> print(ds.x.variable)
<xarray.IndexVariable 'x' (x: 2000000000)> Size: 16GB
array([[         0,          1,          2, ..., 1999999997, 1999999998,
        1999999999]])

into

>>> print(ds.x.variable)
<xarray.IndexVariable 'x' (x: 2000000000)> Size: 16GB
[2000000000 values with dtype=dtype('int64')]

On one hand I like seeing the actual first and last values of a (lazy) range index or coordinate transform. On the other hand I find a bit confusing that is it shown like a plain numpy array.

[benbovy]

Alternatively we could tweak Variable._in_memory such that it returns False when ._data is a PandasIndexingAdapter (only when it wraps a pd.RangeIndex) or a CoordinateTransformIndexingAdapter

This is what I ended up refactoring in 0e5154c, for PandasMultiIndex coordinates too.

I think it makes sense to show those coordinate variables as lazy variables rather than showing them as numpy arrays. Those aren't numpy arrays, coercing them like so may trigger a lot of computation and memory allocation.

A possible new source of confusion, however, is that they now reuse the same repr than for the other lazy variables loaded from disk. One difference is that the index coordinates will still be lazy even after "loading" a dataset, i.e.,

>>> print(ds.load().x.variable)
<xarray.IndexVariable 'x' (x: 2000000000)> Size: 16GB
[2000000000 values with dtype=dtype('int64')]

I'm not sure what would be best.

[keewis]

you could create another lazy array class that inherits from the current one but overrides __repr__ to show that they're different (and if that later turns out to be wrong we don't have to go through a deprecation cycle, as all that machinery is internal anyways)

[dcherian]

I agree that for index variables, it is useful to show (a) the first "n" and last "n" is useful and (b) that they are lazy.

That said, I think this PR is an OK "engineering" fix. We should rethink these reprs a bit.

Shall we add a benchmark to test peak memory so that we don't regress here?