Segmentation Fault on reading StringDType ZarrArray

[FabricioArendTorres]

Zarr version

3.0.8

Numcodecs version

0.15.1

Python Version

3.11.11

Operating System

Linux

Installation

uv pip with global python within Docker based on Image "pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime"

Description

When reading a StringDType zarr array I sometimes obtain segmentation fault.
This depends on the way I index the data (i.e. which index of the array), but I'm always able to print the data of the whole array.

The error is quite hard to catch, and unfortunately makes my data nearly unusuable for now.

Context for the error.

• I created a zarr array with a StringDType.
• For reading the zarr array, I first load it into numpy using the [:] slicing, and then access the data by indexing the resulting numpy array. Copying the numpy array somehow changes where the error
• printing the data of the full numpy array works
• However, sometimes, when slicing it, I obtain a segmentation fault on some (!) indices. And not always on the same - multi-index slicing always throws errors (e.g. array[[6]]), whereas array[6] is a bit inconsistent on whether it works or not.

Odd effects on attempted remedies

Copying the numpy array with new_array = np.copy(zarr_array[:]) at first seemed like a solution,
but in the end I had another segmentation fault pop up somewhere else.
The same goes for other wrappers like np.array(zarr_array) or zarr_array[:].astype(object) etc.
If you try this on my code below, the error will no longer appear, but I can assure you that it still pops up in my codebase at some point.

Reproducibility

I could not reproduce the segfault with synthetic data.
I assume something went wrong during writing of the data.

I have however a zarr array within a LocalStore that reliably reproduces the segfault.
I attached it (manually zipped the directory for upload, you need to manually unzip).

This zarr array has a single chunk. I altered all other indices (to anonymize the data), which did not affect
the offending faulty index.
The indices I overwrote that were faulty before, are not faulty anymore.

I observed the same/a similar segfault on different machines, within different docker environments (same zarr version). As tracking down segfaults is quite annoying in a large pipeline, I can not fully confirm it was the exact same issue / same file that caused it.

# /// script
# requires-python = "==3.11.11"
# dependencies = [
#   "zarr==3.0.8",
#   "numcodecs==0.15.1",
# ]
# ///

import faulthandler

faulthandler.enable()

import numpy as np

[event_description_anony.zip](https://github.com/user-attachments/files/20927786/event_description_anony.zip)

import zarr


def main():
    # Create a temporary directory
    zarr_array = zarr.open("./event_description_anony")

    # I anonymized all other entries. As long as I dont touch index 6, the segmentation fault is reproducible.
    # for i in range(zarr_array.shape[0]):
    #     if i < 6:
    #         zarr_array[i] = "DUMMYTEXT"
    #     if i > 6:
    #         zarr_array[i] = "DUMMYTEXT"

    # this works without issues, direct access to zarr array
    print(f"{zarr_array.dtype=}")
    print(f"{zarr_array[6]=}")
    print(f"{zarr_array[[6]]=}")
    print(f"{zarr_array[[1,2,4,6]]=}")

    # I've only observed this issue when first loading it into numpy

    # np_array = np.copy(zarr_array[:])
    np_array = zarr_array[:]

    # this still works
    for i in range(np_array.shape[0]):
        np_array[i]

    # this fails at [[6]]
    for i in range(np_array.shape[0]):
        print(f"{i}: {np_array[[i]]}")


if __name__ == "__main__":
    main()

Additional output

Stack trace of above code using the python faulthandler.

/root/.cache/uv/environments-v2/debug-zarr-0f1cac5393ffd124/lib/python3.11/site-packages/zarr/codecs/vlen_utf8.py:44: UserWarning: The codec `vlen-utf8` is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  return cls(**configuration_parsed)
zarr_array.dtype=StringDType()
zarr_array[6]=array(['[comment] Look left and right 5 times'], dtype=StringDType())
zarr_array[[6]]=array([['[comment] Look left and right 5 times']], dtype=StringDType())
zarr_array[[1,2,4,6]]=array([['DUMMYTEXT'],
       ['DUMMYTEXT'],
       ['DUMMYTEXT'],
       ['[comment] Look left and right 5 times']], dtype=StringDType())
np_array[1]=array(['DUMMYTEXT'], dtype=StringDType())
np_array[2]=array(['DUMMYTEXT'], dtype=StringDType())
np_array[3]=array(['DUMMYTEXT'], dtype=StringDType())
np_array[4]=array(['DUMMYTEXT'], dtype=StringDType())
np_array[5]=array(['DUMMYTEXT'], dtype=StringDType())
np_array[6]=array(['[comment] Look left and right 5 times'], dtype=StringDType())
Fatal Python error: Segmentation fault

Thread 0x00007f64411db640 (most recent call first):
  File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 81 in _worker
  File "/opt/conda/lib/python3.11/threading.py", line 982 in run
  File "/opt/conda/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/opt/conda/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f64419e0640 (most recent call first):
  File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 81 in _worker
  File "/opt/conda/lib/python3.11/threading.py", line 982 in run
  File "/opt/conda/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/opt/conda/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f64421e5640 (most recent call first):
  File "/opt/conda/lib/python3.11/selectors.py", line 468 in select
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 1898 in _run_once
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 608 in run_forever
  File "/opt/conda/lib/python3.11/threading.py", line 982 in run
  File "/opt/conda/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/opt/conda/lib/python3.11/threading.py", line 1002 in _bootstrap

Current thread 0x00007f647803f740 (most recent call first):
  File "/workspace/debug_zarr.py", line 47 in main
  File "/workspace/debug_zarr.py", line 51 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, yaml._yaml, numcodecs.compat_ext, numcodecs.blosc, numcodecs.zstd, numcodecs.lz4, numcodecs._shuffle, numcodecs.jenkins, crc32c._crc32c, numcodecs.vlen, numcodecs.fletcher32 (total: 12)

[d-v-b]

hi @FabricioArendTorres this sounds like a tricky bug. I'm pretty sure the place to look is in the codec we use for reading and writing StringDType arrays from storage. Since you only see this segfault when reading data, I would look at this function in numcodecs: https://github.com/zarr-developers/numcodecs/blob/4fdb62507376c3896dfb25a02ca36cb4cdd6e913/numcodecs/vlen.pyx#L132-L176

as far as I can tell, there's no part of zarr's codebase that does the kind of direct memory management that can lead to this kind of segfault, so I'm very confident that this is coming from the numcodecs code I linked (which is written in cython).

[FabricioArendTorres]

I'm sorry but I do not have enough insight into the underworkings of Zarr / Numcodecs to thoroughly pin the cause.

However, below is another stack trace that I managed to produce, which might maybe help?

The surrounding code is not easily reproducible, but this essentially happens if I try first copying the data to another numpy array. I trimmed away all the irrelevant parts from my code of the stack trace.

...
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/array.py", line 2441, in __getitem__
    return self.get_orthogonal_selection(pure_selection, fields=fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/array.py", line 2883, in get_orthogonal_selection
    return sync(
           ^^^^^
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/sync.py", line 163, in sync
    raise return_result
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/sync.py", line 119, in _runner
    return await coro
           ^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/array.py", line 1298, in _get_selection
    await self.codec_pipeline.read(
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/codec_pipeline.py", line 464, in read
    await concurrent_map(
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/common.py", line 69, in concurrent_map
    return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/common.py", line 67, in run
    return await func(*item)
           ^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/codec_pipeline.py", line 282, in read_batch
    tmp = chunk_array[chunk_selection]
          ~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/zarr/core/buffer/cpu.py", line 180, in __getitem__
    return self.__class__(np.asanyarray(self._data.__getitem__(key)))
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index 21990232697700 is out of bounds for axis 0 with size 315352
...
RuntimeError: DataLoader worker (pid 169842) is killed by signal: Segmentation fault. 

[d-v-b]

i don't think we can do very much without a reproducible example. I understand that this might be hard to create, so no worries if that's not practical, but a self-contained example is probably the difference between this getting fixed and not getting fixed.

[FabricioArendTorres]

At this point I'm not sure if this is an issue of zarr, numcodecs or numpy..

The only thing I can reproduce from a single code snippet is this weird indexing / slicing behaviour of numpys StringDType, as you can see above this comment.
Weird indexing behaviour and out of range access could in my limited understanding be related, no?

Could it be that this numpy lead to issues within zarr? Or is that a totally separate issue I mixed up?

Maybe we could leave this issue here open until the numpy one is resolved?

[FabricioArendTorres]

I think I might have stumbled upon two separate issues at once.

• The first one from my initial post might be related to the StringDType in numpy. There you have my uploaded file and code snippet with which it is reproducible on multiple devices in different environments. I hope the Numpy fix resolves this.

• The second stacktrace I posted is a separate issue. After rewriting my code to not use any StringDType, I still get SegmentationFaults on reading regular floating point zarr-arrays, but with a more comprehensive error message / stack trace. I could (sometimes) catch an out-of-range index in

  zarr-python/src/zarr/core/buffer/cpu.py

  Line 180 in bbdefac

  def __getitem__(self, key: Any) -> Self:

  .
  I did not manage to pdb into that line, as it occurs rather late (but consistently) in a multi-process application (Pytorch Dataloader workers). It does seemingly not occur in a single-threaded setting.

I will probably not be able to provide a reproducible self-contained example for this second case, and we decided internally to unfortunately change back to V2 in the foreseeable future..