Unable to open group using experimental implementation of Zarr V3 Specification in zarr-python 2.x

[candleindark]

Zarr version

v2.18.7

Numcodecs version

v0.15.1

Python Version

3.11

Operating System

Mac

Installation

Using pip in a virtual environment

Description

It is not possible to open a group with the zarr.open() in zarr-python 2.x with the experimental implementation of Zarr V3 Specification enabled.

Steps to reproduce

1. Given the following file

demo.py

import zarr

path = "path-where-you-want-zarr-v3-array"
z1 = zarr.create(
    (10000, 10000),
    chunks=(100, 100),
    dtype="f8",
    path=path,
    zarr_version=3,
    store='data/example-10.zarr'
)

# Opening as an array is OK
# z2 = zarr.open('data/example-10.zarr', mode="r", zarr_version=3, path=path)

# Opening as a group is not OK
z2 = zarr.open('data/example-10.zarr', mode="r", zarr_version=3)

2. Run

python demo.py

to produce the following error.

 python demo.py
/Users/isaac/.pyenv/versions/dandi-cli/lib/python3.11/site-packages/zarr/_storage/store.py:39: FutureWarning: The experimental Zarr V3 implementation in this version of Zarr-Python is not in alignment with the final V3 specification. This version will be removed in Zarr-Python 3 in favor of a spec compliant version.
  warnings.warn(
Traceback (most recent call last):
  File "/Users/isaac/Library/Application Support/JetBrains/PyCharm2024.3/scratches/demo.py", line 17, in <module>
    z2 = zarr.open('data/example-10.zarr', mode="r", zarr_version=3)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/.pyenv/versions/dandi-cli/lib/python3.11/site-packages/zarr/convenience.py", line 139, in open
    raise PathNotFoundError(path)
zarr.errors.PathNotFoundError: nothing found at path ''

Additional output

No response

[TomAugspurger]

I suspect you'll have more luck using zarr-python 3.x. I'm not sure whether anyone is actively working on the zarr v3 support in 2.x.

[d-v-b]

It's not possible to debug this without knowing what's inside 'data/example-10.zarr'. In any case, I recommend against using the experimental v3 implementation supported by zarr-python 2.x. That implementation is not compliant with the final version of the zarr v3 spec.

Instead I recommend using Zarr-python 3, as it supports both zarr v2 and v3. Is there something in particular blocking you from using zarr-python 3?

[candleindark]

It's not possible to debug this without knowing what's inside 'data/example-10.zarr'.

'data/example-10.zarr' doesn't exist before the execution of the demo.py script. It is created with by the call to `zarr.create()

Instead I recommend using Zarr-python 3, as it supports both zarr v2 and v3. Is there something in particular blocking you from using zarr-python 3?
Yes, a least two. We currently blocked our dependency of hdmf-zarr, hdmf-dev/hdmf-zarr#202, and potentially that zarr-python 3 doesn't support Python 3.9-3.10, as

➜  zarr python3.10 -m venv venv && source venv/bin/activate && pip install "zarr~=3.0"
ERROR: Ignored the following versions that require a different python version: 2.18.4 Requires-Python >=3.11; 2.18.5 Requires-Python >=3.11; 2.18.6 Requires-Python >=3.11; 2.18.7 Requires-Python >=3.11; 3.0.0 Requires-Python >=3.11; 3.0.0a6 Requires-Python >=3.11; 3.0.0a7 Requires-Python >=3.11; 3.0.0b0 Requires-Python >=3.11; 3.0.0b1 Requires-Python >=3.11; 3.0.0b2 Requires-Python >=3.11; 3.0.0b3 Requires-Python >=3.11; 3.0.0rc1 Requires-Python >=3.11; 3.0.0rc2 Requires-Python >=3.11; 3.0.1 Requires-Python >=3.11; 3.0.2 Requires-Python >=3.11; 3.0.3 Requires-Python >=3.11; 3.0.4 Requires-Python >=3.11; 3.0.5 Requires-Python >=3.11; 3.0.6 Requires-Python >=3.11
ERROR: Could not find a version that satisfies the requirement zarr~=3.0 (from versions: 0.2.2, 0.2.3, 0.2.5, 0.2.6, 0.2.7, 0.3.0, 0.4.0, 1.0.0b3, 1.0.0b4, 1.0.0b6, 1.0.0, 1.1.0, 2.0.0a2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.5.0, 2.6.1, 2.7.0, 2.7.1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.10.0, 2.10.1, 2.10.2, 2.10.3, 2.11.0a1, 2.11.0a2, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 2.12.0a1, 2.12.0a2, 2.12.0, 2.13.0a1, 2.13.0a2, 2.13.0, 2.13.1, 2.13.2, 2.13.3, 2.13.6, 2.14.0, 2.14.1, 2.14.2, 2.15.0a1, 2.15.0a2, 2.15.0, 2.16.0, 2.16.1, 2.17.0, 2.17.1, 2.17.2, 2.18.0, 2.18.1, 2.18.2, 2.18.3, 3.0.0a0, 3.0.0a1, 3.0.0a2, 3.0.0a3, 3.0.0a4, 3.0.0a5)
ERROR: No matching distribution found for zarr~=3.0

[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip

@yarikoptic What is the chance for us to drop support of Python 3.9 and 3.10 in https://github.com/dandi/dandi-cli for the support of zarr-python 3?

[d-v-b]

'data/example-10.zarr' doesn't exist before the execution of the demo.py script. It is created with by the call to zarr.create()

aha, I didn't see the included script, thanks for that context. I can't say exactly why that script is failing, but even if it succeeded, as stated elsewhere the zarr v3 implemented in zarr-python 2.18 is essentially a dead end that should not be used.

That script raised a FutureWarning:
The experimental Zarr V3 implementation in this version of Zarr-Python is not in alignment with the final V3 specification. This version will be removed in Zarr-Python 3 in favor of a spec compliant version.

Should we change this message to make the situation more clear?

[candleindark]

I think the FutureWarning is explicit enough.

I encountered the the problem in this issue when I was working on matters related to #2981 and considered it a bug since, with the experimental feature, zarr-python at 2.x is not able to open an object itself created. However, as you mentioned "the zarr v3 implemented in zarr-python 2.18 is essentially a dead end", so it may not be worth fixing the problem. I guess you can close this issue. Nevertheless, thanks for clarification about the experimental zarr v3 format implementation in zarr-python 2.x.

[d-v-b]

It's an unfortunate situation of our own creation -- zarr-python 2 supports zarr version 2 and something called version 3 (but it isn't the real version 3), while zarr-python 3 supports zarr version 2 and zarr version 3.

I think the best thing we can do is help people start using zarr-python 3, but if you have any other ideas for making this easier to understand feel free to let us know. Thanks for your patience here, and sorry for all the confusion we have caused.

[candleindark]

I think you already nailed the point above. You may want to convey the same point in the documentation of zarr-python 2.x.

For us, it was not sufficient to just know that the supported experimental implementation of Zarr V3 format in zarr-python 2.x is deprecated in zarr-python 3.x. It would be much more clear to state that the experimental Zarr V3 format is really a version of its own (quite distant from the final Zarr V3 format) in that zarr-python 2.x is not only unable to write a final/real Zarr V3 format object, it is not able to read any objects of that format either. (We were hoping to use zarr-python 2.x to read final/real Zarr V3 format object while waiting for the unblocking of a dependency).