zarr.array from from an existing zarr.Array

[brokkoli71]

• fixes [v3] zarr.array from from an existing zarr.Array #2410

added concurrent streaming of source array into new array

Restriction

• Only allow concurrent streaming if the chunk shape of the existing and new array match. Otherwise, while streaming the existing array and writing to the new one, we could be writing to the same file in parallel, resulting in a race condition. (Is there some measure to prevent this that I am not aware of?)

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[brokkoli71]

Do we also want concurrency for different chunk sizes?

[normanrz]

Do we also want concurrency for different chunk sizes?

That would be nice, if the chunk sizes are somewhat compatible, i.e. one is a multiple of the other.

[d-v-b]

is the intention for this to work with numpy arrays? because they don't have a chunks attribute. by contrast, dask arrays do have a chunks attribute, but it's a tuple of tuples of ints (because dask chunks can be irregularly sized). So maybe a bit more parsing is needed here.

[brokkoli71]

I was only thinking of zarr arrays as data. But I can generalize it for numpy and dask

[normanrz]

But I can generalize it for numpy and dask

I think that would be great! In the tests, we could use something like zarr.from_array(store=store, data=np.arange(10)) in many places!

[d-v-b]

a lot of my inspiration comes from trying to make tests easier to write :)

it might also make sense to put in a keyword argument that controls whether data is written or not. some users might want to only create the array, and write data to it later with a different method. I think the default should be to avoid IO (i.e., don't write).

[normanrz]

I would do it the other way around: add an optional metadata_only kwarg.
In any case, there would be IO to write the zarr.json.

[d-v-b]

yes creating an array will always involve writing metadata, but users with TB scale datasets generally create the array first, then schedule writing to the array in a separate step. IMO attempting to write TB of data by default is not scalable for large datasets, and we should design around that case.

[normanrz]

I know it is not scalable for large arrays. But from an from_array method, I would expect it to actually create the array with the data from the array. Opt out is fine. Maybe what you are proposing should be called something else. No strong opinion here, though.

[d-v-b]

I would expect from_array to create a zarr array. I would not expect it to fill the array with data, because I routinely use zarr with huge arrays. write-on-default would also be problematic for people with dask arrays, because zarr would not be in a position to decide which dask scheduler to use.

[normanrz]

Ok.

[d-v-b]

• (Is there some measure to prevent this that I am not aware of?)

if you are trying to write K input chunks into M output chunks, you can partition your K chunks into sets, where within each set elements can be written independently from all the other elements. then you write each set one after another. in the worst case scenario there will be 1 set per chunk, but you are guaranteed to avoid write collisions this way.

[normanrz]

Suggested change

	data: Array,
	data: Array | npt.ArrayLike,

As discussed, this function should also work with numpy arrays.