Zarr support

[keller-mark]

Fixes #91

These changes are from both me and @Artur-man

The main public-facing changes here are:

• The ZarrAnnData class
• read_zarr and write_zarr top-level functions
• Support for from_Seurat(output_class="ZarrAnnData")
• Support for from_SingleCellExperiment(output_class="ZarrAnnData")

Internally:

• read_zarr_helpers.R is the zarr analog of read_h5ad_helpers.R
• write_zarr_helpers.R is the zarr analog of write_h5ad_helpers.R
• Test fixtures within inst/extdata/example.zarr (this makes the diff noisy, apologies)
• Lots of tests:
  • test-Zarr-read.R (35 new tests)
  • test-Zarr-write.R (70)
  • test-ZarrAnnData.R (26)
  • test-h5ad-zarr.R (17)

A number of these functions generate warnings in the R console that are intended to be followed up on to improve the code (and should probably be resolved as end users may not appreciate them), but the tests still pass despite these warnings.

Known things that are not implemented here:

• support for recarrays
• usage of mode = c("r", "r+", "a", "w", "w-", "x") parameter value

[rcannood]

Fantastic work @keller-mark and @Artur-man !

I went through the PR for a first time and left some minor comments. I will review the code by running it a couple of times next :)

[rcannood]

where are a lot of linting issues in this file -- could you run lintr::lint_package() and fix any issues that pop up?

[Artur-man]

done ... did a full lint_package check and corrected some R check issues too!

[rcannood]

This file should probably be removed

[Artur-man]

done

[Artur-man]

I have quickly checked if pizzarr utilities could be replaced with https://github.com/grimbough/Rarr, unfortunately there exists a set of limitations to the BioC native package, which

• opening zarr stores and creating groups, see add utilities for creating zarr stores and groups Huber-group-EMBL/Rarr#18
• read/write boolean arrays, see Add support for writing boolean arrays and lower precision numeric Huber-group-EMBL/Rarr#59
• fixing errors when reading from some character arrays, see negative length vectors are not allowed error for read_zarr_array Huber-group-EMBL/Rarr#20

Will be in touch to see if these are resolved in the future, otherwise no zarr R package is currently both in CRAN/BioC and functionally complete yet.

[Artur-man]

There is some progress in the Rarr package, would you guys like a clean PR (since there were so many updates since) or continuing here is fine ?

[lazappi]

There is some progress in the Rarr package, would you guys like a clean PR (since there were so many updates since) or continuing here is fine ?

Probably whatever is easiest for you and @keller-mark/whatever makes the PR easiest to understand. There have been a lot of changes to the package since this was opened so we would need to make sure those are included here.

I saw that {Rarr} is planning to have Zarr v3 support for the next release so I think that makes sense in terms of which backend package to use.

[Artur-man]

I adapted the changes, was fairly easy. I will then continue here and refer back to Hugo if needed again.

[lazappi]

I'm going to make this a draft for now, just to help with our organisation. Please let us know when it's ready to review.

[Artur-man]

Guys, I am close .... I will add a new example.zarr here, however lets add a yml file to allow creating quick environments, so the same files can be generated quickly later, and also ... would you like a separate example_zarr.py file ?

anndataR/inst/scripts/example_h5ad.py

Lines 1 to 7 in 5ffd79a

	# python v3.13.5
	import anndata # anndata v0.11.4
	import scanpy # scanpy v1.11.4
	import numpy # numpy v2.2.6
	import pandas # pandas v2.3.0
	import scipy.sparse # scipy v1.14.1

[lazappi]

We tried to cover as many cases as possible in the example dataset so I would probably modify the script to also output a (tarred/zipped) Zarr as well as an H5AD (and maybe rename it to something like create_example.py) rather than creating a new one.

I noted down the versions just in case it became important later but maybe it makes sense to store them in a separate file. If you are regenerating the dataset I would probably also update the environment. I think there is also a version number in the script that should be bumped and a changelog that should be updated.

[Artur-man]

I have tested this in a couple of anndata examples as well as test datasets from https://github.com/HelenaLC/SpatialData and https://github.com/HelenaLC/SpatialData.data, and it seems to be working.

We are currently waiting for structure data and scalar support from Rarr:

• Reading scalars Huber-group-EMBL/Rarr#80
• Reading structured data types Huber-group-EMBL/Rarr#81