Skip to content

healpix moc index #151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 57 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
eeb3004
create a xarray healpix moc index
keewis Jun 3, 2025
900546c
hack to allow creating a moc index
keewis Jun 3, 2025
fbc7154
depend on `healpix-geo`
keewis Jun 5, 2025
017d1d0
install `healpix-geo` from PyPI
keewis Jun 5, 2025
51d85ea
Merge branch 'main' into moc-index
keewis Jun 5, 2025
d197dfa
install `healpix-geo` in the docs
keewis Jun 5, 2025
4246fac
use a special `values` method to access the actual data
keewis Jun 5, 2025
2b5670b
only support `"nested"` in the MOC index
keewis Jun 5, 2025
3b43dea
special-case the full domain
keewis Jun 5, 2025
86c0a44
remove the duplicate index creation
keewis Jun 5, 2025
19fe087
forward to the wrapped index' `create_variables`
keewis Jun 5, 2025
3315745
dask support in `create_variables`
keewis Jun 6, 2025
d8b327a
support `dask` in `from_cell_ids`
keewis Jun 6, 2025
c88f0c3
wrap the chunked delayed construction in a list
keewis Jun 11, 2025
ffbb4e9
replace the dask loading code with a error message
keewis Jun 11, 2025
c769134
out-of-core implement the RangeMOCIndex
keewis Jun 11, 2025
4d95020
remove the `n_partitions` parameter for now
keewis Jun 11, 2025
ea42c6e
forward the `size` and `nbytes` properties
keewis Jun 11, 2025
e8b689c
check that construction from an array works properly
keewis Jun 11, 2025
b3bea15
dask testing utils
keewis Jun 11, 2025
680ad64
remove the unused `unique` scheme for now
keewis Jun 11, 2025
e0353a5
typo
keewis Jun 11, 2025
15e7633
changelog
keewis Jun 11, 2025
179ae5a
treat `decode` as a method
keewis Jun 11, 2025
2ef036c
document the `index_kind` parameter
keewis Jun 11, 2025
bde552d
pass through the `index_kind` parameter
keewis Jun 11, 2025
19ca9d9
rename `_pd_index` to `_index`
keewis Jun 11, 2025
177ab6c
check `isel`
keewis Jun 12, 2025
15d99c2
raise on irregular chunks
keewis Jun 12, 2025
13ebccc
check `create_variables`
keewis Jun 12, 2025
15de58a
check that irregular chunks are rejected
keewis Jun 12, 2025
6df12bb
more accurate condition
keewis Jun 12, 2025
cc4d983
check more invalid chunks
keewis Jun 12, 2025
c791c93
check more valid chunk sizes
keewis Jun 12, 2025
ed9527c
modify the chunksize if we reach the end
keewis Jun 12, 2025
fc6cb5c
support irregular chunks by tracking them on the index
keewis Jun 12, 2025
92fa1c6
require `numpy>=2`
keewis Jun 12, 2025
f99c045
more (and smaller) tests
keewis Jun 12, 2025
259b927
restructure the `isel` tests
keewis Jun 12, 2025
f26e426
improve the `isel` test
keewis Jun 12, 2025
a786344
fix two bugs in the chunk subsetting
keewis Jun 12, 2025
7bec771
remove the default for the `chunksizes` parameter
keewis Jun 12, 2025
826500b
depth → level
keewis Jun 12, 2025
545557e
tests for `from_variables`
keewis Jun 12, 2025
82b4580
more tests for edge cases
keewis Jun 12, 2025
f88d55b
check that the index is created properly
keewis Jun 12, 2025
ba2a925
check the extracted chunks
keewis Jun 13, 2025
3a28ed0
explicitly raise an error for non-1D cell id coords
keewis Jun 13, 2025
cf12c82
properly extract the chunks
keewis Jun 13, 2025
1b4c8eb
also replace the typing to `xr.Index`
keewis Jun 13, 2025
92c484f
accept index options instead of specific args in `decode`
keewis Jun 13, 2025
c866046
remove support for the `index_kind` kwarg for H3
keewis Jun 13, 2025
512c905
mark the `raise_if_dask_computes` as vendored from `xarray`
keewis Jun 13, 2025
6de6c33
docstrings for the moc index wrapper
keewis Jun 16, 2025
387a9be
point towards `healpix_geo.nested.RangeMOCIndex`
keewis Jun 16, 2025
8a809b9
point out this only works for `nested`
keewis Jun 16, 2025
45842e5
more comments on the role of the different index objects
keewis Jun 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,5 @@ dependencies:
- cdshealpix
- pip
- pip:
- healpix-geo
- -e ..
3 changes: 3 additions & 0 deletions ci/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@ dependencies:
- arro3-core
- cdshealpix
- h3ronpy
- pip
- pip:
- healpix-geo
12 changes: 11 additions & 1 deletion docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ Parameters

Dataset.dggs.grid_info
Dataset.dggs.params

.. autosummary::
:toctree: generated
:template: autosummary/accessor_method.rst

Dataset.dggs.decode


Expand All @@ -62,7 +67,12 @@ Parameters

DataArray.dggs.grid_info
DataArray.dggs.params
DataArray.dggs.decode

.. autosummary::
:toctree: generated
:template: autosummary/accessor_method.rst

Dataset.dggs.decode


Data inference
Expand Down
4 changes: 3 additions & 1 deletion docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@

### New features

- memory-efficient index implementation based on multi-order coverage maps (MOCs) ({pull}`151`)

### Bug fixes

### Documentation

- Documentation Contributer Guide + Github Button ({pull}`137`)
- Documentation Contributor's Guide + Github Button ({pull}`137`)

### Internal changes

Expand Down
1 change: 1 addition & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@
"healpy": ("https://healpy.readthedocs.io/en/latest", None),
"cdshealpix-python": ("https://cds-astro.github.io/cds-healpix-python", None),
"shapely": ("https://shapely.readthedocs.io/en/stable", None),
"healpix-geo": ("https://healpix-geo.readthedocs.io/en/latest", None),
}

# -- myst-nb options ---------------------------------------------------------
Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@ classifiers = [
requires-python = ">=3.10"
dependencies = [
"xarray",
"numpy>=2.0",
"cdshealpix",
"healpix-geo>=0.0.3",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible to make it an optional dependency? I'm concerned by the number of dependencies (compiled packages) xdggs would require as we add more support for various DGGS.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true, I think we should figure out how to resolve this. When we added plotting support we decided to install everything by default (it was easier that way), but that won't scale (see also #128). However, I think we should limit the builtin DGGS to just h3 and healpix, and have separate extension packages for others.

healpix-geo is a bit special in that you will need it to support ellipsoidal healpix, so that may just stay required (or in the same extra as cdshealpix, should we modify the code to make h3ronpy and cdshealpix optional)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, I think we should limit the builtin DGGS to just h3 and healpix

Yes we can limit builtin support to just a few DGGSs (S2 might probably also make sense), but even so users who may be interested in only H3 would need to download a few Healpix dependencies, which is probably not ideal.

We can address that later, though.

"h3ronpy",
"typing-extensions",
"lonboard>=0.9.3",
Expand Down
13 changes: 11 additions & 2 deletions xdggs/accessor.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ def __init__(self, obj: xr.Dataset | xr.DataArray):
self._name = name
self._index = index

def decode(self, grid_info=None, *, name="cell_ids") -> xr.Dataset | xr.DataArray:
def decode(
self, grid_info=None, *, name="cell_ids", index_options=None, **index_kwargs
) -> xr.Dataset | xr.DataArray:
"""decode the DGGS cell ids

Parameters
Expand All @@ -39,6 +41,8 @@ def decode(self, grid_info=None, *, name="cell_ids") -> xr.Dataset | xr.DataArra
the dataset.
name : str, default: "cell_ids"
The name of the coordinate containing the cell ids.
index_options, **index_kwargs : dict, optional
Additional options to forward to the index.

Returns
-------
Expand All @@ -51,7 +55,12 @@ def decode(self, grid_info=None, *, name="cell_ids") -> xr.Dataset | xr.DataArra
if isinstance(grid_info, dict):
var.attrs = grid_info

return self._obj.drop_indexes(name, errors="ignore").set_xindex(name, DGGSIndex)
if index_options is None:
index_options = {}

return self._obj.drop_indexes(name, errors="ignore").set_xindex(
name, DGGSIndex, **(index_options | index_kwargs)
)

@property
def index(self) -> DGGSIndex:
Expand Down
7 changes: 3 additions & 4 deletions xdggs/h3.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
cells_to_wkb_polygons,
coordinates_to_cells,
)
from xarray.indexes import PandasIndex

from xdggs.grid import DGGSInfo, translate_parameters
from xdggs.index import DGGSIndex
Expand Down Expand Up @@ -208,7 +207,7 @@ class H3Index(DGGSIndex):

def __init__(
self,
cell_ids: Any | PandasIndex,
cell_ids: Any | xr.Index,
dim: str,
grid_info: DGGSInfo,
):
Expand All @@ -231,8 +230,8 @@ def from_variables(
def grid_info(self) -> H3Info:
return self._grid

def _replace(self, new_pd_index: PandasIndex):
return type(self)(new_pd_index, self._dim, self._grid)
def _replace(self, new_index: xr.Index):
return type(self)(new_index, self._dim, self._grid)

def _repr_inline_(self, max_width: int):
return f"H3Index(level={self._grid.level})"
Loading