🐛[BUG]: constant Dask arrays don't compute on load (healpix TimeSeriesDataModule) #875
Description
Version
1.0.1
On which installation method(s) does this occur?
No response
Describe the issue
I'm one of Dale's students trying to set up a new pipeline for DLWP/Healpix data and relying on physicsnemo.
This line,
2025-05-07T09:35:49.532Z [rank0]: File "/opt/conda/envs/zephyr-1.1/lib/python3.10/site-packages/physicsnemo/datapipes/healpix/timeseries_dataset.py", line 200, in init
const = self.ds.constants.values
causes an issue when loading our basic dataset.
I was able to workaround the issue with this change:
-
pip install python-snappy
-
at the top of the file: (/conda/envs/zephyr-1.1/lib/python3.10/site-packages/physicsnemo/datapipes/healpix/timeseries_dataset.py )
import dask
-
later on in the same file:
if "constants" in self.ds.data_vars: # Check if constants are a Dask array or an xarray.DataArray if isinstance(self.ds["constants"].data, dask.array.core.Array): const = self.ds["constants"].compute().values else: const = self.ds["constants"].values # .values should work if it's already a NumPy array if self.constant_scaling: const = (const - self.constant_scaling["mean"]) / self.constant_scaling[ "std" ] # transpose to match new format: # [C, F, H, W] -> [F, C, H, W] self.constants = np.transpose(const, axes=(1, 0, 2, 3))
Minimum reproducible example
Relevant log output
2025-05-07T09:35:49.532Z [rank0]: Traceback (most recent call last):
2025-05-07T09:35:49.532Z [rank0]: File "/opt/conda/envs/zephyr-1.1/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
2025-05-07T09:35:49.532Z [rank0]: return _target_(*args, **kwargs)
2025-05-07T09:35:49.532Z [rank0]: File "/opt/conda/envs/zephyr-1.1/lib/python3.10/site-packages/physicsnemo/datapipes/healpix/data_modules.py", line 507, in __init__
2025-05-07T09:35:49.532Z [rank0]: self.setup()
2025-05-07T09:35:49.532Z [rank0]: File "/opt/conda/envs/zephyr-1.1/lib/python3.10/site-packages/physicsnemo/datapipes/healpix/data_modules.py", line 617, in setup
2025-05-07T09:35:49.532Z [rank0]: self.train_dataset = TimeSeriesDataset(
2025-05-07T09:35:49.532Z [rank0]: File "/opt/conda/envs/zephyr-1.1/lib/python3.10/site-packages/physicsnemo/datapipes/healpix/timeseries_dataset.py", line 200, in __init__
2025-05-07T09:35:49.532Z [rank0]: const = self.ds.constants.values
2025-05-07T09:35:49.532Z [rank0]: File "/opt/conda/envs/zephyr-1.1/lib/python3.10/site-packages/xarray/core/common.py", line 302, in __getattr__
2025-05-07T09:35:49.532Z [rank0]: raise AttributeError(
2025-05-07T09:35:49.532Z [rank0]: AttributeError: 'DataArray' object has no attribute 'values'Environment details
Package Version
--------------------------------------- --------------
absl-py 2.1.0
aiobotocore 2.21.1
aiohappyeyeballs 2.6.1
aiohttp 3.11.10
aioitertools 0.12.0
aiosignal 1.3.1
alabaster 1.0.0
alembic 1.15.2
aniso8601 10.0.1
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.9.0
appdirs 1.4.4
arrow 1.3.0
asciitree 0.3.3
astropy 5.3.2
astropy-healpix 1.0.0
asttokens 3.0.0
astunparse 1.6.3
async-timeout 4.0.3
attrs 24.3.0
babel 2.17.0
bcrypt 3.2.2
beautifulsoup4 4.13.4
bidict 0.23.1
black 23.9.1
blinker 1.9.0
blosc 1.11.2
bokeh 3.6.2
botocore 1.37.1
Bottleneck 1.4.2
Brotli 1.0.9
brotlipy 0.7.0
cachetools 5.5.1
Cartopy 0.24.1
cdsapi 0.6.1
certifi 2025.1.31
cffi 1.17.1
cftime 1.6.4
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
colorama 0.4.6
Columnar 1.4.1
comm 0.2.2
contourpy 1.3.1
cramjam 2.10.0
cryptography 44.0.1
cuda-cccl 0.0.1.dev0
cycler 0.11.0
cypari 2.5.5
Cython 3.0.12
cytoolz 1.0.1
dacite 1.9.2
dask 2025.2.0
dask-expr 2.0.0
databricks-sdk 0.49.0
debugpy 1.8.11
decorator 5.2.1
Deprecated 1.2.18
discord 2.3.2
discord.py 2.5.2
distributed 2025.2.0
dlwp_healpix 0.0.0
dm-tree 0.1.9
dnspython 2.7.0
docker 7.1.0
docker-py 1.10.6
docker-pycreds 0.4.0
docutils 0.21.2
einops 0.8.1
email_validator 2.2.0
entrypoints 0.4
eventlet 0.39.1
exceptiongroup 1.2.2
executing 2.1.0
fastapi 0.115.4
fastapi-cli 0.0.5
fasteners 0.17.3
ffmpeg 1.4
filelock 3.13.1
Flask 3.1.0
fonttools 4.55.3
frozenlist 1.5.0
fsspec 2024.12.0
furo 2024.8.6
FXrays 1.3.6
gast 0.6.0
gds-tools 0.1.2
gdspy 1.6.13
geos 0.2.3
gitdb 4.0.12
GitPython 3.1.44
glib 1.0.0
glog 0.3.1
gmp 0.0.17
gmpy2 2.2.1
google-auth 2.38.0
google-auth-oauthlib 1.2.1
gradient-descent-the-ultimate-optimizer 1.0
graphene 3.4.3
graphite2 0.0.1
graphql-core 3.2.6
graphql-relay 3.2.0
greenlet 1.1.2
grpcio 1.71.0
grpcio-tools 1.48.2
gunicorn 23.0.0
h11 0.14.0
h2 4.2.0
healpy 1.16.5
HeapDict 1.0.1
hpack 4.1.0
httpcore 1.0.7
httpx 0.28.1
huggingface_hub 0.30.2
hydra-core 1.3.2
hyperframe 6.1.0
icu 0.0.1
idna 3.7
imageio 2.37.0
imageio-ffmpeg 0.6.0
imagesize 1.4.1
importlib_metadata 8.5.0
importlib_resources 6.4.0
intel-cmplr-lib-ur 2025.1.1
intel-openmp 2025.1.1
ipykernel 6.29.5
ipython 8.35.0
itsdangerous 2.2.0
jedi 0.19.2
Jinja2 3.1.4
jmespath 1.0.1
joblib 1.4.2
jupyter_client 8.6.3
jupyter_core 5.7.2
keyutils 0.6
kiwisolver 1.4.8
knot_floer_homology 1.2.2
krb5 0.7.1
lerc 4.0.1
libclang 18.1.1
libnetcdf 0.0.1
libsodium 2.7.4
libuuid 1.0.0
locket 1.0.0
low_index 1.2.1
lxml 5.4.0
lz4 4.3.2
Mako 1.3.10
Markdown 3.4.1
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.10.0
matplotlib-inline 0.1.7
mdurl 0.1.2
mkl 2025.1.0
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mlflow 2.21.3
mlflow-skinny 2.21.3
moviepy 1.0.3
mpc 0.0.6
mpfr 0.0.1
mpmath 1.3.0
msgpack 1.0.3
msgpack-python 0.5.6
multidict 6.1.0
munkres 1.1.4
mypy-extensions 1.0.0
nest_asyncio 1.6.0
netCDF4 1.6.2
networkx 3.3
nss 3.8.5
numcodecs 0.12.1
numexpr 2.8.4
numpy 1.26.4
nvidia-cublas-cu11 11.11.3.6
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu11 11.8.87
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu11 11.8.89
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu11 11.8.89
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu11 9.1.0.70
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu11 10.9.0.58
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu11 10.3.0.86
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu11 11.4.1.48
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu11 11.7.5.86
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-dali-cuda120 1.35.0
nvidia-nccl-cu11 2.21.5
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu11 11.8.86
nvidia-nvtx-cu12 12.4.127
nvidia-physicsnemo 1.0.1
nvtx 0.2.11
oauthlib 3.2.2
omegaconf 2.3.0
onnx 1.17.0
opentelemetry-api 1.32.0
opentelemetry-sdk 1.32.0
opentelemetry-semantic-conventions 0.53b0
orc 0.1.0
packaging 23.1
pandas 2.2.3
paramiko 3.5.1
parso 0.8.4
partd 1.4.2
pathspec 0.11.2
pcre2 0.5.2
pexpect 4.9.0
pickleshare 0.7.5
pillow 11.0.0
pip 25.0
platformdirs 4.3.7
plink 2.4.6
ply 3.11
pooch 1.8.2
proglog 0.1.10
proj 0.2.0
prometheus_client 0.21.1
prometheus_flask_exporter 0.23.2
prompt_toolkit 3.0.51
propcache 0.3.1
protobuf 3.20.2
psutil 5.9.0
ptyprocess 0.7.0
pure_eval 0.2.3
pyarrow 14.0.2
pyasn1 0.6.1
pyasn1_modules 0.4.1
pycparser 2.21
pycryptodome 3.22.0
pydantic 2.10.3
pydantic_core 2.27.1
pyerfa 2.0.0.3
Pygments 2.19.1
PyJWT 2.10.1
PyNaCl 1.5.0
pyOpenSSL 25.0.0
pyparsing 3.2.0
pypng 0.20220715.0
pyproj 3.6.1
PyQt5 5.15.10
PyQt5_sip 12.13.0
pyshp 2.3.1
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-engineio 4.12.0
python-gflags 3.1.2
python-multipart 0.0.20
python-snappy 0.7.3
python-socketio 5.13.0
pytz 2024.1
pyu2f 0.1.5
PyX 0.16
PyYAML 6.0.2
pyzmq 25.1.0
querystring_parser 1.2.4
recordclass 0.23.1
repoze.lru 0.7
reproject 0.11.0
requests 2.32.3
requests-oauthlib 2.0.0
rich 14.0.0
Routes 2.5.1
rsa 4.9
s3fs 2024.12.0
safetensors 0.5.3
scikit-learn 1.1.1
scipy 1.11.2
sentry-sdk 2.26.1
setproctitle 1.2.3
setuptools 75.8.0
shapely 2.0.6
shellingham 1.5.4
simple-websocket 1.1.0
sip 6.7.12
six 1.16.0
smmap 5.0.2
snappy 3.2
snappy_manifolds 1.2.1
sniffio 1.3.1
snowballstemmer 2.2.0
sonora 0.2.3
sortedcontainers 2.4.0
soupsieve 2.7
spherogram 2.3
Sphinx 8.1.3
sphinx-basic-ng 1.0.0b2
sphinxcontrib-applehelp 2.0.0
sphinxcontrib-devhelp 2.0.0
sphinxcontrib-htmlhelp 2.1.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 2.0.0
sphinxcontrib-serializinghtml 2.0.0
SQLAlchemy 2.0.37
sqlparse 0.5.3
stack_data 0.6.3
starlette 0.41.3
string-color 1.2.3
sympy 1.13.1
tbb 2022.1.0
tblib 1.7.0
tcmlib 1.3.0
tensorboard 2.17.0
tensorboard_data_server 0.7.0
tensorboard-plugin-wit 1.8.1
termcolor 3.0.1
tesseract 0.1.3
threadpoolctl 3.6.0
timm 1.0.15
tk 0.1.0
tkinter-gl 1.0
toml 0.10.2
tomli 2.2.1
toolz 1.0.0
torch 2.6.0+cu118
torchaudio 2.3.1
torchinfo 1.8.0
torchvision 0.21.0+cu118
tornado 6.4.2
tqdm 4.67.1
traitlets 5.14.3
treelib 1.2.5
triton 3.2.0
typer 0.15.2
typer-slim 0.15.2
types-python-dateutil 2.9.0.20241206
typing_extensions 4.12.2
tzdata 2025.2
umf 0.10.0
unicodedata2 15.1.0
urllib3 1.26.20
uvicorn 0.34.1
wandb 0.19.10
wcwidth 0.2.13
websocket-client 1.8.0
websockets 15.0.1
Werkzeug 3.1.3
wheel 0.45.1
wrapt 1.14.1
wsproto 1.2.0
xarray 2024.10.0
xyzservices 2025.1.0
yarl 1.18.0
zarr 2.14.2
zict 3.0.0
zipp 3.21.0
zstandard 0.23.0