Refactor (#1224)

* Move filesystems and version_check to core

* Fix version check tests

* Reorganize distributed, domain_parallel, and begin nn / utils cleanup.

* Move modules and meta to core.  Move registry to core.

No tests fixed yet.

* Add missing init files

* Update build system and specify some deps.

* Reorganize tests.

* Update init files

* Clean up neighbor tools.

* Update testing

* Fix compat tests

* Move core model tests to tests/core/

* Add import lint config

* Relocate layers

* Move graphcast utils into model directory

* Relocating util functionalities.

* Further clean up and organize tests.

* utils tests are passing now

* Cleaning up distributed tests

* Patching tests working again in nn

* Fix sdf test

* Fix zenith angle tests

* Some organization of tests.  Checkpoints is moved into utils.

* Remove launch.utils and launch.config.  Checkpointing is moved to
phsyicsnemo.utils, launch.config is just gone.  It was empty.

* Most nn tests are passing

* Further cleanup.  Getting there!

* Remove constants file

* Add import linting to pre-commit.

* Update crash readme (#1212)

* update license headers- second try

* update readme

* Bump multi-storage-client to v0.33.0 with rust client (#1156)

* Move gnn layers and start to fix several model tests.

* AFNO is now passing.

* Rnn models passing.

* Fix improt

* Healpix tests are working

* Domino and unet working

* Add jaxtyping to requirements.txt for crash sample (#1218)

* update license headers- second try

* Update requirements.txt

* Updating to address some test issues

* MGN tests passing again

* Most graphcast tests passing again

* Move nd conv layers.

* update fengwu and pangu

* Update sfno and pix2pix test

* update tests for figconvnet, swinrnn, superresnet

* updating more models to pass

* Update distributed tests, now passing.

* Domain parallel tests now passing.

* Fix active learning imports so tests pass in refactor

* Fix some metric imports

* Remove deploy package

* Remove unused test file

* unmigrate these files ... again?

* Update import linter.

---------

Co-authored-by: Mohammad Amin Nabian <[email protected]>
Co-authored-by: Yongming Ding <[email protected]>

Author: 3 people

.importlinter

@@ -9,6 +9,7 @@ containers=
     physicsnemo
 layers = 
     experimental
+    diffusion | active_learning
     models  : registry : datapipes : metrics : domain_parallel
     nn
     utils
@@ -44,7 +45,7 @@ type = layers
 containers=
     physicsnemo.utils
 layers = 
-    mesh
+    mesh | insolation | zenith_angle
     profiling
     checkpoint
     capture
@@ -58,10 +59,17 @@ containers=
 layers = 
     fourier_layers | transformer_layers
     dgm_layers | mlp_layers | fully_connected_layers | gnn_layers
-    activations | attention_layers | ball_query | conv_layers  | drop | fft  | fused_silu | interpolation | kan_layers | patching | resample_layers | sdf | siren_layers | spectral_layers | transformer_decoder  | weight_fact | weight_norm
+    activations | attention_layers | ball_query | conv_layers  | drop | fft  | fused_silu | interpolation | kan_layers  | resample_layers | sdf | siren_layers | spectral_layers | transformer_decoder  | weight_fact | weight_norm
     neighbors 
     utils
 
+[importlinter:contract:physicsnemo-diffusion]
+name = Control Dependencies in PhysicsNeMo diffusion
+type = layers
+containers=
+    physicsnemo.diffusion
+layers = 
+    patching
 
 [importlinter:contract:physicsnemo-models]
 name = Prevent Imports between physicsnemo models
@@ -72,5 +80,4 @@ layers =
     mesh_reduced
     afno | dlwp | dlwp_healpix | domino | dpot | fengwu | figconvnet | fno | graphcast | meshgraphnet | pangu | pix2pix | rnn |  srrn | swinvrnn | topodiff | transolver  | vfgn 
     unet | diffusion  | dlwp_healpix_layers 
-    utils
 

CHANGELOG.md

@@ -51,6 +51,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Support passing custom tokenizer, detokenizer, and attention `Module`s in
   experimental DiT architecture
 - Improved Transolver training recipe's configuration for checkpointing and normalization.
+- Bumped `multi-storage-client` version to 0.33.0 with rust client.
 
 ### Deprecated
 

Dockerfile

@@ -213,7 +213,7 @@ RUN pip install --no-cache-dir "black==22.10.0" "interrogate==1.5.0" "coverage==
 RUN pip install --no-cache-dir "numpy-stl" "scikit-image>=0.24.0" "sparse-dot-mkl" "shapely" "numpy<2.0"
 
 # Install MSC
-RUN pip install --no-cache-dir "multi-storage-client[boto3]>=0.14.0"
+RUN pip install --no-cache-dir "multi-storage-client[boto3]>=0.33.0"
 
 # cleanup of stage
 RUN rm -rf /physicsnemo/

docs/img/crash/roof_crash.gif

@@ -25,6 +25,8 @@ profiles:
         region_name: us-west-2
         base_path: cmip6-pds
         signature_version: UNSIGNED
+        rust_client:
+          skip_signature: True
 cache:
   location: /tmp/.cache
   size_mb: 5000

examples/multi_storage_client/msc_config.yaml

@@ -1 +1,2 @@
-multi-storage-client[boto3]
+multi-storage-client[boto3,fsspec]
+zarr

examples/multi_storage_client/requirements.txt

@@ -29,13 +29,6 @@ For an in-depth comparison between the Transolver and MeshGraphNet models and th
 
 </p>
 
-### Roof Crash Modeling
-
-<p align="center">
-  <img src="../../../docs/img/crash/roof_crash.gif" alt="Roof crash animation" width="80%" />
-  
-</p>
-
 ## Quickstart
 
 1) Select your recipe (reader, datapipe, model) in `conf/config.yaml`.

examples/structural_mechanics/crash/README.md

@@ -1,3 +1,4 @@
+jaxtyping==0.3.3
 lasso-python==2.0.3
 torch_geometric==2.6.1
 torch_scatter>=2.1.2

examples/structural_mechanics/crash/requirements.txt

@@ -35,7 +35,6 @@
 from torch.nn.parallel import DistributedDataParallel
 from torch.utils.data import DataLoader, DistributedSampler
 
-from physicsnemo import Module
 from physicsnemo import __version__ as physicsnemo_version
 from physicsnemo.active_learning import protocols as p
 from physicsnemo.active_learning.config import (
@@ -47,6 +46,7 @@
     ActiveLearningLoggerAdapter,
     setup_active_learning_logger,
 )
+from physicsnemo.core import Module
 from physicsnemo.distributed import DistributedManager
 
 

physicsnemo/active_learning/driver.py

@@ -26,11 +26,11 @@
 from torch.utils.data import DataLoader
 from tqdm import tqdm
 
-from physicsnemo import Module
 from physicsnemo.active_learning import protocols as p
+from physicsnemo.core import Module
 from physicsnemo.distributed import DistributedManager
-from physicsnemo.launch.logging import LaunchLogger
 from physicsnemo.utils.capture import StaticCaptureEvaluateNoGrad, StaticCaptureTraining
+from physicsnemo.utils.logging import LaunchLogger
 
 __all__ = ["DefaultTrainingLoop"]