Skip to content

Conversation

@saikrishnanc-nv
Copy link
Collaborator

PhysicsNeMo Pull Request

Description

This PR adds a Zarr reader for Crash. A corresponding Zarr preprocessor/writer has been merged to PhysicsNeMo-Curator already.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.
  • The CHANGELOG.md is up to date with these changes.
  • An issue is linked to this pull request.

Dependencies

Review Process

All PRs are reviewed by the PhysicsNeMo team before merging.

Depending on which files are changed, GitHub may automatically assign a maintainer for review.

We are also testing AI-based code review tools (e.g., Greptile), which may add automated comments with a confidence score.
This score reflects the AI’s assessment of merge readiness and is not a qualitative judgment of your work, nor is
it an indication that the PR will be accepted / rejected.

AI-generated feedback should be reviewed critically for usefulness.
You are not required to respond to every AI comment, but they are intended to help both authors and reviewers.
Please react to Greptile comments with 👍 or 👎 to provide feedback on their accuracy.

@saikrishnanc-nv saikrishnanc-nv self-assigned this Nov 13, 2025
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Nov 13, 2025

Greptile Overview

Greptile Summary

Adds a Zarr reader for crash simulation data to complement existing d3plot and VTP readers. The reader loads pre-processed Zarr stores created by PhysicsNeMo-Curator, where computational work (node filtering, edge building) has already been done during ETL.

Key changes:

  • New zarr_reader.py with Reader class that loads mesh_pos, edges, and point data features from Zarr stores
  • Comprehensive validation for 1D [N] and 2D [N,K] feature arrays (addresses previous review feedback)
  • Natural sorting of Zarr stores for consistent ordering
  • Full test coverage including edge cases, validation logic, and 2D feature support
  • Updated documentation with Zarr format examples and PhysicsNeMo-Curator workflow
  • New Hydra configuration file conf/reader/zarr.yaml

All previously flagged issues with the validation logic have been resolved. The code now correctly handles both 1D and 2D feature arrays as documented.

Confidence Score: 5/5

  • This PR is safe to merge with no issues found
  • The implementation is clean, well-tested, and follows existing patterns in the codebase. All previous validation issues have been resolved, comprehensive tests cover edge cases and 2D features, documentation is thorough, and the code integrates seamlessly with the existing reader infrastructure
  • No files require special attention

Important Files Changed

File Analysis

Filename Score Overview
examples/structural_mechanics/crash/zarr_reader.py 5/5 New Zarr reader implementation with proper validation for 1D and 2D feature arrays, comprehensive error handling, and natural sorting
examples/structural_mechanics/crash/tests/test_zarr_reader.py 5/5 Comprehensive test suite covering edge cases, validation logic, 2D features, natural sorting, and error conditions
examples/structural_mechanics/crash/README.md 5/5 Updated documentation to describe Zarr format, reader usage, and PhysicsNeMo-Curator ETL workflow

Sequence Diagram

sequenceDiagram
    participant User
    participant Reader
    participant process_zarr_data
    participant find_zarr_stores
    participant load_zarr_store
    participant zarr as Zarr Store

    User->>Reader: __call__(data_dir, num_samples)
    Reader->>process_zarr_data: process_zarr_data(data_dir, num_samples, logger)
    process_zarr_data->>find_zarr_stores: find_zarr_stores(data_dir)
    find_zarr_stores->>find_zarr_stores: List *.zarr directories
    find_zarr_stores->>find_zarr_stores: Natural sort by name
    find_zarr_stores-->>process_zarr_data: List of zarr_paths
    
    loop For each zarr_path (up to num_samples)
        process_zarr_data->>load_zarr_store: load_zarr_store(zarr_path)
        load_zarr_store->>zarr: Open Zarr store
        load_zarr_store->>zarr: Read mesh_pos [T,N,3]
        load_zarr_store->>zarr: Read edges [E,2]
        load_zarr_store->>zarr: Read point data fields (thickness, etc.)
        zarr-->>load_zarr_store: mesh_pos, edges, point_data_dict
        load_zarr_store-->>process_zarr_data: mesh_pos, edges, point_data_dict
        
        process_zarr_data->>process_zarr_data: Validate mesh_pos shape [T,N,3]
        process_zarr_data->>process_zarr_data: Validate edges shape [E,2]
        process_zarr_data->>process_zarr_data: Validate point data (1D [N] or 2D [N,K])
        process_zarr_data->>process_zarr_data: Validate edge indices in bounds
        process_zarr_data->>process_zarr_data: Extract src, dst from edges
        process_zarr_data->>process_zarr_data: Build record with coords + features
    end
    
    process_zarr_data-->>Reader: srcs, dsts, point_data_all
    Reader-->>User: srcs, dsts, point_data
Loading

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@saikrishnanc-nv
Copy link
Collaborator Author

/blossom-ci

Copy link
Collaborator

@mnabian mnabian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@saikrishnanc-nv
Copy link
Collaborator Author

/blossom-ci

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@saikrishnanc-nv saikrishnanc-nv merged commit 7277097 into NVIDIA:main Nov 14, 2025
1 check passed
@saikrishnanc-nv saikrishnanc-nv deleted the saikrishnanc/update_crash_zarr branch November 14, 2025 22:33
coreyjadams added a commit that referenced this pull request Nov 18, 2025
* Move filesystems and version_check to core

* Fix version check tests

* Reorganize distributed, domain_parallel, and begin nn / utils cleanup.

* Move modules and meta to core.  Move registry to core.

No tests fixed yet.

* Add missing init files

* Update build system and specify some deps.

* Reorganize tests.

* Update init files

* Clean up neighbor tools.

* Update testing

* Fix compat tests

* Move core model tests to tests/core/

* Add import lint config

* Relocate layers

* Move graphcast utils into model directory

* Relocating util functionalities.

* Further clean up and organize tests.

* utils tests are passing now

* Cleaning up distributed tests

* Patching tests working again in nn

* Fix sdf test

* Fix zenith angle tests

* Some organization of tests.  Checkpoints is moved into utils.

* Remove launch.utils and launch.config.  Checkpointing is moved to
phsyicsnemo.utils, launch.config is just gone.  It was empty.

* Most nn tests are passing

* Further cleanup.  Getting there!

* Remove constants file

* Add import linting to pre-commit.

* Update crash readme (#1212)

* update license headers- second try

* update readme

* Bump multi-storage-client to v0.33.0 with rust client (#1156)

* Move gnn layers and start to fix several model tests.

* AFNO is now passing.

* Rnn models passing.

* Fix improt

* Healpix tests are working

* Domino and unet working

* Add jaxtyping to requirements.txt for crash sample (#1218)

* update license headers- second try

* Update requirements.txt

* Updating to address some test issues

* Replace 'License' link with 'Dev blog' link (#1215)

Co-authored-by: Corey adams <[email protected]>

* MGN tests passing again

* Most graphcast tests passing again

* Move nd conv layers.

* update fengwu and pangu

* Update sfno and pix2pix test

* update tests for figconvnet, swinrnn, superresnet

* updating more models to pass

* Update distributed tests, now passing.

* Validation fu added to examples/structural_mechanics/crash/train.py (#1204)

* validation added: works for multi-node job.

* rename and rearrange validation function

* validate_every_n_epochs, save_ckpt_every_n_epochs added in config

* corrected bug (args of model) in inference

* args in validation code updated

* val path added and args name changed

* validation split added -> write_vtp=False

* fixed inference bug

* bug fix: write_vtp

* Domain parallel tests now passing.

* Fix active learning imports so tests pass in refactor

* Fix some metric imports

* Remove deploy package

* Remove unused test file

* unmigrate these files ... again?

* Update import linter.

* Add saikrishnanc-nv to github actors (#1225)

* Integrate Curator instructions to the Crash example (#1213)

* Integrate Curator instructions

* Update docs

* Formatting changes

* Adding code of conduct (#1214)

* Adding code of conduct

Adopting the code of conduct from the https://www.contributor-covenant.org/

* Update CODE_OF_CONDUCT.MD

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Create .markdownlintignore

* Revise README for PhysicsNeMo resources and guidance

Updated the 'Getting Started' section and added new resources for learning AI Physics.

* Update README.md

---------

Co-authored-by: Mohammad Amin Nabian <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Corey adams <[email protected]>

* Cleaning up diffusion models. Not quite done yet.

* Restore deleted files

* Updating more tests.

* Fixed minor bug in shape validation in SongUNet (#1230)

Signed-off-by: Charlelie Laurent <[email protected]>

* Add Zarr reader for Crash (#1228)

* Add Zarr reader for Crash

* Update README

* Update validation logic of point data in Zarr reader

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update examples/structural_mechanics/crash/zarr_reader.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Add a test for 2D feature arrays

* Update examples/structural_mechanics/crash/zarr_reader.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Further updates to tests.  Datapipes almost working.

* update import paths

* Starting to clean up dependency tree.

---------

Signed-off-by: Charlelie Laurent <[email protected]>
Co-authored-by: Mohammad Amin Nabian <[email protected]>
Co-authored-by: Yongming Ding <[email protected]>
Co-authored-by: ram-cherukuri <[email protected]>
Co-authored-by: Deepak Akhare <[email protected]>
Co-authored-by: Sai Krishnan Chandrasekar <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Charlelie Laurent <[email protected]>
coreyjadams added a commit that referenced this pull request Nov 19, 2025
* Move filesystems and version_check to core

* Fix version check tests

* Reorganize distributed, domain_parallel, and begin nn / utils cleanup.

* Move modules and meta to core.  Move registry to core.

No tests fixed yet.

* Add missing init files

* Update build system and specify some deps.

* Reorganize tests.

* Update init files

* Clean up neighbor tools.

* Update testing

* Fix compat tests

* Move core model tests to tests/core/

* Add import lint config

* Relocate layers

* Move graphcast utils into model directory

* Relocating util functionalities.

* Further clean up and organize tests.

* utils tests are passing now

* Cleaning up distributed tests

* Patching tests working again in nn

* Fix sdf test

* Fix zenith angle tests

* Some organization of tests.  Checkpoints is moved into utils.

* Remove launch.utils and launch.config.  Checkpointing is moved to
phsyicsnemo.utils, launch.config is just gone.  It was empty.

* Most nn tests are passing

* Further cleanup.  Getting there!

* Remove constants file

* Add import linting to pre-commit.

* Update crash readme (#1212)

* update license headers- second try

* update readme

* Bump multi-storage-client to v0.33.0 with rust client (#1156)

* Move gnn layers and start to fix several model tests.

* AFNO is now passing.

* Rnn models passing.

* Fix improt

* Healpix tests are working

* Domino and unet working

* Add jaxtyping to requirements.txt for crash sample (#1218)

* update license headers- second try

* Update requirements.txt

* Updating to address some test issues

* Replace 'License' link with 'Dev blog' link (#1215)

Co-authored-by: Corey adams <[email protected]>

* MGN tests passing again

* Most graphcast tests passing again

* Move nd conv layers.

* update fengwu and pangu

* Update sfno and pix2pix test

* update tests for figconvnet, swinrnn, superresnet

* updating more models to pass

* Update distributed tests, now passing.

* Validation fu added to examples/structural_mechanics/crash/train.py (#1204)

* validation added: works for multi-node job.

* rename and rearrange validation function

* validate_every_n_epochs, save_ckpt_every_n_epochs added in config

* corrected bug (args of model) in inference

* args in validation code updated

* val path added and args name changed

* validation split added -> write_vtp=False

* fixed inference bug

* bug fix: write_vtp

* Domain parallel tests now passing.

* Fix active learning imports so tests pass in refactor

* Fix some metric imports

* Remove deploy package

* Remove unused test file

* unmigrate these files ... again?

* Update import linter.

* Add saikrishnanc-nv to github actors (#1225)

* Integrate Curator instructions to the Crash example (#1213)

* Integrate Curator instructions

* Update docs

* Formatting changes

* Adding code of conduct (#1214)

* Adding code of conduct

Adopting the code of conduct from the https://www.contributor-covenant.org/

* Update CODE_OF_CONDUCT.MD

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Create .markdownlintignore

* Revise README for PhysicsNeMo resources and guidance

Updated the 'Getting Started' section and added new resources for learning AI Physics.

* Update README.md

---------

Co-authored-by: Mohammad Amin Nabian <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Corey adams <[email protected]>

* Cleaning up diffusion models. Not quite done yet.

* Restore deleted files

* Updating more tests.

* Fixed minor bug in shape validation in SongUNet (#1230)

Signed-off-by: Charlelie Laurent <[email protected]>

* Add Zarr reader for Crash (#1228)

* Add Zarr reader for Crash

* Update README

* Update validation logic of point data in Zarr reader

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update examples/structural_mechanics/crash/zarr_reader.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Add a test for 2D feature arrays

* Update examples/structural_mechanics/crash/zarr_reader.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Further updates to tests.  Datapipes almost working.

* update import paths

* Starting to clean up dependency tree.

* Add AR RT and OT schemes to Crash FIGConvNet (#1232)

* Add AR and OT schemes for FIGConvNet

* Add tests

* Soothe the linter

* Fix the tests

* Fixing and adjusting a broad suite of tests.

* Update test/domain_parallel/conftest.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Minor fix

---------

Signed-off-by: Charlelie Laurent <[email protected]>
Co-authored-by: Mohammad Amin Nabian <[email protected]>
Co-authored-by: Yongming Ding <[email protected]>
Co-authored-by: ram-cherukuri <[email protected]>
Co-authored-by: Deepak Akhare <[email protected]>
Co-authored-by: Sai Krishnan Chandrasekar <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Charlelie Laurent <[email protected]>
Co-authored-by: Alexey Kamenev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants