Skip to content

Move central run_etl into each example #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 21, 2025
Merged

Move central run_etl into each example #38

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bc089b4
Refactored out run_etl.py
saikrishnanc-nv Nov 18, 2025
fda5485
Modified pytest section in CI
saikrishnanc-nv Nov 18, 2025
37bf178
Updated documentation
saikrishnanc-nv Nov 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@ doctest:

pytest:
pip install -e ".[dev]" && \
pytest
Copy link
Collaborator

@Alexey-Kamenev Alexey-Kamenev Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can also do pytest ./tests/?

Copy link
Collaborator Author

@saikrishnanc-nv saikrishnanc-nv Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha - This is a good point to discuss.
Previously that's what was in effect happening, and that worked fine, because the imports were very explicit (full paths relative to the root of the repo).
Now however, that doesn't work because, we don't add examples to the PYTHONPATH anymore.
And therefore, the paths are being modified using sys.path.
Multiple examples have files with the same name (data_sources.py for example). And even if I remove the sys.path modifications at what I think are the right locations - This somehow creates some path pollution.
So my options were:

  1. Ensure file names are unique (hard to guarantee given that users might only care about one example)
  2. Separate the tests out and run them one module at a time

I opted for the 2nd option.
However, if you know of a way to overcome this, I'd be very grateful 😄

pytest tests/test_etl/
pytest tests/test_examples/test_external_aerodynamics/
pytest tests/test_examples/test_structural_mechanics/

coverage:
echo "Not implemented"
Expand Down
22 changes: 11 additions & 11 deletions examples/external_aerodynamics/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,9 +134,9 @@ default processing.

```yaml
surface_preprocessing:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsSurfaceTransformation
_target_: data_transformations.ExternalAerodynamicsSurfaceTransformation
surface_processors:
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.normalize_surface_normals
- _target_: external_aero_surface_data_processors.normalize_surface_normals
_partial_: true
```

Expand All @@ -145,7 +145,7 @@ surface_preprocessing:
```yaml
surface_preprocessing:
surface_processors:
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.non_dimensionalize_surface_fields
- _target_: external_aero_surface_data_processors.non_dimensionalize_surface_fields
_partial_: true
air_density: 1.205 # kg/m³
stream_velocity: 30.0 # m/s
Expand All @@ -157,15 +157,15 @@ surface_preprocessing:
surface_preprocessing:
surface_processors:
# 1. Normalize normals
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.normalize_surface_normals
- _target_: external_aero_surface_data_processors.normalize_surface_normals
_partial_: true
# 2. Then non-dimensionalize fields
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.non_dimensionalize_surface_fields
- _target_: external_aero_surface_data_processors.non_dimensionalize_surface_fields
_partial_: true
air_density: 1.205
stream_velocity: 30.0
# 3. Finally convert to float32
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.update_surface_data_to_float32
- _target_: external_aero_surface_data_processors.update_surface_data_to_float32
_partial_: true
```

Expand All @@ -174,7 +174,7 @@ surface_preprocessing:
You can create your own processors by writing functions that follow this signature:

```python
from examples.external_aerodynamics.schemas import ExternalAerodynamicsExtractedDataInMemory
from schemas import ExternalAerodynamicsExtractedDataInMemory

def my_custom_processor(
data: ExternalAerodynamicsExtractedDataInMemory,
Expand Down Expand Up @@ -236,22 +236,22 @@ for a complete example with filtering enabled:
surface_preprocessing:
surface_processors:
# ... other processors ...
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.non_dimensionalize_surface_fields
- _target_: external_aero_surface_data_processors.non_dimensionalize_surface_fields
_partial_: true
air_density: 1.205
stream_velocity: 30.0
# Apply filter AFTER non-dimensionalization
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.validate_surface_sample_quality
- _target_: external_aero_surface_data_processors.validate_surface_sample_quality
_partial_: true
statistical_tolerance: 7.0 # Filter outliers beyond mean ± 7σ
pressure_max: 4.0 # Max non-dimensional pressure

volume_preprocessing:
volume_processors:
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.non_dimensionalize_volume_fields
- _target_: external_aero_volume_data_processors.non_dimensionalize_volume_fields
_partial_: true
# Apply filter AFTER non-dimensionalization
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.validate_volume_sample_quality
- _target_: external_aero_volume_data_processors.validate_volume_sample_quality
_partial_: true
statistical_tolerance: 7.0 # Filter outliers beyond mean ± 7σ
velocity_max: 3.5 # Max non-dimensional velocity magnitude
Expand Down
28 changes: 14 additions & 14 deletions examples/external_aerodynamics/config/external_aero_etl_drivaerml.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,13 @@ etl:
args: {}

validator:
_target_: examples.external_aerodynamics.dataset_validator.ExternalAerodynamicsDatasetValidator
_target_: dataset_validator.ExternalAerodynamicsDatasetValidator
_convert_: all

validation_level: "fields" # One of "structure" or "fields"

source:
_target_: examples.external_aerodynamics.data_sources.ExternalAerodynamicsDataSource
_target_: data_sources.ExternalAerodynamicsDataSource
_convert_: all

input_dir: ??? # Path to input dataset directory (required)
Expand All @@ -51,57 +51,57 @@ etl:
# These take in the raw geometry, volume and surface data, and apply DoMINO specific preprocessing transformations.
# Variable names are piped in from the variables/drivaerml.yaml or variables/ahmedml.yaml file.
stl_preprocessing:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsSTLTransformation
_target_: data_transformations.ExternalAerodynamicsSTLTransformation
_convert_: all
# Regardless of whether there are any additional geometry processors,
# we always apply the default geometry processing.
# This will ensure that the bare minimum criteria for geometry data is met.
# That is - The geometry data (vertices, faces, areas and centers) are present.
geometry_processors:
- _target_: examples.external_aerodynamics.external_aero_geometry_data_processors.update_geometry_data_to_float32
- _target_: external_aero_geometry_data_processors.update_geometry_data_to_float32
_partial_: true

surface_preprocessing:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsSurfaceTransformation
_target_: data_transformations.ExternalAerodynamicsSurfaceTransformation
_convert_: all
# Regardless of whether there are any additional surface processors,
# we always apply the default surface processing.
# This will ensure that the bare minimum criteria for surface data is met.
# That is - The surface data (mesh centers, normals, areas and fields) are present.
surface_processors:
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.decimate_mesh
- _target_: external_aero_surface_data_processors.decimate_mesh
_partial_: true
algo: decimate_pro # can be one of {decimate_pro, decimate}
reduction: 0.0 # 0 means no decimation.
preserve_topology: false
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.normalize_surface_normals
- _target_: external_aero_surface_data_processors.normalize_surface_normals
_partial_: true
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.non_dimensionalize_surface_fields
- _target_: external_aero_surface_data_processors.non_dimensionalize_surface_fields
_partial_: true
air_density: 1.205 # kg/m³
stream_velocity: 30.00 # m/s
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.update_surface_data_to_float32
- _target_: external_aero_surface_data_processors.update_surface_data_to_float32
_partial_: true

volume_preprocessing:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsVolumeTransformation
_target_: data_transformations.ExternalAerodynamicsVolumeTransformation
_convert_: all
# Regardless of whether there are any additional volume processors,
# we always apply the default volume processing.
# This will ensure that the bare minimum criteria for volume data is met.
# That is - The volume data (mesh centers and fields) are present.
volume_processors:
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.non_dimensionalize_volume_fields
- _target_: external_aero_volume_data_processors.non_dimensionalize_volume_fields
_partial_: true
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.update_volume_data_to_float32
- _target_: external_aero_volume_data_processors.update_volume_data_to_float32
_partial_: true
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.shuffle_volume_data
- _target_: external_aero_volume_data_processors.shuffle_volume_data
_partial_: true

write_ready_transformation: ${override_transformations.write_ready_transformation}

sink:
_target_: examples.external_aerodynamics.data_sources.ExternalAerodynamicsDataSource
_target_: data_sources.ExternalAerodynamicsDataSource
_convert_: all

output_dir: ??? # Path to output directory (required)
Expand Down
32 changes: 16 additions & 16 deletions examples/external_aerodynamics/config/external_aero_etl_drivaerml_filtering.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,13 @@ etl:
args: {}

validator:
_target_: examples.external_aerodynamics.dataset_validator.ExternalAerodynamicsDatasetValidator
_target_: dataset_validator.ExternalAerodynamicsDatasetValidator
_convert_: all

validation_level: "fields" # One of "structure" or "fields"

source:
_target_: examples.external_aerodynamics.data_sources.ExternalAerodynamicsDataSource
_target_: data_sources.ExternalAerodynamicsDataSource
_convert_: all

input_dir: ??? # Path to input dataset directory (required)
Expand All @@ -51,68 +51,68 @@ etl:
# These take in the raw geometry, volume and surface data, and apply DoMINO specific preprocessing transformations.
# Variable names are piped in from the variables/drivaerml.yaml or variables/ahmedml.yaml file.
stl_preprocessing:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsSTLTransformation
_target_: data_transformations.ExternalAerodynamicsSTLTransformation
_convert_: all
# Regardless of whether there are any additional geometry processors,
# we always apply the default geometry processing.
# This will ensure that the bare minimum criteria for geometry data is met.
# That is - The geometry data (vertices, faces, areas and centers) are present.
geometry_processors:
- _target_: examples.external_aerodynamics.external_aero_geometry_data_processors.update_geometry_data_to_float32
- _target_: external_aero_geometry_data_processors.update_geometry_data_to_float32
_partial_: true

surface_preprocessing:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsSurfaceTransformation
_target_: data_transformations.ExternalAerodynamicsSurfaceTransformation
_convert_: all
# Regardless of whether there are any additional surface processors,
# we always apply the default surface processing.
# This will ensure that the bare minimum criteria for surface data is met.
# That is - The surface data (mesh centers, normals, areas and fields) are present.
surface_processors:
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.decimate_mesh
- _target_: external_aero_surface_data_processors.decimate_mesh
_partial_: true
algo: decimate_pro # can be one of {decimate_pro, decimate}
reduction: 0.0 # 0 means no decimation.
preserve_topology: false
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.normalize_surface_normals
- _target_: external_aero_surface_data_processors.normalize_surface_normals
_partial_: true
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.non_dimensionalize_surface_fields
- _target_: external_aero_surface_data_processors.non_dimensionalize_surface_fields
_partial_: true
air_density: 1.205 # kg/m³
stream_velocity: 30.00 # m/s
# Validate surface sample quality has to be applied after non-dimensionalization.
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.validate_surface_sample_quality
- _target_: external_aero_surface_data_processors.validate_surface_sample_quality
_partial_: true
pressure_max: 4.0
statistical_tolerance: 7.0
- _target_: examples.external_aerodynamics.external_aero_surface_data_processors.update_surface_data_to_float32
- _target_: external_aero_surface_data_processors.update_surface_data_to_float32
_partial_: true

volume_preprocessing:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsVolumeTransformation
_target_: data_transformations.ExternalAerodynamicsVolumeTransformation
_convert_: all
# Regardless of whether there are any additional volume processors,
# we always apply the default volume processing.
# This will ensure that the bare minimum criteria for volume data is met.
# That is - The volume data (mesh centers and fields) are present.
volume_processors:
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.non_dimensionalize_volume_fields
- _target_: external_aero_volume_data_processors.non_dimensionalize_volume_fields
_partial_: true
# Validate volume sample quality has to be applied after non-dimensionalization.
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.validate_volume_sample_quality
- _target_: external_aero_volume_data_processors.validate_volume_sample_quality
_partial_: true
velocity_max: 3.5
pressure_max: 4.0
statistical_tolerance: 7.0
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.update_volume_data_to_float32
- _target_: external_aero_volume_data_processors.update_volume_data_to_float32
_partial_: true
- _target_: examples.external_aerodynamics.external_aero_volume_data_processors.shuffle_volume_data
- _target_: external_aero_volume_data_processors.shuffle_volume_data
_partial_: true

write_ready_transformation: ${override_transformations.write_ready_transformation}

sink:
_target_: examples.external_aerodynamics.data_sources.ExternalAerodynamicsDataSource
_target_: data_sources.ExternalAerodynamicsDataSource
_convert_: all

output_dir: ??? # Path to output directory (required)
Expand Down
2 changes: 1 addition & 1 deletion examples/external_aerodynamics/config/override_transformations/numpy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,5 @@
# limitations under the License.

write_ready_transformation:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsNumpyTransformation
_target_: data_transformations.ExternalAerodynamicsNumpyTransformation
_convert_: all
2 changes: 1 addition & 1 deletion examples/external_aerodynamics/config/override_transformations/zarr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# limitations under the License.

write_ready_transformation:
_target_: examples.external_aerodynamics.data_transformations.ExternalAerodynamicsZarrTransformation
_target_: data_transformations.ExternalAerodynamicsZarrTransformation
_convert_: all
compression_method: "zstd"
compression_level: 5
Expand Down
13 changes: 6 additions & 7 deletions examples/external_aerodynamics/data_sources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,18 @@
import pyvista as pv
import vtk
import zarr

from physicsnemo_curator.etl.data_sources import DataSource
from physicsnemo_curator.etl.processing_config import ProcessingConfig

from .constants import DatasetKind, ModelType
from .paths import get_path_getter
from .schemas import (
from constants import DatasetKind, ModelType
from paths import get_path_getter
from schemas import (
ExternalAerodynamicsExtractedDataInMemory,
ExternalAerodynamicsMetadata,
ExternalAerodynamicsNumpyDataInMemory,
ExternalAerodynamicsZarrDataInMemory,
)

from physicsnemo_curator.etl.data_sources import DataSource
from physicsnemo_curator.etl.processing_config import ProcessingConfig


class ExternalAerodynamicsDataSource(DataSource):
"""Data source for reading and writing External Aerodynamics simulation data."""
Expand Down
23 changes: 10 additions & 13 deletions examples/external_aerodynamics/data_transformations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,32 +19,29 @@
from typing import Callable, Optional

import numpy as np
from numcodecs import Blosc

from examples.external_aerodynamics.external_aero_geometry_data_processors import (
from constants import PhysicsConstants
from external_aero_geometry_data_processors import (
default_geometry_processing_for_external_aerodynamics,
)
from examples.external_aerodynamics.external_aero_surface_data_processors import (
from external_aero_surface_data_processors import (
default_surface_processing_for_external_aerodynamics,
)
from examples.external_aerodynamics.external_aero_volume_data_processors import (
from external_aero_utils import to_float32
from external_aero_volume_data_processors import (
default_volume_processing_for_external_aerodynamics,
)
from physicsnemo_curator.etl.data_transformations import DataTransformation
from physicsnemo_curator.etl.processing_config import ProcessingConfig

from .constants import PhysicsConstants
from .external_aero_utils import (
to_float32,
)
from .schemas import (
from numcodecs import Blosc
from schemas import (
ExternalAerodynamicsExtractedDataInMemory,
ExternalAerodynamicsNumpyDataInMemory,
ExternalAerodynamicsNumpyMetadata,
ExternalAerodynamicsZarrDataInMemory,
PreparedZarrArrayInfo,
)

from physicsnemo_curator.etl.data_transformations import DataTransformation
from physicsnemo_curator.etl.processing_config import ProcessingConfig


class ExternalAerodynamicsNumpyTransformation(DataTransformation):
"""Transforms External Aerodynamics data for NumPy storage format (legacy support)."""
Expand Down
8 changes: 2 additions & 6 deletions examples/external_aerodynamics/dataset_validator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,20 +22,16 @@
from typing import Optional

import vtk
from constants import DatasetKind, ModelType
from paths import get_path_getter

from examples.external_aerodynamics.constants import (
DatasetKind,
ModelType,
)
from physicsnemo_curator.etl.dataset_validators import (
DatasetValidator,
ValidationError,
ValidationLevel,
)
from physicsnemo_curator.etl.processing_config import ProcessingConfig

from .paths import get_path_getter


class ExternalAerodynamicsDatasetValidator(DatasetValidator):
"""Validator for External Aerodynamics datasets."""
Expand Down
7 changes: 2 additions & 5 deletions examples/external_aerodynamics/external_aero_geometry_data_processors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,8 @@
import logging

import numpy as np

from examples.external_aerodynamics.external_aero_utils import to_float32
from examples.external_aerodynamics.schemas import (
ExternalAerodynamicsExtractedDataInMemory,
)
from external_aero_utils import to_float32
from schemas import ExternalAerodynamicsExtractedDataInMemory

logging.basicConfig(
format="%(asctime)s - Process %(process)d - %(levelname)s - %(message)s",
Expand Down
Loading