small error in hazus bridge autopopulation#102
Open
jinyan1214 wants to merge 35 commits intoNHERI-SimCenter:developfrom
Open
small error in hazus bridge autopopulation#102jinyan1214 wants to merge 35 commits intoNHERI-SimCenter:developfrom
jinyan1214 wants to merge 35 commits intoNHERI-SimCenter:developfrom
Conversation
(will be superseded by a custom download option next)
This module provides a robust solution for downloading and managing damage and loss model files from the DLML repository. It was created to streamline the process of keeping local model files in sync with the specific versions available in the DamageAndLossModelLibrary repository. The module gives users control over which version of the the damage and loss models they want to use. Main Features - Flexible Version Selection: Download models from specific release versions, the latest release, specific commits, or the latest commit - Efficient Caching: Avoid redundant downloads by caching files and only downloading changed content - Progress Visualization: Visual progress bar showing download status and current file - Commit Validation: Validation of commit SHA format to prevent errors - Command-Line Interface: Simple CLI for easy integration with workflows and scripts - Error Handling: Robust error handling for network issues and invalid inputs
Slightly refactor DLML module to facilitate testing.
Fix release_notes rst formatting to avoid Sphinx warnings
Merge developments into master and release v3.7.1
# Conflicts: # pelicun/resources/DamageAndLossModelLibrary # setup.py
It was in setup.py, but got lost in a merge with the most recent develop branch.
- Move dlml.py from pelicun/ to pelicun/tools/ following tools structure - Replace dlml main() function with dlml_update() for better integration - Add DLML folder to gitignore - Add dlml subcommand to CLI with comprehensive usage examples - Update test imports and references to new module location This change aligns dlml with the existing tools pattern (regional_sim) and takes one more step towards a unified CLI interface for all pelicun functionality.
Implement automatic DLML data management during pelicun import to eliminate manual setup requirements while providing helpful update notifications. Key changes: - Auto-download DLML data on first import if missing - Daily version checking with 24-hour caching to minimize API calls - User warnings for available updates using pelicun's warning system - Enhanced cache system with version metadata tracking - Centralized logging configuration in dlml module and CLI logging with automatic stdout output and optional file logging The system gracefully handles network errors and provides clear guidance for manual resolution when needed. All existing CLI functionality is preserved for backward compatibility. Files modified: - pelicun/__init__.py: Added import-time DLML initialization - pelicun/tools/dlml.py: Added check_dlml_version() and check_dlml_data() - pelicun/cli.py: Added CLI logging with --log option for file output - pyproject.toml: Added packaging dependency for semantic versioning - Comprehensive test coverage: 28 new unit and integration tests This eliminates the need for manual "pelicun dlml update" setup while maintaining user control over updates.
Apply comprehensive code quality improvements addressing ruff linting issues while maintaining Python 3.9 compatibility. Key improvements: - Add comprehensive type hints to all functions in dlml.py and cli.py - Add return type annotations (-> None) to all test functions - Fix import sorting and organization across modules - Improve docstring formatting with imperative mood - Fix all EM102 errors by assigning f-strings to variables before exceptions - Add noqa suppressions for Python 3.9 compatibility (UP007) - Simplify nested with statements and clean up unused variables
- Replace os.path usage with pathlib.Path throughout dlml.py and test files - Add module-level path constants (PELICUN_ROOT_DIR, DLML_DATA_DIR) for better maintainability - Update function signatures to accept Union[str, Path] for improved type safety - Refactor test mocking strategy to use pathlib.Path methods instead of os.path functions - Add pytest fixtures for common test setup to reduce code duplication This change improves code maintainability, follows modern Python best practices, and makes the codebase more robust and readable.
Resolve all ruff linting errors through targeted fixes addressing security vulnerabilities, code quality issues, and type safety improvements. Security fixes: - Replace MD5 with SHA256 for secure file hashing (S324) - Add timeout=10 to all requests.get() calls to prevent hanging (S113) - Implement proper exception chaining with 'from' clauses (B904) Code quality improvements: - Replace broad exception handling with specific exception types (BLE001) - Add comprehensive type annotations to test functions and fixtures - Modernize type hints: Dict -> dict, add Generator types - Fix variable naming (N806) - Remove trailing whitespace from blank lines (W293) - Refactor exception messages to use variables instead of literals (EM101) Test improvements: - Add specific match parameters to pytest.raises calls (PT011) - Refactor pytest.raises blocks to single statements (PT012) - Add proper noqa comments for acceptable test patterns (S607, ANN401, DOC402) - Move imports to module level where appropriate (PLC0415) Style and documentation: - Add missing exception documentation to docstrings (DOC501) - Remove extraneous exception references from docstrings (DOC502) - Combine nested with statements for better readability (SIM117) - Add noqa comments for configuration functions with boolean parameters (FBT001/FBT002) This commit resolves 81 ruff errors, bringing the codebase to zero linting violations while maintaining full functionality and improving overall code quality, security, and maintainability.
… to the official NHERI-SimCenter.
- Add GitHub token authentication to dlml.py API calls to increase rate limits from 60 to 1,000 requests/hour when run as part of GitHub Actions - Pin DLML data version to v2.0.0 via .dlml-version file for reproducible builds - Add pytest session-scoped fixture to mock network calls during testing - Consolidate separate workflow files into unified ci.yml with setup job pattern - Implement DLML data caching strategy to prevent download race conditions - Remove obsolete workflow files (tests.yml, lint.yml, format_check.yml, etc.) Resolves CI failures due to GitHub API rate limiting while maintaining test isolation.
…vements - Add pytest-socket to development dependencies for network call blocking - Configure global socket blocking by default in pytest configuration - Fix test isolation issues by implementing intelligent conftest.py fixture: - Mocks DLML data checks globally to prevent network calls - Excludes specific DLML integration tests from global mock - Uses function-scoped fixture with explicit test exclusion list - Enable CI/CD pipeline to run on all pushes and pull requests All 417 tests pass with proper network isolation and improved CI coverage.
- Pin Ruff version to 0.7.0 for consistent linting - Fix Sphinx documentation build path - Upgrade pip in deployment workflow - Add type annotations and improve formatting - Standardize string quotes in dlml.py - Enhance test fixture with proper typing
- Created comprehensive release notes in unreleased.rst - Set version to 3.8.0.dev0 in pyproject.toml
…t-setup Develop dlml at setup
Refactor DLML test suite with comprehensive test coverage including parametrized tests, new test classes for version resolution and file change detection, and extensive error handling scenarios. Enhance DLML module with better CLI detection, improved error formatting, and more robust fallback mechanisms for version resolution. - Add TestResolveVersionToCommitSHA class with comprehensive test coverage - Add TestGetChangedFiles class for file change detection testing - Convert individual tests to parametrized format for better maintainability - Improve error handling with _format_initial_download_error function - Add CLI detection logic to prevent update conflicts - Replace logger with print statements for better user feedback - Update pyproject.toml to skip doc/_build directory in coverage
Complete release preparation for Pelicun v3.8.0 including creation of comprehensive CHANGELOG.md with full project history, release notes migration, and version updates. This establishes a complete documentation foundation for the new DLML integration release. - Create CHANGELOG.md with complete release history from v1.1.0 to v3.8.0 - Move v3.8.0 release notes from unreleased.rst to dedicated file - Update README.md to reference new CHANGELOG.md - Update version from 3.8.0.dev0 to 3.8.0 in pyproject.toml
…t-setup Final PR for v3.8.0: DLML Integration and Comprehensive Documentation
Update version from 3.8.0 to 3.9.0.dev0 to begin development cycle for the next minor release. This prepares the codebase for new feature development and testing in the v3.9 release cycle.
chore: bump version to 3.9.0.dev0
This commit makes the number of nearest neighbors (NNR) used in the Event-to-Building mapping a configurable parameter. The previous hardcoded value of 8 neighbors caused failures when running with event grids containing fewer than 8 points, such as in the new test case. The `regional_sim` tool now reads the `neighbors` value from the input configuration file, allowing for greater flexibility and ensuring the successful execution of tests with small mock datasets. Additionally, this commit introduces a comprehensive integration test for the tool to establish a testing baseline before further refactoring. The new test simulates the end-to-end earthquake workflow and achieves 98% code coverage for the `regional_sim.py` module.
This commit introduces a feature to filter the building inventory based the a string provided in the configuration file. Pelicun recognizes the same comma-separated list of IDs and ranges (e.g., "1, 3-5, 8") that is used in rWHALE. The implementation includes robust parsing and validation, raising a `ValueError` for malformed filter strings or for building IDs that are not found in the inventory. This new functionality is fully covered by a comprehensive, parametrized test suite that validates successful filtering, invalid syntax, and missing ID error cases.
This major refactoring generalizes the regional simulation and NNR scripts to support configurable, non-earthquake hazards like hurricanes. The hardcoded logic for earthquake-specific Intensity Measures (IMs) and loss assessment has been replaced with a more flexible, data-driven workflow. Key changes in `regional_sim.py`: - **Dynamic IM Handling**: The script now reads a list of IMs and their units from the config file, validates input data against them, and dynamically generates a multi-IM demand sample. - **Conditional Loss Assessment**: The loss calculation is now conditional. It uses a dedicated function for the complex Hazus Earthquake logic and a new, efficient 1-to-1 mapping path for methods like Hazus Hurricane Wind. - **Robust Data Processing**: The output logic is now sequential, saving results for demand, damage, etc., one by one. Logic to upsample demand realizations (e.g., from a single deterministic input) has been added. Key changes in `NNR.py`: - **Enhanced Functionality**: The "expected value" mode can now operate on each realization of a 3D input array independently. - **Performance**: The 2D expected value calculation was vectorized, replacing a slow for-loop with a dot product. - **Code Quality**: The function is now fully type-hinted and has a comprehensive docstring. Testing: - A new end-to-end test for the hurricane wind scenario (`PWS`) has been added, driven by a new, dedicated pytest fixture. - Fixtures were refactored to create hazard-specific building inventories. Finally, numerous `ruff` linting issues were resolved across all modified files, improving overall code quality and maintainability.
This commit updates the project's documentation with the new developments. - The docstring for the `NNR` function has been reformatted to be fully compliant with the Sphinx/NumPy docstring standard, resolving all parsing errors that were causing the documentation build to fail. - The `unreleased.rst` file has been populated with a comprehensive summary of the new multi-hazard features and other enhancements included in this development cycle.
…urricane Generalize regional simulation for multi-hazard support
The `dlml` module and its entire test suite had several issues related to testability, network reliability, and user-facing command robustness. Tests were brittle due to a reliance on module-level constants and excessive mocking of the filesystem and internal functions.
This commit introduces a significant refactoring of the `dlml` module, a complete overhaul of its unit and integration tests, and related project configuration changes to address these issues.
Key changes to the source code (`dlml.py`):
- **Improved Testability**:
- Removed module-level constants for configuration to be replaced by new `get_*()` helper functions (e.g., `get_api_repo_url()`, `get_dlml_data_dir()`) that resolve configuration from environment variables at runtime.
- **Increased Robustness**:
- Increased `requests` timeouts from a flat 10s to a (10s connect, 60s read) tuple to better handle slow network conditions.
- Made the recommended `dlml update` command in user prompts more robust by programmatically building it with `shlex.quote(sys.executable)`, ensuring it works on all platforms.
- Improved the logic that detects when the script is being run by the `dlml update` CLI to prevent redundant operations.
Key changes to the test suite (`test_dlml.py` and `test_dlml_integration.py`):
- **Complete Overhaul**: Modernized the entire test suite for the `dlml` module, replacing brittle tests that heavily mocked the filesystem and internal functions with robust tests that validate real behavior.
- **Realistic Filesystem Interaction**: Replaced all filesystem mocking (e.g., `patch('pathlib.Path')`) in both unit and integration tests with interaction with a real temporary filesystem via the `tmp_path` fixture.
- **Smarter Network Mocks**: In the integration suite, replaced simple `requests.get` mocks with a smarter `mock.side_effect` pattern to provide URL-specific responses. This fixed a critical bug where the application logic would overwrite the test's local `model_files.txt`.
- **New End-to-End Test**: Added a true end-to-end test that uses `pytest-httpserver` to simulate the entire download workflow against a live local server.
- **Final Polish**: Standardized docstrings, added comprehensive type hints, and refactored tests to use consistent patterns (e.g., `@patch` decorators) across both files.
Build & Configuration Changes:
- Bumped the project version to `3.8.1` in `pyproject.toml`.
- Added `pytest-httpserver` as a new development dependency.
- Updated the default DLML data version to `v2.1.0`.
- Made the `cli.py` script directly executable.
refactor(dlml): Improve testability, robustness, and test suite
Merge changes in release 3.8.1 into master
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.