[WIP] - Add SpectralTemplateMatcher class for spectral template matching #318
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements SNID-like spectral template matching functionality: - Match observed spectra against template library with redshift search - Support for Pearson correlation and chi-squared matching methods - Built-in blackbody templates for Type Ia, II, and Ib/c SNe - Load custom templates from CSV/DAT files - Classification with confidence metrics and type probabilities - Visualization of matched templates vs observed spectra Key features: - Redshift grid search over configurable range - Handles wavelength overlap and interpolation - Proper error handling for chi-squared calculations - Extensible template library with add_template method
New template source methods: - get_available_template_sources(): List all available public template repositories - download_templates_from_osc(): Download from Open Supernova Catalog API - from_sesn_templates(): Auto-download METAL/SESNtemple stripped-envelope SN templates - from_snid_template_directory(): Load SNID .lnw template files - download_github_templates(): Generic GitHub repo downloader with caching - parse_snid_template_file(): Parse SNID-format template files Additional features: - save_templates(): Save templates to disk for reuse - filter_templates(): Filter by type and phase range - Local caching in ~/.redback/spectral_templates/ - Automatic metadata extraction from SNID filenames Supported sources: - SNID templates v2.0 (Blondin & Tonry) - Super-SNID (Magill et al. 2025) - METAL/SESNtemple (Williamson et al. 2023) - Open Supernova Catalog API - WISeREP repository
Documentation: - docs/spectral_template_matching.txt: Comprehensive guide covering template sources, loading methods, matching, classification, and API - docs/analysis.txt: Updated to reference spectral template matching - docs/examples.txt: Added link to new example script - docs/index.rst: Added spectral_template_matching to table of contents Example: - examples/spectral_template_matching_example.py: Complete example showing: - Basic template matching with default templates - Full classification with type probabilities - Visualization of matches - Custom template management - Template filtering and saving - Chi-squared vs correlation matching - Advanced usage patterns
Test coverage includes: - Initialization with default, custom, and file-based templates - Template addition and normalization - Spectrum matching with correlation and chi-squared methods - Classification with type probabilities - Plotting functionality - Template I/O (save/load CSV and DAT formats) - SNID template file parsing - Template filtering by type and phase - Available template sources information - Download functionality (with mocking) - Blackbody flux calculation - Edge cases and error handling - Integration tests for complete workflows Total: 50+ test cases across 11 test classes covering: - TestSpectralTemplateMatcherInit - TestSpectralTemplateMatcherAddTemplate - TestSpectralTemplateMatcherMatching - TestSpectralTemplateMatcherClassification - TestSpectralTemplateMatcherPlotting - TestSpectralTemplateMatcherTemplateIO - TestSpectralTemplateMatcherFiltering - TestSpectralTemplateMatcherAvailableSources - TestSpectralTemplateMatcherDownload - TestSpectralTemplateMatcherBlackbodyFlux - TestSpectralTemplateMatcherEdgeCases - TestSpectralTemplateMatcherIntegration
Code fixes (redback/analysis.py): - save_templates: Write CSV with proper format (metadata comments, then header, then data) instead of using np.savetxt comments parameter - _load_templates: Count and skip comment lines (#) plus header row correctly when loading CSV files, and extract metadata from comments Test fixes (test/spectral_template_matcher_test.py): - Use non-constant flux (linear gradient) instead of constant flux in tests to avoid undefined Pearson correlation coefficients - Affects TestSpectralTemplateMatcherClassification, TestSpectralTemplateMatcherPlotting, and TestSpectralTemplateMatcherIntegration test classes All 56 tests now pass successfully.
…8dN8USB5dodxSm8Fs1L
Significantly expanded test coverage from 56 to 97 tests across 14 test classes. New test categories added: TestSpectralTemplateMatcherCoverageExtensions (40+ tests): - DAT format loading with comments and metadata - Filename parsing edge cases (single part, invalid phase, + in phase) - DAT format saving with metadata headers - NaN handling in flux arrays - Chi2 matching without/with errors - Exception handling in Pearson correlation - Negative correlation handling in classification - SNID template parsing for all SN types (Ia, Ib, Ic, II, IIn, etc.) - Empty SNID file error handling - Metadata extraction from comment lines - Mixed whitespace parsing - Template directory loading (.txt files, invalid files) - Plot template matching by type/phase fallback - Scale factor usage in plotting - Blackbody flux at extreme temperatures - GitHub template caching behavior - Corrupted file handling in template loading TestSpectralTemplateMatcherRealWorldScenarios (8 tests): - Realistic Type Ia spectrum matching with noise - Pipeline processing of multiple spectra - Redshift search precision (coarse vs fine grid) - Template library round-trip save/load - Combined correlation+chi2 analysis - Narrow wavelength range spectra - High redshift matching TestSpectralTemplateMatcherConstantsAndPhysics (4 tests): - Blackbody unit consistency - Overflow protection in Planck function - Template normalization verification - Redshift calculation correctness This significantly improves code coverage across all code paths including: - Error handling branches - Edge cases in file parsing - Different matching methods - Template I/O operations - Physical calculations
Add 38 new comprehensive tests covering: - from_sesn_templates() class method with mocking - download_templates_from_osc() edge cases (network errors, parsing, missing data) - download_github_templates() actual download flow and error handling - _load_templates() error handling for malformed metadata - parse_snid_template_file() edge cases (empty values, lowercase metadata) - match_spectrum() with missing attributes and extreme redshifts - Property-based tests for template consistency and structure Total test count: 97 -> 135 tests (38 new tests) All tests pass successfully.
Add 30 new tests that execute actual code paths without mocking: - parse_snid_template_file filename parsing (numeric phases, ValueError branches) - _load_templates metadata override, CSV/DAT format handling - match_spectrum 'both' method, chi2 scaling, zero flux masking - classify_spectrum type probability calculations - from_snid_template_directory fallback pattern matching - plot_match template lookup fallback paths - filter_templates boundary conditions - save_templates directory creation These tests increase actual code coverage by exercising real implementation paths rather than mocking external dependencies. Test count: 135 -> 165 tests
Include the new test file in Group 2 of the CI test matrix so that the 165 tests are actually run and contribute to coverage metrics. Without this, the test file exists but is not executed by CI, causing the coverage to not reflect the actual test coverage.
Add 34 more tests (165 -> 199) focusing on executing real code paths in analysis.py for improved coverage: - Tests for actual OSC download processing with mocked network - Tests for GitHub download full flow with proper zip extraction - Tests for parse_snid_template_file covering all branches (ValueError, IndexError, lowercase metadata, malformed lines) - Tests for _load_templates covering CSV and DAT metadata override paths - Tests for match_spectrum edge cases (no overlap, insufficient points, correlation exceptions) - Tests for classify_spectrum probability calculation - Tests for from_snid_template_directory fallback pattern matching - Tests for save_templates format verification - Tests for filter_templates returning new matcher instance These tests use minimal mocking (only network I/O) to maximize actual code execution for better coverage metrics.
…8dN8USB5dodxSm8Fs1L
Prioritize master's workflow configuration: - Restore test-group: [1, 2, 3, 4, 5] with Group 5 - Restore full Group 4 with all test files from master - Restore coverage-db artifact handling (not coverage-data) - Restore proper .coverage file combining logic - Restore COVERALLS_FLAG_NAME: combined Keep only the addition of spectral_template_matcher_test.py to Group 2.
…8dN8USB5dodxSm8Fs1L
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements SNID-like spectral template matching functionality:
Key features: