Add stages to compute and plot Delta Sigma - #428
Open
joezuntz wants to merge 79 commits into
Open
Conversation
Merge DESI Ingestor into WIP Delta-Sigma branch
… real DESI catalog as well
joezuntz
marked this pull request as ready for review
July 14, 2026 12:05
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces new pipeline stages and utilities to ingest DESI/Flagship inputs, build joint masks, compute Delta-Sigma (excess surface density) using dsigma, generate a corresponding theory curve (via dsf), and plot the results, integrating these stages into example pipelines.
Changes:
- Added
TXDeltaSigma,TXDeltaSigmaTheory, andTXDeltaSigmaPlotsstages plus example pipeline/config wiring. - Added DESI ingestion/selection stages and new mask/catalog-cutting helpers to support DESI + shear joint analyses.
- Centralized SACC metadata/provenance writing (
SACCFile.add_metadata) and adjusted SACC loading behavior.
Reviewed changes
Copilot reviewed 20 out of 22 changed files in this pull request and generated 17 comments.
Show a summary per file
| File | Description |
|---|---|
| txpipe/utils/conversion.py | Adds photometry conversion/SNR helper utilities used by new ingestion. |
| txpipe/twopoint.py | Uses centralized SACC metadata writer; replaces sys.exit with exceptions. |
| txpipe/source_selection/simple.py | Adjusts band-variant selection catalog type for “simple” shear catalogs. |
| txpipe/source_selection/base.py | Removes an unused/required input tag from the source selector base stage. |
| txpipe/random_cats.py | Updates depth-map pixelization handling; adds an nside consistency check. |
| txpipe/masks.py | Adds joint-mask stage(s) and catalog footprint cutting stages. |
| txpipe/lens_selector.py | Adds DESI selector stages (FITS + Parquet mocks + multi-mock globbing). |
| txpipe/ingest/flagship_mocks.py | Adds Flagship mock shear catalog + mask ingestion stages for DESI mock workflows. |
| txpipe/ingest/dp1.py | Adds a TXIngestDataPreview2 alias stage. |
| txpipe/ingest/desi.py | Adds DESI ingest stages for selected DESI catalogs and DESI randoms. |
| txpipe/ingest/init.py | Exposes new ingest stages via package imports. |
| txpipe/delta_sigma.py | Implements Delta-Sigma measurement, theory generation, and plotting stages. |
| txpipe/data_types.py | Adds SACCFile.add_metadata; switches SACC read path to sacc.Sacc.load. |
| txpipe/init.py | Exposes new stages at top-level import. |
| examples/metadetect/pipeline.yml | Wires Delta-Sigma stages into the metadetect example pipeline. |
| examples/metadetect/config.yml | YAML boolean normalization for TXShearCalibration. |
| examples/desi_mocks/pipeline.yml | New example pipeline for DESI mocks + Delta-Sigma flow. |
| examples/desi_mocks/config.yml | New config for DESI mocks pipeline (selection, ingest, Delta-Sigma). |
| examples/desi_dr1_stage3/pipeline.yml | New example pipeline wiring for DESI DR1 stage-3 + Delta-Sigma. |
| examples/desi_dr1_stage3/config.yml | New config for DESI DR1 stage-3 example. |
| bin/environment-perlmutter.yml | Bumps healsparse/sacc; adds dsigma and dsf dependency sources. |
| bin/environment-local.yml | Bumps healsparse/sacc; adds dsigma and dsf dependency sources. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Collaborator
|
I commented in some of the copilot reviews, but I think you've addressed most of them in later pushes. So I think it all looks ok now? |
empEvil
reviewed
Aug 19, 2026
empEvil
left a comment
empEvil
left a comment
Contributor
There was a problem hiding this comment.
Looks great, but there are a lot of todo's that we might want to address first
| ("lens_photoz_stack", QPNOfZFile), | ||
| ] | ||
|
|
||
| # TODO: CHANGE CONFIG OPTIONS |
| zbin = np.digitize(z, zbin_edges) - 1 | ||
| zbin[zbin == nbin_lens] = -1 | ||
|
|
||
| # # can select on any other criterion here, e.g. |
|
|
||
| # Add columns to two tables in-place to do most of the pre-computation work. | ||
| # We should look at using the n_jobs multiprocessing option here | ||
| # but I don't know if it will play well with MPI on NERSC that we are |
|
|
||
| def load_source_table(self, bin_index): | ||
| """ | ||
| Load the lens table for a given bin index |
|
|
||
| print("CCL Cosmology:", cosmo) | ||
|
|
||
| # TODO expose these as config options |
| x = sacc_data.get_tag("rp", tracers=(f"source_{s}", f"lens_{l}")) | ||
| x = np.array(x) | ||
| y = sacc_data.get_mean(tracers=(f"source_{s}", f"lens_{l}")) | ||
| # raw_y = sacc_data.get_tag("raw_value", tracers=(f"source_{s}", f"lens_{l}")) |
Contributor
There was a problem hiding this comment.
if not used can we delete?
| axes[l, 0].errorbar( | ||
| x * shift[s], y * x, yerr=error * x, fmt="+", label=rf"$\Delta\Sigma$ s={s}" | ||
| ) | ||
| # Also plot the raw value with boost/randoms |
| Select lens galaxies from a DESI spectroscopic FITS catalog. | ||
|
|
||
| Reads a raw DESI FITS catalog and applies: | ||
| TODO: ADJUST NECESSARY CUTS FOR REAL DATA |
Contributor
There was a problem hiding this comment.
Will this be part of this PR or a subsequent one?
| with self.open_input("mask", wrapper=True) as f: | ||
| self.mask = f.read_mask("mask") | ||
| self.mask_nside = f.read_map_info("mask")["nside"] | ||
| # self.mask_fmt = f.read_map_info("mask")["nest"] |
Contributor
There was a problem hiding this comment.
If we are not using lets remove this?
| start = end | ||
|
|
||
|
|
||
| class TXDESIMultiMockSelector(TXDESIMockSelector): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Someone who knows more about this area should check what I have done here!