AnaCal ingestion - #446
Conversation
2. correct for the typo in calibration_calibrators.py, det_respponse_e1 3. in conversion.py f was not defined, changed to flux
|
It seems that i do not have permission to modify the pr here; therefore i opened a pr to correct for some bugs/typo in this pr: |
Two equivalent bookkeeping conventions give an unbiased shear:
(A) <e> = Σ(wsel·e) / N
R = Σ(de/dg·wsel + dw/dg·e) / N
→ R_sel normalized by count.
(B) <e> = Σ(wsel·e) / N / <wsel>
R = Σ(de/dg·wsel + dw/dg·e) / N / <wsel>
→ R_sel normalized by sum(wsel).
The rest of the code — R_shape, R_weight, and every mean-e
computation — uses convention (B), so R_sel must match with a
sum_weights denominator, not count.
Also: nanojansky_err_to_mag_ab returns NaN for flux <= 0 via
np.divide(where=flux>0) instead of producing negative sigma_mag
and RuntimeWarnings.
anacal ingest bug correction
|
@mr-superonion I've added you to the team that gives you permissions for future things. |
as a second stage after ingest, with a matching config.yml block:
delta_gamma: 0.01, mask_threshold: 40, s2n_cut: 5.0, T_cut: 0.1,
5-bin source_zbin_edges, input_pz: true.
2. Ingested photo-z point estimates. TXIngestAnacal now reads
zmode_{0,1p,1m,2p,2m} from the merged catalog and exposes them on the
shear catalog as mean_z, mean_z_{1p,1m,2p,2m} — baseline for
tomographic binning, the four variants for the R_sel bin-migration
term.
3. Built ±γ variants of every quantity the selector cuts on. In
TXSourceSelectorAnaCal.apply_simple_redshift_cut, injected
s2n_{1p,1m,2p,2m} = s2n ± δγ · ds2n_dg{1,2}, and analogous variants
of m00, m20 (from dm00_dg, dm20_dg) and zbin (from mean_z_*). All
four cut quantities now respond to shear via a single uniform
injection step.
4. Unified the calculator's cut logic under _DataWrapper. Deleted the
ad-hoc get_submask size-cut machinery in AnaCalCalculator.add_data.
The selector now applies mask + s2n + size + zbin in one place; the
calculator just runs the same selector on five wrappers (baseline +
_1p/_1m/_2p/_2m), and the wrapper's suffix-lookup routes the shifted
columns automatically. AnaCalCalculator.__init__ dropped its
mask_threshold argument as a consequence.
5. Consistent response formulae. Rewrote R_shape, R_weight, R_sel_{1,2}
in the calculator so all three share the same code shape: Σ(w · X) /
sum_weights for R_shape and R_weight, and [Σ(w·e)|sel_+ −
Σ(w·e)|sel_−] / (2 δγ · sum_weights) for R_sel. R_shape switched from
a ParallelMean to a plain scalar sum for parity; det_response_*
renamed to weight_response_* to match the "R_weight" label; a small
_reduce helper replaced 30 lines of MPI reduce boilerplate.
6. Promoted the size threshold to config. The AnaCal size cut,
previously hard-coded as (m00 + m20) / m00 > 0.1, is now > T_cut —
driven by the base selector's T_cut config option that was previously
unused for AnaCal. TXSourceSelectorAnaCal.config_options gives it a
default of 0.1 alongside a new s2n_cut default of 5.0, so a minimal
yaml block still runs; both can be overridden in the yaml for
tighter/looser cuts.
mr-superonion
left a comment
mr-superonion
left a comment
There was a problem hiding this comment.
I made a few update to include the selection bias correction for s2n and add selection stage into the example. I will check over the dp1 data to see whether i can get consistent result to the hsc code.
| pz_data["s2n_1m"] = s2n - dg * shear_data["ds2n_dg1"] | ||
| pz_data["s2n_2p"] = s2n + dg * shear_data["ds2n_dg2"] | ||
| pz_data["s2n_2m"] = s2n - dg * shear_data["ds2n_dg2"] | ||
|
|
There was a problem hiding this comment.
here is the code to create shifted columns for s2n.
| shear_cols += ["mean_z"] | ||
| # Baseline photo-z + the four shifted variants used to derive | ||
| # zbin_{1p,1m,2p,2m} for the selection-response calculation. | ||
| shear_cols += ["mean_z", "mean_z_1p", "mean_z_1m", "mean_z_2p", "mean_z_2m"] |
There was a problem hiding this comment.
Here we need columns for the shifted redshifts to get the selection bias due to the binning.
| data_1p = _DataWrapper(data, "_1p") | ||
| data_1m = _DataWrapper(data, "_1m") | ||
| data_2p = _DataWrapper(data, "_2p") | ||
| data_2m = _DataWrapper(data, "_2m") |
There was a problem hiding this comment.
data wrapper is used here to get different shifted version of the catalog
| The indicies of the objects selected from this chunk of data | ||
| """ | ||
| data = _DataWrapper(data, "") | ||
| select = self.selector(data, *args, **kwargs) |
There was a problem hiding this comment.
thanks for the explanation
- i am implementing it here (e112dea#r3592442390) to create
s2n_1p,s2n_1m,s2n_2p,s2n_2m - then the data wrapper is used here (e112dea#r3592459804) to create different shifted version
| # (zmode_{1p,1m,2p,2m}) in xlens' photoZPipe (currently ±0.01 in | ||
| # DISTORTIONS). | ||
| delta_gamma: 0.01 | ||
|
|
There was a problem hiding this comment.
i add the selection stage in the example.
There was a problem hiding this comment.
But we don't want to run them at the same time. Sadly from how things are set up on NERSC, we want to run the ingestion stage in one environment and then the rest of the pipeline should be run in a dedicated environment for TXPipe. But we want to be able to run the Selection stage later, where we might be able to use different photo-z's for example
There was a problem hiding this comment.
thanks for the comment and i have separated the ingestion from the rest of the task in the examples folder (see commit a21049e)
|
@mr-superonion Thank you for the additions, I however do not think we should move away from parallel mean if we can avoid it! I am a little confused about what has been moved from the calibration stage to the source-selection, and why? |
2. use consistent definition with paper e_mea = e_raw * wsel and g= e_mea / R_total wher <R_total> = sum (R_shape + R_detect) / Ngal + R_sel; and weight is set to 1. This ensures that treecorr get the correct correlation <e_mea_a e_mea_b> / <R_total>^2
|
Hi @empEvil , thanks for your comments:
Yes. good point and i revert it back to use parallel for all the average computations for shear responses.
my understanding is that the selection was applied twice: once in the selector (for mask_value and s2n cut), once again in the calculator (for trace cut). The trace cut's shear response was computed; however the s2n cut's selection shear response was not included. My update puts all the code to create sheared variant columns into the selection stage (including variants for s2n, trace and redshift bins). The selection for different sheared variants are in calculator to compute the selection shear response using the shear variant columns provided by the selector. I think this makes it easy to maintain since all the shear variants preparation code is in one function (#446 (comment)) and we can use the existing wrapper for the selection response calculation following the suggestion: #446 (comment) |
| zbin variants are only produced under ``input_pz`` (truth-z has no | ||
| shear response); s2n and moment variants are always emitted since | ||
| the selector applies those cuts in every mode. | ||
| """ |
There was a problem hiding this comment.
Here is the function to add sheared variant columns that are used to compute selection bias
response); 2. add selection on |e| (with selection response); 3. the magnitude is calculated from anacal code with smooth truncation at 40;
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end support for ingesting and calibrating DP2-style AnaCal shear catalogs in TXPipe, including a new ingest stage, a source selector/tomography stage, and a corresponding shear calibration calculator/calibrator.
Changes:
- Added a new AnaCal ingest stage that reads merged AnaCal outputs and writes a TXPipe-format shear catalog.
- Added an AnaCal source selector that builds ±γ selection variants and computes AnaCal calibration statistics.
- Added AnaCal calibration plumbing (calculator + calibrator), plus tests and example pipelines/config.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| txpipe/utils/conversion.py | Makes magnitude-error conversion robust to non-positive flux values. |
| txpipe/utils/init.py | Reformats conversion imports (no behavior change). |
| txpipe/test/test_cal.py | Adds unit tests covering AnaCalCalculator response accumulation behavior. |
| txpipe/source_selection/anacal.py | Introduces AnaCal-specific source selection, shear-variant injection, and response output wiring. |
| txpipe/source_selection/init.py | Exposes TXSourceSelectorAnacal from the package. |
| txpipe/shear_calibration/calibrators.py | Adds AnaCalibrator and registers it in Calibrator.load. |
| txpipe/shear_calibration/calibration_calculators.py | Adds AnaCalCalculator implementation. |
| txpipe/shear_calibration/init.py | Re-exports AnaCalCalibrator/AnaCalCalculator. |
| txpipe/ingest/dp1_details.py | Adds DP1 tract/field constants used by ingest selection. |
| txpipe/ingest/anacal.py | Adds TXIngestAnacal stage to ingest AnaCal merged catalogs into TXPipe HDF5. |
| txpipe/ingest/init.py | Exposes TXIngestAnacal from the ingest package. |
| txpipe/data_types.py | Adds primary shear-column mapping for catalog_type == "anacal". |
| txpipe/calibrate.py | Adds calibrate-stage dispatch for catalog_type == "anacal". |
| txpipe/init.py | Exposes TXSourceSelectorAnacal from top-level txpipe. |
| examples/anacal/pipeline.yml | Adds an example pipeline running the AnaCal source selector. |
| examples/anacal/ingest.yml | Adds an example pipeline running only AnaCal ingest. |
| examples/anacal/config.yml | Adds an example config for AnaCal ingest + selection. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def __init__(self, R, mu, mu_is_weighted=True): | ||
| self.R = R | ||
| if mu_is_weighted: | ||
| self.mu = np.array(mu) | ||
| else: | ||
| self.mu = np.array(mu) / R | ||
| assert("Anacal needs an already calibrated mu.") |
| if not subtract_mean: | ||
| g1, g2 = weights * [g1, g2] / self.R | ||
| elif np.isscalar(g1): | ||
| g1, g2 = weights * [g1, g2] / self.R - self.mu | ||
| else: | ||
| g1, g2 = weights * [g1, g2] / self.R - self.mu[:, np.newaxis] | ||
| return g1, g2 |
| outfile = super().setup_output() | ||
| n = outfile["tomography/bin"].size | ||
| group = outfile.create_group("response") | ||
| group.create_dataset("R", (n, 1, 1), dtype="f") | ||
| group.create_dataset("R_2d", (1,), dtype="f") | ||
| return outfile |
| def write_tomography(self, outfile, start, end, source_bin, r): | ||
| super().write_tomography(outfile, start, end, source_bin, r) | ||
| group = outfile["response"] | ||
| group["R"][start:end] = r | ||
|
|
| print("Trimming shear columns:") | ||
| for col in shear_data.keys(): | ||
| print(" ", col) | ||
| h5py_shorten(shear_outfile["shear"], col, shear_end) | ||
|
|
||
| shear_outfile.close() | ||
|
|
|
So I noticed we are still not ingesting g1/g2 and T for the PSF? @mr-superonion you added those to the catalog right? |
This is a pull request for ingesting the DP2 version of AnaCal. currently it is ingesting a DP1 version of the catalog