Skip to content

Metadetect aliasing - #491

Open
empEvil wants to merge 27 commits into
masterfrom
Metadetect_aliasing
Open

Metadetect aliasing#491
empEvil wants to merge 27 commits into
masterfrom
Metadetect_aliasing

Conversation

@empEvil

@empEvil empEvil commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

This is an update to the ingestion of MetaDetect, to include the full list of columns, and then providing an aliasing list that then adds the aliases to the TXPipe needed columns

@joezuntz

Copy link
Copy Markdown
Collaborator

I'm not sure we want absolutely all the columns - there are millions of them!

I thought the idea was just to make sure that all the columns we did import get an alias to their old name?

@empEvil

empEvil commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

So I was unsure, it was suggested in PO that it was better to do the other way around. I.e. get all columns, and then add our relevant aliases.
there will be 67 columns in DP1/DP2

@empEvil
empEvil requested a review from joezuntz August 19, 2026 10:35

@joezuntz joezuntz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good, just minor changes!

Comment thread txpipe/ingest/dp_info.py
Comment thread txpipe/diagnostics.py Outdated
Comment thread txpipe/diagnostics.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Rubin MetaDetect ingestion to optionally retain the full input column set and then add TXPipe-friendly column aliases into the output shear catalog. It also refactors MetaDetect column selection logic and tweaks diagnostics plotting.

Changes:

  • Add an all_columns stage option and pass it through to process_metadetect_data to control whether all input columns are written.
  • Add a post-write aliasing step to link TXPipe-standard column names to their original MetaDetect column names.
  • Adjust structured-array sanitization allocation and make small diagnostic plot scale tweaks.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
txpipe/ingest/metaDetect.py Adds all_columns config, changes Butler get path, and adds an aliasing post-pass on the output HDF5.
txpipe/ingest/lsst.py Refactors process_metadetect_data to emit either full columns or a reduced set, and adjusts sanitize structured-array allocation.
txpipe/ingest/dp1_info.py Introduces the alias mapping dictionary for TXPipe column names.
txpipe/diagnostics.py Minor plotting scale/comment tweaks for diagnostic plots.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread txpipe/ingest/lsst.py Outdated
Comment thread txpipe/ingest/metaDetect.py Outdated
Comment thread txpipe/ingest/metaDetect.py
Comment thread txpipe/ingest/dp1_info.py Outdated
@empEvil
empEvil requested a review from joezuntz August 19, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants