Binospec updates and first draft of IFU support#2080
Conversation
|
just ran this on the binospec MOS dataset i've been using for benchmarks and there is a definite speed-up! wall time dropped from 3691 sec to 2958 sec. both cases using python 3.13 with the macos Accelerate framework hooks for |
|
in terms of dithering, here's the blurb from the instrument website:
|
|
going to have to change tack on how the fiber illumination correction is being handled. need to fold it into the pixelflat or create an illumflat so that it gets folded properly into the sky model. more changes incoming... |
something about the IFU data is messing up overscan subtraction. sky fiber traces in some regions are getting significantly over-subtracted. the regions look normal in the raw data and in the IDL pipeline processed data. claude came up with a plan to port the way IDL handles overscan over. pypeit was doing it in a relatively rudimentary way that hadn't been touched in years. |
|
the binospec bad pixel mask was badly, badly out of date. the code comments even said so! fixed that here because it was messing up making IFU maps. |
|
re-ran the binospec dev suite tests for all modes and they all pass. and they run quicker than they used to. inspected the results by hand and they look good, at least to my eye. the improvement from updated BPM and better overscan handling is apparent when compared to old dev suite reductions i had on hand. maybe not so much in the final 1D products, but definitely in the 2D data. |
|
converted this back to draft because i ran into limits of the current approach of treating each fiber as a slit. pypeit's assumptions about slits having spatial information runs pretty deep. trying a different tack where each group of fibers is treated like a MOS slit and then the fibers manually extracted as objects. |
tbowers7
added
the
New Spectrograph
Load a pre-built static BPM from the IDL pipeline calibration data (badpix_binospec.fits plus hard-coded bad columns and detector trap regions from bino_mosaic.pro) instead of the previous hard-coded per-detector lists. Companion to PR pypeit#2102.
Rewrite binospec_read_amp() overscan subtraction to match the IDL pipeline (bino_mosaic.pro): Y-axis postscan first, then X-axis prescan+postscan, using sigma-clipped mean with outlier cleaning. Add per-amplifier nonlinearity correction using degree-4 polynomial coefficients from the IDL calibration file scicam_bino_sep2017.fits. Add clean_overscan_vector() helper in procimg. Companion to PR pypeit#2101.
Extend mmt_binospec.py with MMTBINOSPECIFUSpectrograph, the Fiber-pypeline instrument class for the Binospec IFU mode. Covers: mosaic geometry and detector container, instrument/config metadata, fiber reference profile matching, fiber metadata lookup (MASKDEF_ID / MASKDEF_OBJNAME), block-slit edge definition from the reference profile, sky-line-based fiber illumination correction, scattered-light subtraction in inter-slit gaps, and the calibration-data FITS files (fiber_ref_profile, fiber_illumination). Add a ``"gaps"`` option to ScatteredLightPar valid methods and wire it through RawImage.subtract_scattlight() so Binospec science frames can measure scattered light per-frame from inter-slit gap pixels. Optimize WaveTilts.fit2tiltimg_work to evaluate the 2D tilt polynomial only at pixels inside each slit rather than over the full image. Needed for Binospec's 42 block-slits, which would otherwise blow out memory.
Introduce a new pypeline targeted at fiber-fed spectrographs where each fiber IS the object (no peak-detection object finding). The Fiber pypeline shares its calibration flow with SlicerIFU but implements its own object-finding, extraction, and flat-fielding: - FiberFindObjects (find_objects.py): one SpecObj per fiber with the trace at the fiber center (midpoint of slit edges), using joint sky subtraction inherited from SlicerIFUFindObjects. Handles per-block narrow-fiber masking across the joint sky fit. - FiberExtract (extraction.py): boxcar and Horne (1986) optimal extraction built from empirical flat-derived spatial profiles, using the global sky model directly (no local sky subtraction). - FiberFlatField + FiberFlatImages (flatfield.py, datamodel.py): superflat from science fibers only, per-fiber throughput captured in a separate fiberflat. Dispatched from IFUCalibrations.get_flats() in calibrations.py. - get_fiber_metadata() and get_arc_extract_center() hooks on Spectrograph, letting instruments map detected traces to instrument- defined fiber IDs/names/types (populating MASKDEF_ID / MASKDEF_OBJNAME on each SpecObj) and snap the initial wavelength-calibration extraction centers to the nearest fiber instead of the block-slit midpoint. - Register ``Fiber`` in the pypeline dispatch validation lists in specobj.py, specobjs.py, slittrace.py, pypeit_steps.py, and show_2dspec.py. - Tests for FiberFlatField and block-slit fiber extraction.
Add a dedicated script that builds a 3D datacube from spec1d outputs for the Binospec IFU, using the instrument's sky-fiber layout and per-fiber throughput corrections from the Fiber pypeline. Register the script in scripts/__init__.py and drop the required support hooks into coadd3d, core/datacube, and core/flat.
Add mmt_binospec.rst and mmt_binospec_pipeline_comparison.rst, update spectrographs.rst, and expand the 2.1.0dev release notes with Fiber pypeline and Binospec IFU entries. Update CLAUDE.md with the Fiber pypeline and Binospec IFU architecture notes. Exclude claude_docs from packaging (MANIFEST.in, pyproject.toml) and refresh environment.yml.
| # The following are added for SlicerIFU spectrographs, as they are | ||
| # needed by the coadd3d routine | ||
| if self.pypeline == "SlicerIFU": | ||
| if self.pypeline in ["SlicerIFU", "Fiber"]: |
There was a problem hiding this comment.
Might it be a good idea to relabel this as "FiberIFU", to distinguish it in the future from Fiber based MOS?
There was a problem hiding this comment.
i went with Fiber because in terms of the raw 2D data, fiber-based spectrographs are largely the same. the IFU part only comes into play at the end when dealing with extracted 1D spectra (e.g. cube-building, co-adding regions).
tepickering
force-pushed
the
binospec_updates
branch
from
a989515 to
279c899
Compare
# Conflicts: # pypeit/flatfield.py # pypeit/wavetilts.py
New CLI script that opens an interactive matplotlib GUI showing the MMT
Binospec IFU as a hex grid of fibers colored by per-fiber integrated
flux. Users select fibers via rectangle drag, circle drag, or
individual click and the combined extracted 1D spectrum is written as a
OneSpec FITS file compatible with pypeit_show_1dspec, pypeit_coadd_1dspec,
and other downstream tools.
Implementation in pypeit/scripts/binospec_ifu_extract.py:
- Helpers (all unit-tested via fakes, no disk I/O required):
* _sky_line_mask, _SKY_LINE_WAVELENGTHS -- bright optical sky lines
excluded from the hex-display flux integration so that residual
sky-subtraction errors don't bias the visualization.
* _project_to_sky -- TAN WCS projection matching binospec_ifu_cube,
so the hex view aligns with the corresponding datacube.
* _compute_fiber_fluxes -- per-fiber wavelength-range integration on
each fiber's native grid, with sky-line masking.
* _resample_and_combine -- common-grid linear resample + inverse-
variance combine of selected fibers; raises PypeItError on no
overlap.
* _load_fibers -- reads SpecObjs into per-fiber dicts, drops sky/
unmatched fibers, picks up sky positions from load_sky_layout.
* _write_onespec -- writes the result as a OneSpec FITS file.
- BinospecIFUExtract(scriptbase.ScriptBase) -- CLI parser + main()
with --boxcar (BOX instead of OPT) and -o/--output options. Default
output filename replaces spec1d_ with extract1d_.
- _ExtractGUI -- matplotlib widget GUI with rectangle/circle/click
selection modes, wavelength RangeSlider with live preview, sky-line-
masked hex coloring, and Extract/Save/Reset buttons. matplotlib is
imported lazily so --help is fast.
Includes 25 unit tests covering all helpers (sky-line masking, sky
projection, per-fiber flux integration, resample-and-combine variance
propagation, fiber loader sky/unmatched filtering, OneSpec round-trip,
parser defaults). GUI behavior is exercised manually since the
matplotlib widgets don't lend themselves to unit testing.
Sky-line masking is applied only to the hex-display flux integration;
extraction itself uses the full wavelength range. Sky fibers are
filtered out before fiber records are built.
Registered as pypeit_binospec_ifu_extract in pyproject.toml (GUI
scripts section). Documented in doc/spectrographs/mmt_binospec.rst
and noted in doc/releases/2.1.0dev.rst under Script Changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…x fallback for fiber-flat filenames; fix rst bug; ruff cleanup of some unused variables and imports
…ames in spec2d files are generated so that SCIIMG-SKYMODEL is meaningful; add function to help mitigate some of the nasty CRs that show up in binospec that lacosmic can't handle
The previous _long_diagonal_arc_cr_mask Hough-binning detector was narrowly targeted at the JADES DET02 trail. Replace it with a row-median subtraction followed by LA Cosmic on the residual (sigclip=10, maxiter=2, grow=2.0, objlim=0, remove_compact_obj=False). This is direction-agnostic, multi-trail-capable, both-detector, and works on arc and tilt frames. See claude_docs/specs/2026-05-15-binospec-arc-lacosmic-residual-design.md.
…peItPar The row-median + LA Cosmic CR cleanup applies to both MOS and IFU Binospec modes, so move clean_calibration_image from MMTBINOSPECIFUSpectrograph to the base MMTBINOSPECSpectrograph. Expose all four tunables (sigclip, lamaxiter, grow, cr_median_width) via the standard ProcessImagesPar keys on par['calibrations']['arcframe']['process'] and ['tiltframe']['process'] so users can override them in .pypeit files. Add new key cr_median_width to ProcessImagesPar; 0 disables the median pre-step. The base Spectrograph.clean_calibration_image hook now accepts an optional process_par; calibrations.py passes the relevant frame's process par at both call sites. Docs updated to describe the cleanup in the general Binospec section (not the IFU-specific one), and a changelog entry added.
Move the row-local-median + LA Cosmic residual algorithm from mmt_binospec.clean_calibration_image into a general core function pypeit.core.procimg.lacosmic_spatial_median_residual, and dispatch to it from PypeItImage.build_crmask when par['cr_median_width'] > 1. Masked pixels are filled with the row-local median. Drops the spectrograph.clean_calibration_image hook and its call sites in calibrations.py. Any frame type (arc, tilt, science, ...) can now opt in via the standard [...][process] block by setting mask_cr=True and cr_median_width to an odd window width. Binospec arc/tilt defaults updated to enable the new path; behavior is unchanged. Test moved to test_procimg.py and exercises the core function directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New core function pypeit.core.procimg.lacosmic_spatial_median_residual: subtracts a row-local median filter (size=(1, cr_median_width)) and runs L.A.Cosmic on the residual. Catches the bodies of long, narrow CR trails that the standard 3x3 Laplacian misses; well-suited to line-rich frames such as arcs and tilts. PypeItImage.build_crmask dispatches to it when par['cr_median_width'] > 1, filling masked pixels with the row-local median. Standard lacosmic path is unchanged for cr_median_width=0. New ProcessImagesPar.cr_median_width parameter; usable on any frame type (arc/tilt/science/...) via the standard [...][process] block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… settings. still clips the very brightest lines in a few places, but overall the best compromise that still gets rids of the nastiest crs
The residual-CR path used to replace flagged pixels in `self.image` with the row-local median, while standard L.A.Cosmic only sets the CR bit. That divergence let the residual path mask a latent bug: `BuildWaveCalib` and `BuildWaveTilts` build their bad-pixel mask from BPM only, so when standard L.A.Cosmic flagged CRs the arc/tilt centroiders still saw the contaminated pixel values. Bring both CR paths in line and fix the consumer side: - `lacosmic_spatial_median_residual` now returns just `crmask`; the caller no longer overwrites `self.image`. Both CR paths set only the CR bit in `fullmask`. - `BuildWaveCalib.__init__` and `BuildWaveTilts._setup_slits` mask `['BPM', 'CR']` so flagged pixels are excluded from line/tilt centroiding, resolving the existing TODO in `wavetilts.py`. - Tests updated to the new single-return signature and to check that bright arc rows stay mostly unflagged after the row-local median pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
# Conflicts: # doc/releases/2.1.0dev.rst
Adopts the mask-only residual-CR design from row_median_cr_reject: build_crmask and lacosmic_spatial_median_residual now only set the CR bit in fullmask; image pixel values are left unmodified, and the auto-merged wavecalib/wavetilts changes from row_median_cr_reject teach those callers to respect the CR flag. - Resolved procimg.py, pypeitimage.py, pypeitpar.py, test_procimg.py in favor of the mask-only API (drop the (crmask, median_image) tuple return and the in-place image fill). - Replaced the duplicate verbose changelog entry under Instrument-specific Updates with a one-line note that the residual-CR path is enabled by default (cr_median_width=31) for MMT/Binospec arc and tilt frames, pointing at the new :ref:`image_proc-cr` section. - Updated the stale "masked pixels are filled with the row-local median" comment in mmt_binospec.py to describe the mask-only behavior.
The right-side spectrum panel now tracks the slider window: xlim is set to (current_wave_min, current_wave_max) and ylim is rescaled from the flux values inside that window, so an off-window outlier no longer dominates the y-axis. Triggered both when a spectrum is freshly extracted and when the slider moves after extraction. Extraction itself is unchanged — it still spans the selected fibers' full overlap range, so the saved OneSpec FITS contains the full spectrum regardless of the displayed window.
The merge from upstream/develop (22515d7) reverted the one-line addition from 93d2b6e that registered 'gaps' as a valid scattered light method. The 'gaps' handler in rawimage.py and the Binospec config_specific_par defaults both survived the merge, so config_specific_par produced method='gaps' only for the validator to reject it, breaking test_ifu_config_specific_keeps_standard_tilts in CI across every platform.
In the Fiber pypeline, bad detector rows (BPM-masked) and blue-end low-throughput regions produced wavelength-locked spike artifacts in the reconstructed 2D SKYMODEL. The fix has three parts: - flatfield.py: rescale moment1d boxcar sums by box_width/npix so a half-masked aperture no longer returns half the true flux; mark heavily-masked bins NaN, and preserve those NaN entries in the saved FiberFlat instead of refilling with 1.0. - find_objects.py: in _apply_flat_correction, fill NaN bins in normflat by linear interpolation across the index axis before np.interp, so flat_corr stays smooth across bad-row wavelengths and no longer creates a multiplicative cliff that injects a wavelength-locked spike into the SKYMODEL. Mask BOX_MASK and BOX_COUNTS_IVAR only at BOX_WAVE entries within half a sample of an original bad NORMFLAT bin. - extraction.py: fall back to a uniform aperture profile in _build_empirical_profiles when fewer than half the aperture pixels are good, preventing positive flat outliers in low-throughput regions from concentrating the projected sky on 1-2 pixels. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace wavelength-dependent flat division with a single per-fiber throughput scalar (fiber_throughput); spectral flattening is deferred to flux calibration downstream. Build a genuine per-pixel 2D B-spline SKYMODEL fit jointly over sky and science fibers, with two-pass per-fiber wavelength refinement. Add joint_fit_use_sci and sci_exclude_radius parameters to SkySubPar to tune the joint fit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the Binospec IFU docs to match the new pipeline: rewrite the fiber throughput correction and sky subtraction section, document the joint_fit_use_sci and sci_exclude_radius parameters with .pypeit examples, and refresh the pipeline comparison. Register FiberFlatImages in the datamodel generator, add its generated datamodel table, and add the missing astropy.io.fits.FITS_rec link target. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…for Y axis of extracted spectrum plot
…og scales and to mask areas around bright sky lines in the displayed spectrum
…the overlays are not usable and make the gui unusably slow.
|
this is now working much more to my satisfaction. ended up refactoring how sky subtraction is done so that it works in 2D space. this made it much easier to visually see what was going on (and going wrong). an optimization was done to work with blocks of fibers as pseudo-slits. doing the full wavelength calibration on every fiber was very slow. doing it once per block and fitting tilts is a factor of several faster. however, the tilt fitting isn't quite accurate enough to model the residual variations due to slight mechanical offsets of the fiber heads. using the tilt model as a starting point, per-fiber offsets are now calculated and applied. an option has been added to use both science and sky fibers to build the sky model. this is controlled by the spectra extraction gui was ported over from the IDL pipeline and adapted to load spec1d files and output the co-added spectra in spec1d-format. it's implemented in pure-matplotlib so it doesn't incur any additional dependencies. an example screenshot of an observation of NGC 2392 (the eskimo nebula):
this will be ready to leave draft status once the spin-off PR #2127 is merged. |
3 participants
With a big assist from Claude Opus 4.6, this is the first cut at adding support for Binospec's fiber-based IFU. This is an ok starting point and provides something to build on and improve. The new
Fiberpypelineshould make it pretty straightforward to support other fiber-based spectrographs, both IFU and multi-object. The Binospec documentation has been updated to include information on the new functionality as well as a detailed comparison between the CfA-maintained IDL-based pipeline and PypeIt.The changes made elsewhere in the code were largely some missed validity checks that were tripped over during testing and development. A more significant change in
flatfield.pyis to fix an issue where 3 GB temporary arrays were being constructed for all 360 slits per detector. That obviously got pretty ugly, but the fix here did the trick to alleviate that.