FIX: reduce memory in flatfield by evaluating tilts only at slit pixels

[tepickering]

Replace full-frame meshgrid tilt evaluation with per-slit-pixel evaluation using PypeItFit.eval directly. For spectrographs with many slits (e.g., fiber-fed IFUs with hundreds of fibers), the previous approach allocated a full-frame tilts array per slit, causing excessive memory usage.

This was originally implemented in #2080 and was required to get the original implementation there to work. However, it's a massive improvement for any multi-slit mode.

[kbwestfall]

I'm approving this as it is, given that it reduces memory so substantially. But I had some questions that I hope we can think through before we merge this.

[kbwestfall]

3 questions:

• Instead of adding this here, could we instead pass onslit_padded to tracewave.fit2tilts and essentially get the same thing?
• Do we need to explicitly delete the "work" arrays, or can we lean on garbage collection?
• I'm wondering if there's a way we could minimize the number of times we need to create the tilts array, and/or try to use the same memory block.

[tepickering]

3 questions:

• Instead of adding this here, could we instead pass onslit_padded to tracewave.fit2tilts and essentially get the same thing?

maybe. i'll take a deeper look.

• Do we need to explicitly delete the "work" arrays, or can we lean on garbage collection?

i don't see any downside to just manually deleting them if we know they're not going to be used. especially given what look like GC issues with python 3.14.

• I'm wondering if there's a way we could minimize the number of times we need to create the tilts array, and/or try to use the same memory block.

given the scale of the bug being fixed here, i am sure there are plenty of places in the code that can be streamlined and made more efficient.

[tepickering]

asked claude and got this response:

1. Could we pass onslit_padded to fit2tilts instead? — Yes, this is a clean approach. fit2tilts currently creates a full nspec x nspat meshgrid (line 893), which is the memory problem. Adding an optional mask parameter would let it evaluate only at masked pixels and return either a sparse full-frame array or just the 1D values. There are only 2 other callers (wavetilts.py:154 and wavetilts.py:817), so backward compatibility is easy with a default mask=None. This would also benefit those callers if they ever need it.

2. Explicit del vs garbage collection — The del statements are a belt-and-suspenders measure. In a tight loop over hundreds of slits, it ensures the previous iteration's arrays are freed before allocating the next. GC would eventually collect them, but in CPython the reference counting means del triggers immediate deallocation. Given the memory-sensitive context, it's cheap insurance.

3. Reuse the tilts array across iterations — Currently tilts = np.zeros(...) allocates a new array each iteration. You could allocate once before the loop and tilts[:] = 0 each iteration to reuse the memory block. However, spec_coo is derived from tilts on the next line, so both would need coordinated handling. Modest win for standard spectrographs, bigger win for many-slit cases.

claude's point about cpython is a good one since the objects in question are numpy arrays.

[tepickering]

went ahead and implemented @kbwestfall's idea of doing the fix at the fit2tilts level.

[kbwestfall]

At the risk of micro-managing, can we do this instead:

if slit_mask is None:
    spat_pix, spec_pix = map(
        lambda x : x.ravel(), np.meshgrid(np.arange(nspat), np.arange(nspec))
    )
else:
    spec_pix, spat_pix = np.where(slit_mask)

tilts_vals = pypeitFit.eval(spec_pix / xnspecmin1, x2=(spat_pix - _spat_shift) / xnspatmin1)
tilts_vals = np.fmax(np.fmin(tilts_vals, 1.2), -0.2)
tilts = np.zeros(shape, dtype=float)
tilts[(spec_pix,spat_pix)] = tilts_vals
del tilts_vals, spec_pix, spat_pix

return tilts

[tepickering]

done.

[debora-pe]

Looks good. I only have one comment, please take a look at it.

Let's also wait on the tests result.

[debora-pe]

Should the slit_mask parameter be added here too?

[tepickering]

i think so, but we'll need to fix the following check to limit it to the masked region. otherwise nanmin will always come up as 0. i'll go ahead and make that change.

[kbwestfall]

Dev-suite running.

[kbwestfall]

There are a number of test failures:

Test Summary
--------------------------------------------------------
--- PYTEST PYPEIT UNIT TESTS PASSED  330 passed, 450 warnings in 290.87s (0:04:50) ---
--- PYTEST UNIT TESTS FAILED  1 failed, 157 passed, 1449 warnings in 764.25s (0:12:44) ---
--- PYTEST VET TESTS FAILED  11 failed, 63 passed, 1591 warnings in 4441.06s (1:14:01) ---
--- PYPEIT DEVELOPMENT SUITE FAILED 4/291 TESTS  ---
Failed tests:
    vlt_xshooter/VIS_manual pypeit_coadd_2dspec
    gemini_gnirs_echelle/32_SB_SXD pypeit_coadd_2dspec
    gemini_gnirs_echelle/32_SB_SXD pypeit_tellfit
    keck_nires/ABBA_nostandard_faint pypeit_coadd_2dspec
Skipped tests:
    gemini_gnirs_echelle/32_SB_SXD pypeit_tellfit
Total disk usage: 292.628 GiB
Testing Started at 2026-05-05T18:39:50.721950
Testing Completed at 2026-05-06T06:06:59.646555
Total Time: 11:27:08.924605

All the script test failures have to do with the afterburn scripts, and the vet tests may be associated with that. @tepickering , I'll post the full report in Slack.

[kbwestfall]

Blocking merging until we can get the tests fixed.

[tepickering]

it looks like this change is the culprit:

  # OLD                                                                                                                                               
  if np.nanmax(self.tilts) - np.nanmin(self.tilts) < 0.8:                                                                                                                       
                                                                                                                                                                                
  # NEW (slit-scoped)                                                                                                                                                           
  _slit_tilts = self.tilts[thismask_science]                                               
  if np.nanmax(_slit_tilts) - np.nanmin(_slit_tilts) < 0.8:

the old way was doing a full-frame evaluation of the tilts so it was always fine. the 3 failures in the test run all trip on the new slit-scoped test. one option would be to set that to something smaller like 0.1 or 0.2. or add more code to set a smarter threshold based on spec_min_max.

[tepickering]

we should do another full dev suite run to test the change i just made.

[kbwestfall]

I've restarted the tests!

[kbwestfall]

Tests pass!

Test Summary
--------------------------------------------------------
--- PYTEST PYPEIT UNIT TESTS PASSED  331 passed, 448 warnings in 143.20s (0:02:23) ---
--- PYTEST UNIT TESTS FAILED  1 failed, 157 passed, 1449 warnings in 428.54s (0:07:08) ---
--- PYTEST VET TESTS PASSED  74 passed, 1664 warnings in 2220.95s (0:37:00) ---
--- PYPEIT DEVELOPMENT SUITE PASSED 292/292 TESTS  ---
Total disk usage: 225.114 GiB
Testing Started at 2026-05-07T15:06:26.577755
Testing Completed at 2026-05-07T20:44:18.201105
Total Time: 5:37:51.623350

The unit test failure is the known LDT one.