Unusually long orbitfit execution time for certain MBAs

[awilson110]

LINUX ONLY

When running orbitfit for the MBA 1119 T-1 (using only its Rubin DP2 r band observations; if needs be I can send these to anyone who is a rubin data rights holder), it stalls for an unusually long time. Total execution takes ~6 minutes. This object is a normal MBA on a very circular orbit (e=0.04), there are 14 observations being used with an arc of over 1 month (2025-07-18 to 2025-08-27). During execution, 3 errors appear:

Error! The requested time is outside the coverage provided by the ephemeris file.
This error repeated tens of times

Error: cannot interpolate beyond timestep bounds (h=-nan).
This error repeats 4 times, where h is a different value each time

Warning! At least 10 predictor corrector loops in IAS15 did not converge. This is typically an indication of the timestep being too large.
This error appears once.

However, these errors are not recorded in the .log or .err files. I believe these errors come from ASSIST. This object does not noticeably stall on MacOS, and fits without error.

MACOS AND LINUX

When running orbitfit for the MBA 2015 PE134 (again only its Rubin DP2 r band observations), it also stalls but for different times on Mac and Linux. On Mac it takes 10 minutes to execute, on Linux it takes 40 minutes, and in both cases the fitting fails (flag 3). However, running with the IOD auto takes twice as long but does successfully fit. Again this object is an MBA with a slightly elliptical (e=0.11) orbit, with 14 observations spanning more than 1 month (2025-07-01 to 2025-08-15).

No errors appear for this object.

For both of these objects, I would have expected them to be fit a lot faster, or error out faster if they can't be fit.

Some environment info (for both OS's):

OS = MacOS Tahoe 26.6.2; Ubuntu 22.04.5

layup=0.1.dev722+gc860acffe (Ubuntu) 0.1.dev708+gbec27479c.d20260727 (MacOS)
sorcha=1.2.1
assist=1.2.3
rebound=4.5.0

[mschwamb]

@awilson110 is sending @matthewholman the observations that were failing to time out for debugging

[matthewholman]

Thanks!

…

On Mon, Aug 24, 2026 at 11:47 AM Meg Schwamb ***@***.***> wrote: *mschwamb* left a comment (Smithsonian/layup#465) <#465 (comment)> @awilson110 <https://github.com/awilson110> is sending @matthewholman <https://github.com/matthewholman> the observations that were failing to time out for debugging — Reply to this email directly, view it on GitHub <#465?email_source=notifications&email_token=AA5CVWIZFKFHDKHBNJC2N4D5LRPSPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZZG43TANJZGQ4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5397705948>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5CVWNF73JFK3LAQ2KTJCL5LRPSPAVCNFSNUABFKJSXA33TNF2G64TZHM4TCNRSHAYTGMRUHNEXG43VMU5TKMRQGUZDKMJYGY3KC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AA5CVWON2H2ELJDLM3YGT3T5LRPSPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZZG43TANJZGQ4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AA5CVWKTEKAFSMEHKP6SZAL5LRPSPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZZG43TANJZGQ4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Matthew J. Holman, PhD Senior Astrophysicist Center for Astrophysics | Harvard & Smithsonian 60 Garden Street, MS #51 Cambridge, MA 02138 (617) 496-7775

[matthewholman]

Reopened — closed by accident, not resolved.

Status, so the thread has it in one place:

Ruled out. adaptive_mode — the fix for the old close-Earth grind already landed (g_ias15_adaptive_mode = 2, applied after assist_attach) and is in the build @awilson110 reported against.

Largely ruled out. The IOD residual pre-filter silently switching off when assist.Ephem fails to load. Measured directly by forcing that failure and timing both ways: the penalty is 1.4x on a 6-day / 14-observation arc and at most ~3.6x on longer ones — and in one case disabling the pre-filter was faster, since it has its own cost. That cannot turn a sub-second fit into 6–40 minutes.

Current leading explanation: this is platform-linked rather than seed-linked. Three independent signals: 1119 T-1 is marked LINUX ONLY and fits fine on macOS; test_iod_auto.py:119 is permanently skipped with a reason describing the same pathology and naming Linux nondeterminism (~80 min on one Ubuntu leg against ~12 min on its twin); and it will not reproduce on macOS here even after clearing two blockers that were masking the relevant test.

A candidate axis is the if(APPLE) / else() split in CMakeLists.txt — Linux links libassist/librebound directly, macOS dlopens them with RTLD_GLOBAL (#459). Different code path in exactly the library that is grinding.

What would settle it is a seed-quality → integrator-work curve run on Linux: seeds at controlled fractional error in heliocentric distance, each perturbed both inward and outward, recording wallclock and LM iterations. The inward/outward asymmetry is the discriminating measurement — a 54%-wrong seed for a 2.5 au object lands at either ~1.1 or ~3.9 au, and only one of those sits on Earth's orbit. If inward is far worse at equal error, the driver is close approaches rather than seed quality, and the fix is a cheap guard rather than better seeds.

Note that #446 (BK-IOD's dropped perspective denominator) is now fixed and merged, and it improves the seed substantially — but it likely does not close this: the failing path here is iod='gauss', which never reaches BK-IOD.

[matthewholman]

I worked through the observations @awilson110 sent (8 objects, 88 observations, Rubin DP2 via X05). Summary first, because the two halves of this issue turn out to have different answers:

• The failure reproduces, and it is an IOD problem, not a platform problem. 7 of the 8 objects fail on macOS with current main. --iod auto fixes all 7.
• The slowness does not reproduce at all. Those same fits take 4–6 seconds here, against the 2–40 minutes reported.

1. What current main does with the file

Fitting each object separately on macOS, layup 0.1.dev750, default settings:

	default (--iod gauss)	--iod auto
7 of 8 objects	flag = 3, no state written	flag = 0, a = 2.31–2.99 au
2002 VU19	flag = 0, a = −1.4e-05 au	same

Every one of the seven takes 4–6 seconds either way. flag = 3 is set where no IOD root converges on the primary interval (orbitfit.py:981), so this is a seeding failure — Gauss finds nothing usable — rather than the Levenberg-Marquardt refinement struggling.

With --iod auto, all seven converge to sensible main-belt semimajor axes in the same 4–6 seconds. That matches your own note above that auto succeeds.

2015 PE134 is the useful cross-check, because it is in both your report and the file. You report 10 minutes on macOS and flag 3. I reproduce the flag 3 exactly, in 5.2 seconds, and --iod auto fits it in 5.2 seconds at a = 2.93 au.

2. Why the timings probably differ

layup-hergit-method branches from ccda6339 (2026-02-09) and is 364 commits behind main. That merge base does not contain adaptive_mode.

PR #324 set IAS15's adaptive_mode = 2 specifically to remove a close-encounter grind that took more than 120 seconds and dropped it to sub-second. Your timeout is set at exactly 120 seconds. That is a strong match, and it would explain why the same objects are minutes for you and seconds here.

Worth confirming rather than assuming: which build produced these timings — your branch, or main? If it was the branch, updating it is likely the whole answer to the timing half of this issue. (M.J.H. has asked separately about updating it.)

3. Two hypotheses I checked and discarded

Astrometric uncertainty units. rmsRA/rmsDec in the file are in degrees; the ADES specification says arcsec. The median is 4.2e-06, which read as arcsec is 4 microarcsec — not physically meaningful. Read as degrees it is 0.015 arcsec, which is right for Rubin.

Converting them changes nothing about this issue: identical flags, identical timings. But it is worth fixing at the source anyway, because it makes every weight wrong by a factor of 3600² and will distort chi-square and the reported covariance on fits that do converge.

Near-opposition degeneracy. Ruled out — these sit at 42–93° solar elongation, nowhere near opposition, so the known curvature-below-noise mechanism does not apply here.

4. Split off separately

2002 VU19 is a different bug and is now #485. It has 8 observations spanning 12.3 minutes and returns flag = 0 — success — with a hyperbolic orbit at 267× the circular velocity. That one reproduces on the bundled demo data too, so it needed no restricted data to file.

5. What is still open here

• The Linux-only half. The 1119 T-1 ephemeris errors (requested time is outside the coverage provided by the ephemeris file, cannot interpolate beyond timestep bounds (h=-nan)) are not in this file — it contains eight different objects. Reproducing that still needs 1119 T-1's observations, and it remains the one part that genuinely needs a Linux run.
• Why Gauss fails to seed these at all. Seven of eight is not a marginal rate, and auto rescuing all seven says the observations are fine. That is the substantive question underneath this issue, and it is worth answering even after the timing half resolves.

[mschwamb]

@awilson110 were you using the latest version of layup and not the hergit branch?

[matthewholman]

A reproducible slow fit on public MPC astrometry — no Rubin data rights needed.

While testing whether auto should become the default IOD (layup#486), I cut 450 short discovery-like arcs out of the cached full-history MPC corpus (48,868 numbered objects) and fitted each one separately. In the 7-day group, 106 of 108 arcs fitted in 4-5 seconds. Two did not finish at all.

That gives this issue something it did not have: a slow case anyone can reproduce. The observations Adam supplied are Rubin DP2, which stays restricted to data rights holders until the two-year proprietary period expires, so it can only be shared with rights holders. These are ordinary MPC observations from Pan-STARRS, Catalina and Mount Lemmon, and the whole input is 9 and 19 lines.

The two cases

object	obs	arc	epoch	stations
(818273)	9	4.9 d	2020-09-18	F52, G96
(134584)	19	6.0 d	2015-01-16	703, G45, G96

Against a 4-second median for the other 106 arcs in the same group, one of these ran more than 11 minutes without completing before I stopped it, and both were still running at a 90-second cap on a re-check. So this is at least two to three orders of magnitude above the median, which puts it in the same range Adam reported (2-40 minutes).

Neither produced an output file, so they are grinding rather than converging slowly.

To reproduce

Save as slow_134584.obs80:

D4584         C2015 01 16.39204 08 17 07.47 +15 38 46.7          19.9 rR~1KWPG45
D4584         C2015 01 16.39800 08 17 07.01 +15 38 47.4          19.6 rR~1KWPG45
D4584         C2015 01 16.40221 08 17 06.75 +15 38 48.0          19.9 rR~1KWPG45
D4584         C2015 01 17.29947 08 16 07.08 +15 41 30.3          19.2 Vq~1HWiG96
D4584         C2015 01 17.30505 08 16 06.69 +15 41 31.4          19.0 Vq~1HWiG96
D4584         C2015 01 17.31063 08 16 06.32 +15 41 32.5          19.1 Vq~1HWiG96
D4584         C2015 01 17.31623 08 16 05.93 +15 41 33.4          19.1 Vq~1HWiG96
D4584         C2015 01 17.38229 08 16 01.33 +15 41 45.3          19.8 rR~1KWPG45
D4584         C2015 01 17.38653 08 16 01.02 +15 41 45.8          19.2 rR~1KWPG45
D4584         C2015 01 17.39246 08 16 00.62 +15 41 47.3          19.5 rR~1KWPG45
D4584         C2015 01 17.39654 08 16 00.35 +15 41 47.8          19.8 rR~1KWPG45
D4584         C2015 01 18.38380 08 14 54.23 +15 44 48.2          19.5 rR~1KWPG45
D4584         C2015 01 18.38966 08 14 53.80 +15 44 49.2          19.6 rR~1KWPG45
D4584         C2015 01 18.39374 08 14 53.52 +15 44 49.4          19.4 rR~1KWPG45
D4584         C2015 01 18.39788 08 14 53.22 +15 44 50.5          19.5 rR~1KWPG45
D4584         C2015 01 22.36122 08 10 27.83 +15 57 12.0          18.9 Vq~1HWi703
D4584         C2015 01 22.36625 08 10 27.47 +15 57 12.9          19.6 Vq~1HWi703
D4584         C2015 01 22.37127 08 10 27.13 +15 57 13.3          19.5 Vq~1HWi703
D4584         C2015 01 22.37631 08 10 26.73 +15 57 15.2          19.7 Vq~1HWi703

Save as slow_818273.obs80:

~0pZxK13K17N  C2020 09 18.48356 00 38 55.081-06 40 58.78         21.5 wU~49uqF52
~0pZxK13K17N  C2020 09 18.49520 00 38 54.631-06 41 06.28         21.2 wU~49uqF52
~0pZxK13K17N  C2020 09 21.43450 00 37 03.774-07 13 28.37         21.0 wU~49uqF52
~0pZxK13K17N  C2020 09 21.45893 00 37 02.816-07 13 44.38         21.3 wU~49uqF52
~0pZxK13K17N  C2020 09 21.47116 00 37 02.322-07 13 52.38         21.4 wU~49uqF52
~0pZxK13K17N  C2020 09 23.35189 00 35 49.69 -07 34 21.2          21.4 GV~49uqG96
~0pZxK13K17N  C2020 09 23.35759 00 35 49.44 -07 34 25.2          21.2 GV~49uqG96
~0pZxK13K17N  C2020 09 23.36326 00 35 49.20 -07 34 29.1          21.0 GV~49uqG96
~0pZxK13K17N  C2020 09 23.36894 00 35 48.98 -07 34 32.5          21.3 GV~49uqG96

then

layup orbitfit slow_134584.obs80 MPC80col -i gauss -n 1 -o out_134584
layup orbitfit slow_818273.obs80 MPC80col -i gauss -n 1 -o out_818273

Measured on macOS with layup 0.1.dev750 (main @ af6273c), one worker, nothing else running.

What this does and does not settle

Does: there exist individual short arcs, in ordinary MPC data, that make orbitfit grind for minutes where comparable arcs take seconds. It is not specific to Rubin data, not specific to DP2, and not specific to Adam's machine.

Does not: explain the whole of this issue. Two arcs at a few minutes each do not account for a batched run of all 108 arcs in a single call stalling for over an hour without producing output. That looks like a separate problem in the batch path — chunking or the worker pool — and it is the one that matters for catalogue-scale runs. I have not isolated it and am not claiming the two are the same fault.

Still open, and worth doing next

• Does --iod auto fix the slowness, or only the failures? On Adam's DP2 sample auto converted 7 of 8 flag=3 failures into good fits at no time cost. Whether it also rescues these two is unknown — my probe was stopped before the auto runs. If it does, that is a much stronger argument for layup#486 than convergence counts alone.
• Where does the time actually go? A profile of one of these two would answer it directly, and both are small enough to profile comfortably.

One practical note for anyone debugging this

layup dispatches by scanning PATH for layup-<verb> executables and shelling out, so the running process is layup-orbitfit, not layup orbitfit. pkill -f "layup orbitfit" silently matches nothing — I left an orphan running for over an hour that way. It is also why a stale sibling installation earlier on PATH silently wins over the build under test.

[awilson110]

@awilson110 were you using the latest version of layup and not the hergit branch?

I'll do a fresh install now to check as I'm not sure how outdated my layup version was, but I wasn't on the Herget branch

[awilson110]

Update: I used a fresh install of layup and I'm getting the same results as @matthewholman ; the fitting is now taking seconds and 7/8 of the objects are failing to be fit (with the exception of 2002 VU19). Apologies, I should have done this before reporting the bug.

[matthewholman]

Excellent!

[matthewholman]

Traced where the time goes on (818273), one of the two public reproducible cases above. Short version: one Gauss root sits 0.0022 au from Earth, and the fit hangs on it. Three plausible mechanisms were tested and ruled out, so the cause is not yet pinned down.

What is established.

Gauss returns three roots. Two are ordinary; the third is the spurious near-Earth solution:

root	|r| helio	a	topocentric distance
0	3.227 au	+3.145	2.24 au
1	1.000 au	+0.996	0.0022 au
2	0.827 au	+2.276	1.77 au

Fitting from root 0 converges to a = 3.148 in under 10 ms. Fitting from root 1 does not return even with iter_max = 1 — a single LM iteration is enough to hang it. So there is no wandering to explain; the cost is present immediately.

What was ruled out.

• Unphysical seed. All three roots are bound with sensible e and q. Root 1 is wrong, not absurd.
• The Cartesian LM loop. --engine bk_native hangs identically, and the two engines share only the IOD and the integration.
• The legacy IAS15 controller. Root 1 costs 1,244,045 steps at adaptive_mode = 1 and 7 steps at mode 2. But layup already defaults to mode 2 and applies it after every assist_attach, so it is not running the slow controller.
• Variational particles. The natural suspect, since the variation grows as √(GM⊕/d³) near Earth and IAS15 sets its step from derivatives across all particles. Measured: root 1 integrates in 7 steps at mode 2 both bare and with six variations attached.

What is left. A bare ASSIST integration of root 1 over the arc is fast; one LM iteration from it is not. The difference is layup's own machinery around the integration — light-time iteration, the per-observation integrate calls, or the min-dt floor. At 0.0022 au the light-time correction and the topocentric direction are both extremely sensitive, which makes the light-time loop the next thing I would look at.

Getting further needs profiling the C++ rather than probing from Python.

Worth noting regardless of the cause. A near-Earth Gauss root on a short arc is a predictable hazard, and the fit has no cheap way to decline it. Rejecting a root whose topocentric distance is implausibly small, before integrating it, would sidestep this whole class — the check costs nothing and the root is wrong anyway. That is independent of #492's step budget, and probably complementary to it.