test: assert release model eval scores on sample data - #1413
Closed
proofbyhuman wants to merge 1 commit into
Closed
Conversation
Add a regression test that pins box_precision and box_recall of the release model (weecology/deepforest-tree) on the OSBS_029 sample data, which ships with ground truth. This detects drift in the model outputs over time, unlike the existing loose-bound evaluation tests. A small pytest.approx tolerance catches a single changed detection while tolerating float noise across platforms. Refs weecology#1233 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1413 +/- ##
==========================================
+ Coverage 86.61% 86.67% +0.06%
==========================================
Files 26 28 +2
Lines 3736 4069 +333
==========================================
+ Hits 3236 3527 +291
- Misses 500 542 +42
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
proofbyhuman
marked this pull request as draft
July 30, 2026 22:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a regression test
test_release_model_regressionintests/test_evaluate.pythat runs the release model (weecology/deepforest-tree) on theOSBS_029sample data (which ships with ground truth) and pins the evaluation scores, so that drift in the model outputs over time is detected.The existing evaluation tests only use loose bounds (e.g.
box_recall > 0.5), so a genuine change in the model outputs would not be caught. This test pins, atiou_threshold=0.4:box_precision == 0.80box_recall == 0.72These values come from integer match counts (44/55 and 44/61), so they are stable across platforms. They are asserted with
pytest.approx(..., abs=0.01)so that a single changed detection (~1/61 ≈ 0.016) trips the test while float noise does not. The expected values are meant to be updated intentionally when the release model changes.Testing:
pytest tests/test_evaluate.py::test_release_model_regression -vpasses locally (Python 3.12, CPU). Test-only addition — no breaking changes.Related Issue(s)
Related to #1233
This overlaps with the earlier draft #1249 by @musaqlain, which has been inactive for ~2 months. Opened as a small, self-contained alternative in case that one is stalled — happy to defer to it or align with whatever the maintainers prefer.
AI-Assisted Development
AI tools used (if applicable):
Claude (Anthropic) helped investigate the codebase and draft the test; I reviewed the code and ran it locally.