Skip to content

test: assert release model eval scores on sample data - #1413

Closed
proofbyhuman wants to merge 1 commit into
weecology:mainfrom
proofbyhuman:test-1233-assert-sample-data
Closed

test: assert release model eval scores on sample data#1413
proofbyhuman wants to merge 1 commit into
weecology:mainfrom
proofbyhuman:test-1233-assert-sample-data

Conversation

@proofbyhuman

Copy link
Copy Markdown

Description

Adds a regression test test_release_model_regression in tests/test_evaluate.py that runs the release model (weecology/deepforest-tree) on the OSBS_029 sample data (which ships with ground truth) and pins the evaluation scores, so that drift in the model outputs over time is detected.

The existing evaluation tests only use loose bounds (e.g. box_recall > 0.5), so a genuine change in the model outputs would not be caught. This test pins, at iou_threshold=0.4:

  • box_precision == 0.80
  • box_recall == 0.72

These values come from integer match counts (44/55 and 44/61), so they are stable across platforms. They are asserted with pytest.approx(..., abs=0.01) so that a single changed detection (~1/61 ≈ 0.016) trips the test while float noise does not. The expected values are meant to be updated intentionally when the release model changes.

Testing: pytest tests/test_evaluate.py::test_release_model_regression -v passes locally (Python 3.12, CPU). Test-only addition — no breaking changes.

Related Issue(s)

Related to #1233

This overlaps with the earlier draft #1249 by @musaqlain, which has been inactive for ~2 months. Opened as a small, self-contained alternative in case that one is stalled — happy to defer to it or align with whatever the maintainers prefer.

AI-Assisted Development

  • I used AI tools (e.g., GitHub Copilot, ChatGPT, etc.) in developing this PR
  • I understand all the code I'm submitting
  • I have reviewed and validated all AI-generated code

AI tools used (if applicable):
Claude (Anthropic) helped investigate the codebase and draft the test; I reviewed the code and ran it locally.

Add a regression test that pins box_precision and box_recall of the
release model (weecology/deepforest-tree) on the OSBS_029 sample data,
which ships with ground truth. This detects drift in the model outputs
over time, unlike the existing loose-bound evaluation tests. A small
pytest.approx tolerance catches a single changed detection while
tolerating float noise across platforms.

Refs weecology#1233

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.67%. Comparing base (d2851ec) to head (eab1016).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1413      +/-   ##
==========================================
+ Coverage   86.61%   86.67%   +0.06%     
==========================================
  Files          26       28       +2     
  Lines        3736     4069     +333     
==========================================
+ Hits         3236     3527     +291     
- Misses        500      542      +42     
Flag Coverage Δ
unittests 86.67% <ø> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@proofbyhuman
proofbyhuman marked this pull request as draft July 30, 2026 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant