Make EfficientEvaluationLoops hookable for Python callbacks - #331
Closed
tachyonicClock wants to merge 1 commit into
Closed
Make EfficientEvaluationLoops hookable for Python callbacks#331tachyonicClock wants to merge 1 commit into
tachyonicClock wants to merge 1 commit into
Conversation
Add HookableEvaluationLoop, a test-then-train loop that notifies registered callbacks (OnLabel per instance, OnWindowClose per window) so CapyMOA can observe predictions and scores as they happen, unblocking score-based metrics like AUC without paying the cost of the slow pure-Python path. Hooks receive normalized predicted probabilities; evaluators still get raw votes so PrequentialResult is unchanged. The trailing partial window is always closed and handed to OnWindowClose hooks. Buffering only occurs when such hooks are registered. EfficientEvaluationLoops.PrequentialEvaluation now delegates to the new loop, preserving its signature and behavior for backwards compatibility. Test parity verified against pre-change baseline: 70.8% accuracy on 1000 RandomTreeGenerator instances with NaiveBayes (seed 1), 10 windows of 100. All evaluator-related tests still pass. Addresses adaptive-machine-learning/backlog#145. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the prequential evaluation loop hookable so CapyMOA can observe predictions, scores, and window boundaries as they happen.
HookableEvaluationLoopwithOnLabel(per-instance) andOnWindowClose(per-window) callbacksEfficientEvaluationLoops.PrequentialEvaluationnow delegates to preserve backwards compatibilityUnblocks scoring-based metrics like AUC that need class probabilities.
Addresses adaptive-machine-learning/backlog#145.
Test Plan
HookableEvaluationLoopTest(10 tests): parity, hook firing, window buffering, zero-vote handling, null evaluatorsEfficientEvaluationLoopsTest(existing): regression checkFollow-up
moa.jarmust be rebuilt and republished (moa_urlin CapyMOA'sinvoke.yml) before Python work can use this.Benchmarking jpype crossing cost deferred to Python-side work.
🤖 Generated with Claude Code