Skip to content

Add Fold Weights for Variable Resample Weighting #1007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

tjburch
Copy link

@tjburch tjburch commented May 23, 2025

Closes #990

This PR implements support for variable fold weights in hyperparameter tuning. This is useful in cases where folds may have differing numbers of observations, and you want proportional contribution to hyperparameter selection.

The implementation adds two main functions: add_fold_weights() to attach custom weights to rset objects, and calculate_fold_weights() to automatically compute weights proportional to fold sizes. Weights are stored as .fold_weights attributes and should flow through the existing tuning pipeline.

Core changes are in estimate_tune_results() which now detects weights and uses weighted statistics (weighted mean, weighted standard deviation, effective sample size) when aggregating metrics. Implementation should be backwards compatible and non-breaking.

@topepo
Copy link
Member

topepo commented Jun 4, 2025

Hey @tjburch. Thanks for the PR.

We're doing a pretty invasive update the this package that will take some time. We'll look at the PR after things are settled there but it might be another 2-3 weeks.

Is this something time-sensitive for you?

@tjburch
Copy link
Author

tjburch commented Jun 4, 2025

Nope. Just had some bandwidth staying awake on paternity leave. Review at your leisure, let me know if I can assist otherwise.

@topepo
Copy link
Member

topepo commented Jun 4, 2025

Just had some bandwidth staying awake on paternity leave

I estimate that 5% of all my work has been while waiting at the bustop or for some sort of practice to end 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Variable Fold Weights
2 participants