docs: Add SKD005 underrepresented classes pitfall example - #3147
Open
moujanrastgoo wants to merge 5 commits into
Open
docs: Add SKD005 underrepresented classes pitfall example#3147moujanrastgoo wants to merge 5 commits into
moujanrastgoo wants to merge 5 commits into
Conversation
Contributor
Contributor
Author
|
i'm working on removing the warnings and also on improving the last part |
glemaitre
reviewed
Jul 17, 2026
glemaitre
left a comment
Member
There was a problem hiding this comment.
So here, I don't think that I'll go into details in the section because what you got is what the literature is advertising but it is rather wrong. I think that the notebook that I forwarded can just correct and make the narrative towards what we should be advocating for.
Comment on lines
+14
to
+17
| - use per-class metrics (precision, recall, F1 per class), | ||
| - resample the dataset, | ||
| - use class weights in the estimator, | ||
| - collect more data for the underrepresented classes if possible. |
Member
There was a problem hiding this comment.
Here, we should follow the what we advocate in the notebook here: https://probabl-ai.github.io/calibration-cost-sensitive-learning/content/notebooks/imbalanced_classification.html
From what is written:
- do not use F1: it is an average between precision and recall and usually it is never a good choice by default
- precision and recall refer to thresholded metrics and in the imbalanced case, you probably want to look at the estimate of the probability because rare classes will lead to rare probability so one should probably look at metric related to assess how good those estimate are
- do not use class weight. Instead post tune the classifier once you are sure it is calibrated to optimize the thresholded metric of your choice.
- collecting more data is good, but only from underrepresented might change your distribution (the prevalence will be different from the general population) and thus there is a need to correct for this bias to have a properly calibrated learner before to take decision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change description
Related to #2622 and #3134
sphinx/conf.pyContribution checklist
with
pre-commit run --all-files)to a preview of the documentation to review it visually)
here)
here)
AI usage disclosure
AI tools were involved for:
The use of AI regarding documentation was mostly for formatting the doc files and checking for errors and debugging, not for creating the examples themselves.