Skip to content

docs: Add SKD005 underrepresented classes pitfall example - #3147

Open
moujanrastgoo wants to merge 5 commits into
probabl-ai:mainfrom
moujanrastgoo:docs/add-skd005-pitfall-example
Open

docs: Add SKD005 underrepresented classes pitfall example#3147
moujanrastgoo wants to merge 5 commits into
probabl-ai:mainfrom
moujanrastgoo:docs/add-skd005-pitfall-example

Conversation

@moujanrastgoo

@moujanrastgoo moujanrastgoo commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Change description

Related to #2622 and #3134

  • Adds new sphinx-gallery subsection Pitfalls and solutions
  • Registers subsection in sphinx/conf.py
  • Adds SKD005 check exmaple and links from automated checks user guide

Contribution checklist

  • Unit tests were added or updated (if necessary)
  • Documentation was added or updated (if necessary)
  • The code passes our style conventions (you can check this by running pre-commit on your code
    with pre-commit run --all-files)
  • All the tests pass (please test locally before pushing)
  • The documentation builds and renders properly (if it does, our bot will add a comment linking
    to a preview of the documentation to review it visually)
  • All the commits in the PR are signed (more information
    here)
  • The pull request title respects the Conventional Commits convention (more information
    here)

AI usage disclosure

AI tools were involved for:

  • Code generation (e.g., when writing an implementation or fixing a bug)
  • Test/benchmark generation
  • Documentation (including examples)
  • Research and understanding

The use of AI regarding documentation was mostly for formatting the doc files and checking for errors and debugging, not for creating the examples themselves.

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Documentation preview @ fdddeac

@moujanrastgoo

Copy link
Copy Markdown
Contributor Author

i'm working on removing the warnings and also on improving the last part

@glemaitre glemaitre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So here, I don't think that I'll go into details in the section because what you got is what the literature is advertising but it is rather wrong. I think that the notebook that I forwarded can just correct and make the narrative towards what we should be advocating for.

Comment on lines +14 to +17
- use per-class metrics (precision, recall, F1 per class),
- resample the dataset,
- use class weights in the estimator,
- collect more data for the underrepresented classes if possible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, we should follow the what we advocate in the notebook here: https://probabl-ai.github.io/calibration-cost-sensitive-learning/content/notebooks/imbalanced_classification.html

From what is written:

  • do not use F1: it is an average between precision and recall and usually it is never a good choice by default
  • precision and recall refer to thresholded metrics and in the imbalanced case, you probably want to look at the estimate of the probability because rare classes will lead to rare probability so one should probably look at metric related to assess how good those estimate are
  • do not use class weight. Instead post tune the classifier once you are sure it is calibrated to optimize the thresholded metric of your choice.
  • collecting more data is good, but only from underrepresented might change your distribution (the prevalence will be different from the general population) and thus there is a need to correct for this bias to have a properly calibrated learner before to take decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants