docs: Add SKD003 inconsistent performance pitfall example by moujanrastgoo · Pull Request #3145 · probabl-ai/skore

moujanrastgoo · 2026-07-15T11:40:18Z

Change description

Related to #2622 and #3134

Adds new sphinx-gallery subsection Pitfalls and solutions
Registers subsection in sphinx/conf.py
Adds SKD003 check exmaple and links from automated checks user guide

Contribution checklist

Unit tests were added or updated (if necessary)
Documentation was added or updated (if necessary)
The code passes our style conventions (you can check this by running pre-commit on your code
with pre-commit run --all-files)
All the tests pass (please test locally before pushing)
The documentation builds and renders properly (if it does, our bot will add a comment linking
to a preview of the documentation to review it visually)
All the commits in the PR are signed (more information
here)
The pull request title respects the Conventional Commits convention (more information
here)

AI usage disclosure

AI tools were involved for:

Code generation (e.g., when writing an implementation or fixing a bug)
Test/benchmark generation
Documentation (including examples)
Research and understanding

The use of AI regarding documentation was mostly for formatting the doc files and checking for errors and debugging, not for creating the examples themselves.

github-actions · 2026-07-15T11:48:34Z

Documentation preview @ 2cc7c40

Co-authored-by: Cursor <cursoragent@cursor.com>

GaetandeCast

Here are my general thoughts on this check and a few other minor remarks

GaetandeCast

Some remarks

GaetandeCast · 2026-07-29T15:52:22Z

+problem is understood. We focus on SKD003 throughout and pass
+``ignore=["SKD008"]`` so Covertype's constant one-hot soil columns do not
+drown the summary in correlated-feature noise.


If we are not exploiting anything specific from this dataset (I think we don't but correct me if I'm wrong), let's just switch to a dataset that has no constant columns to avoid that concern altogether.

sure i will look into that!

GaetandeCast · 2026-07-29T15:54:10Z

+outlier_report.metrics.summarize(data_source="test").frame()
+
+# %%
+outlier_report.metrics.precision_recall().plot()


I think people are more used to roc plots so I'd rather use one here. It is easy to show that something is wrong on a roc plot since the plot will follow the diagonal "chance level" line.

Indeed. Can you also catch the output:

_ = outlier_report.metrics.roc().plot()

GaetandeCast · 2026-07-29T15:55:46Z

+report.metrics.summarize(data_source="test").frame(aggregate=None)
+
+# %%
+# ``SKD003`` should flag split #0.


Suggested change

# ``SKD003`` should flag split #0.

# ``SKD003`` correctly flags split #0.

Here and for other occurences.

GaetandeCast · 2026-07-29T16:00:14Z

+X_arr, y_arr = make_classification(
+    n_samples=n_time,
+    n_features=6,
+    n_informative=2,
+    n_redundant=0,
+    weights=[0.5, 0.5],
+    random_state=0,
+)


I would rather we add a fake datetime column to the same dataset used in the previous sections for consistency, just like how we created fake groups, and then change the distribution in the last temporal group.

glemaitre

Here are a couple of comments.

glemaitre · 2026-07-30T08:06:25Z

+===============================================
+
+:ref:`SKD003 <skd003-inconsistent-performance>` flags folds whose test metrics
+diverge sharply from the median on a :class:`~skore.CrossValidationReport`.


We should focus on the stats words instead of the Python object.

Suggested change

diverge sharply from the median on a :class:`~skore.CrossValidationReport`.

diverge sharply from the median during a cross-validation evaluation.

glemaitre · 2026-07-30T08:29:01Z

+- a much easier or harder group in one test fold under
+  :class:`~sklearn.model_selection.GroupKFold`,
+- temporal drift under :class:`~sklearn.model_selection.TimeSeriesSplit`
+  (e.g. an easy class that becomes rarer),


I would change the example:

Suggested change

(e.g. an easy class that becomes rarer),

(e.g. fraudsters finding workarounds to circumvent controls),

glemaitre · 2026-07-30T08:30:10Z

+:ref:`SKD003 <skd003-inconsistent-performance>` flags folds whose test metrics
+diverge sharply from the median on a :class:`~skore.CrossValidationReport`.
+With a proper splitter this is often a diagnostic check: the data have structure
+(groups, time, or a corrupted batch) that shuffled CV would hide.


Let's use whenever possible "cross-validation" instead of "CV"

glemaitre · 2026-07-30T08:36:29Z

+# subsample. SKD003 needs CV, so we use ``splitter=5`` or an explicit five-fold
+# splitter below.


To early to speak about the cross-validation. Let's only discuss it when we call skore.evaluate.

glemaitre · 2026-07-30T08:43:43Z

+# Randomly reassign labels on the first fifth of rows (mislabelled batch,
+# logging bug, or merge mix-up). ``splitter=5`` is unshuffled
+# :class:`~sklearn.model_selection.KFold`, so that block stays in fold #0 and
+# SKD003 can flag it.


Suggested change

# Randomly reassign labels on the first fifth of rows (mislabelled batch,

# logging bug, or merge mix-up). ``splitter=5`` is unshuffled

# :class:`~sklearn.model_selection.KFold`, so that block stays in fold #0 and

# SKD003 can flag it.

# Below, we will use an unshuffled 5-fold cross-validation. Here, we permute

# labels on the first fold to break any association between `X` and `y`. We expect

# the score of this fold to be low due to this corruption.

glemaitre · 2026-07-30T09:49:40Z

+    size=int(bad_batch.sum()),
+)
+
+group_splits = list(GroupKFold(n_splits=5).split(X, y_batch, groups=groups))


We should not have to create a list of index here. But because it is an advance use case we need to use a skrub DataOp. We probably need to have some narrative to explain that to the user by:

generating the data

explaining the creation of the skrub data op and for which reason

make the evaluation and explain the problem at hand

The implementation of the skrub DataOp here will look something like:

# %% # Bad group in the test fold # ========================== # # With group ids (site, patient, batch), a grouped splitter keeps each group on # one side of every split. A much harder or easier group in one test fold # triggers SKD003: expected, the splitter did its job. Shuffled CV would smear # that group across folds, hide the gap, and overestimate performance. # # Below we invent batch ids and corrupt labels for batch ``0`` only. Instead of # materializing a list of train/test indices, we keep features, target, and # group ids in a single dataframe: the group ids then follow the rows through # every split. n_batches = 10 batch_id = np.minimum(np.arange(len(X)) // (len(X) // n_batches), n_batches - 1) y_batch = y_clean.copy() bad_batch = batch_id == 0 y_batch.iloc[bad_batch] = np.random.RandomState(1).choice( y_clean.unique(), size=int(bad_batch.sum()), ) df_batch = X.assign(batch_id=batch_id, cover_type=y_batch) # %% # A skrub :class:`~skrub.DataOp` declares the split on the data itself: # :meth:`~skrub.DataOp.skb.mark_as_X` takes the splitter through ``cv`` and the # group ids through ``split_kwargs``. The resulting learner carries its own # cross-validation scheme, so :func:`~skore.evaluate` needs no ``splitter``. import skrub from sklearn.model_selection import GroupKFold data = skrub.var("data", df_batch) groups = data["batch_id"] X_op = data.drop(columns=["batch_id", "cover_type"]).skb.mark_as_X( cv=GroupKFold(n_splits=5), split_kwargs={"groups": groups}, ) y_op = data["cover_type"].skb.mark_as_y() learner = X_op.skb.apply(model, y=y_op).skb.make_learner() # %% report_grouped = skore.evaluate(learner, data={"data": df_batch}, n_jobs=4) report_grouped.metrics.summarize(data_source="test").frame(aggregate=None)

glemaitre · 2026-07-30T09:50:28Z

+# ===============================================
+#
+# Under :class:`~sklearn.model_selection.TimeSeriesSplit`, later windows can
+# diverge from early training data (e.g. an easy class becomes rarer). Earlier


I prefer the frauds cases.

glemaitre · 2026-07-30T09:51:04Z

+#
+# Build a balanced task with :func:`~sklearn.datasets.make_classification`,
+# treat row order as time, and replace labels only in the final test window
+# with a rare positive class so that one late fold stands out. No shuffle.


Same remarks about the imperative tense :)

glemaitre · 2026-07-30T09:51:44Z

+)
+X_time = pd.DataFrame(X_arr, columns=[f"f{i}" for i in range(6)])
+y_time = pd.Series(y_arr, name="label")
+y_time.iloc[hard_start:] = rng.choice([0, 1], size=n_time - hard_start, p=[0.96, 0.04])


Can you separate the data generative process from the evaluation.

glemaitre · 2026-07-30T09:52:33Z

+y_time = pd.Series(y_arr, name="label")
+y_time.iloc[hard_start:] = rng.choice([0, 1], size=n_time - hard_start, p=[0.96, 0.04])
+
+time_splits = list(TimeSeriesSplit(n_splits=n_splits).split(X_time))


You should not have to materialize as a list. Pass directly the splitter:

time_splitter = TimeSeriesSplit(n_splits=n_splits) skore.evaluate(..., splitter=time_splitter, ...)

I'm thinking that since we introduce a skrub DataOp earlier, we could actually create a timestamp as index, and create our own TimeSeriesSplit that is based on the time. Then, in the data generative process, we could make that for instance the last fold that will be the year "2026", drift compared to previous year.

I find that showing the dataframe and the time and potentially making associated plot will be easier to read.

For the dataset, I think that the bike sharing (e.g. https://scikit-learn.org/stable/auto_examples/applications/plot_time_series_lagged_features.html) has this kind of drift: it is a 2 year collection data and the second year as more demand than the first one. Therefore, learning the trend on the first year will not help to compensate the trend of the second year.

docs: add SKD003 inconsistent performance pitfall example

c2018b8

moujanrastgoo and others added 2 commits July 15, 2026 14:07

fix(docs): remove stray cell marker in SKD003 example

5529411

Use question heading without bold in SKD003 prose.

013ecfe

Co-authored-by: Cursor <cursoragent@cursor.com>

glemaitre reviewed Jul 17, 2026

View reviewed changes

docs: address SKD003 review feedback

d03c1d1

GaetandeCast reviewed Jul 23, 2026

View reviewed changes

docs: restructure SKD003 example around structured CV diagnostics

2cc7c40

This was referenced Jul 29, 2026

fix(skore): Metrics summary crash for cv report with report n_job != 1 #3175

Closed

Changing the number of jobs in a cv report creates a recursion error in metrics.summarize() #3177

Closed

GaetandeCast reviewed Jul 29, 2026

View reviewed changes

glemaitre self-requested a review July 30, 2026 08:01

glemaitre reviewed Jul 30, 2026

View reviewed changes

	# ``SKD003`` should flag split #0.
	# ``SKD003`` correctly flags split #0.

	diverge sharply from the median on a :class:`~skore.CrossValidationReport`.
	diverge sharply from the median during a cross-validation evaluation.

	(e.g. an easy class that becomes rarer),
	(e.g. fraudsters finding workarounds to circumvent controls),

		# subsample. SKD003 needs CV, so we use ``splitter=5`` or an explicit five-fold
		# splitter below.

Uh oh!

Conversation

moujanrastgoo commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change description

Contribution checklist

AI usage disclosure

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GaetandeCast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GaetandeCast left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glemaitre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

moujanrastgoo commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 15, 2026 •

edited

Loading