You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is the remaining Gap B from #734. PR #742 completed Gap A (structured logging) and closed #734 via its Fixes reference, but schema-drift detection was intentionally deferred to a separate contribution.
SmartPredictor.check_dataset_features() validates feature names and compatible dtypes, but structurally valid input can still drift materially from the reference data used to create the predictor. Examples include a large increase in missing values, numeric ranges or quantiles moving outside the reference distribution, or categorical frequencies changing substantially. These inputs pass structural validation and may produce predictions and explanations whose behavior differs from what operators expect.
Overview of the Solution:
Implement a lightweight, dependency-free drift mechanism tailored to SmartPredictor, following the direction agreed in #734:
use skrub's per-column summary ideas as inspiration only; do not add it as a dependency;
store schema_distribution inside the SmartPredictor pickle, not in the external manifest sidecar;
preserve backward compatibility for older predictors without this attribute;
compute compact reference summaries for numeric and categorical columns;
compare incoming data in add_input() against those reference summaries;
Numeric: missing-value rate, min/max, median and selected quantiles.
Categorical: missing-value rate, cardinality and top-K category frequencies.
The first version should warn rather than reject input. A strict error mode can be considered later if operational use cases require it.
Examples:
predictor.add_input(x=current_batch)
# UserWarning: Potential schema drift detected for 'age':# missing rate changed from 0.02 to 0.18; median moved outside the# reference interquartile range.
The same event should be available to configured logging handlers on shapash.smartpredictor, including the schema fingerprint when available.
Blockers:
No known blockers. The storage and dependency decisions were discussed in #734: dependency-free implementation, with distribution summaries stored as a SmartPredictor attribute.
Definition of Done:
Reference numeric and categorical summaries are stored in SmartPredictor.schema_distribution.
add_input() compares incoming data with the stored summaries.
Material drift emits actionable per-column UserWarning messages.
Drift events are emitted on shapash.smartpredictor with the schema fingerprint when available.
Predictors created before this feature remain loadable and skip drift checks when no reference summary exists.
Tests cover numeric, categorical and missing-rate drift, stable input, logging, and backward compatibility.
All new code has type hints and introduces no new runtime dependency.
Description of Problem:
This is the remaining Gap B from #734. PR #742 completed Gap A (structured logging) and closed #734 via its
Fixesreference, but schema-drift detection was intentionally deferred to a separate contribution.SmartPredictor.check_dataset_features()validates feature names and compatible dtypes, but structurally valid input can still drift materially from the reference data used to create the predictor. Examples include a large increase in missing values, numeric ranges or quantiles moving outside the reference distribution, or categorical frequencies changing substantially. These inputs pass structural validation and may produce predictions and explanations whose behavior differs from what operators expect.Overview of the Solution:
Implement a lightweight, dependency-free drift mechanism tailored to SmartPredictor, following the direction agreed in #734:
schema_distributioninside theSmartPredictorpickle, not in the external manifest sidecar;add_input()against those reference summaries;UserWarningper materially shifted column and log the drift result through theshapash.smartpredictorlogger introduced by feat(smartpredictor): add structured logging on the shapash.smartpredictor logger #742.Proposed initial summaries:
The first version should warn rather than reject input. A strict error mode can be considered later if operational use cases require it.
Examples:
The same event should be available to configured logging handlers on
shapash.smartpredictor, including the schema fingerprint when available.Blockers:
No known blockers. The storage and dependency decisions were discussed in #734: dependency-free implementation, with distribution summaries stored as a SmartPredictor attribute.
Definition of Done:
SmartPredictor.schema_distribution.add_input()compares incoming data with the stored summaries.UserWarningmessages.shapash.smartpredictorwith the schema fingerprint when available.References: #734, #742, #722, #711.