Skip to content

Add lightweight schema-drift detection to SmartPredictor (Gap B follow-up to #734) #755

Description

@immu4989

Description of Problem:

This is the remaining Gap B from #734. PR #742 completed Gap A (structured logging) and closed #734 via its Fixes reference, but schema-drift detection was intentionally deferred to a separate contribution.

SmartPredictor.check_dataset_features() validates feature names and compatible dtypes, but structurally valid input can still drift materially from the reference data used to create the predictor. Examples include a large increase in missing values, numeric ranges or quantiles moving outside the reference distribution, or categorical frequencies changing substantially. These inputs pass structural validation and may produce predictions and explanations whose behavior differs from what operators expect.

Overview of the Solution:

Implement a lightweight, dependency-free drift mechanism tailored to SmartPredictor, following the direction agreed in #734:

  • use skrub's per-column summary ideas as inspiration only; do not add it as a dependency;
  • store schema_distribution inside the SmartPredictor pickle, not in the external manifest sidecar;
  • preserve backward compatibility for older predictors without this attribute;
  • compute compact reference summaries for numeric and categorical columns;
  • compare incoming data in add_input() against those reference summaries;
  • emit UserWarning per materially shifted column and log the drift result through the shapash.smartpredictor logger introduced by feat(smartpredictor): add structured logging on the shapash.smartpredictor logger #742.

Proposed initial summaries:

  • Numeric: missing-value rate, min/max, median and selected quantiles.
  • Categorical: missing-value rate, cardinality and top-K category frequencies.

The first version should warn rather than reject input. A strict error mode can be considered later if operational use cases require it.

Examples:

predictor.add_input(x=current_batch)
# UserWarning: Potential schema drift detected for 'age':
# missing rate changed from 0.02 to 0.18; median moved outside the
# reference interquartile range.

The same event should be available to configured logging handlers on shapash.smartpredictor, including the schema fingerprint when available.

Blockers:

No known blockers. The storage and dependency decisions were discussed in #734: dependency-free implementation, with distribution summaries stored as a SmartPredictor attribute.

Definition of Done:

  • Reference numeric and categorical summaries are stored in SmartPredictor.schema_distribution.
  • add_input() compares incoming data with the stored summaries.
  • Material drift emits actionable per-column UserWarning messages.
  • Drift events are emitted on shapash.smartpredictor with the schema fingerprint when available.
  • Predictors created before this feature remain loadable and skip drift checks when no reference summary exists.
  • Tests cover numeric, categorical and missing-rate drift, stable input, logging, and backward compatibility.
  • All new code has type hints and introduces no new runtime dependency.

References: #734, #742, #722, #711.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions