diff --git a/README.md b/README.md index b3cdab4..35a51b0 100644 --- a/README.md +++ b/README.md @@ -14,21 +14,27 @@ # Leakly: Leakage checks for any machine-learning pipeline -`Leakly` uses label permutation to test whether a pipeline still performs above -chance when no true signal is present. +`Leakly` uses label permutation to test whether a machine-learning pipeline -If it does, the pipeline may be leaking test-set information -through preprocessing, feature selection, tuning, or another step. +performs above chance when no true signal is present. -## Principle +Above-chance performance after permutation may indicate leakage from + +preprocessing, feature selection, tuning, or another step of the pipeline. + +## How it works + +1. Permute labels to remove the real feature-label association. + +2. Run the full pipeline exactly as in the original analysis. + +3. Compare the permuted score distribution with chance level. -1. Permute labels to remove real signal. -2. Run the full pipeline exactly as a user would run it. -3. Compare the score distribution with chance level. 4. Above-chance permuted performance suggests possible leakage. -Leakly includes example configurations for a leaky pipeline and a non-leaky -pipeline so users can see the effect immediately. +`Leakly` includes example configurations for a leaky pipeline and a non-leaky + +pipeline so users can inspect the effect directly. ![Example permutation AUC summary](https://raw.githubusercontent.com/DeMONLab-BioFINDER/Leakly/main/assets/AUC.png) @@ -52,9 +58,7 @@ cd Leakly pip install -e . ``` -## Quick Start - -### Open example.ipynb in Colab +## Quick Start on Colab: Open example.ipynb in Colab ### Key Python snippet @@ -63,38 +67,44 @@ from leakly import ( MLPipeline, SummaryPlotter, load_example_leakage_config, - permute_label, -) + permute_label) scores = [] for seed in range(100): - permuted_y = permute_label(data.y, random_state=seed) + permuted_y = permute_label(y, random_state=seed) score = ( - # user could replace with any pipeline + # Replace with any user-defined pipeline MLPipeline( - data.X, + X, permuted_y, - covariates=data.covariates, + covariates=covariates, config=load_example_leakage_config(), - ).fit()).evaluate() + ).fit() + ).evaluate() scores.append(score) -SummaryPlotter(scores, chance_level=0.5).plot("assets/AUC.png") +SummaryPlotter(scores, chance_level=0.5).plot() ``` ## FAQ -**Can Leakly check my own pipeline?** +**Can `Leakly` check my own pipeline?** -Yes. Leakly can evaluate any pipeline that takes `X`, `y`, optional `covariates`, and returns a test score. The key is to run the full pipeline exactly as in the real analysis, including preprocessing, feature selection, tuning, and evaluation. +Yes. `Leakly` can evaluate any pipeline that takes `X`, `y`, +optional `covariates`, and returns a test score. +The key is to run the full pipeline exactly as in the real analysis, +including preprocessing, feature selection, tuning, and evaluation. **Why can a leaky pipeline score well on permuted labels?** -After label permutation, there should be no real biological, clinical, or statistical link between features and outcomes. A valid pipeline should therefore perform near chance. - -A leaky pipeline may still score well if information from the full dataset enters the analysis before the train/test split or outside the cross-validation loop. Common sources include feature selection, scaling, imputation, covariate adjustment, dimensionality reduction, or hyperparameter tuning performed on all samples. +If leakage occurs, information from test samples can enter the analysis before +the train/test split or outside the cross-validation loop. Common sources +include feature selection, scaling, imputation, covariate adjustment, +dimensionality reduction, or hyperparameter tuning performed on all samples. -This is especially problematic in high-dimensional data, such as neuroimaging, omics, or biomarker studies, where random label-specific patterns can appear meaningful by chance. If the test set influences preprocessing or feature selection, the model may "remember" these random patterns and show inflated performance. +In high-dimensional data such as omics and neuroimaging, random features can +appear predictive by chance. If a pipeline can retain these spurious patterns, +it may perform above chance even after labels are permuted. **How many permutations should I run?** diff --git a/assets/AUC.png b/assets/AUC.png index f06e8f0..adcea18 100644 Binary files a/assets/AUC.png and b/assets/AUC.png differ diff --git a/example.ipynb b/example.ipynb index 7980b94..cfdf3e2 100644 --- a/example.ipynb +++ b/example.ipynb @@ -64,7 +64,9 @@ "metadata": {}, "source": [ "## 1. Synthesizing Data\n", - "We will start by synthesizing a dataset that simulates a classification problem. We will use the `simulate_dataset` function from the `Leakly` package to create a dataset with a specified configuration." + "We will start by synthesizing a dataset that simulates a classification problem. We will use the `simulate_dataset` function from the `Leakly` package to create a dataset with a specified configuration.\n", + "\n", + "User can modify the parameters of the dataset to create different scenarios for testing leakage. For example, you can specify the number of samples, features, and the effect size of the features across groups.\n" ] }, { @@ -80,11 +82,11 @@ "simulated_data = simulate_dataset(\n", " SimulationConfig(\n", " n_samples=200, # number of samples in the dataset\n", - " n_features=1000, # number of features in the dataset\n", + " n_features=1000, # number of features => mimic high-dimensional data\n", " n_covariates=3, # number of covariates in the dataset\n", - " effect_fraction=0.1, # fraction of features that have an effect\n", - " effect_size=0.5, # size of the effect for the informative features\n", - " class_balance=0.5, # proportion of positive class\n", + " effect_fraction=0.3, # fraction of features that have an effect\n", + " effect_size=0.3, # size of the effect for the informative features\n", + " class_balance=0.3, # proportion of positive class\n", " random_state=42, # random seed for reproducibility\n", " )\n", ")\n", @@ -218,7 +220,7 @@ "from leakly import SummaryPlotter\n", "\n", "leakage_plotter = SummaryPlotter(leakage_permuted_scores, chance_level=0.5)\n", - "leakage_plotter.plot()\n", + "leakage_plotter.plot('./assets/AUC.png')\n", "\n", "nonlocal_plotter = SummaryPlotter(nonleakage_permuted_scores, chance_level=0.5)\n", "nonlocal_plotter.plot()"