Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 3 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,27 +14,18 @@

# Leakly: Leakage checks for any machine-learning pipeline

`Leakly` uses label permutation to test whether a machine-learning pipeline
`Leakly` uses label permutation to test whether a machine-learning pipeline performs above chance when no true signal is present.

performs above chance when no true signal is present.

Above-chance performance after permutation may indicate leakage from

preprocessing, feature selection, tuning, or another step of the pipeline.
Above-chance performance after permutation may indicate leakage from preprocessing, feature selection, tuning, or another step of the pipeline.

## How it works

1. Permute labels to remove the real feature-label association.

2. Run the full pipeline exactly as in the original analysis.

3. Compare the permuted score distribution with chance level.

4. Above-chance permuted performance suggests possible leakage.

`Leakly` includes example configurations for a leaky pipeline and a non-leaky

pipeline so users can inspect the effect directly.
`Leakly` includes example configurations for a leaky pipeline and a non-leaky pipeline so users can inspect the effect directly.

![Example permutation AUC summary](https://raw.githubusercontent.com/DeMONLab-BioFINDER/Leakly/main/assets/AUC.png)

Expand All @@ -44,20 +35,6 @@ pipeline so users can inspect the effect directly.
pip install Leakly
```

For notebook environments that need the optional notebook dependencies:

```bash
pip install "Leakly[notebook]"
```

For the current GitHub checkout:

```bash
git clone https://github.com/DeMONLab-BioFINDER/Leakly.git
cd Leakly
pip install -e .
```

## Quick Start on Colab: <a href="https://colab.research.google.com/github/DeMONLab-BioFINDER/Leakly/blob/main/example.ipynb"><img alt="Open example.ipynb in Colab" src="https://img.shields.io/badge/Open-example.ipynb-F9AB00?logo=googlecolab&logoColor=white" height="28"></a>

### Key Python snippet
Expand Down
2 changes: 1 addition & 1 deletion leakly/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
from .simulation import SimulatedDataset, simulate_dataset
from .summary import SummaryPlotter

__version__ = "0.1.1"
__version__ = "0.1.2"

__all__ = [
"__version__",
Expand Down
Loading