Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
63fdb38
refine mock predictors
zacharopoulou Apr 29, 2026
f79ad3a
Add miRDB and TEC-miTarget to benchmark
zacharopoulou May 10, 2026
f0ad0ce
Improve benchmark result directory names
zacharopoulou May 10, 2026
840bff9
Refine benchmark reporting outputs
zacharopoulou May 10, 2026
4db978d
Add report bundle audit helper
zacharopoulou May 10, 2026
218ad60
Cover report bundle audit helper
zacharopoulou May 10, 2026
841e377
Split benchmark config helpers
zacharopoulou May 10, 2026
fc1a691
Split benchmark report helpers
zacharopoulou May 10, 2026
2844b8a
Slim benchmark entrypoint
zacharopoulou May 10, 2026
a95f271
Remove report bundle audit helper
zacharopoulou May 10, 2026
e6a4588
Remove report bundle audit tests
zacharopoulou May 10, 2026
2627a89
Use date-only benchmark run directories
zacharopoulou May 10, 2026
953782f
Write top prediction CDF plots by default
zacharopoulou May 10, 2026
db6f520
Enable top prediction CDFs in default config
zacharopoulou May 10, 2026
aa8c57a
Update benchmark tests for date runs and CDF plots
zacharopoulou May 10, 2026
b7feba2
Add publication comparison plot repair
zacharopoulou May 10, 2026
8e3be4b
Wire publication common plot repair into benchmark
zacharopoulou May 10, 2026
0c0ef3b
Add publication run PDF report writer
zacharopoulou May 10, 2026
4696b3c
Use publication run PDF report writer
zacharopoulou May 10, 2026
d4ff315
Cover publication common plot repair
zacharopoulou May 10, 2026
995a67f
Add publication predictor PDF reports
zacharopoulou May 10, 2026
ce19cf0
Rewrite predictor PDFs with publication layout
zacharopoulou May 10, 2026
c22b06d
Use landscape figure pages in run report
zacharopoulou May 10, 2026
368ebb4
Configure publication figure DPI
zacharopoulou May 10, 2026
c565080
Add predictor combination frontier analysis
zacharopoulou May 10, 2026
796a74e
Write predictor combination outputs in benchmark
zacharopoulou May 10, 2026
2a0e0e2
Include predictor combination frontier in report
zacharopoulou May 10, 2026
fbb467a
Cover predictor combination frontier analysis
zacharopoulou May 10, 2026
cca2792
Add common prediction coverage summaries
zacharopoulou May 10, 2026
c73446c
Report common prediction percentages in predictor PDFs
zacharopoulou May 10, 2026
a5fbc15
Replace correlation outputs with common prediction summaries
zacharopoulou May 10, 2026
1a41873
Cover common prediction summaries
zacharopoulou May 10, 2026
ca011d1
Add common prediction percentages to run report
zacharopoulou May 10, 2026
a75d20f
Silence correlation cleanup during benchmark runs
zacharopoulou May 10, 2026
9c311c2
Remove correlation cleanup helper
zacharopoulou May 10, 2026
769b04b
Remove correlation cleanup from benchmark
zacharopoulou May 10, 2026
39a7ae0
Rename predictor combination module
zacharopoulou May 10, 2026
c902339
Rename common comparison plot module
zacharopoulou May 10, 2026
d557d97
Add neutral dataset report wrapper
zacharopoulou May 10, 2026
4c44619
Add neutral run report wrapper
zacharopoulou May 10, 2026
4e26cc3
Use neutral report and plot module names
zacharopoulou May 10, 2026
ba6a92d
Rename predictor combination tests
zacharopoulou May 10, 2026
1508e2e
Rename common comparison plot tests
zacharopoulou May 10, 2026
b067703
Remove old publication combination test name
zacharopoulou May 10, 2026
bd22145
Remove old publication plot test name
zacharopoulou May 10, 2026
a72f8fa
Use neutral common prediction summary names
zacharopoulou May 10, 2026
691f369
Use report common coverage argument name
zacharopoulou May 10, 2026
048fffa
Use neutral report configuration keys
zacharopoulou May 10, 2026
74b3cf6
Remove old publication combination module name
zacharopoulou May 10, 2026
97f7671
Add numbered step logging to miRDB pipeline.
stephaniesamm May 7, 2026
3d9f9db
Improve benchmark result directory names
zacharopoulou May 10, 2026
05acd98
Add report bundle audit helper
zacharopoulou May 10, 2026
c317669
Cover report bundle audit helper
zacharopoulou May 10, 2026
42e99be
Handle empty common prediction report inputs
zacharopoulou May 10, 2026
02d5244
Remove predictor correlation heatmap plot
zacharopoulou May 10, 2026
99c000f
Split cross-dataset evaluation outputs
zacharopoulou May 10, 2026
b298d36
include pos recovery by prediction count
zacharopoulou May 11, 2026
f4cf8a1
Refine publication plots and reports
dimostzim May 23, 2026
6b2cffc
extra info for microt-cnn included
zacharopoulou May 25, 2026
661778b
Include time in benchmark run directory names
zacharopoulou Jun 1, 2026
8ae9939
Allow benchmark experiments from local files or Zenodo
zacharopoulou Jun 1, 2026
4146c37
Allow effect-only benchmark threshold configuration
zacharopoulou Jun 1, 2026
b2f1281
Rename benchmark effect threshold config key
zacharopoulou Jun 1, 2026
d6f5744
Read effect threshold benchmark config key
zacharopoulou Jun 1, 2026
68cdc65
Allow effect-only experiment validation
zacharopoulou Jun 1, 2026
0becb7b
Support effect-only evaluation without FDR filtering
zacharopoulou Jun 1, 2026
430d9b5
Use effect-only benchmark configuration
zacharopoulou Jun 1, 2026
cd94f51
Describe effect-only rule in dataset reports
zacharopoulou Jun 1, 2026
098f23c
Drop invalid FDR rows during validation
zacharopoulou Jun 1, 2026
81431b6
update predictors and experiments
zacharopoulou Jun 1, 2026
02a505c
clean print on reports
zacharopoulou Jun 1, 2026
b53b5d9
Optimize joined table construction and add join timing logs
zacharopoulou Jun 1, 2026
13f5a85
Log detailed join timings during benchmark runs
zacharopoulou Jun 1, 2026
1d22aa9
Improve heatmap gene label readability
zacharopoulou Jun 1, 2026
d7a26e0
Use readable heatmap plotting helpers
zacharopoulou Jun 1, 2026
d0f7cd3
rename scripts
zacharopoulou Jun 1, 2026
b822ea8
Export readable heatmap helper functions
zacharopoulou Jun 1, 2026
e95d659
Keep heatmap methodology and add labeled heatmap pages
zacharopoulou Jun 1, 2026
c292e76
Render top positive heatmap as one unlabeled figure
zacharopoulou Jun 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,9 +297,12 @@ normal workflow is just to edit or uncomment filters.

## Outputs

After a benchmark run, `results/` contains one new run folder, for example:
After a benchmark run, `results/` contains one new run folder named with the run date. For example:

- `results/tag-demo__exp3__pred2__oe1__ko1__kd0__cell3/`
- `results/20260510/`

Detailed dataset, miRNA, predictor, perturbation, cell-line, and evaluation-threshold metadata is
recorded inside the run folder.

Inside each run folder you get:

Expand All @@ -312,7 +315,7 @@ Inside each run folder you get:
- `datasets/<dataset_id>/reports/`: per-dataset Markdown/PDF reports and correlation TSVs
- `tables/per_experiment/`: per-experiment metric tables
- `tables/combined/`: cross-dataset predictor summary table
- `plots/combined/metrics/`, `plots/combined/coverage/`, `plots/combined/ranks/`: cross-dataset comparison plots grouped by theme
- `plots/combined/metrics/`, `plots/combined/ranks/`, `plots/combined/combinations/`: cross-dataset comparison plots grouped by theme
- `summary.json`: run summary

When 2 or more predictors are selected, each dataset gets:
Expand All @@ -321,7 +324,7 @@ When 2 or more predictors are selected, each dataset gets:
- one combined PR curve on common scored pairs
- one combined ROC curve on common scored pairs
- one algorithms-vs-genes heatmap
- one predictor-correlation heatmap
- common-prediction overlap summaries for comparable predictor sets

## Commands

Expand Down
13 changes: 8 additions & 5 deletions benchmark.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ predictions_tsv: metadata/predictions_info.tsv
# The default repo config ships with a small 3-dataset demo selection.
# Edit or comment these active rows to benchmark other subsets.
experiments:
id: [GSE109725_OE_miR_204_5p, GSE118315_KO_miR_124_3p, GSE210778_OE_miR_375_3p]
id: [GSE124411_OE_miR_150_5p_2, GSE129076_OE_miR_450a_5p_1]
# mirna_name: [hsa-miR-375-3p]
# mirna_sequence: [UUUGUUCGUUCGGCUCGCGUGA]
# article_pubmed_id: [https://pubmed.ncbi.nlm.nih.gov/36074222]
Expand All @@ -25,9 +25,9 @@ experiments:

# Filter predictors by ANY column in the predictions TSV.
# The default repo config compares two random baselines, the cheating and perfect
# demo predictors, and TargetScan.
# demo predictors, TargetScan, miRDB, and TEC-miTarget.
predictors:
tool_id: [random, random_3000, cheating, perfect, targetscan, microt_cnn]
tool_id: [random, random_3000, cheating, perfect, targetscan, mirdb_mirtarget, tec-mitarget, microt_cnn, mirbind2, miraw]
# official_name: [Random, Random (3000 per dataset), Cheating, Perfect]
# organism: [Homo sapiens]
# score_type: [probability]
Expand All @@ -40,8 +40,11 @@ predictors:
# predictor_output_path: [data/predictions/random/random_standardized.tsv]

evaluation:
fdr_threshold: 0.05
abs_logfc_threshold: 1.0
fdr_threshold: null
effect_threshold: 1.0
predictor_top_fraction: 0.10
write_top_prediction_cdfs: true
report_min_common_coverage: 0.10
figure_dpi: 600

out_dir: results/

This file was deleted.

16,524 changes: 0 additions & 16,524 deletions data/experiments/processed/18745741/GSE112859_edger_out_oe_hsa_miR_574_3p_oe_3p.tsv

This file was deleted.

15,767 changes: 0 additions & 15,767 deletions data/experiments/processed/18745741/GSE112859_edger_out_oe_hsa_miR_574_5p_oe_5p.tsv

This file was deleted.

18,919 changes: 0 additions & 18,919 deletions data/experiments/processed/18745741/GSE118315_edger_out_ko_hsa_miR_124_3p_2.tsv

This file was deleted.

This file was deleted.

This file was deleted.

21,403 changes: 0 additions & 21,403 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_let_7c_5p_oe_7c5p.tsv

This file was deleted.

21,037 changes: 0 additions & 21,037 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_107_oe_107.tsv

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

21,062 changes: 0 additions & 21,062 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_133b_oe_133b.tsv

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

22,697 changes: 0 additions & 22,697 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_16_5p_oe_165p.tsv

This file was deleted.

22,337 changes: 0 additions & 22,337 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_17_5p_oe_175p.tsv

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

22,137 changes: 0 additions & 22,137 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_206_oe_206.tsv

This file was deleted.

This file was deleted.

22,845 changes: 0 additions & 22,845 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_21_5p_oe_215p.tsv

This file was deleted.

22,602 changes: 0 additions & 22,602 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_31_5p_oe_315p.tsv

This file was deleted.

This file was deleted.

21,532 changes: 0 additions & 21,532 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_9_3p_oe_93p.tsv

This file was deleted.

21,826 changes: 0 additions & 21,826 deletions data/experiments/processed/18745741/GSE124530_edger_out_oe_hsa_miR_9_5p_oe_95p.tsv

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

18,226 changes: 0 additions & 18,226 deletions data/experiments/processed/18745741/GSE154968_edger_out_oe_hsa_miR_21_3p.tsv

This file was deleted.

14,943 changes: 0 additions & 14,943 deletions data/experiments/processed/18745741/GSE160736_edger_out_oe_hsa_miR_642a_5p.tsv

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

16,768 changes: 0 additions & 16,768 deletions data/experiments/processed/18745741/GSE165469_edger_out_ko_hsa_miR_340_5p_ko.tsv

This file was deleted.

17,286 changes: 0 additions & 17,286 deletions data/experiments/processed/18745741/GSE165469_edger_out_oe_hsa_miR_340_5p_oe.tsv

This file was deleted.

Loading