Learning curve Train/test #427
Description
Hello,
First of all, thank you for all the work that has gone into building this tool and for the excellent documentation 🥳.
With @saraltamargo, we are analysing some very low-quality data that we still need to try to salvage due to its clinical and biological value. We have encountered several cases where the training curve is above the learning curve, and conceptually we don’t quite understand how this can happen or what it means. I’m attaching an image of one of the samples along with the err output. (between 50-100 epoch)
cellbender:remove-background: Command: cellbender remove-background --cuda --input /mnt/beegfs/mguerrero/.../GEX_Pt3_232HMLLT4_cellbender.h5 --epochs 150 --fpr 0.01 --learning-rate 1e-5 --z-dim 100 --expected-cells 20000 --low-count-threshold 60 --total-droplets-included 40000 cellbender:remove-background: CellBender 0.3.0 cellbender:remove-background: (Workflow hash 230373e805) cellbender:remove-background: 2025-09-24 10:07:37 cellbender:remove-background: Running remove-background cellbender:remove-background: Loading data from /mnt/beegfs/mguerrero/CD19rel_mguerrero/.../GEX_Pt3_232HMLLT4/outs/raw_feature_bc_matrix cellbender:remove-background: CellRanger v3 format cellbender:remove-background: Features in dataset: 60671 Gene Expression cellbender:remove-background: Trimming features for inference. cellbender:remove-background: 34074 features have nonzero counts. cellbender:remove-background: Prior on counts for cells is 4261 cellbender:remove-background: Prior on counts for empty droplets is 163 cellbender:remove-background: Excluding 7975 features that are estimated to have <= 0.1 background counts in cells. cellbender:remove-background: Including 26099 features in the analysis. cellbender:remove-background: Trimming barcodes for inference. cellbender:remove-background: Excluding barcodes with counts below 81 cellbender:remove-background: Using 20000 probable cell barcodes, plus an additional 20000 barcodes, and 110554 empty droplets. cellbender:remove-background: Largest surely-empty droplet has 163 UMI counts. cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmpbn6pvtg7 cellbender:remove-background: Successfully unpacked tarball to /tmp/tmpbn6pvtg7
Thank you very much for your help 💖!