Option to select individual for upload #111

dnil · 2022-12-16T13:06:00Z

This PR adds | fixes:

adds an option to the case load to specify which individual to load for multi individual vcfs

How to prepare for test:

ssh to ...
Install on stage:
bash servers/resources/SERVER.scilifelab.se/update-[THIS_TOOL]-stage.sh [THIS-BRANCH-NAME]

How to test:

Expected outcome:

[ ]

Review:

Code approved by
Tests executed by
"Merge and deploy" approved by

This version is a:

MAJOR - when you make incompatible API changes
MINOR - when you add functionality in a backwards compatible manner
PATCH - when you make backwards compatible bug fixes or documentation/instructions

dnil · 2022-12-16T14:54:26Z

Hmm, not quite. Errors are out, but it appears only SVs loaded ok. Could be the qual + GT call often 0/0 for somatic, but will test some more...

[daniel.nilsson@hasta:~] [S_base] 5s $ loqusdb-somatic load --case-id grandmarmot --variant-file /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz --sv-variants /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SV.somatic.grandmarmot.svdb.vcf.gz --check-profile /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.germline.tumor.dnascope.vcf.gz --gq-treshold 10 --hard-threshold 0.95 --soft-threshold 0.9 --select_individual TUMOR
2022-12-16 15:43:56 hasta.scilifelab.se loqusdb.commands.cli[165904] INFO Running loqusdb version 2.6.9
2022-12-16 15:43:56 hasta.scilifelab.se mongo_adapter.client[165904] INFO Connecting to uri:mongodb://loqusdb-stage:******@cg-mongo-stage.scilifelab.se:27030
2022-12-16 15:43:56 hasta.scilifelab.se mongo_adapter.client[165904] INFO Connection established
2022-12-16 15:43:56 hasta.scilifelab.se mongo_adapter.adapter[165904] INFO Use database loqusdb-somatic-stage
2022-12-16 15:43:56 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Check if vcf is on correct format...
[W::hts_idx_load3] The index file is older than the data file: /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz.tbi
2022-12-16 15:43:59 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Vcf file /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz looks fine
2022-12-16 15:43:59 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Nr of variants in vcf: 223798
2022-12-16 15:43:59 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Type of variants in vcf: snv
2022-12-16 15:43:59 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Check if vcf is on correct format...
[W::vcf_parse] FILTER 'GERMLINE' is not defined in the header
2022-12-16 15:44:00 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Vcf file /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SV.somatic.grandmarmot.svdb.vcf.gz looks fine
2022-12-16 15:44:00 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Nr of variants in vcf: 17580
2022-12-16 15:44:00 hasta.scilifelab.se loqusdb.utils.vcf[165904] INFO Type of variants in vcf: sv
[W::hts_idx_load3] The index file is older than the data file: /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz.tbi
[W::hts_idx_load3] The index file is older than the data file: /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz.tbi
Inserting variants  [####################################]  100%          2022-12-16 15:44:10 hasta.scilifelab.se loqusdb.utils.load[165904] INFO Inserted 0 variants of type snv
Inserting variants  [#-----------------------------------]    3%  00:00:44[W::vcf_parse] FILTER 'GERMLINE' is not defined in the header
Inserting variants  [####################################]  100%          2022-12-16 15:45:02 hasta.scilifelab.se loqusdb.utils.load[165904] INFO Inserted 17580 variants of type sv

dnil · 2022-12-16T14:59:00Z

Yeah, it was the genotype quality:

[daniel.nilsson@hasta:~] [S_base] 5s $ loqusdb-somatic load --case-id grandmarmot --variant-file /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz --sv-variants /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SV.somatic.grandmarmot.svdb.vcf.gz --check-profile /home/proj/stage/housekeepermarmot/2022-12-05/SNV.germline.tumor.dnascope.vcf.gz --gq-treshold 0 --hard-threshold 0.95 --soft-threshold 0.9 --select_individual TUMOR

2022-12-16 15:56:43 hasta.scilifelab.se loqusdb.commands.cli[175675] INFO Running loqusdb version 2.6.9
2022-12-16 15:56:43 hasta.scilifelab.se mongo_adapter.client[175675] INFO Connecting to uri:mongodb://loqusdb-stage:******@cg-mongo-stage.scilifelab.se:27030
2022-12-16 15:56:43 hasta.scilifelab.se mongo_adapter.client[175675] INFO Connection established
2022-12-16 15:56:43 hasta.scilifelab.se mongo_adapter.adapter[175675] INFO Use database loqusdb-somatic-stage
2022-12-16 15:56:43 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Check if vcf is on correct format...
[W::hts_idx_load3] The index file is older than the data file: /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz.tbi
2022-12-16 15:56:47 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Vcf file /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz looks fine
2022-12-16 15:56:47 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Nr of variants in vcf: 223798
2022-12-16 15:56:47 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Type of variants in vcf: snv
2022-12-16 15:56:47 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Check if vcf is on correct format...
[W::vcf_parse] FILTER 'GERMLINE' is not defined in the header
2022-12-16 15:56:47 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Vcf file /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SV.somatic.grandmarmot.svdb.vcf.gz looks fine
2022-12-16 15:56:47 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Nr of variants in vcf: 17580
2022-12-16 15:56:47 hasta.scilifelab.se loqusdb.utils.vcf[175675] INFO Type of variants in vcf: sv
[W::hts_idx_load3] The index file is older than the data file: /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz.tbi
[W::hts_idx_load3] The index file is older than the data file: /home/proj/stage/housekeeper-bundles/grandmarmot/2022-12-05/SNV.somatic.grandmarmot.tnscope.vcf.gz.tbi
Inserting variants  [####################################]  100%          2022-12-16 15:57:21 hasta.scilifelab.se loqusdb.utils.load[175675] INFO Inserted 223798 variants of type snv
Inserting variants  [#-----------------------------------]    3%  00:00:43[W::vcf_parse] FILTER 'GERMLINE' is not defined in the header
Inserting variants  [####################################]  100%          2022-12-16 15:58:13 hasta.scilifelab.se loqusdb.utils.load[175675] INFO Inserted 17580 variants of type sv
2022-12-16 15:58:13 hasta.scilifelab.se loqusdb.commands.load[175675] INFO Nr variants inserted: 241378
2022-12-16 15:58:13 hasta.scilifelab.se loqusdb.commands.load[175675] INFO Time to insert variants: 0:01:30.178186
2022-12-16 15:58:13 hasta.scilifelab.se loqusdb.plugins.mongo.adapter[175675] INFO All indexes exists

henrikstranneheim

Nice 👍 .

loqusdb/build_models/case.py

loqusdb/build_models/variant.py

Co-authored-by: Henrik Stranneheim <[email protected]>

Bumps [setuptools](https://github.com/pypa/setuptools) from 59.2.0 to 65.5.1. - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/CHANGES.rst) - [Commits](pypa/setuptools@v59.2.0...v65.5.1) --- updated-dependencies: - dependency-name: setuptools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>

dnil added 2 commits December 16, 2022 14:00

Option to select individual for upload

fe84dd1

relax profile and match check for missing samples in multi vcf

f6586c2

henrikstranneheim approved these changes Jan 10, 2023

View reviewed changes

loqusdb/build_models/case.py Show resolved Hide resolved

loqusdb/build_models/case.py Outdated Show resolved Hide resolved

loqusdb/build_models/case.py Show resolved Hide resolved

loqusdb/build_models/variant.py Outdated Show resolved Hide resolved

dnil and others added 11 commits February 8, 2023 09:31

A couple of type hints and hopefully fixing an old bug.

ea08564

Update loqusdb/build_models/case.py

6a466b1

Co-authored-by: Henrik Stranneheim <[email protected]>

imports for typehints

84464af

temp reverting for tests

cee622b

actually just weird naming - adding another typehint

8602e07

Merge branch 'master' into select_dup_check_ind

e04af12

Bump version: 2.7.0 → 2.7.1 [skip ci]

bd9328c

Option to select individual for upload

90001cc

imports for typehints

907c86f

Merge branch 'master' into select_dup_check_ind

835c0eb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Option to select individual for upload #111

Option to select individual for upload #111

Uh oh!

dnil commented Dec 16, 2022

Uh oh!

dnil commented Dec 16, 2022 •

edited

Loading

Uh oh!

dnil commented Dec 16, 2022

Uh oh!

henrikstranneheim left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Option to select individual for upload #111

Are you sure you want to change the base?

Option to select individual for upload #111

Uh oh!

Conversation

dnil commented Dec 16, 2022

This PR adds | fixes:

How to test:

Expected outcome:

Review:

Uh oh!

dnil commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dnil commented Dec 16, 2022

Uh oh!

henrikstranneheim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dnil commented Dec 16, 2022 •

edited

Loading