Better parallelization for metrics/cms

[LuLeom]

metrics/cms is a computationally demanding metric which requires parallelization.

As of now (PR #79) I am using the library BiocParallel which allows me to specify the following (line 41 script.R):

cms<- cms(
...
BPPARAM = MulticoreParam(workers = 8)

Although, this is an arbitrary number of workers hardcoded by me.
I believe there could be a better strategy to dynamically set this value when running the full pipeline on the cloud.

I was thinking of something like

cores_avail <- parallel::detectCores() - 1 #leaving one core free
cores_to_use <- min(trsh, cores_avail)
cms<- cms(
...
BPPARAM = MulticoreParam(workers = cores_to_use)

However, I do not know the effect of trying to use all available cores (except one) when multiple nextflow workflows are running, so we might need a maximum threshold trsh of cores to use (?).

Any input is appreciated!

[lazappi]

I think using all the available cores should be ok when running the workflow as Nextflow restricts the number of cores that are available for each process. You can check this by having the component print out the number of cores it is using to make sure. This could still be an issue if the component is run outside the workflow though.

[LuLeom]

When running scripts/run_benchmark/run_test_local.sh the following error is returned

  Error in reducer$value.cache[[as.character(idx)]] <- values :
    wrong args for environment subassignment
  Calls: cms ... .bploop_impl -> .collect_result -> .reducer_add -> .reducer_add
  In addition: Warning message:
  In parallel::mccollect(wait = FALSE, timeout = 1) :
    1 parallel job did not deliver a result
  Execution halted

The BBPARAM option has been temporarily disabled (PR #83).

[lazappi]

This looks like it might be an R error, does it work when you run the component separately outside the workflow?

Here is a script for another project that runs CMS in case it's useful for reference https://github.com/theislab/atlas-feature-selection-benchmark/blob/main/bin/metric-cms.R.

[LuLeom]

Thanks for the hint @lazappi .

I see that the script is using parallelly::availableCores() instead of parallel::detectCores().
I did some tests and both seem to work fine if run separately outside the workflow (e.g. via viash test config.vsh.yaml).

However, running the full workflow locally fails (with the same error) even if parallelly::availableCores() is used.

[lazappi]

Ok, I guess you need to look more into the error then. Does it happen with every dataset?

This thread suggests it might be caused but running out of memory sneumann/xcms#437. That could be the case if you have lots of cores but not enough memory for each of them. You could test this by reducing the number of cores. If that is the problem, you might want to try a different parallel backend (I think some of them can share memory).

[ghar1821]

I was able to run the method with <= 5 cores locally. Anything >= 7 will give the same error. I've also tried using SnowParam, thinking the problem is with the forking nature of MulticoreParam. That didn't help. Atm, I'm limiting it to just 5 cores or less (if the system can only spare < 5 cores)