Extremely slow reference-based refinement

[cbiertue]

Hi,
I am testing the tutorial workflow for the ribosome data set EMPIAR-10304 (12 tomos) in nextPYP. I am stuck with an extremely slow reference-based refinement (4th step, 11267 particles) with the default parameters from the workflow. I am using 80 CPUs, and it looks like it would take 34 days:

pyp/system/mpi.py:95 | Running 1,092 command(s) (pyp/external/CSP/csp frealign/maps/tilt8_r01.cistem frealign/maps/tilt8_r01_extended.cistem 5 0 0 1 frealign/tilt8.mrc frealign/tilt8_stack.mrc > tilt8_csp_000000_000000.log)
2% 21:22:36 < 820:03:45 2761.67s/it

Any ideas what can be wrong or what can speed up the process?
Thanks!

[abartesaghi]

For reference, the tutorial was benchmarked using 124 threads per tilt-series, with all 12 tilt-series running in parallel. Even with that setup, the reference-based refinement step still takes around 14 hours to complete. So what you’re observing is unfortunately not entirely unexpected, because you are using a total of 80 threads vs. the 1488 we used.

The reference-based refinement step is currently a known bottleneck in nextPYP. We are actively working on improving its performance and expect to release significant optimizations in the next couple of months.

In the meantime, version 0.7.3 (which has not being officially released but is already usable), gives you the option to do traditional sub-volume averaging to obtain initial orientations, which is significantly faster and does not require alignment to an external reference. However, we have not yet published a step-by-step tutorial for that workflow, but it is high on our list of priorities.

[cbiertue]

Thank you Alberto, it is useful to know that we do not do anything wrong. We are looking into accessing more computational resources to circumvent this bottleneck...and looking forward to your improvements. We are also testing the traditional sub-volume averaging for another project.