Issue with mdt-model-fit NODDI, getting stuck at "We will use the optimizer Powell with default settings"

[DarvinShehu]

Hi Robbert,

We are running into an issue when trying to run the mdt-model-fit NODDI command in the MDT pipeline. The process seems to get stuck at the following line:
"We will use the optimizer Powell with default settings"

The pipeline works perfectly on a Fedora 39 workstation equipped with an Nvidia Quadro P5000 GPU and driver version 555.58.02. However, when we run the same command on an Ubuntu 24 virtual machine with an Nvidia GeForce RTX 3090 and driver version 565.57.01, the process gets stuck at the optimizer initialization.
We have tried a workaround by running the first part of the pipeline on the Fedora machine—up to the BallStick_r1 model fitting step—and then continuing the rest on the Ubuntu VM. This approach works, which suggests that the issue is specifically related to the BallStick_r1 model fitting, more precisely during the optimization phase, in the Ubuntu environment.

We were wondering:

Do you have any suggestions for troubleshooting this optimizer-related issue?

Is there a way to skip or modify the first part of the pipeline so that, for example, it uses the BallStick_r2 model instead of BallStick_r1? From what we understand, the pipeline runs a sequence of models in a cascade, but we haven't figured out exactly how it selects which models to use.

We’re also attaching screenshots from both machines to provide more context.

Thanks in advance for your help, and thank you for your great work on MDT!

[robbert-harms]

Hi DarvinShehu, I apologies that at the moment I don't have the time to give you a proper response. I am very occupied.

The optimizer related issue is a common one and is typically a compiler issue. On some versions of the Nvidia OpenCL drivers, the Powell routine fails to compile properly and then it hangs. You could try one of the other optimization routines. Or try to run it on the CPU instead? Try POCL?

It is possible to skip the whole Cascading, although I have to look up how again. Perhaps it is in the documentation? The cascades do improve results typically.

One day I hope to fix all open issues.

Best, Robbert

[DarvinShehu]

Hi Robbert,

I’ve successfully modified the optimization routine, and the NODDI model now fits correctly. We're currently moving on to the MCMC sampling step, but I've run into an issue: when executing mdt.sample_model() from a Python script, the process hangs—similarly to what happened earlier with the Powell optimization routine.

Do you have any insights or suggestions on what might be causing this or how we could resolve it?

Thanks in advance!

[robbert-harms]

Hi Darvin, probably the batch size is too large. You can make a configuration file in your home directory and include there a batch size (i.e. the number of voxels in one run). Decreasing this typically works wonders. (see https://mdt-toolbox.readthedocs.io/en/latest_release/configuration.html#advanced-configuration to creating a default config).