BUG: `sample_smc` can lead to compilation halting

[tvwenger]

Describe the issue:

I upgraded to pymc 5.27.0 and noticed that my models do not sample with sample_smc nearly as efficiently. I just confirmed this on a fresh install and running the model from https://www.pymc.io/projects/examples/en/latest/samplers/SMC2_gaussians.html

Using pymc=5.23.0, each chain starts up right away and completes sampling within 10 seconds. The output is:

Chain 0  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 1  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 2  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 3  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 4  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 5  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 6  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 7  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 8  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 9  ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 10 ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000
Chain 11 ⠸ -:--:-- / 0:00:09 Stage: 0 Beta: 1.000

Using pymc=5.27.0, chains take almost 1 minute to start, and they do not all start at the same time, and they take longer to finish. The output is

Chain 0  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 1  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 2  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 3  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 4  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 5  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 6  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 7  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 8  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 9  ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 10 ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000
Chain 11 ⠦ -:--:-- / 0:01:39 Stage: 0 Beta: 1.000

Reproduceable code example:

https://www.pymc.io/projects/examples/en/latest/samplers/SMC2_gaussians.html

Error message:

PyMC version information:

5.27.0

Context for the issue:

There is a 10x runtime increase in sample_smc between 5.23.0 and 5.27.0.

[ricardoV94]

Can you narrow down the culprit version more?

[tvwenger]

The solution was deleting the ~/.pytensor directory... Now every version has similar sampling times...

[ricardoV94]

That's odd, hopefully somebody will figure out why one day

[tvwenger]

@ricardoV94 I found the solution by

1. creating a fresh install with 5.20.0. Everything worked fine/fast.
2. ditto for 5.20.1
3. Then, starting with 5.21.0, the model would never even start sampling. It just hangs until I killed it after a minute or two. Same for 5.22.0, and again for 5.23.0.
4. I thought I was losing my mind, because it had worked fine on 5.23.0 when I originally posted this issue. But now on a fresh 5.23.0 install it failed.
5. Then I tried it in my original 5.27.0 environment and same problem -- the model wouldn't even start sampling.
6. I considered ripping open my PC in frustration.
7. While I was looking for a hammer, either the 5.23.0 or 5.27.0 (I can't remember) sampler threw an error related to a lock file in ~/.pytensor.
8. So I nuked ~/.pytensor and voila, every version runs fine.

I'm not sure if this is enough info to hunt down whatever was going in ~/.pytensor, but I can post an issue there too if you think it would be helpful.

[ricardoV94]

That would be extremely helpful. Sounds like some cache issue with multiprocessing, which is not unheard off, but with a version twist. How many chains were you running in parallel btw?

[tvwenger]

Alright, I'll post an issue in pytensor.

I was running 12 chains in parallel.

[ricardoV94]

In general pytensor compilation is not thread safe, specifically you can hit lock race conditions. We may not have thought about that well when implementing SMC, which compiles the functions in each process, unlike MCMC which compiles once and then pickles/unpickles. This was attempted in #7472