Chasing the cause of diffs in testpackage Flowthrough_damr

[ykempf]

It has been observed for a year or two that Flowthrough_damr yields relative diffs of order 1e-6 in testpackage runs even in cases when a branch being tested only changes things that do not affect the computations directly, e.g. #1099 in recent weeks for me. As discussed at #1099 and #1161 I dug deeper and this issue is to keep notes of what I observed, and how to reproduce for further investigation.

The key new observation is that I get diffs even when running Flowthrough_damr serially, meaning that our previous interpretation that these were the result of threading/summation differences especially at AMR and MPI domain boundaries is at best incomplete. Of course threading etc will amplify diffs possibly, but the setup documented here yields diffs serially, again.

Another key observation is that the diffs jump up significantly (sometimes more than x10 increase) when AMR happens, but not at plain LB, which I confirmed by setting LB to occur every 5 steps, and AMR to occur every 20 steps (so faster LB but same AMR intervals as the default Flowthrough_amr). I did not see strong jumps in diffs that are not just after an AMR.

Things that had little or no impact on the overall level of diffs include

• changing the LB algorithm (not relevant given the serial diffs observation)
• compiling with O0
• disabling jemalloc
• using short pencils
• using fallback vectorisation
• disabling the randomised order of acceleration directions
• disabling fsgrid filtering
• 1st order space and time

Steps to reproduce and see the diffs:

• build dev and a branch, e.g. the current state of FsGrid & fieldsolver refactoring #1099
• run Flowthrough_damr (increase IO cadence to write a file every 0.8 s i.e. every 10 steps)
• run vlsvdiff_DP for proton/vg_rho
• the diffs go from relative 1e-12 at the beginning to 1e-6 about half-way through the test.

Changing some of the parameters I tested above may change e.g. how early the first big jump in diffs occurs, but none got rid of that jump and subsequent increase in diffs, as opposed to what I would see as acceptable levels of ~1e-12 relative diffs throughout without such massive jumps.

It would seem that short pencils "help" in bringing the abrupt jumps to the fore.

[ykempf]

Oh. Progress. I will stop after this!!
If I run the same Flowthrough_damr cfg serially on dev with default setting or short pencils, it refines the first single cell at tstep 80 and the next file has vg_rho diffs of 1.25 (absolute) / 6.27e-07 (relative). After clean zeros before that.

Now it's well possible I'm not up2date on this and short pencils are not supported or something, in which case this is a harmless red herring, but if they're supposed to behave the same this is a big problem.

[ykempf]

This is where the first diffs appear:

[ykempf]

More specifically, writing out every step, at tstep 80 we refine, we write a file immediately after that (file index 81) and that's still got clean diffs, and then it goes awry:

80      0       0                                                                                                                                                                                                                           
81      0       0                                                                                                                                                                                                                           
82      4.66e-10        2.33e-16                                                                                                                                                                                                            
83      0.000278        1.39e-10                                                                                                                                                                                                            
84      0.0016  8.01e-10                                                                                                                                                                                                                    
85      0.012   5.98e-09                                                                                                                                                                                                                    
86      0.0398  1.99e-08                                                                                                                                                                                                                    
87      0.11    5.48e-08                                                                                                                                                                                                                    
88      0.279   1.39e-07                                                                                                                                                                                                                    
89      0.518   2.59e-07                                                                                                                                                                                                                    
90      0.835   4.17e-07                                                                                                                                                                                                                    
91      1.25    6.27e-07                                                                                                                                                                                                                    
92      1.78    8.88e-07                                                                                                                                                                                                                    
93      2.45    1.22e-06                                                                                                                                                                                                                    
94      3.26    1.63e-06                                                                                                                                                                                                                    
95      4.31    2.16e-06                                                                                                                                                                                                                    
96      5.75    2.88e-06                                                                                                                                                                                                                    
97      7.58    3.79e-06                                                                                                                                                                                                                    
98      9.85    4.93e-06                                                                                                                                                                                                                    
99      12.8    6.38e-06                                                                                                                                                                                                                    
100     16.4    8.18e-06    

[ykempf]

1e-16 level rel. diffs (acceptable, possibly)

1e-10 diffs, growing thereafter

[ykempf]

After discussing with @markusbattarbee I am now testing whether the diffs I obtain are only the sign of having "too many"/"a large count of"/"realistic numbers of" v-space blocks which lead to summation differences even serially e.g. when running short vs long pencils.

I now tested

• dev vs branch with Flowthrough_damr, serially, and halving the block count per dimension
• dev with Flowthrough_damr, short vs long pencils, dividing the block count per dimension by three (and six, so 20³ and 10³ blocks)

and for all these cases diffs start growing somewhat later, yet they reach relative 1e-6 or 1e-5 in proton/vg_rho very soon and apparently they even rise faster than in the base case (that might be less of a surprise).

[ykempf]

To me this smells like there's something not quite up to date at some stage in relation to AMR with existing/required neighbouring blocks which works "well-enough" when the VDFs are not changing too drastically, but goes somewhat off-track in this setup of very steep/quick changes. If Markus' argument of "so many blocks that summations suffer" were holding, I'd expect less diffs with coarser v-space more in line with other tests in the package. Right?

[ykempf]

Running with RANDOM load balance and 16 tasks, if I rerun the same cfg and binary (so Flowthrough_damr, dev branch) I get similar diffs. (diff of vg_rank confirms the LB changes slightly) And that's independent of short and long pencils, or reducing v-space size.

My gut feeling is that there is some subtle blunder lurking, but I can't find where it would be hiding. It might be standard summation issues, but I have a hard time believing it's "only" that. Ärsyttää...