A major finding while looking into #723 was that
- the rate limiter implementations we use for task reconciliation do not use jitter
- natural system jitter can be very small; and lock-step behavior can actually result in unnecessary amount of conflicts
For more detail, see #723 (comment)
Conclusion: at least for the CD daemon's reconciliation loop we have to implement a simple jitter to spread/smear out the updates performed by individual nodes a little more evenly across time.