inference extra latency outside of a continuous loop

[SpitchAG]

Hello
I would like to understand where comes an extra latency when executing a model inference:
if I run for (N times) model.infer(); sleep(1s); repeat ...
the very first model.infer() (within a loop) has always a penalty (instead of 60ms, it goes up to 110ms). Without the sleep(), no penalty (the first of the first loop always has, but then the repeated loops dont show this latency).
is this expected? how to mitigate ? (without doing busy dummy infer)
is that related to how tbb worker threads are rescheduled/initialized ? something else?
thanks!