CD daemon: process manager: backoff on daemon restarts

[jgehrcke]

For config failures and similar, right now we busy-spin on daemon restarts managed by process manager.

Let's introduce an adaptive wait time here:

k8s-dra-driver-gpu/cmd/compute-domain-daemon/process.go

Line 195 in 7b5e2cd

klog.Warningf("Watchdog: start process again")

Example, showing unreasonably high rate:

W1014 09:04:42.467224       1 process.go:189] Watchdog: child terminated unexpectedly
[...]
W1014 09:04:42.467236       1 process.go:195] Watchdog: start process again
[...]
W1014 09:04:43.467276       1 process.go:189] Watchdog: child terminated unexpectedly
[...]
W1014 09:04:44.467334       1 process.go:195] Watchdog: start process again

[jgehrcke]

I misread the logs -- the timestamps above show that there's a one second delay between attempts. That's expected. We're not busy-spinning.

We can add a back-off later, of course. But there's nothing to do here for the 25.8.0 release.

[klueska]

Yes, that is what I pointed out in our call yesterday.

[jgehrcke]

what I pointed out in our call yesterday.

Misunderstanding. I thought we were both seeing two restarts within the same millisecond.