feat(dir): add wait check to dependencies for apiserver

[arpad-csepi]

Both the API server and the reconciler previously assumed their dependencies (PostgreSQL and the OCI registry) were already reachable the moment the process started. In Kubernetes environments where pod scheduling is not strictly ordered, this caused crash-loops during initial deploy/rolling restarts: the service failed on first connection attempt and the pod restarted before dependencies came up.

This PR introduces a configurable startup-wait mechanism. Before establishing any connections, each process probes its required dependencies and retries with exponential backoff until they become ready or a configurable timeout is reached. The behavior is opt-in via Helm values now and can be enabled per-dependency but should not broke any existing deployment workflow if opt-out (needs feedback, my preference opt-out).

An alternative solution can be implemented with Kubernetes init-containers but this approach is environment independent.

[codecov]

Codecov Report

❌ Patch coverage is 79.85075% with 27 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
server/startup/dependencies.go	81.1%	9 Missing and 8 partials ⚠️
reconciler/main.go	0.0%	6 Missing ⚠️
server/server.go	0.0%	2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[ramizpolic]

Although I understand the rationale, but is this really an issue? Kubernetes will sometimes deploy services not in order and will cause API server to crash, but it will be restarted until it's deployed and actually ready.
So even if we have initial crashes, this is resolved after some restarts. I do not see this as an issue or problematic behavior since it is eventually consistent behavior.

[arpad-csepi]

Although I understand the rationale, but is this really an issue? Kubernetes will sometimes deploy services not in order and will cause API server to crash, but it will be restarted until it's deployed and actually ready. So even if we have initial crashes, this is resolved after some restarts. I do not see this as an issue or problematic behavior since it is eventually consistent behavior.

In this case when Kubernetes deployment is ongoing, it cause unnecessary noise in form of log spamming and restart count increase. It is not ideal specially on version upgrade to have these kind of distractions, but I can understand the eventually consistent concept behind Kubernetes too.

However this solution covers deployment outside of Kubernetes where this consistent concept are not there, like docker, native, etc...

Do you think this "standardized" health check is worth it to have (in every environment) even if not aligned with Kubernetes consistent concept? Do you favor to handle the health checks for every type of environment instead?

[tkircsi]

I think a "retry" mechanism could be useful, not only during initialization but also while the program is running. That's why I would make it much more generic. The logic/process itself is almost the same for any resource.

1. input parameters: timeouts, attempts, probeFn
2. run probeFn
3. ok/nok (retry)

[arpad-csepi]

I think a "retry" mechanism could be useful, not only during initialization but also while the program is running. That's why I would make it much more generic. The logic/process itself is almost the same for any resource.

1. input parameters: timeouts, attempts, probeFn

2. run probeFn

3. ok/nok (retry)

@tkircsi
Make sense to want to be more generic solution. Is it a good solution if a healthcheck interface is provided, the probeFn is in the directory code for each module where needed and the rest of parameters are coming from config/helm values?

@ramizpolic
Would this work for you as well?