[Feature]: Use Common Controller/Event Watcher Framework across controllers

[ivelichkovich]

Prerequisites

• I searched existing issues

Feature Summary

Currently, most of the actors on the health events like fault-quarantine, fault-remediation, node-drainer etc all have similar EventWatcher needs. Each has their own slightly different implementation. We should solidified to a single shared EventWatcher/Reconciler/Controller thing that each of the modules can reuse.

Problem/Use Case

Useful for maintainability and readability.

Proposed Solution

Common shared eventwatcher package. It looks like one already exists in store-client but we just need to move to using it. So maybe this work is in progress already? But creating this issue for tracking.

Component

Multiple Components

[ivelichkovich]

Theoretically it seems like it should be possible to use a controller-runtime controller for all these components if we can find a clever way to plugin the datasource stream (mongo/postgres) to the controller-runtime. That would come with a ton of benefits if we can figure it out

[ivelichkovich]

Seems like it's pretty well supported actually to just pass a channel to ctrl-runtime. There would be some changes to the reconcilers needed to fit into the ctrl-runtime patterns but imho it's a good direction to take. Let me know if you want to discuss offline. This is another item we'd be happy to help out with.

[lalitadithya]

I'm really glad that you brought this up. Yes, using controller runtime's external source mechanism was something on my radar as well, but we never really could get to it. From what I have read, it is possible to hook up any external source using channels, so in theory it should be possible to route any data source via controller runtime.

I think it is a good direction, let's talk about it on the Slack channel. I think we should take a simple reconciler and do a POC and try out our use cases and any edge cases we may run into. The fault remediation one might be the simplest and might be good to do a POC on.

[ivelichkovich]

Sure I can take a swing at that as a POC and we can see how it looks!