Support multiple datastores per TenantControlPlane

[synthe102]

Hi 👋,

My team is planning to run very large clusters using Kamaji as control plane provider. We would like to store k8s Events in a secondary ETCD cluster, which would require to set this flag --etcd-servers-overrides="/events#<etcd_urls>".
While we could use ControlPlaneExtraArgs to set this flag, we think it might be better to add a new field to TenantControlPlaneSpec named DataStoreOverrides:

type DataStoreOverride struct {
	Resource  string
	DataStore string
}

type TenantControlPlaneSpec struct {
	DataStoreOverrides []DataStoreOverride `json:"dataStoreOverrides,omitempty"`
	...
}

This field would be used to generate the --etcd-servers-overrides flag value.

Some of the possible limitations are:

• Datastore migration: maybe it could be blocked it if DataStoreOverride is set ?
• ETCD auth: the client certificate used by the kube-apiserver need to work for all the Datastores

What's your opinion on this feature ?

[prometherion]

The --etcd-server-overrides has been the hidden gem for running Kubernetes at scale. Kamaji's scope has been making Control Plane management easier in regard to Cluster Sprawl, and we have never focused so much on large clusters.

But this is not a limitation.

I would be happy to help shape this feature to simplify it and make it more accessible.

• Datastore migration: maybe it could be blocked it if DataStoreOverride is set ?

The overrides are mostly used for Events, Leases, and disposable resources: I would ignore them in case of migration to a different datastore, unless we track in the state which Datastore is, and migrate data accordingly as we're doing with the Datastore.

• ETCD auth: the client certificate used by the kube-apiserver need to work for all the Datastores

The Kubernetes API Server ignores authentication for those etcd overrides: although we strived for a security-first implementation in Kamaji, Kubernetes ditched this feature. AFAIK, most of the adopters taking advantage of the etcd overrides feature, are using a separate etcd with no credentials at all.

[synthe102]

Great ! Thanks for the feedback.
I think that ignoring the migration for DataStoreOverrides makes sense.
When using a driver different from ETCD, kine runs as sidecar on each apiserver Pod. How should we handle kine with DataStoreOverrides ? Run additional kine sidecars or allow DataStoreOverrides to only use ETCD datastores ?

[prometherion]

When Kamaji creates a TenantControlPlane backed by a DataStore with a non etcd driver, it creates the kine sidecar container, etc.

We could do the same here, but as a first iteration, I would focus in supporting only etcd to simplify things: WDYT?

[synthe102]

That works for me !