Cluster Authentication via Workload Identity for Generic Provider

[Preisschild]

Reference: fluxcd/pkg#960, https://github.com/fluxcd/flux2/blob/main/rfcs/0010-multi-tenant-workload-identity/README.md#api-changes-and-feature-gates, fluxcd/kustomize-controller#1476
Slack thread: https://cloud-native.slack.com/archives/C01A402P0R2/p1752066566113149

Description

In the currently proposed RFC-0010 implementation, it is not supported to authenticate against the target cluster using workload identity if you are using a generic kubernetes cluster

It would be useful to cover this use case for generic clusters too. This could be possible by creating a serviceaccount token with the audience set to the target cluster's address and then using this token to authenticate against the target cluster. The target cluster needs to have the source cluster's service account issuer discovery endpoints configured as an additional kube-apiserver jwt issuer in the target cluster.

Example kube-apiserver AuthenticationConfiguration:

---
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
    url: https://<hub cluster domain>:6443
    audiences:
    - https://<target cluster domain>:6443
    certificateAuthority: |-
      <hub cluster apiserver CA>
  claimMappings:
    username:
    claim: "sub"
    prefix: "hub-cluster:"

If Flux controllers then create a token for the serviceaccount referenced in .spec.kubeConfig.serviceAccountName of the kustomization / helmrelease, with the audience set to .spec.kubeConfig.address using the TokenRequest API in the source cluster it can then authenticate against the target cluster using this serviceaccount.

For example, if you'd create a Kustomization like this:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sync
  namespace: tenant-a
spec:
  kubeConfig:
    address: https://<target cluster domain>:6443
    serviceAccountName: reconciler
    

it should authenticate towards the management cluster using a user named hub-cluster:system:serviceaccount:tenant-a:reconciler

kube-apiserver CA cert

Most generic clusters have a self signed kube-apiserver certificate, so we'd also need to have an additional kubeConfig.certSecretRef field so that the CA can be referenced too.

[matheuscscp]

So we discussed this in the open dev meeting today, and we decided to go with a different API:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sync
  namespace: tenant-a
spec:
  kubeConfig:
    configMapRef:
      name: my-kube-config

Where my-kube-config references a ConfigMap like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-kube-config
  namespace: tenant-a
data:
  provider: generic # or 'aws', 'azure' or 'gcp'
  cluster: ... # fully qualified name of the cluster resource. for 'aws', 'azure' or 'gcp' only (required if one of .address or ca.crt is not specified)
  address: https://<api server endpoint> # used by all providers. will be the default audience for the 'generic' provider if .audiences is not specified. optional for 'aws', 'azure' and 'gcp'
  serviceAccountName: tenant-a-sa # optional. defaults to the controller SA
  ca.crt: | # optional. used by all providers
    <PEM-encoded cluster CA>
  audiences: | # optional. used by all providers
    - aud-1
    - aud-2

When both data.address and data["ca.crt"] are specified, the aws, azure and gcp providers can avoid an API call to the cloud provider. If data.address is specified but not data["ca.crt"], the address must match one of the addresses returned by the cloud provider API. If data["ca.crt"] is specified but not data.address, the address will be the first one returned by the cloud provider API and the CA returned by the cloud provider API will be ignored.

The reasons for this API change are:

• Many of these values may not be available at the time the Kustomization/HelmRelease is applied, for example, if CAPI is being used to provision the cluster in another Kustomization/HelmRelease this Kustomization/HelmRelease would depend on. By referencing a ConfigMap, there's space for the ConfigMap being created after the Cluster is fully reconciled, and both Kustomization/HelmRelease objects can be applied together (the one deploying the cluster and the one deploying things inside it).
• If there are many Kustomization/HelmRelease objects in a given namespace targeting the same cluster with the same ServiceAccount, they can all share this ConfigMap.
• We want to emphasize that the CA is not a secret. Workload identity is about secret-less authentication.