How to manage remote GKE clusters using Workload Identity?

[2hamed]

Hi,

What I want to do is, to use a single instance of FluxV2 to manage workloads on multiple remote clusters. We managed to do this before by manually creating user tokens on the remote clusters and putting them inside the kubeconfig manifest for those in the central Flux.

Now what I want to do is basically the same, but instead of static tokens I want to use Workload Identity.

I have created the necessary SA bindings and have also added the necessary annotation on the helm-controller and kustomize-controller KSAs.

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: kustomize-controller
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/version: v2.2.2
  annotations:
    iam.gke.io/gcp-service-account: "[email protected]"
  name: kustomize-controller
  namespace: flux-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: helm-controller
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/version: v2.2.2
  annotations:
    iam.gke.io/gcp-service-account: "[email protected]"
  name: helm-controller
  namespace: flux-system

Now this is how a kubeconfig manifest generally looks like:

apiVersion: v1
kind: Config
clusters:
- name: ${cluster_name}
  cluster:
    certificate-authority-data: ${ca}
    server: ${server}
contexts:
- name: default
  context:
    cluster: ${cluster_name}
    namespace: default
    user: flux
current-context: default
users:
- name: flux
  user:
    token: ${token}

I wonder if there is a way to make Flux controllers authenticate via the serviceAccount token instead of this user mentioned in the kubeconfig?

[bm1216]

Is this possible wrt flux? Looking at the source-code, it doesn't seem to be as the kubeconfig is used to generate a vanilla k8s client.

If GCP WiF is possible with respect to storage it should be allowed for GKE Service Accounts. Argo support it - https://github.com/argoproj/argo-cd/blob/master/cmd/argocd-k8s-auth/commands/gcp.go as well.

Would be a lovely feature to have!

[matheuscscp]

We will investigate a solution for this after the implementation of RFC-0010, you can track it here: #5022

[matheuscscp]

This is being introduced here and will be released in Flux 2.7

[matheuscscp]

Implemented in kustomize-controller here: fluxcd/kustomize-controller#1476
Implemented in helm-controller here: fluxcd/helm-controller#1249