[RFC-0010] Full multi-tenancy lockdown support

[matheuscscp]

Part of: #5022

In multi-tenant clusters, users normally set the optional controller flag --default-service-account in kustomize-controller and helm-controller. This dictates that any Flux Kustomization and HelmRelease objects must either specify the field .spec.serviceAccountName or use the value of --default-service-account as the name of the ServiceAccount in the same namespace of the object that needs to be impersonated for the reconciliation.

After RFC-0010, Flux now supports .spec.serviceAccountName for most APIs and also .spec.decryption.serviceAccountName for Kustomization. For KubeConfig Workload Identity in the Kustomization and HelmRelease APIs, the ConfigMap pointed at by .spec.kubeConfig.configMapRef may specify the field .data.serviceAccountName. This creates three classes of ServiceAccount name fields that need to be locked down on multi-tenant clusters:

• --default-service-account must be implemented across all Flux controllers, for enforcing multi-tenancy lockdown over .spec.serviceAccountName fields. Only kustomize-controller and helm-controller currently implement this flag.
• --default-decryption-service-account must be implemented in kustomize-controller, for enforcing multi-tenancy lockdown over the .spec.decryption.serviceAccountName field of the Kustomization API.
• --default-kubeconfig-service-account must be implemented in both kustomize-controller and helm-controller, for enforcing multi-tenancy lockdown over the .data.serviceAccountName field in ConfigMaps pointed at by the .spec.kubeConfig.configMapRef field present in the Kustomization and HelmRelease APIs.

For the implementation we must define three constants in the auth package, one for each controller flag. The file can be controller_flags.go (we should rename feature_gate.go to feature_gates.go). The main.go of each controller must import and use the relevant constants accordingly.

[cappyzawa]

I'll work on this! 🚀

Based on the discussion with @matheuscscp, here's the implementation plan:

Implementation Approach

pkg/auth changes:

• Create controller_flags.go with constants like ControllerFlagDefaultServiceAccount = "default-service-account"
• Add unexported globals and setter/getter functions (similar to feature_gate.go pattern)
• Implement lockdown logic in access_token.go:32 and restconfig.go:94

Controller updates:

• KC/HC: Use constants for flag binding, keep existing logic for --default-service-account (it's for resource impersonation, not workload identity)
• Other controllers: Integrate with pkg/auth for workload identity lockdown

PR Strategy:
One PR per repository in this order:

• pkg: [RFC-0010] auth: add lockdown support for multi-tenant workload identity pkg#1008
• image-automation-controller: [RFC-0010] Add default-service-account for lockdown image-automation-controller#952
• kustomize-controller: [RFC-0010] Add multi-tenancy lockdown for decryption and kubeconfig kustomize-controller#1495
• helm-controller: [RFC-0010] Add multi-tenancy lockdown for kubeconfig helm-controller#1284
• source-controller: [RFC-0010] Add default-service-account for lockdown source-controller#1872
• notification-controller: [RFC-0010] Add default-service-account for lockdown notification-controller#1161
• image-reflector-controller: [RFC-0010] Add default-service-account for lockdown image-reflector-controller#807
• website: [RFC-0010] Add default service account flags for lockdown website#2323

[matheuscscp]

In the website PR we need to update https://fluxcd.io/flux/security/best-practices/#multi-tenancy-lock-down and the controller flags (they are/should be in alphabetical order):

• https://fluxcd.io/flux/components/kustomize/options/
• https://fluxcd.io/flux/components/helm/options/
• https://fluxcd.io/flux/components/source/options/
• https://fluxcd.io/flux/components/notification/options/
• https://fluxcd.io/flux/components/image/options/