Skip to content

Commit 65a7127

Browse files
matheuscscpmatheuscscp
committed
[RFC-0010] Introduce workload identity auth for remote clusters
Signed-off-by: Matheus Pimenta <[email protected]>
1 parent a342d00 commit 65a7127

30 files changed

+422
-98
lines changed

‎api/go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ go 1.24.0
44

55
require (
66
github.com/fluxcd/pkg/apis/kustomize v1.10.0
7-
github.com/fluxcd/pkg/apis/meta v1.12.0
7+
github.com/fluxcd/pkg/apis/meta v1.13.1-0.20250703213016-549b769de3a4
88
k8s.io/apiextensions-apiserver v0.33.0
99
k8s.io/apimachinery v0.33.0
1010
sigs.k8s.io/controller-runtime v0.21.0

‎api/go.sum‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
44
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
55
github.com/fluxcd/pkg/apis/kustomize v1.10.0 h1:47EeSzkQvlQZdH92vHMe2lK2iR8aOSEJq95avw5idts=
66
github.com/fluxcd/pkg/apis/kustomize v1.10.0/go.mod h1:UsqMV4sqNa1Yg0pmTsdkHRJr7bafBOENIJoAN+3ezaQ=
7-
github.com/fluxcd/pkg/apis/meta v1.12.0 h1:XW15TKZieC2b7MN8VS85stqZJOx+/b8jATQ/xTUhVYg=
8-
github.com/fluxcd/pkg/apis/meta v1.12.0/go.mod h1:+son1Va60x2eiDcTwd7lcctbI6C+K3gM7R+ULmEq1SI=
7+
github.com/fluxcd/pkg/apis/meta v1.13.1-0.20250703213016-549b769de3a4 h1:p8nA1j2PL7U60HT1FLsSnEZByurYkSNPtwKZjleG0lg=
8+
github.com/fluxcd/pkg/apis/meta v1.13.1-0.20250703213016-549b769de3a4/go.mod h1:+son1Va60x2eiDcTwd7lcctbI6C+K3gM7R+ULmEq1SI=
99
github.com/fxamacker/cbor/v2 v2.8.0 h1:fFtUGXUzXPHTIUdne5+zzMPTfffl3RD5qYnkY40vtxU=
1010
github.com/fxamacker/cbor/v2 v2.8.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
1111
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=

‎api/v1/zz_generated.deepcopy.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/v1beta2/zz_generated.deepcopy.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎config/crd/bases/kustomize.toolkit.fluxcd.io_kustomizations.yaml‎

Lines changed: 116 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -256,16 +256,47 @@ spec:
256256
a controller level fallback for when KustomizationSpec.ServiceAccountName
257257
is empty.
258258
properties:
259+
address:
260+
description: |-
261+
Address is the optional address of the Kubernetes API server.
262+
Not supported for the generic provider, optional for the
263+
other providers. The address is used to select among a list
264+
of endpoints in the cluster resource. If not set, the first
265+
endpoint on the list is used. If none of the addresses in the
266+
cluster resource match a provided address, the controller will
267+
error out and the reconciliation will fail. Must be a valid
268+
HTTPS endpoint, e.g. "https://api.example.com:6443".
269+
pattern: ^https://.*
270+
type: string
271+
cluster:
272+
description: |-
273+
Cluster is the optional fully qualified resource name of the
274+
Kubernetes cluster in the cloud provider to connect to.
275+
Not supported for the generic provider, required for the
276+
other providers.
277+
type: string
278+
provider:
279+
default: generic
280+
description: |-
281+
Provider is the optional name of the cloud provider that should be used
282+
to authenticate to the Kubernetes API server. Can be one of "aws",
283+
"azure", "gcp", or "generic". Defaults to "generic".
284+
enum:
285+
- aws
286+
- azure
287+
- gcp
288+
- generic
289+
type: string
259290
secretRef:
260291
description: |-
261-
SecretRef holds the name of a secret that contains a key with
292+
SecretRef holds an optional name of a secret that contains a key with
262293
the kubeconfig file as the value. If no key is set, the key will default
263294
to 'value'.
264295
It is recommended that the kubeconfig is self-contained, and the secret
265296
is regularly updated if credentials such as a cloud-access-token expire.
266297
Cloud specific `cmd-path` auth helpers will not function without adding
267298
binaries and credentials to the Pod that is responsible for reconciling
268-
Kubernetes resources.
299+
Kubernetes resources. Supported only for the generic provider.
269300
properties:
270301
key:
271302
description: Key in the Secret, when not specified an implementation-specific
@@ -277,9 +308,32 @@ spec:
277308
required:
278309
- name
279310
type: object
280-
required:
281-
- secretRef
311+
serviceAccountName:
312+
description: |-
313+
ServiceAccountName is the optional name of the Kubernetes
314+
ServiceAccount in the same namespace that should be used
315+
to authenticate to the Kubernetes API server. If not set,
316+
the controller ServiceAccount will be used. Not supported
317+
for the generic provider.
318+
type: string
282319
type: object
320+
x-kubernetes-validations:
321+
- message: .secretRef is required for the 'generic' .provider
322+
rule: has(self.secretRef) || (has(self.provider) && self.provider
323+
!= 'generic')
324+
- message: .secretRef is not supported for the specified .provider
325+
rule: '!has(self.secretRef) || !has(self.provider) || self.provider
326+
== ''generic'''
327+
- message: .serviceAccountName is not supported when .provider is
328+
empty or 'generic'
329+
rule: '!has(self.serviceAccountName) || (has(self.provider) && self.provider
330+
!= ''generic'')'
331+
- message: .cluster is not supported when .provider is empty or 'generic'
332+
rule: '!has(self.cluster) || (has(self.provider) && self.provider
333+
!= ''generic'')'
334+
- message: .address is not supported when .provider is empty or 'generic'
335+
rule: '!has(self.address) || (has(self.provider) && self.provider
336+
!= ''generic'')'
283337
namePrefix:
284338
description: NamePrefix will prefix the names of all managed resources.
285339
maxLength: 200
@@ -1347,16 +1401,47 @@ spec:
13471401
a controller level fallback for when KustomizationSpec.ServiceAccountName
13481402
is empty.
13491403
properties:
1404+
address:
1405+
description: |-
1406+
Address is the optional address of the Kubernetes API server.
1407+
Not supported for the generic provider, optional for the
1408+
other providers. The address is used to select among a list
1409+
of endpoints in the cluster resource. If not set, the first
1410+
endpoint on the list is used. If none of the addresses in the
1411+
cluster resource match a provided address, the controller will
1412+
error out and the reconciliation will fail. Must be a valid
1413+
HTTPS endpoint, e.g. "https://api.example.com:6443".
1414+
pattern: ^https://.*
1415+
type: string
1416+
cluster:
1417+
description: |-
1418+
Cluster is the optional fully qualified resource name of the
1419+
Kubernetes cluster in the cloud provider to connect to.
1420+
Not supported for the generic provider, required for the
1421+
other providers.
1422+
type: string
1423+
provider:
1424+
default: generic
1425+
description: |-
1426+
Provider is the optional name of the cloud provider that should be used
1427+
to authenticate to the Kubernetes API server. Can be one of "aws",
1428+
"azure", "gcp", or "generic". Defaults to "generic".
1429+
enum:
1430+
- aws
1431+
- azure
1432+
- gcp
1433+
- generic
1434+
type: string
13501435
secretRef:
13511436
description: |-
1352-
SecretRef holds the name of a secret that contains a key with
1437+
SecretRef holds an optional name of a secret that contains a key with
13531438
the kubeconfig file as the value. If no key is set, the key will default
13541439
to 'value'.
13551440
It is recommended that the kubeconfig is self-contained, and the secret
13561441
is regularly updated if credentials such as a cloud-access-token expire.
13571442
Cloud specific `cmd-path` auth helpers will not function without adding
13581443
binaries and credentials to the Pod that is responsible for reconciling
1359-
Kubernetes resources.
1444+
Kubernetes resources. Supported only for the generic provider.
13601445
properties:
13611446
key:
13621447
description: Key in the Secret, when not specified an implementation-specific
@@ -1368,9 +1453,32 @@ spec:
13681453
required:
13691454
- name
13701455
type: object
1371-
required:
1372-
- secretRef
1456+
serviceAccountName:
1457+
description: |-
1458+
ServiceAccountName is the optional name of the Kubernetes
1459+
ServiceAccount in the same namespace that should be used
1460+
to authenticate to the Kubernetes API server. If not set,
1461+
the controller ServiceAccount will be used. Not supported
1462+
for the generic provider.
1463+
type: string
13731464
type: object
1465+
x-kubernetes-validations:
1466+
- message: .secretRef is required for the 'generic' .provider
1467+
rule: has(self.secretRef) || (has(self.provider) && self.provider
1468+
!= 'generic')
1469+
- message: .secretRef is not supported for the specified .provider
1470+
rule: '!has(self.secretRef) || !has(self.provider) || self.provider
1471+
== ''generic'''
1472+
- message: .serviceAccountName is not supported when .provider is
1473+
empty or 'generic'
1474+
rule: '!has(self.serviceAccountName) || (has(self.provider) && self.provider
1475+
!= ''generic'')'
1476+
- message: .cluster is not supported when .provider is empty or 'generic'
1477+
rule: '!has(self.cluster) || (has(self.provider) && self.provider
1478+
!= ''generic'')'
1479+
- message: .address is not supported when .provider is empty or 'generic'
1480+
rule: '!has(self.address) || (has(self.provider) && self.provider
1481+
!= ''generic'')'
13741482
patches:
13751483
description: |-
13761484
Strategic merge and JSON patches, defined as inline YAML objects,

‎docs/spec/v1/kustomizations.md‎

Lines changed: 84 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -791,14 +791,42 @@ with:
791791
kustomize.toolkit.fluxcd.io/force: enabled
792792
```
793793

794-
### KubeConfig reference
794+
### KubeConfig (Remote clusters)
795+
796+
With the `.spec.kubeConfig` field a Kustomization
797+
can apply and manage resources on a remote cluster.
798+
799+
The field `.spec.kubeConfig.provider` determines the type of remote cluster
800+
and the authentication method used to connect to it. The following providers
801+
are supported:
802+
803+
- `generic`: The default. Uses a KubeConfig stored in the Kubernetes Secret
804+
referenced by `.spec.kubeConfig.secretRef`.
805+
- `aws`: Secret-less authentication for remote EKS clusters with IAM Roles.
806+
- `azure`: Secret-less authentication for remote AKS clusters with Managed
807+
Identities.
808+
- `gcp`: Secret-less authentication for remote GKE clusters with GCP Service
809+
Accounts or Workload Identity Federation.
810+
811+
When both `.spec.kubeConfig` and `.spec.ServiceAccountName` are specified,
812+
the controller will impersonate the ServiceAccount on the target cluster,
813+
i.e. a ServiceAccount with name `.spec.serviceAccountName` must exist in
814+
the target cluster inside a namespace with the same name as the namespace
815+
of the Kustomization. For example, if the Kustomization is in the namespace
816+
`apps` of the cluster where Flux is running from, then the ServiceAccount
817+
must be in the `apps` namespace of the target remote cluster, and have the
818+
name `.spec.serviceAccountName`. In other words, the namespace of the
819+
Kustomization must exist both in the cluster where Flux is running from
820+
and in the target remote cluster where Flux will apply resources.
821+
822+
#### Secret-based authentication
795823

796824
`.spec.kubeConfig.secretRef.Name` is an optional field to specify the name of
797825
the secret containing a KubeConfig. If specified, objects will be applied,
798826
health-checked, pruned, and deleted for the default cluster specified in that
799827
KubeConfig instead of using the in-cluster ServiceAccount.
800828

801-
The secret defined in the `kubeConfig.SecretRef` must exist in the same
829+
The secret defined in the `.spec.kubeConfig.secretRef` must exist in the same
802830
namespace as the Kustomization. On every reconciliation, the KubeConfig bytes
803831
will be loaded from the `.secretRef.key` key (default: `value` or `value.yaml`)
804832
of the Secret’s data , and the Secret can thus be regularly updated if
@@ -824,11 +852,61 @@ This matches the constraints of KubeConfigs from current Cluster API providers.
824852
KubeConfigs with `cmd-path` in them likely won't work without a custom,
825853
per-provider installation of kustomize-controller.
826854

827-
When both `.spec.kubeConfig` and `.spec.ServiceAccountName` are specified,
828-
the controller will impersonate the service account on the target cluster.
829-
830855
For more information, see [remote clusters/Cluster-API](#remote-clusterscluster-api).
831856

857+
#### Secret-less authentication
858+
859+
For the `aws`, `azure` and `gcp` providers, the controller supports
860+
secret-less authentication to remote EKS, AKS and GKE clusters
861+
respectively.
862+
863+
When `.spec.kubeConfig.provider` is set to `aws`, `azure` or `gcp`,
864+
the field `.spec.kubeConfig.cluster` becomes required and must be
865+
set to the fully qualified name of the cluster resource in the
866+
respective cloud provider:
867+
868+
* `aws`: `arn:<partition>:eks:<region>:<account-id>:cluster/<cluster-name>`
869+
* `azure`: `/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ContainerService/managedClusters/<cluster-name>`
870+
* `gcp`: `projects/<project-id>/locations/<location>/clusters/<cluster-name>`
871+
872+
Users can also optionally specify the address of the remote cluster
873+
API server with the `.spec.kubeConfig.address` field. This field is
874+
necessary in case the remote cluster has multiple addresses, e.g.
875+
public and private addresses. If not specified, the controller will
876+
select the first address available. If specified, the address has
877+
to match at least one of the addresses of the remote cluster,
878+
otherwise the controller will return an error.
879+
880+
The optional `.spec.kubeConfig.serviceAccountName` field can be used to
881+
specify a ServiceAccount in the same namespace as the Kustomization for
882+
object-level workload identity. Leaving this field empty will cause the
883+
controller to use its identity for getting access to the remote cluster.
884+
885+
For complete guides on workload identity and setting up permissions for
886+
this feature, see the following docs:
887+
888+
* [EKS](/flux/integrations/aws/#for-amazon-elastic-kubernetes-service)
889+
* [AKS](/flux/integrations/azure/#for-azure-kubernetes-service)
890+
* [GKE](/flux/integrations/gcp/#for-google-kubernetes-engine)
891+
892+
Example for an EKS cluster:
893+
894+
```yaml
895+
apiVersion: kustomize.toolkit.fluxcd.io/v1
896+
kind: Kustomization
897+
metadata:
898+
name: apps
899+
namespace: flux-system
900+
spec:
901+
... # other fields omitted for brevity
902+
kubeConfig:
903+
provider: aws
904+
cluster: arn:aws:eks:eu-central-1:123456789012:cluster/my-cluster
905+
address: https://my-prod-cluster.us-east-1.eks.amazonaws.com # optional
906+
serviceAccountName: apps-iam-role # optional. maps to a cloud identity. used for authentication
907+
serviceAccountName: apps-sa # optional. must exist in the target cluster. user for impersonation
908+
```
909+
832910
### Decryption
833911

834912
Storing Secrets in Git repositories in plain text or base64 is unsafe,
@@ -1355,7 +1433,7 @@ specified will use the service account name provided by
13551433

13561434
### Remote clusters/Cluster-API
13571435

1358-
With the [`.spec.kubeConfig` field](#kubeconfig-reference) a Kustomization can be fully
1436+
With the [`.spec.kubeConfig` field](#kubeconfig-remote-clusters) a Kustomization can be fully
13591437
reconciled on a remote cluster. This composes well with Cluster API bootstrap
13601438
providers such as CAPBK (kubeadm), CAPA (AWS) and others.
13611439

0 commit comments

Comments
 (0)