Multiple Clusters with single cluster definition in read only.

[Klaven]

Ok, so here is what I want to do. I want to spin up hundreds of clusters and I want all of these clusters too look exactly the same. My thought was to make a single git repo with a single cluster definition. And then point each cluster/flux to that single git repo and have flux sync the cluster. A simple diagram of roughly what I mean (although with many more clusters)

I'm thinking I'm just "doing something wrong" or there is a better way of what I would like to do. I want flux to keep all these clusters in sync with each other as far as deployed applications. Making an update to the repo would then effectively update all of the clusters.

I did quite a bit of googling (and maybe I just failed to search for the right term but I found: #1819
But no matter what I tried I found it hard to impossible to link a bunch of clusters to the same "cluster" definition. I also would like to manage flux by using that repo (this is more optional) because I would like all the clusters to have the same version of flux. Anyhow. I'm hope i'm just missing a bit of documentation! Thanks for your time!

[kingdonb]

I think that you can do this, currently only if you are managing SSH keys by hand.

When you run flux bootstrap, each cluster gets its own Deploy key which is managed based on the path that bootstrap is ran against. This way Flux enforces that each bootstrap refers to an individual specific cluster only, where that cluster is represented by a path that belongs to it (example: ./clusters/my-cluster). You can of course rewrite the gotk-sync.yaml to use an access method that does not require publishing a new Deploy key onto the repo, like connecting with an HTTPS endpoint and a PAT, or even a public HTTPS repo which is not protected by access controls and is read-only for the many identical Flux clients. You will have to do this with a patch if you want to avoid overwriting it with subsequent upgrades done via flux bootstrap later on. Each deploy key can be added manually to the Git remote, and included in the Flux config for that cluster via flux create secret or another way.

I have not tested this, but I believe it would work. You can manage multiple SSH deploy keys by hand on a repo and populate the flux-system secret for the flux-system gitrepository by hand in each replicated cluster. You have to be sure that multiple replicas that are identical do not all run their own Image Update Automation, which seems to be the main sticking point in documenting this mode of working, since we would want to provide that Image Update Automation support and never suggest that it should be avoided. They (the replica clusters) should each use Flux in read-only mode, to prevent a race between different update agents that might all decide independently to reconcile Flux update automation at different times and generate conflicts between each other. But one Flux "updater" cluster would be in charge of writing updates. This could work within the current design, without coloring too far outside of the lines I think.

You can find more information about identical environments in the https://github.com/fluxcd/flux2-kustomize-helm-example#identical-environments heading of the flux2-kustomize-helm-example doc.

Please note though, that this method describes something different than what you are attempting to accomplish here. The documented method assumes that you will inevitably want some replicas to diverge, either temporarily or permanently, and so it provides that each cluster can be bootstrapped separately and upgraded independently, with most of the common manifests coming from a common kustomize overlay base. Maybe you already found this and ruled it out because of the differences.

What you're trying to do specifically is not documented. I am not sure if there's any reason we couldn't support it as a use case and document it separately, but you should know that you are heading into undocumented territory, you may find unexpected surprise behaviors or you may encounter inconsistencies that require a work-around.

The most obvious one is actually described above -- if one cluster has read-write credentials so that it can run ImageUpdateAutomation and the other clusters do not run Image Update Controllers, then you really have at least two different cluster specs, and they are not identical replicas anymore. (In a way that is accounted for in the example from flux2-kustomize-helm-example, but still not described all that fully.)

The two cluster specs would diverge in gotk-sync.yaml as described above. The read-only clusters can all be replicas of a "read-only environment" replica of your base manifests. So you need at least two separate git manifest "environments" even if you are managing SSH Deploy keys by hand, or using a separate trick like public access to deliver read-only access to git for every downstream replicated cluster.

[kingdonb]

I got side-tracked, but today is for sure. I am building in parallel:

One local (bare-metal) cluster that runs Flux, on a private network and for development only, but with Flux as a production service

One GKE cluster, which has some infrastructure services (Traefik, perhaps Flagger, Prometheus, ... only the basics...)
One AKS cluster, which is a mirror of the GKE cluster.

An explicit goal is to have a lever which I can turn, that allows GKE or AKS clusters to be a perfect replica, or a slightly divergent mirror that has only a few things different. Since you should be able to flip between these modes, if one cluster represents Staging while the other represents Production, and perhaps they can be swapped in Multi-Cloud fashion, you will usually want both clusters to be a perfect mirror, but occasionally might want to make your application or policies in Staging a bit different, ahead of some major Production deployment say.

I hope that I don't need to enter any Cloud-specific things in the common part of the repo that my applications live in, but I anticipated that I would need to make some Cloud-specific things in the infrastructure part. But also, inevitably, there will be cloud-specific things that creep into our applications, but for now I'm going to assume those are not the things that we will make different from one cluster to another, and we are only using this for Staging/pre-release testing, and perhaps once in a while for A-B testing or something like that.

The main thing that is common regardless of whether you have identical clusters or "nearly identical clusters" with separate overlays, is that you will have a namespace where kubeconfig secrets live. This is what I was alluding to as "thing that CAPI would provide" -- if you are managing multiple Flux clusters from a single cluster, you would have a directory full of kubeconfig secrets and manage them all on a single (centrally controlled) Flux management cluster.

This might not be what you had in mind, I think you were talking more about setting up a separate Flux instance on each cluster. I will try to keep both perspectives in mind as I scale out this example to cover a few minor things that our existing docs maybe don't cover so well.

[Klaven]

Interesting @kingdonb initially you are right. was talking about separate flux instances all pointing back to the same git. For now my account for "multi-cloud" was going to be have a separate repo for each cloud provider (not optimal but I was not trying to solve this part just yet), but having some type of overlay for the different bits would work just fine... and honestly be a much better solution.

That being said my architecture does use cluster-api so this would not be out of scope. Though i honestly did not know that flux could be run as a "flux management cluster". I would assume that this "management cluster" way of running would work similarly to rancher fleet? Well maybe not the awkward double pull stuff. Anyhow I have not found documentation on that, I'm sorry if i'm missing something really dumb and right on the front page of the documentation but my searching did not reveal this configuration too me, I also looked though the flux command line and did not see anything that would lead me to run flux this way.

Thanks for the detailed conversation, i'm learning a lot from it! 🙇 And thanks for helping me investigate it!

[yebyen]

My immediate plan is for Flux on the management cluster to simply install Flux (a bootstrapped instance) on each remote cluster. But my understanding is spec.kubeConfig in Flux Kustomization can be used to manage workloads on a remote cluster, and it should work out-of-the-box with CAPI. You're right this is not well exposed by the docs. There is a PR open now to call more attention to it, (I'll find it and link it when I'm not on mobile)

(This is @kingdonb on mobile)

[kingdonb]

I got it started, this is the progress on the example I was promising:

https://github.com/kingdonb/bootstrap-repo

There are two clusters running Flux branched out from one central cluster running Flux here. The kubeconfig secrets are kept on a local cluster that runs on a desktop computer inside of my house. Normally I would recommend keeping this repo private, but everything is encrypted with SOPS in here, secrets/cluster-api -- so it is safe. Note that this is not really a CAPI provider, just a collection of kubeconfig files that I would have to rotate manually, or find some other way to keep refreshing. Doing the SOPS encryption was easy.

There is some sleight of hand, because I had to go to some lengths to export a kubeconfig from GKE that would not require a gcloud client authenticator. It has to have a service account token embedded in it. The encrypted secrets in secrets/cluster-api represent our kubeconfig for each cluster, one on AKS and another on GKE. 😄 So there is a script for what I added as a flux-bootstrap user, which I then escalated to have the crd-controller-flux-system and the cluster-admin roles required for bootstrapping Flux.

Flux bootstraps more Fluxes. Then users can step in on the target clusters and add any gitrepository or kustomization resources, or helmreleases what else have you... in this example I have installed podinfo onto both clusters with a flux-sync directory there, that I would just apply manually the first time. Then each cluster manages its own applications internally, as a basic "Hard Multitenancy".

I am also duplicating all of the config across both clusters. They are identical, take my word for it or look it over with a diff tool... combining these into a single base is the next exercise. They should draw each config from a common base, or better, each sync from the exact same directory. I am done changing it for tonight, I will have to implement those later...

This example has some rough edges still. There is a lot that can be improved. I will iterate on it again tomorrow!

[Klaven]

So I made an attempt, I don't am pretty sure mine is not nearly as sophisticated as yours.
steps:

1. created 2 kind clusters
2. bootstrapped flux on cluster 1 flux bootstrap github --owner=klaven --repository=flux-text-repo --path=clusters/test-clusters --personal
3. installed flux on cluster 2 - flux install
4. make repo public (easiest way for test/demo... I think)
5. added the git repo as readonly on second cluster. flux create source git --url=https://github.com/Klaven/flux-text-repo.git --branch=main
6. added podinfo source and kustomization to the /clusters/test-clusters/ dir in the flux-text-repo

They both have a gitrepository with the following spec

  spec:
    gitImplementation: go-git
    interval: 1m0s
    ref:
      branch: main
    secretRef:
      name: flux-system
    timeout: 20s
    url: ssh://[email protected]/klaven/flux-text-repo

the second one has an https url but everything else is the same.
Everything seems to be working on both (both say they are working) But only the initial cluster, the one I used bootstrap on is actually syncing with the repo. The other one, while it says the gitrepository is up to date and working, does not pick up the new cluster spec.

So then I took to hacking a little and added the flux-system kustomization, as I think this is the one that manages the cluster. Updated the flux-system kustomization to point to the git-repository I added to the cluster.

And now both seem to be mirrors of each other! :) so its working. now I guess I might try with cluster-api. This did not have a "flux management cluster" I'm still foggy on how that might work. :) But i'm happy I got something working! Again I want to thank you for all the help you have provided so far. 100% I would not be this far just reading the documentation. If people would like it I don't mind taking the time to help write up the documentation on how to do this, though i'm not sure what I did is that useful without some updates to flux's cli to better support it. I'm also interested in helping this not suck so much, it's a little bit of a pain to set up "correctly"