[Bug]: tfstate issues lead to flux-system being deleted

[ionutleca]

Describe the bug

We use the tf-controller to manage flux on multiple clusters via the tf-provider.
We had an issue when the tf-runner crashed before it got to write the state but after it managed to push the initial commits.

We missed the tf-runner crash and thought everything was good (as flux synced everything), but when terraform ran again it triggered the bootstrap process and this logic seems to have caused the entire flux-system namespace to go into Terminating.

Steps to reproduce

1. Run the terraform (with a kustomization_override set).
2. Delete the state.
3. Rerun terraform.

Expected behavior

It would help if there was check for existing files in git before directly overwriting them with empty files.

Screenshots and recordings

No response

Terraform and provider versions

terraform {
  required_version = ">=1.1.5"

  required_providers {
    flux = {
      source = "fluxcd/flux"
      version = "1.3.0"
    }
    github = {
      source  = "integrations/github"
      version = "5.25.1"
    }
  }
}

Terraform provider configurations

provider "flux" {
  kubernetes = {
   ***
  }
  git = {
    url    = "ssh://git@github.com/${var.github_org}/${var.github_repository}.git"
    branch = var.github_branch
    ssh    = {
      username    = "git"
      private_key = tls_private_key.flux.private_key_pem
    }
  }

flux_bootstrap_git resource

resource "flux_bootstrap_git" "this" {
  depends_on = [github_repository_deploy_key.main]

  path                   = var.github_path
  version                = var.flux_version
  kustomization_override = templatefile("${path.module}/kustomization.yaml", {
                                             ***
                                         })
  registry               = var.registry
}

Flux version

v2.4.0

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Would you like to implement a fix?

None

[stefanprodan]

I recommend switching to Flux Operator which does not require pushing the Flux manifest to Git. This would simplify your setup and avoid this issue all together. You can find here a Terraform example https://github.com/controlplaneio-fluxcd/flux-operator/tree/main/config/terraform

[ionutleca]

Thanks, we will check it out. From a quick look, it mentions the need for a repo token next to the operator. What was nice for our previous flow was that only a central "management cluster" had our GitHub token and terraform used it to send a deployment key to the target cluster that was bootstrapped. Is there a similar option to generate "per cluster credentials" with the operator?

[stefanprodan]

The operator supports any type of source (GitRepo, OCIRepo, Bucket) and auth. If you want to use deploy keys, you can fully automate this from Terraform and create the flux-system secret in each cluster with the SSH private key as showed here

terraform-provider-flux/examples/github-self-managed-ssh-keypair/main.tf

Line 77 in f0f7542

resource "kubernetes_secret" "ssh_keypair" {

[ionutleca]

That might work for us, thank you! 🙏🏼
As far as I'm concerned, this issue can be closed. Might be just worth documenting it as a possible warning or a reason to move to the operator.