[Bug]: Error While Upgrading Flux, Bootstrapped Through GitLab: could not create git client: scheme "" is not supported

[satyamsareen007]

Describe the bug

Hi Team, I am facing a strange error when trying to upgrade Flux through GitLab via the flux_bootstrap_git resource:

Error: could not create git client: scheme "" is not supported

Background:

We have a lot of Flux installations bootstrapped through GitLab running on multiple EKS clusters.

Our relatively newer Flux installations bootstrapped through GitLab use the following config. The GitLab project creation & Flux provider initialisation happen successfully in a single terraform apply:
Flux version: 2.4.0
Flux provider version: 1.4.0
Gitlab provider version: 17.5.0

provider "flux" {
  kubernetes = {
    config_path = "${path.root}/../config.yaml"
  }

  git = {
    url = gitlab_project.flux_bootstrap.web_url
    http = {
      username = var.gitlab_token
      password = var.gitlab_token
    }
  }
}

resource "gitlab_project" "flux_bootstrap" {
  name                   = local.gitlab_project_name
  namespace_id           = var.gitlab_namespace_id
  initialize_with_readme = true
}

resource "flux_bootstrap_git" "bootstrap" {
  embedded_manifests   = true
  path                 = var.eks_cluster_name
  namespace            = "flux-cd"
  version              = var.flux_version
  watch_all_namespaces = true
  keep_namespace       = true
  depends_on           = [kubernetes_namespace.flux_cd]
}

Our relatively older Flux installation is on version 2.3.0, which was bootstrapped through GitLab using the SSH scheme.
Flux version: 2.3.0
Flux provider version: 1.3.0

provider "flux" {
  kubernetes = {
    config_path = "${path.root}/../config.yaml"
  }
  git = {
    url         = "ssh://git@gitlab.com/${data.terraform_remote_state.gitlab_deploy_key.outputs.gitlab_project_path_with_namespace}.git"
    branch      = "main"
    author_name = "${var.project_name}-flux-${var.region}-${var.environment}"
    ssh = {
      username    = "git"
      private_key = data.terraform_remote_state.gitlab_deploy_key.outputs.gitlab_deploy_key_tls_private_key
    }
  }
}
resource "flux_bootstrap_git" "bootstrap" {
  path                 = var.eks_cluster_name
  namespace            = "flux-cd"
  version              = var.flux_version
  watch_all_namespaces = false
  keep_namespace       = true
  depends_on = [kubernetes_namespace.flux_cd]
}

Now I am trying to upgrade it to version 2.5.1 but using the HTTP scheme as we are trying to use HTTP across the board for all of our Flux installations using the same config we saw above
Flux version: 2.5.1
Flux provider version: 1.5.1
Gitlab provider version: 17.5.0

provider "flux" {
  kubernetes = {
    config_path = "${path.root}/../config.yaml"
  }

  git = {
    url = gitlab_project.flux_bootstrap.web_url
    http = {
      username = var.gitlab_token
      password = var.gitlab_token
    }
  }
}

resource "gitlab_project" "flux_bootstrap" {
  name                   = local.gitlab_project_name
  namespace_id           = var.gitlab_namespace_id
  initialize_with_readme = true
}

resource "flux_bootstrap_git" "bootstrap" {
  embedded_manifests   = true
  path                 = var.eks_cluster_name
  namespace            = "flux-cd"
  version              = var.flux_version
  watch_all_namespaces = true
  keep_namespace       = true
  depends_on           = [kubernetes_namespace.flux_cd]
}

But I get the following error during terraform plan: could not create git client: scheme "" is not supported

This error looks like, is saying that the GitLab project creation & Flux provider installation cannot happen parallely, which is why the scheme is an empty string in place of HTTP, as the attribute gitlab_project.flux_bootstrap.web_url is not populated yet.

But then, how our other Flux installations, which we saw above, that are running on 2.4.0 are able to get bootstrapped in a single terraform apply, and the project creation also happens in the same job.

When I provide the GitLab project URL as a hardcoded string, the plan succeeds. I am seeing some examples provided in this repo as well, which showcase provider initialisation & project creation as part of the same tf config: https://github.com/fluxcd/terraform-provider-flux/blob/main/examples/gitlab-via-ssh/providers.tf#L11

If GitLab project creation & Flux provider initialisation can't happen in a single tf plan/apply, shouldn't this behaviour have happened in all of our Flux installations?

Can you please help us fix this error?

Steps to reproduce

Install Flux version 2.3.0 using the following config :

provider "flux" {
  kubernetes = {
    config_path = "${path.root}/../config.yaml"
  }
  git = {
    url         = "ssh://git@gitlab.com/${data.terraform_remote_state.gitlab_deploy_key.outputs.gitlab_project_path_with_namespace}.git"
    branch      = "main"
    author_name = "${var.project_name}-flux-${var.region}-${var.environment}"
    ssh = {
      username    = "git"
      private_key = data.terraform_remote_state.gitlab_deploy_key.outputs.gitlab_deploy_key_tls_private_key
    }
  }
}
resource "flux_bootstrap_git" "bootstrap" {
  path                 = var.eks_cluster_name
  namespace            = "flux-cd"
  version              = var.flux_version
  watch_all_namespaces = false
  keep_namespace       = true
  depends_on = [kubernetes_namespace.flux_cd]
}

And then try to upgrade to 2.5.1 using the following config:

provider "flux" {
  kubernetes = {
    config_path = "${path.root}/../config.yaml"
  }

  git = {
    url = gitlab_project.flux_bootstrap.web_url
    http = {
      username = var.gitlab_token
      password = var.gitlab_token
    }
  }
}

resource "gitlab_project" "flux_bootstrap" {
  name                   = local.gitlab_project_name
  namespace_id           = var.gitlab_namespace_id
  initialize_with_readme = true
}

resource "flux_bootstrap_git" "bootstrap" {
  embedded_manifests   = true
  path                 = var.eks_cluster_name
  namespace            = "flux-cd"
  version              = var.flux_version
  watch_all_namespaces = true
  keep_namespace       = true
  depends_on           = [kubernetes_namespace.flux_cd]
}

Expected behavior

Flux should successfully get upgraded to 2.5.1 from 2.3.0

Screenshots and recordings

Terraform and provider versions

Provider versions for flux 2.3.0:

Flux version: 2.3.0
Flux provider version: 1.3.0

from which we are trying to upgrade to 2.5.1 with the provider versions:

Flux version: 2.5.1
Flux provider version: 1.5.1
Gitlab provider version: 17.5.0

Terraform provider configurations

Provider configuration with Flux 2.3.0:

provider "flux" {
  kubernetes = {
    config_path = "${path.root}/../config.yaml"
  }
  git = {
    url         = "ssh://git@gitlab.com/${data.terraform_remote_state.gitlab_deploy_key.outputs.gitlab_project_path_with_namespace}.git"
    branch      = "main"
    author_name = "${var.project_name}-flux-${var.region}-${var.environment}"
    ssh = {
      username    = "git"
      private_key = data.terraform_remote_state.gitlab_deploy_key.outputs.gitlab_deploy_key_tls_private_key
    }
  }
}

from which we are trying to upgrade to 2.5.1 with the provider config:

provider "flux" {
  kubernetes = {
    config_path = "${path.root}/../config.yaml"
  }

  git = {
    url = gitlab_project.flux_bootstrap.web_url
    http = {
      username = var.gitlab_token
      password = var.gitlab_token
    }
  }
}

flux_bootstrap_git resource

resource "flux_bootstrap_git" "bootstrap" {
  embedded_manifests   = true
  path                 = var.eks_cluster_name
  namespace            = "flux-cd"
  version              = var.flux_version
  watch_all_namespaces = true
  keep_namespace       = true
  depends_on           = [kubernetes_namespace.flux_cd]
}

Flux version

2.5.1

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Would you like to implement a fix?

None

[satyamsareen007]

@swade1987 for viz

[mloskot]

@satyamsareen007

But then, how our other Flux installations, which we saw above, that are running on 2.4.0 are able to get bootstrapped in a single terraform apply, and the project creation also happens in the same job.

IMHO, by a sheer flux or accident or something else, but most likely because previously you set flux.git.url = "ssh://... with literal and now you set it with flux.git.url = gitlab_project.flux_bootstrap.web_url which is a dynamically obtained value unknown at the point of the provider configuration (it's not data source, is it).

You have to split your all-in-one Terraform module into two:

1. Module setting up project setup and sticking the URL inoutput
2. Module bootstrapping Flux which reads the URL from outputs of the previous module

that is, two separate steps, two separate terraform plan and terraform apply executions.

Terraform modules doing all-in-one are not quite Terraformic, but rather PITA.

@swade1987 for viz

It's not a good idea to mention-to-nudge maintainers of any OSS project, no matter how urgent an issue is to you. Besides, see

• Stepping down as a maintainer #731

[satyamsareen007]

@mloskot office example from this repo also, bootstrap flux & create GitLab projects in a single terraform apply

pls see: https://github.com/fluxcd/terraform-provider-flux/blob/main/examples/gitlab-via-ssh/providers.tf#L11

[mloskot]

pls see: https://github.com/fluxcd/terraform-provider-flux/blob/main/examples/gitlab-via-ssh/providers.tf#L11

Your examples does not quite do the same thing as you do, does it?

Besides, you probably should have a look at the docs, as web_url is not quite the thing you expect it to be meant for, it is not git clone url.

Anyway, this is not the right space to discuss usage of Terraform provider for GitLab.
IMHO, your problem is not due to a bug in this provider but an issue somewhere in your own code.

[satyamsareen007]

It fails with http_url_to_repo as well 🙂, I think it is indeed a bug within the provider.

[mloskot]

Have you considered inspecting the actual value of gitlab_project.flux_bootstrap.web_url, with outputs, terraform_data, whatever?