Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions tofu/kubernetes/BACKEND.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
## Setup Environment

### Using infisical

1. Setup [gcloud cli](/DEVCONTAINER.md).
1. Setup TF_VAR_proxmox_api_token, TF_VAR_tofu_encryption_passphrase and save them to infisical.
1. Setup .env file in root folder and commit it to git.
1. Follow devcontainers docs [here](/DEVCONTAINER.md). If done properly, all secrets from infisical will be available in the container environment.

Note: By default all secrets in /tofu folder will be populated. /tofu_rw is the folder where secrets are written. Also checkout default branch to env mapping.

## Setup Backend

### Local

```shell
# Local Backend
cp samples/backend_local.tofu.sample ./backend.tofu
```

```shell
# Local Backend
tofu init
```

**Note: If your are using local backend with dev devcontainers and git repo, your state file will be deleted when the container is removed. So be very careful.**

### R2

1. Follow instructions in [Cloudflare R2](../remote_state/cf/README.md) to setup R2 bucket for remote state.

```shell
# R2 Backend
cp samples/backend_r2.tofu.sample ./backend.tofu
```

```shell
# Initialize tofu
tofu init
```

### GCS

1. Follow instructions in [Google Cloud](../state/gcs/README.md) to setup GCS bucket for remote state.

```shell
# GCS Backend
cp samples/backend_gcs.tofu.sample ./backend.tofu
```

```shell
# Initialize tofu
tofu init
```
49 changes: 6 additions & 43 deletions tofu/kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,35 +3,18 @@
Read [Talos Kubernetes on Proxmox using OpenTofu](https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/) for
a more thorough explanation of how everything works.

## Install pre-requisites
## Install pre-requisites - Pre-installed in devContainer

1. [tofu](https://opentofu.org/docs/intro/install/)
2. [talosctl](https://www.talos.dev/v1.9/talos-guides/install/talosctl/)
3. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)

## Initialize tofu

```shell
tofu init
```

## Proxmox

### Environment variable

```shell
export TF_VAR_proxmox_api_token="<YOUR_API_TOKEN>"
```

### Optional External Secrets Manager / Other methods
One cluster/state per branch.

**Bitwarden Secrets Manager** - Name your secret TF_VAR_proxmox_api_token in bws.

```shell
bws run -- tofu ...
```

Note: By default, the shell is sh. Change with --shell if required.
1. Setup and initialize [remote backend](BACKEND.md).
1. Keep the environment populated with [required secrets](BACKEND.md) when running `tofu plan/apply`.

## Sealed-secrets

Expand Down Expand Up @@ -59,32 +42,12 @@ tofu output -raw talos_config

## Upgrading Talos and Kubernetes

[Upgrade](https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/#upgrading-the-cluster) talos nodes one by
one.

1. Set talos_image.auto.tfvars -> image -> update_version to the required update version.
2. Set talos_cluster.auto.tfvars -> talos_cluster_config -> kubernetes_version to the required kubernetes version.
3. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_1 -> update = true and run tofu apply.
4. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_2 -> update = true, leave the previous nodes update = true and
run tofu apply.
5. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_3 -> update = true, leave the previous nodes update = true and
run tofu apply.
6. ...
7. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_n -> update = true, leave the previous nodes update = true and
run tofu apply.
8. After upgrading all nodes, Set talos_image.auto.tfvars -> image -> version to match the update version and set
update = false for all nodes.

## Upgrading Talos Schematic

1. Create a new schematic file.
2. Same process as above instead of `image.version` and `image.update_version`, change `image.schematic` and
`image.update_schematic`, in `talos_image.auto.tfvars`.
Follow these [instructions](UPGRADE.md).

## Reuse machine secrets

```shell
tofu state rm module.talos.talos_machine_secrets.this
tofu import module.talos.talos_machine_secrets.this output/talos-machine-secrets.yaml
tofu apply --refresh=false
```
```
42 changes: 0 additions & 42 deletions tofu/kubernetes/REMOTE_BACKEND.md

This file was deleted.

22 changes: 22 additions & 0 deletions tofu/kubernetes/UPGRADE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
## Upgrading Talos
[Upgrade](https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/#upgrading-the-cluster) talos nodes one by
one.

1. Set talos_image.auto.tfvars -> image -> update_version to the required update version.
2. Set talos_cluster.auto.tfvars -> talos_cluster_config -> kubernetes_version to the required kubernetes version.
3. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_1 -> update = true and run tofu apply.
4. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_2 -> update = true, leave the previous nodes update = true and
run tofu apply.
5. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_3 -> update = true, leave the previous nodes update = true and
run tofu apply.
6. ...
7. Set talos_nodes.auto.tfvars -> talos_nodes -> $node_n -> update = true, leave the previous nodes update = true and
run tofu apply.
8. After upgrading all nodes, Set talos_image.auto.tfvars -> image -> version to match the update version and set
update = false for all nodes.

## Upgrading Talos Schematic

1. Create a new schematic file.
2. Same process as above instead of `image.version` and `image.update_version`, change `image.schematic` and
`image.update_schematic`, in `talos_image.auto.tfvars`.
5 changes: 5 additions & 0 deletions tofu/kubernetes/backend.tofu
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
terraform {
backend "local" {
path = "${var.gcs_prefix_env}.tfstate"
}
}
3 changes: 3 additions & 0 deletions tofu/kubernetes/infisical/git_branch.tofu
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
data "external" "git-branch" {
program = ["/bin/bash", "-c", "jq -n --arg branch `git rev-parse --abbrev-ref HEAD` '{\"branch\":$branch}'"]
}
25 changes: 25 additions & 0 deletions tofu/kubernetes/infisical/infisical.tofu
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
locals {
infisical_environment = lookup(var.infisical.branch_env_mapping, data.external.git-branch.result.branch, "dev")
}

provider "infisical" {
host = var.infisical_domain
auth = {
universal = {
client_id = var.infisical_client_id
client_secret = var.infisical_client_secret
}
}
}

resource "infisical_secret" "created_secrets" {
# Create one resource per entry in the input map
for_each = var.secrets_to_create

name = each.key
value = each.value

env_slug = local.infisical_environment
folder_path = var.infisical_rw_secrets_path
workspace_id = var.infisical_project_id
}
48 changes: 48 additions & 0 deletions tofu/kubernetes/infisical/variables.tofu
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
variable "infisical_domain" {
description = "Infisical Domain"
type = string
default = "https://app.infisical.com"
}

variable "infisical_client_id" {
description = "Infisical Client ID"
type = string
default = null
}

variable "infisical_project_id" {
description = "Infisical Project ID"
type = string
default = null
}

variable "infisical_rw_secrets_path" {
description = "Infisical Client Secret"
type = string
default = "/tofu_rw"
}

variable "infisical_branch_env_mapping" {
description = "Infisical Branch Environment Mapping"
type = map(string)
default = {
"main" = "prod"
"prod" = "prod"
"staging" = "staging"
"dev" = "dev"
}
}

variable "infisical_client_secret" {
description = "Infisical Client Secret"
type = string
sensitive = true
default = null
}

variable "secrets_to_create" {
description = "A map of secrets to create in Infisical. Key: secret name, Value: secret content."
type = map(string)
default = {}
sensitive = true
}
41 changes: 41 additions & 0 deletions tofu/kubernetes/infisical_variables.tofu
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
variable "infisical_domain" {
description = "Infisical Domain"
type = string
default = "https://app.infisical.com"
}

variable "infisical_client_id" {
description = "Infisical Client ID"
type = string
default = null
}

variable "infisical_project_id" {
description = "Infisical Project ID"
type = string
default = null
}

variable "infisical_rw_secrets_path" {
description = "Infisical Client Secret"
type = string
default = "/tofu_rw"
}

variable "infisical_branch_env_mapping" {
description = "Infisical Branch Environment Mapping"
type = map(string)
default = {
"main" = "prod"
"prod" = "prod"
"staging" = "staging"
"dev" = "dev"
}
}

variable "infisical_client_secret" {
description = "Infisical Client Secret"
type = string
sensitive = true
default = null
}
24 changes: 24 additions & 0 deletions tofu/kubernetes/main.tofu
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,27 @@ module "volumes" {
proxmox_api = var.proxmox
volumes = var.kubernetes_volumes
}

module "infisical_secrets" {
# Conditionally create infisical module based on where var.infisical is set in infisical.auto.tfvars
count = (var.infisical_project_id != null && var.infisical_client_id != null) ? 1 : 0

source = "./infisical"

infisical_domain = var.infisical_domain
infisical_client_id = var.infisical_client_id
infisical_project_id = var.infisical_project_id
infisical_rw_secrets_path = var.infisical_rw_secrets_path
infisical_branch_env_mapping = var.infisical_branch_env_mapping
infisical_client_secret = var.infisical_client_secret

secrets_to_create = {
# Create map entries only for non-null values
for k, v in {
"kubeconfig" = module.talos.kube_config.kubeconfig_raw
"talos_config" = module.talos.client_configuration.talos_config
"kube_certificate" = file("${path.root}/${var.sealed_secrets_config.certificate_path}")
"kube_certificate_key" = file("${path.root}/${var.sealed_secrets_config.certificate_key_path}")
} : k => v if v != null
}
}
Loading