namespace secrets and postgresql databases

[thomas-maschler]

Ok, here a POC of how this could work. (still WIP)

1. Database module

• Creates database, role and random password.
• Stores random password in secret manager

2. Refactor k8s_namespaces module.

• Module creates name space and associated secrets
• In main.tf define one module per namespace
• Secrets can be injected via env var (store secrets centrally in a vault such as GH, or AWS secret manager and write a script to retrieve secrets and load as env vars)
• To avoid destruction of current namespaces, we would need to manually deregister namespace from terraform and then reimport them using the new module name.

As a Tier 2 or 3 item, we should look at all the AWS Access keys listed as secrets. Now that everything is on AWS, there is no need of this anymore and we should switch to using IAM roles instead.

[tiagojsag]

Overall LGTM 👍

Some potential improvement points/concerns:

• I see we are naming the secrets 1 by 1 here, and the logic to load them from elsewhere is missing. A "nice to have" going forward would be a flexible solution that would allow the secrets to be stored elsewhere as an array of sorts, and this logic here would handle it as such, without having to be aware of individual secrets. This would make adding/removing secrets a more flexible process
• A concern I am increasingly having with the general direction of moving everything to TF and GH actions is the agility penalty that's adding up. For example, depending on how this PR evolves, and using standard workflow, modifying a secret in production would now require 3 PRs, 3 TF plan + builds. I am aware that this sort of operation does not happen every day, but do find myself often working around this workflow at times, while developing and experimenting, just to not have to wait for GH actions to do it's thing, and then reconciling state at the end of the day. Not something urgent, and it's a negative trade off compared to the several positive things that TF/GH actions bring to the table, just leaving it here as food for thought mostly.

[thomas-maschler]

Thanks,

in regards to point one.

• If we make this repo private we could write the secrets into the terraform-{ENV}.tfvars file. In that case , these values would be considered when referencing a variable. If we don't specify a variable we could instruct terraform to create a random password instead.

Point two,
I agree, this can become lengthly. One approach would be to create hotfixes for small configuration changes and deploy directly production. Aside from that, we cannot deploy the k8s terraform directory using GH, since we need to be inside the VPC to make these changes.