Skip to content

Oppdater guide for migrering av postgres fra FSS til GCP #897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
torkjels wants to merge 1 commit into
base: main
Choose a base branch
from
+98 −93
Open

Oppdater guide for migrering av postgres fra FSS til GCP #897

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 98 additions & 93 deletions docs/persistence/cloudsql/how-to/migrating-databases-to-gcp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,99 +58,104 @@ Cons:

This method is suitable for applications that can have the database in read-only or application that allow for some downtime. It requires that the database instance and DDLs are created up front \(i.e. deploy your application in GCP and let flyway create DDLs\):

Use docker container image with psql and cloudsdk: [GCP migration image](https://github.com/navikt/gcp-migrering). This image let you do all the following actions from one place.

You need to create a secret in your namespace containing the Google SA you want to use to do the migration. Add the token to the secret.yaml file and apply it in your namespace.

Deploy the pod into on-premise cluster that can connect to the database

```shell
kubectl apply -f https://raw.githubusercontent.com/navikt/gcp-migrering/main/gcloud.yaml
```

`exec` into that pod

```shell
kubectl exec -it gcloud -- /bin/bash
```

Log in to gcloud with your own NAV-account

```shell
gcloud auth login
```

Configure the project id \(find project id with `gcloud projects list --filter <team>`\)

```shell
gcloud config set project <project id>
```

Set readable directory as $HOME

```shell
export HOME=/tmp
```

Create a GCP bucket.

```shell
gsutil mb -l europe-north1 gs://<bucket name>
```

Find the GCP service account e-mail \(the instance id is specified in your `nais.yaml` file\)

```shell
gcloud sql instances describe <CloudSQL instance id> | grep serviceAccountEmailAddress
```

Set the objectAdmin role for the bucket \(with the previous e-mail\)

```shell
gsutil iam ch serviceAccount:<GCP service account e-mail>:objectAdmin gs://<bucket name>/
```

Use `pg_dump` to create the dump file. Notes:

- Make sure that you stop writes to database before running `pg_dump`.
- Get a [database user from Vault](https://github.com/navikt/database-iac#how-to-manually-login-to-a-database-to-make-changes-or-read-data).
- If the database in GCP already has the `flyway_schema_history` table,
you might want to exclude the equivalent table in the dump by using the `--exclude-table=flyway_schema_history` option.

```shell
pg_dump \
-h <postgreSQL on-premise host name> \
-d <database instance name> \
-U <database user name to connect with> \
--format=plain --no-owner --no-acl --data-only -Z 9 > dump.sql.gz
```

Copy the dump file to GCP bucket

```shell
gsutil -o GSUtil:parallel_composite_upload_threshold=150M -h "Content-Type:application/x-gzip" cp dump.sql.gz gs://<bucket name>/
```

Import the dump into the GCP postgreSQL database. Notes:

- You need the `roles/cloudsql.admin` IAM role in order to perform the import.
- The `user` in the command below should be a GCP SQL Instance user.
- If the GCP Postgres database has any existing tables or sequences, make sure that the `user` has all required grants for these.

```shell
gcloud sql import sql <Cloud SQL instance id> gs://<bucket name>/dump.sql.gz \
--database=<database instance name> \
--user=<database instance user name>
```

Verify that the application is behaving as expected and that the data in the new database is correct. Finally we need to switch loadbalancer to route to the GCP application instead of the on-premise equivalent.

Delete the bucket in GCP after migration is complete

```shell
gsutil -m rm -r gs://<bucket name>
gsutil rb gs://<bucket name>
```
Use docker container image with psql and cloudsdk: [GCP migration image](https://github.com/navikt/gcp-migrering/blob/main/gcloud.yaml). This image let you do all the following actions from one place.

1. Deploy the pod into on-premise cluster that can connect to the database
```shell
kubectl apply -f https://raw.githubusercontent.com/navikt/gcp-migrering/main/gcloud.yaml
```

2. Find the name of the pod using `kubectl get pods` and `exec` into that pod
```shell
kubectl exec -it <name of gcloud pod> -- /bin/bash
```

3. Log in to gcloud with your own Nav-account
```shell
gcloud auth login
```

4. Configure the project id \(find project id with `gcloud projects list --filter <team>`\)
```shell
gcloud config set project <project id>
```

5. Set readable directory as $HOME
```shell
export HOME=/tmp
```

6. Set current directory to that new directory
```shell
cd /tmp
```

7. Choose a name for the GCP bucket you want to create to store the SQL dump
```shell
export MIGRATION_BUCKET=<name you want to give to the bucket>
```

8. Create a GCP bucket.
```shell
gsutil mb -l europe-north1 "gs://MIGRATION_BUCKET"
```

9. Find the GCP service account e-mail \(the instance id is specified in your `nais.yaml` file\)
```shell
export SERVICE_ACCOUNT_EMAIL=$(gcloud sql instances describe <CloudSQL instance id> --format='value(serviceAccountEmailAddress)')
```

10. Set the objectAdmin role for the bucket \(with the previous e-mail\)
```shell
gsutil iam ch "serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin" "gs://MIGRATION_BUCKET/"
```

11. Use `pg_dump` to create the dump file. Notes:

- Make sure that you stop writes to database before running `pg_dump`.
- Get a [database user from Vault](https://github.com/navikt/database-iac#how-to-manually-login-to-a-database-to-make-changes-or-read-data).
- If the database in GCP already has the `flyway_schema_history` table,
you might want to exclude the equivalent table in the dump by using the `--exclude-table=flyway_schema_history` option.

```shell
pg_dump \
-h <postgreSQL on-premise host name> \
-d <database instance name> \
-U <database user name to connect with> \
--format=plain --no-owner --no-acl --data-only -Z 9 > dump.sql.gz
```

12. Copy the dump file to GCP bucket
```shell
gsutil -o GSUtil:parallel_composite_upload_threshold=150M -h "Content-Type:application/x-gzip" cp dump.sql.gz "gs://MIGRATION_BUCKET/"
```

13. Import the dump into the GCP postgreSQL database. Notes:
- You need the `roles/cloudsql.admin` IAM role in order to perform the import.
- The `user` in the command below should be a GCP SQL Instance user, for instance your own personal GCP username.
- If the GCP Postgres database has any existing tables or sequences, make sure that the `user` has all required grants for these.

```shell
gcloud sql import sql <Cloud SQL instance id> "gs://MIGRATION_BUCKET/dump.sql.gz" --database=<database instance name> --user=<GCP-user>
```

14. Verify that the application is behaving as expected and that the data in the new database is correct. Finally we need to switch loadbalancer to route to the GCP application instead of the on-premise equivalent.

15. Delete the bucket in GCP after migration is complete
```shell
gsutil -m rm -r "gs://MIGRATION_BUCKET"
gsutil rb "gs://MIGRATION_BUCKET" # This might give you a BucketNotFoundException, which is OK, it only means that the bucket was deleted by the previous command.
```

16. Exit the pod
```shell
exit
```

17. Delete the gcloud pod
```shell
kubectl delete -f https://raw.githubusercontent.com/navikt/gcp-migrering/main/gcloud.yaml
```

Pros:

Expand Down