Introduce automated backups

[OmeGak]

To do (roughly):

• Find a way specify at playbook level which DBs and file system volumes need to be backed up.
• Figure out where to store back ups (probably in the S3-compatible Vultr Object Storage).
• Figure out which tool(s) to use for scheduled backups.

For now, the only machine that stores persistent information is matrix.bonfire.link. It stores the Discord-bridged rooms and the +bonfire.link:bonfire.link community.

[OmeGak]

The plan:

• Move postgres-related tasks to a dedicated postgres role.
• Install and configure autopostgresqlbackup with the postgres role.
• Add a backups role that will:
  • Create/mount a volume for backups locally.
  • Install and configure borg to periodically back up files to the backup volume.
  • Install and configure rclone to backup the backups in our S3 object storage in Vultr.

Considerations:

• Backups should be encrypted at rest in the object storage.

[definitivelynotsimon]

About backups.

The goal is to have automated encrypted backups of everything relevant to a secure location.

What we have for now is a running borg backup configuration on the matrix server that backs up matrix database and files to an S3 container. It works but it's experimental.

The next steps to make it more stable are :
Define a strategy for what needs to be backed up on every server.
Create a backup role in ansible to implement that strategy.
Deploy the role and test recoveries.