diff --git a/README.md b/README.md index 6baa36c1..8626be24 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,7 @@ The table below is written from `apps/web/content/docs.manifest.json` by | Using your board | [`membership-guide.md`](./docs/membership-guide.md) | Taking the subs through the board: plans, discount codes, gifting, the memberships desk and the ledger — Dues end to end, without a terminal. | | Running the server | [`operating.md`](./docs/operating.md) | The operator handbook: configuration, the CLI, permissions, themes, plugins, mail, spam controls, backups, and troubleshooting. | | Running the server | [`upgrading.md`](./docs/upgrading.md) | Moving a board between versions: the upgrade command, how far you can jump, and the behaviour changes each release brings. | +| Running the server | [`disaster-recovery.md`](./docs/disaster-recovery.md) | The runbook for the day the server is gone: what recovery consumes, the order of operations from provisioning to DNS, and the rehearsal that turns it from a hope into a plan. | | Running the server | [`single-sign-on.md`](./docs/single-sign-on.md) | Two-factor authentication, federated sign-in and passkeys: what each means for your members, how to configure them, and the record of what has opened an account. | | Running the server | [`internationalisation.md`](./docs/internationalisation.md) | How a page picks its language, how to add one, and how a theme or plugin ships its own words. | | Running the server | [`web-push.md`](./docs/web-push.md) | Notifications that reach a member who does not have the board open, and the manifest that makes the board installable — what it costs their privacy, and how to turn it on. | diff --git a/apps/web/content/docs.manifest.json b/apps/web/content/docs.manifest.json index 70574727..d480a5b1 100644 --- a/apps/web/content/docs.manifest.json +++ b/apps/web/content/docs.manifest.json @@ -116,6 +116,15 @@ "generated": false, "primary": false }, + { + "slug": "disaster-recovery", + "file": "disaster-recovery.md", + "section": "running", + "title": "Disaster recovery", + "blurb": "The runbook for the day the server is gone: what recovery consumes, the order of operations from provisioning to DNS, and the rehearsal that turns it from a hope into a plan.", + "generated": false, + "primary": false + }, { "slug": "single-sign-on", "file": "single-sign-on.md", diff --git a/docs/README.md b/docs/README.md index f45692c2..50ac2deb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -26,6 +26,7 @@ somebody has already set the board up for you, start with | Run the server day to day | [Running a board](./operating.md) | | Run more than one web container | [Scaling out](./scaling.md) | | Move a board to a new version | [Upgrading a board](./upgrading.md) | +| Recover a board whose server is gone | [Disaster recovery](./disaster-recovery.md) | | Add 2FA, SSO or passkeys | [Signing in](./single-sign-on.md) | | Run a board in another language | [Languages](./internationalisation.md) | | Push notifications to a phone | [Web push](./web-push.md) | @@ -68,6 +69,7 @@ For whoever minds the machine. |---|---| | [`operating.md`](./operating.md) | The operator handbook: configuration, the CLI, permissions, themes, plugins, mail, spam controls, backups, and troubleshooting. | | [`upgrading.md`](./upgrading.md) | Moving between versions: the upgrade command, how far you can jump, and the behaviour changes each upgrade brings. | +| [`disaster-recovery.md`](./disaster-recovery.md) | The full-loss runbook: what recovery consumes, the order of operations from provisioning to DNS cutover, and the rehearsal that measures your recovery time. | | [`single-sign-on.md`](./single-sign-on.md) | Everything beyond a password: two-factor authentication, federated sign-in, passkeys, sessions, and the sign-in activity log. | | [`internationalisation.md`](./internationalisation.md) | How a page picks its language, how to add one, and how a theme or plugin ships its own words. | | [`web-push.md`](./web-push.md) | Notifications that reach a member who does not have the board open, the privacy that costs, and the manifest that makes the board installable. | diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md new file mode 100644 index 00000000..8be9bc84 --- /dev/null +++ b/docs/disaster-recovery.md @@ -0,0 +1,163 @@ +# Disaster recovery + +This page is the runbook for the bad day: the server is gone — seized, +dead, deleted, or unreachable in a way that is not coming back — and the +board has to exist again somewhere else. +[Backup and restore](./operating.md#backup-and-restore) is the everyday +half of this story: what to take, how to take it, and how to restore one +piece. This page is the order of operations when *everything* has to be +restored at once, written to be followed under stress. Read it once on a +calm day; the last section asks you to do slightly more than read it. + +## What recovery consumes + +Recovery is assembled from three artifacts, and the reason each backup +guide says **copy them off the machine** is that this page assumes the +machine is gone: + +| Artifact | Without it | +|---|---| +| The database dump | There is no board to recover. Everything the board knows — accounts, posts, settings, permissions — is here. | +| The uploads — a volume archive, or the S3 bucket | Every post keeps its text and loses its images; every member loses their avatar. A board on [object storage](./operating.md#where-uploads-go) skips this step entirely: the bucket never lived on the machine. | +| The environment — your `.env`, or the secrets the panel generated | The board boots with new secrets, but `AUTH_SECRET` seals members' two-factor secrets: lose it and every enrolled authenticator app is stranded, and every unsubscribe link in already-sent mail dies. Sessions survive either way — they are random tokens stored hashed in the database. | + +The code is not on the list. It is in git, pinned by the release tag the +board was running — which the dump itself can tell you: the recorded +version is in the database, and the admin panel showed it every day. + +## The order + +Each step exists because a later one assumes it. Resist reordering under +pressure — restoring uploads before the database wastes no work, but +serving traffic before verifying does. + +### 1. Provision + +A server like the one you lost: the +[requirements](./self-hosting.md#what-you-need) have not changed because +the old machine died. Install Docker, clone the repository, and check out +**the release the board was running** — never a newer one, never `main`: + +```sh +git clone https://github.com/meith-dev/meith && cd meith/docker +git checkout v0.12.0 +``` + +Recovering and upgrading are two changes; make them one at a time. Once +the board is verified and serving, upgrade the ordinary way — +[Upgrading a board](./upgrading.md) — with this recovery as the backup +you just proved restorable. + +### 2. Write the environment + +Recreate `.env` from your copy — the same `POSTGRES_PASSWORD`, the same +`AUTH_SECRET`, the same `TICK_SECRET`, the same `S3_*` and mail values. +This step comes before the database because the compose stack will not +start without it, and the migrator refuses to run unnamed. + +If a secret truly is lost, the table above says what each one costs. +`TICK_SECRET` is the cheap one — generate a new value and update whatever +external scheduler presents it, if anything does. `AUTH_SECRET` is the +expensive one; regenerating it means telling your members to re-enrol +their authenticator apps, so exhaust the places a copy might be first. + +### 3. Restore the database + +Bring up Postgres alone, restore into it, and keep the board down until +the data is in: + +```sh +docker compose up -d postgres +gunzip -c board-2026-08-20.sql.gz | docker compose exec -T postgres psql -U community community +``` + +(For a `--format=custom` dump, `pg_restore --no-owner --no-privileges` +into the same place — +[Restoring](./operating.md#restoring) has the variants.) The compose +stack's `migrate` service will run before anything serves; on a dump +taken from the same version it reports nothing to do, and that silence is +itself a check. + +### 4. Restore the uploads + +On local disk, unpack the archive into the named volume before the board +comes up: + +```sh +docker compose up -d --no-start web +docker run --rm -v meith_uploads:/u -v "$PWD":/backup alpine \ + tar xzf /backup/uploads-2026-08-20.tar.gz -C /u +``` + +(`docker volume ls` for the real volume name — Compose prefixes it with +the project directory.) On S3 there is nothing to restore: confirm the +credential in `.env` still works and move on. This asymmetry is most of +the argument for +[moving uploads to object storage](./operating.md#moving-a-board-from-local-disk-to-s3) +on a calm day. + +### 5. Boot and verify + +```sh +docker compose up -d --build +``` + +Then the same three checks a +[restore rehearsal](./operating.md#rehearse-it) uses, plus two this +situation adds: + +1. `select count(*) from posts;` — the content is there. +2. Sign in as an administrator — the credentials survived. +3. `community upgrade --dry-run` — it reports nothing to do. +4. Open a thread with attachments and a page with avatars — the uploads + restore actually met the database restore. +5. `/admin/settings?group=mail` → **Send a test message** — mail is the + subsystem that fails silently, and the provider may be seeing a new IP + address. + +Verify on the new machine directly — `curl` against localhost, or a +hosts-file entry for your domain — while the world still resolves to the +old address. Nothing here needs DNS. + +### 6. Cut over + +Point the domain at the new machine. The reverse proxy obtains its +certificate on the first request after DNS moves ([put a proxy in +front](./self-hosting.md#5-put-a-proxy-in-front)); until propagation +finishes, some visitors reach the corpse and some the recovery — which is +another reason the old address should serve nothing rather than something +stale. If you can plan ahead at all, a short TTL on the record turns this +step from hours into minutes. + +### 7. Resume the backups + +The new machine has no cron, and the recovery you just finished consumed +a backup rather than producing one. Re-create the schedule from +[taking one](./operating.md#taking-one), run it once by hand, and copy +the result off the machine — the next disaster does not care how recent +the last one was. + +## Partial losses are smaller pages + +Full loss is rare; most bad days are one of these, and each has a shorter +answer than this runbook: + +| What happened | Do this | +|---|---| +| A bad upgrade | Restore the pre-upgrade backup — [Downgrades](./upgrading.md#downgrades) explains why that is the whole answer. | +| The database is fine, uploads are gone | Restore the uploads archive alone (step 4); the board can serve, with broken images, while it runs. | +| Uploads are fine, the database is gone | Steps 3 and 5. Do not skip the verification because the machine survived. | +| The Valkey/Redis cache server died | Nothing. It held [cache entries only](./scaling.md#what-redis-is-and-is-not-used-for); restart it and the cache warms on the next requests. | +| The disk is full | Not a restore at all — free space (old backups on the server are the usual culprit, which is its own lesson) and the board resumes. | + +## Rehearse it, and write the number down + +The [backup page's advice](./operating.md#rehearse-it) — a backup nobody +has restored is a file, not a backup — applies to this whole page: a +runbook nobody has run is a hope, not a plan. Once, on a scratch server +or a laptop, run steps 1 through 5 against last week's real backups and +time it. That number is your recovery time; the gaps you hit are this +page's errata for your board — a panel-generated secret you never copied, +a volume name you guessed wrong. Fix what you find, note the time +somewhere that is not on the server, and repeat after anything about the +deployment changes shape. diff --git a/docs/operating.md b/docs/operating.md index 8af20cbc..af945c43 100644 --- a/docs/operating.md +++ b/docs/operating.md @@ -152,9 +152,69 @@ real disk mounted as a volume. DigitalOcean Spaces — which also switches the client to path-style addressing. +**The bucket stays private.** The board never hands a visitor a bucket +URL: every avatar, attachment and logo is fetched by the server and +served through the board's own routes, behind the board's own permission +checks. So there is no public-access setting to get right, no CORS to +configure, and a leaked bucket listing is the only thing a public bucket +would have added. Create the bucket closed and leave it closed. + +The endpoint and region are the only parts that vary by provider: + +| Provider | `S3_REGION` | `S3_ENDPOINT` | +|---|---|---| +| AWS S3 | the bucket's region | unset | +| Cloudflare R2 | `auto` | `https://.r2.cloudflarestorage.com` | +| DigitalOcean Spaces | the Space's region, e.g. `ams3` | `https://ams3.digitaloceanspaces.com` (the region's endpoint, **not** the bucket's own hostname) | +| MinIO | whatever your server declares, `us-east-1` by default | your MinIO URL | + +The credential should be scoped to this one bucket — the board needs to +read, write and delete objects in it, and nothing else about your +account. + On local disk, the uploads directory is the second thing to back up; on an object store the bucket has its own backup story. See -[backup and restore](#backup-and-restore). +[backup and restore](#backup-and-restore) — and the +[disaster-recovery runbook](./disaster-recovery.md), which is the page +this choice makes shorter. + +### Moving a board from local disk to S3 + +The store's keys are the same in both drivers — a file at +`.uploads/avatars/7.webp` becomes the object `avatars/7.webp` — so the +migration is a copy, a cutover, and a second copy to catch the gap. +Nothing in the database changes: it stores keys, not URLs. + +1. **Create the bucket and credentials**, per the table above, and put + the four (or five) `S3_*` values where your deployment reads its + environment. Leave `FILESTORE_DRIVER` alone for now. +2. **Copy the uploads into the bucket** while the board runs. Any + S3-capable copier works; from the compose host with + [rclone](https://rclone.org): + + ```sh + docker run --rm -v meith_uploads:/u:ro -v "$PWD/rclone.conf":/config/rclone/rclone.conf \ + rclone/rclone copy /u remote:your-bucket + ``` + + (Check the volume's real name with `docker volume ls` — Compose + prefixes it with the project directory, Coolify with the resource's + UUID.) `aws s3 sync` or MinIO's `mc mirror` do the same job. +3. **Cut over**: set `FILESTORE_DRIVER=s3` and redeploy. Set it on the + **web and worker** both — the worker touches the store too. +4. **Copy again.** Anything uploaded between step 2 and the redeploy + exists only on the volume; a second `copy` (it is incremental) closes + the window. Until it runs, those few files 404 — minutes-old avatars, + nothing older. +5. **Verify before deleting anything**: open a page with avatars, + download an attachment, upload a new file and download that too. The + old volume then becomes your fallback — keep it until the first + [restore rehearsal](#rehearse-it) with the new arrangement has + succeeded. + +Moving the other way — S3 back to local — is the same copy in reverse, +with `FILESTORE_DRIVER` returned to `local` and the files landed in the +volume the compose file mounts. ### Settings from the command line @@ -2244,6 +2304,10 @@ versions — is [Upgrading a board](./upgrading.md). > recovery procedure — which is why it is worth rehearsing before you > need it. +This section is the everyday half: what to take and how to put one piece +back. The order of operations for the day the whole machine is gone is +its own page — the [disaster-recovery runbook](./disaster-recovery.md). + ### What to back up Two things, and only one of them is the database: diff --git a/docs/scaling.md b/docs/scaling.md index 22f2932c..73256d7d 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -143,8 +143,10 @@ there is still only one instance to watch. If boot fails naming `FILESTORE_DRIVER=s3` and the `S3_*` variables, and copy the existing uploads across — the files under the uploads volume keep their keys, so a `rclone` or `aws s3 sync` of the volume's contents into the bucket is the -whole job. Skip this step while every instance shares one machine and one -volume. +whole job; +[moving a board from local disk to S3](./operating.md#moving-a-board-from-local-disk-to-s3) +is the step-by-step. Skip this step while every instance shares one +machine and one volume. **4. Add instances.** On Coolify, raise the web service's replica count. On Compose, the `--scale web=3` shape above. Watch the board for a diff --git a/docs/self-hosting.md b/docs/self-hosting.md index 62e8c879..1101bb8b 100644 --- a/docs/self-hosting.md +++ b/docs/self-hosting.md @@ -341,7 +341,9 @@ it: - **Backups are yours.** Nobody else is taking one — the database *and* the uploads volume. See - [backup and restore](./operating.md#backup-and-restore). + [backup and restore](./operating.md#backup-and-restore), and the + [disaster-recovery runbook](./disaster-recovery.md) for the day they + are all you have. - **Certificates are yours.** Caddy makes it a solved problem, but it is a problem you now own. - **Security updates are yours.** `unattended-upgrades` for the host; a