docs(operating): object storage first-class, and a disaster-recovery runbook

README.md

@@ -115,6 +115,7 @@ The table below is written from `apps/web/content/docs.manifest.json` by
 | Using your board | [`membership-guide.md`](./docs/membership-guide.md) | Taking the subs through the board: plans, discount codes, gifting, the memberships desk and the ledger — Dues end to end, without a terminal. |
 | Running the server | [`operating.md`](./docs/operating.md) | The operator handbook: configuration, the CLI, permissions, themes, plugins, mail, spam controls, backups, and troubleshooting. |
 | Running the server | [`upgrading.md`](./docs/upgrading.md) | Moving a board between versions: the upgrade command, how far you can jump, and the behaviour changes each release brings. |
+| Running the server | [`disaster-recovery.md`](./docs/disaster-recovery.md) | The runbook for the day the server is gone: what recovery consumes, the order of operations from provisioning to DNS, and the rehearsal that turns it from a hope into a plan. |
 | Running the server | [`single-sign-on.md`](./docs/single-sign-on.md) | Two-factor authentication, federated sign-in and passkeys: what each means for your members, how to configure them, and the record of what has opened an account. |
 | Running the server | [`internationalisation.md`](./docs/internationalisation.md) | How a page picks its language, how to add one, and how a theme or plugin ships its own words. |
 | Running the server | [`web-push.md`](./docs/web-push.md) | Notifications that reach a member who does not have the board open, and the manifest that makes the board installable — what it costs their privacy, and how to turn it on. |

apps/web/content/docs.manifest.json

@@ -116,6 +116,15 @@
       "generated": false,
       "primary": false
     },
+    {
+      "slug": "disaster-recovery",
+      "file": "disaster-recovery.md",
+      "section": "running",
+      "title": "Disaster recovery",
+      "blurb": "The runbook for the day the server is gone: what recovery consumes, the order of operations from provisioning to DNS, and the rehearsal that turns it from a hope into a plan.",
+      "generated": false,
+      "primary": false
+    },
     {
       "slug": "single-sign-on",
       "file": "single-sign-on.md",

docs/README.md

@@ -26,6 +26,7 @@ somebody has already set the board up for you, start with
 | Run the server day to day | [Running a board](./operating.md) |
 | Run more than one web container | [Scaling out](./scaling.md) |
 | Move a board to a new version | [Upgrading a board](./upgrading.md) |
+| Recover a board whose server is gone | [Disaster recovery](./disaster-recovery.md) |
 | Add 2FA, SSO or passkeys | [Signing in](./single-sign-on.md) |
 | Run a board in another language | [Languages](./internationalisation.md) |
 | Push notifications to a phone | [Web push](./web-push.md) |
@@ -68,6 +69,7 @@ For whoever minds the machine.
 |---|---|
 | [`operating.md`](./operating.md) | The operator handbook: configuration, the CLI, permissions, themes, plugins, mail, spam controls, backups, and troubleshooting. |
 | [`upgrading.md`](./upgrading.md) | Moving between versions: the upgrade command, how far you can jump, and the behaviour changes each upgrade brings. |
+| [`disaster-recovery.md`](./disaster-recovery.md) | The full-loss runbook: what recovery consumes, the order of operations from provisioning to DNS cutover, and the rehearsal that measures your recovery time. |
 | [`single-sign-on.md`](./single-sign-on.md) | Everything beyond a password: two-factor authentication, federated sign-in, passkeys, sessions, and the sign-in activity log. |
 | [`internationalisation.md`](./internationalisation.md) | How a page picks its language, how to add one, and how a theme or plugin ships its own words. |
 | [`web-push.md`](./web-push.md) | Notifications that reach a member who does not have the board open, the privacy that costs, and the manifest that makes the board installable. |

docs/disaster-recovery.md

@@ -0,0 +1,163 @@
+# Disaster recovery
+
+This page is the runbook for the bad day: the server is gone — seized,
+dead, deleted, or unreachable in a way that is not coming back — and the
+board has to exist again somewhere else.
+[Backup and restore](./operating.md#backup-and-restore) is the everyday
+half of this story: what to take, how to take it, and how to restore one
+piece. This page is the order of operations when *everything* has to be
+restored at once, written to be followed under stress. Read it once on a
+calm day; the last section asks you to do slightly more than read it.
+
+## What recovery consumes
+
+Recovery is assembled from three artifacts, and the reason each backup
+guide says **copy them off the machine** is that this page assumes the
+machine is gone:
+
+| Artifact | Without it |
+|---|---|
+| The database dump | There is no board to recover. Everything the board knows — accounts, posts, settings, permissions — is here. |
+| The uploads — a volume archive, or the S3 bucket | Every post keeps its text and loses its images; every member loses their avatar. A board on [object storage](./operating.md#where-uploads-go) skips this step entirely: the bucket never lived on the machine. |
+| The environment — your `.env`, or the secrets the panel generated | The board boots with new secrets, but `AUTH_SECRET` seals members' two-factor secrets: lose it and every enrolled authenticator app is stranded, and every unsubscribe link in already-sent mail dies. Sessions survive either way — they are random tokens stored hashed in the database. |
+
+The code is not on the list. It is in git, pinned by the release tag the
+board was running — which the dump itself can tell you: the recorded
+version is in the database, and the admin panel showed it every day.
+
+## The order
+
+Each step exists because a later one assumes it. Resist reordering under
+pressure — restoring uploads before the database wastes no work, but
+serving traffic before verifying does.
+
+### 1. Provision
+
+A server like the one you lost: the
+[requirements](./self-hosting.md#what-you-need) have not changed because
+the old machine died. Install Docker, clone the repository, and check out
+**the release the board was running** — never a newer one, never `main`:
+
+```sh
+git clone https://github.com/meith-dev/meith && cd meith/docker
+git checkout v0.12.0
+```
+
+Recovering and upgrading are two changes; make them one at a time. Once
+the board is verified and serving, upgrade the ordinary way —
+[Upgrading a board](./upgrading.md) — with this recovery as the backup
+you just proved restorable.
+
+### 2. Write the environment
+
+Recreate `.env` from your copy — the same `POSTGRES_PASSWORD`, the same
+`AUTH_SECRET`, the same `TICK_SECRET`, the same `S3_*` and mail values.
+This step comes before the database because the compose stack will not
+start without it, and the migrator refuses to run unnamed.
+
+If a secret truly is lost, the table above says what each one costs.
+`TICK_SECRET` is the cheap one — generate a new value and update whatever
+external scheduler presents it, if anything does. `AUTH_SECRET` is the
+expensive one; regenerating it means telling your members to re-enrol
+their authenticator apps, so exhaust the places a copy might be first.
+
+### 3. Restore the database
+
+Bring up Postgres alone, restore into it, and keep the board down until
+the data is in:
+
+```sh
+docker compose up -d postgres
+gunzip -c board-2026-08-20.sql.gz | docker compose exec -T postgres psql -U community community
+```
+
+(For a `--format=custom` dump, `pg_restore --no-owner --no-privileges`
+into the same place —
+[Restoring](./operating.md#restoring) has the variants.) The compose
+stack's `migrate` service will run before anything serves; on a dump
+taken from the same version it reports nothing to do, and that silence is
+itself a check.
+
+### 4. Restore the uploads
+
+On local disk, unpack the archive into the named volume before the board
+comes up:
+
+```sh
+docker compose up -d --no-start web
+docker run --rm -v meith_uploads:/u -v "$PWD":/backup alpine \
+  tar xzf /backup/uploads-2026-08-20.tar.gz -C /u
+```
+
+(`docker volume ls` for the real volume name — Compose prefixes it with
+the project directory.) On S3 there is nothing to restore: confirm the
+credential in `.env` still works and move on. This asymmetry is most of
+the argument for
+[moving uploads to object storage](./operating.md#moving-a-board-from-local-disk-to-s3)
+on a calm day.
+
+### 5. Boot and verify
+
+```sh
+docker compose up -d --build
+```
+
+Then the same three checks a
+[restore rehearsal](./operating.md#rehearse-it) uses, plus two this
+situation adds:
+
+1. `select count(*) from posts;` — the content is there.
+2. Sign in as an administrator — the credentials survived.
+3. `community upgrade --dry-run` — it reports nothing to do.
+4. Open a thread with attachments and a page with avatars — the uploads
+   restore actually met the database restore.
+5. `/admin/settings?group=mail` → **Send a test message** — mail is the
+   subsystem that fails silently, and the provider may be seeing a new IP
+   address.
+
+Verify on the new machine directly — `curl` against localhost, or a
+hosts-file entry for your domain — while the world still resolves to the
+old address. Nothing here needs DNS.
+
+### 6. Cut over
+
+Point the domain at the new machine. The reverse proxy obtains its
+certificate on the first request after DNS moves ([put a proxy in
+front](./self-hosting.md#5-put-a-proxy-in-front)); until propagation
+finishes, some visitors reach the corpse and some the recovery — which is
+another reason the old address should serve nothing rather than something
+stale. If you can plan ahead at all, a short TTL on the record turns this
+step from hours into minutes.
+
+### 7. Resume the backups
+
+The new machine has no cron, and the recovery you just finished consumed
+a backup rather than producing one. Re-create the schedule from
+[taking one](./operating.md#taking-one), run it once by hand, and copy
+the result off the machine — the next disaster does not care how recent
+the last one was.
+
+## Partial losses are smaller pages
+
+Full loss is rare; most bad days are one of these, and each has a shorter
+answer than this runbook:
+
+| What happened | Do this |
+|---|---|
+| A bad upgrade | Restore the pre-upgrade backup — [Downgrades](./upgrading.md#downgrades) explains why that is the whole answer. |
+| The database is fine, uploads are gone | Restore the uploads archive alone (step 4); the board can serve, with broken images, while it runs. |
+| Uploads are fine, the database is gone | Steps 3 and 5. Do not skip the verification because the machine survived. |
+| The Valkey/Redis cache server died | Nothing. It held [cache entries only](./scaling.md#what-redis-is-and-is-not-used-for); restart it and the cache warms on the next requests. |
+| The disk is full | Not a restore at all — free space (old backups on the server are the usual culprit, which is its own lesson) and the board resumes. |
+
+## Rehearse it, and write the number down
+
+The [backup page's advice](./operating.md#rehearse-it) — a backup nobody
+has restored is a file, not a backup — applies to this whole page: a
+runbook nobody has run is a hope, not a plan. Once, on a scratch server
+or a laptop, run steps 1 through 5 against last week's real backups and
+time it. That number is your recovery time; the gaps you hit are this
+page's errata for your board — a panel-generated secret you never copied,
+a volume name you guessed wrong. Fix what you find, note the time
+somewhere that is not on the server, and repeat after anything about the
+deployment changes shape.

docs/operating.md

@@ -152,9 +152,69 @@ real disk mounted as a volume.
 DigitalOcean Spaces — which also switches the client to path-style
 addressing.
 
+**The bucket stays private.** The board never hands a visitor a bucket
+URL: every avatar, attachment and logo is fetched by the server and
+served through the board's own routes, behind the board's own permission
+checks. So there is no public-access setting to get right, no CORS to
+configure, and a leaked bucket listing is the only thing a public bucket
+would have added. Create the bucket closed and leave it closed.
+
+The endpoint and region are the only parts that vary by provider:
+
+| Provider | `S3_REGION` | `S3_ENDPOINT` |
+|---|---|---|
+| AWS S3 | the bucket's region | unset |
+| Cloudflare R2 | `auto` | `https://<account-id>.r2.cloudflarestorage.com` |
+| DigitalOcean Spaces | the Space's region, e.g. `ams3` | `https://ams3.digitaloceanspaces.com` (the region's endpoint, **not** the bucket's own hostname) |
+| MinIO | whatever your server declares, `us-east-1` by default | your MinIO URL |
+
+The credential should be scoped to this one bucket — the board needs to
+read, write and delete objects in it, and nothing else about your
+account.
+
 On local disk, the uploads directory is the second thing to back up; on an
 object store the bucket has its own backup story. See
-[backup and restore](#backup-and-restore).
+[backup and restore](#backup-and-restore) — and the
+[disaster-recovery runbook](./disaster-recovery.md), which is the page
+this choice makes shorter.
+
+### Moving a board from local disk to S3
+
+The store's keys are the same in both drivers — a file at
+`.uploads/avatars/7.webp` becomes the object `avatars/7.webp` — so the
+migration is a copy, a cutover, and a second copy to catch the gap.
+Nothing in the database changes: it stores keys, not URLs.
+
+1. **Create the bucket and credentials**, per the table above, and put
+   the four (or five) `S3_*` values where your deployment reads its
+   environment. Leave `FILESTORE_DRIVER` alone for now.
+2. **Copy the uploads into the bucket** while the board runs. Any
+   S3-capable copier works; from the compose host with
+   [rclone](https://rclone.org):
+
+   ```sh
+   docker run --rm -v meith_uploads:/u:ro -v "$PWD/rclone.conf":/config/rclone/rclone.conf \
+     rclone/rclone copy /u remote:your-bucket
+   ```
+
+   (Check the volume's real name with `docker volume ls` — Compose
+   prefixes it with the project directory, Coolify with the resource's
+   UUID.) `aws s3 sync` or MinIO's `mc mirror` do the same job.
+3. **Cut over**: set `FILESTORE_DRIVER=s3` and redeploy. Set it on the
+   **web and worker** both — the worker touches the store too.
+4. **Copy again.** Anything uploaded between step 2 and the redeploy
+   exists only on the volume; a second `copy` (it is incremental) closes
+   the window. Until it runs, those few files 404 — minutes-old avatars,
+   nothing older.
+5. **Verify before deleting anything**: open a page with avatars,
+   download an attachment, upload a new file and download that too. The
+   old volume then becomes your fallback — keep it until the first
+   [restore rehearsal](#rehearse-it) with the new arrangement has
+   succeeded.
+
+Moving the other way — S3 back to local — is the same copy in reverse,
+with `FILESTORE_DRIVER` returned to `local` and the files landed in the
+volume the compose file mounts.
 
 ### Settings from the command line
 
@@ -2244,6 +2304,10 @@ versions — is [Upgrading a board](./upgrading.md).
 > recovery procedure — which is why it is worth rehearsing before you
 > need it.
 
+This section is the everyday half: what to take and how to put one piece
+back. The order of operations for the day the whole machine is gone is
+its own page — the [disaster-recovery runbook](./disaster-recovery.md).
+
 ### What to back up
 
 Two things, and only one of them is the database:

docs/scaling.md

@@ -143,8 +143,10 @@ there is still only one instance to watch. If boot fails naming
 `FILESTORE_DRIVER=s3` and the `S3_*` variables, and copy the existing
 uploads across — the files under the uploads volume keep their keys, so a
 `rclone` or `aws s3 sync` of the volume's contents into the bucket is the
-whole job. Skip this step while every instance shares one machine and one
-volume.
+whole job;
+[moving a board from local disk to S3](./operating.md#moving-a-board-from-local-disk-to-s3)
+is the step-by-step. Skip this step while every instance shares one
+machine and one volume.
 
 **4. Add instances.** On Coolify, raise the web service's replica count.
 On Compose, the `--scale web=3` shape above. Watch the board for a

docs/self-hosting.md

@@ -341,7 +341,9 @@ it:
 
 - **Backups are yours.** Nobody else is taking one — the database *and*
   the uploads volume. See
-  [backup and restore](./operating.md#backup-and-restore).
+  [backup and restore](./operating.md#backup-and-restore), and the
+  [disaster-recovery runbook](./disaster-recovery.md) for the day they
+  are all you have.
 - **Certificates are yours.** Caddy makes it a solved problem, but it is
   a problem you now own.
 - **Security updates are yours.** `unattended-upgrades` for the host; a