feat(major): new authentication layer between couch DB and app by PeterBaker0 · Pull Request #2211 · FAIMS/FAIMS3

PeterBaker0 · 2026-07-26T08:22:11Z

Summary

Closes the long-standing per-document read gap on Pouch↔Couch sync by putting couch-auth-proxy in front of project data DBs and aligning the FAIMS write path, migrations, and AWS topology with a single ACL model.

Until now, PROJECT_GUEST (and anyone with only READ_MY_PROJECT_RECORDS) could replicate an entire data-* database: Couch _security is DB-wide, while API/UI already enforced my/all. This PR makes DB access match that policy without inventing a second permission language.

The problem

Layer	My vs all enforced?
Permission model (`READ_MY_` / `READ_ALL_`)	Intended policy
Conductor Records CRUD / `canReadRecord`	Yes
App `shouldDisplayRecord`	UI filter only
Couch `_design/permissions` VDU	Writes yes
Couch `_security` + Pouch sync	No — full DB readable to members

Guests must remain DB members to sync their own graph. Couch alone cannot answer “which docs?” — only “may talk to this DB?”.

What is couch-auth-proxy?

A reverse proxy (Docker/GHCR service, not an npm dep) in front of CouchDB that:

Authenticates with Couch-native session resolution (Authorization / Cookie → GET /_session). FAIMS RSA JWT validation stays in Couch (jwt_keys); Conductor JWT issuance is unchanged.
Enforces per-document grants on sync traffic (_changes, _bulk_get, _all_docs, attachments, etc.) using doc fields creator / owners / acl / parent, plus a bucket overlay on _design/acl (dbacl.{_r,_w,_d}).
Owns the ACL protocol design doc (map + VDU). FAIMS never vendors that code, we pin the image (1.7.0 in compose/CDK) and only patch project-scoped dbacl.

Role in FAIMS: the sync read boundary for apps. Conductor keeps admin Basic on COUCHDB_INTERNAL_URL and bypasses the proxy.

App (Pouch + JWT) ──COUCHDB_PUBLIC_URL──► couch-auth-proxy ──► CouchDB
Conductor (admin) ─COUCHDB_INTERNAL_URL─────────────────────► CouchDB

Auth / ACL model (FAIMS + proxy)

One policy source: @faims3/data-model RBAC. dbacl role lists are derived from necessaryActionToCouchRoleList for *_ALL_PROJECT_RECORDS — not hand-maintained role literals.

Clean write path (always stamped; no “legacy mode” on hot path):

Doc	ACL fields
`rec-*`	`creator` = record owner (alongside existing `created_by`)
`frev-` / `avp-` / attachments	`creator` = writer; `parent` = `rec-*` id (inherit record ACL)

Mapping:

Capability	Sync enforcement
`READ_MY_PROJECT_RECORDS`	`creator` / `parent` on the record graph
`READ_ALL_PROJECT_RECORDS`	`dbacl._r`
Edit/delete my/all	`creator` / `dbacl` + existing `_design/permissions` VDU

Defense in depth stays: proxy (sync reads) + permissions VDU (writes) + API canReadRecord + UI filter. Layering is documented in AclValidationLayering.md.

Unstamped docs remain member-readable (r-*) by proxy semantics — which is why migrate/repair must follow deploy promptly.

What’s in this PR

data-model: ACL stamp helpers, types, engine/attachment stamping, _design/faims_acl_shape, DATA migration v1→v2
api: warm proxy + patch dbacl, repair-data-db-acl, integration suite
app: remote URL re-point on public-URL changeover (no automatic local IndexedDB wipe)
compose / localdev: proxy on :5985; COUCHDB_PUBLIC_URL → proxy
AWS CDK: always-on proxy on shared ALB (couch.* → proxy; Couch VPC-only); Conductor public vs internal URL split
e2e: guest-record-isolation.e2e.ts + sync probe harness
docs: handover, cutover runbook, CDK design, permission-model sync section

Migration / cutover path

Deploy-then-migrate (intentional, not multi-phase idle flip):

Deploy stack that always routes public sync through the proxy (compose already; AWS CDK always-on — deploy is the cutover).
Migrate all data-* DBs to version 2 (migrate-with-keys / Conductor startup migrate).
pnpm --filter=@faims3/api run repair-data-db-acl (warm _design/acl, patch dbacl, ensure FAIMS shape ddoc).
Validate guest isolation (test:couch-auth-proxy + e2e).

Until migrate stamps legacy docs, effective access matches today’s public Couch (not a new privacy regression). After migrate, guests lose other users’ graphs on the wire.

Known limitation: already-activated clients may retain pre-cutover local docs until refresh / re-activate / clear site data. Wire isolation is guaranteed; local wipe was deliberately not implemented.

Rollback: re-point public hostname / COUCHDB_PUBLIC_URL at Couch → re-opens the old read gap. ACL fields and _design/acl can remain harmlessly. There is no enabled: false flag on AWS.

Still open: DigitalOcean public→proxy / Couch-internal split (compose + AWS are the reference).

Risks vs rewards

Rewards	Risks / costs
Guests can no longer pull others’ records/revisions/AVPs/attachments over sync	New mandatory runtime dependency (proxy availability = sync availability)
Sync semantics finally match API/UI and product roles	Cutover requires prompt migrate + repair; missed migrate leaves `r-*` gaps
Idiomatic split: FAIMS owns policy, proxy owns filter protocol	Extra ECS service + SG path in AWS; image pin must stay in sync across compose/CDK
Conductor admin path unchanged; same public hostname on AWS (no client rebuild)	Local leftover docs on old devices; validate with clean clients
Fail-closed creates (`ACL_REQUIRE_CREATOR`) + orphan stamp for missing `created_by`	Proxy constraints (e.g. no multipart doc writes, limited reduce) — FAIMS paths already compatible
Clear ops runbook + repair tool + integration/e2e proof	DO/custom deploys must not leave Couch publicly reachable beside the proxy

Net: this is the architectural fix for a real confidentiality hole in offline-first sync. The main operational bet is always-on proxy + immediate DATA v2 migrate, accepted as clearer and safer than a long dual-mode period.

Docs

CouchAuthProxyHandover.md — design brief
CouchAuthProxyCutover.md — operator runbook
AclValidationLayering.md — ownership / VDU split
CouchAuthProxyAwsCdk.md — AWS topology

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

…rd name Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Close the Couch sync read gap for PROJECT_GUEST by stamping creator/parent on the clean write path, provisioning _design/acl with dbacl from the permission model, adding DATA v1→v2 backfill, and wiring compose so COUCHDB_PUBLIC_URL points at couch-auth-proxy while Conductor keeps the internal Couch URL. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Preserve creator/parent through hydrated revision updates, stamp the legacy upsert/merge/attachment write paths, and fix migrationService tests that assumed DB_MIGRATIONS[0] was PEOPLE v1→v2. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Keep creator/parent optional in Zod/legacy types so pre-migration fixtures still parse, while write paths continue to stamp. Wait for the proxy ACL follower before assertions, pin the compose image to the published sha tag, and exclude the integration suite from the default mocha glob. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Bring feat/couch-auth-proxy-integration-413d onto feat/proxy-auth-integration as-is from main. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Close remaining handover gaps: keep DATA defaultVersion at 1 so orphan data DBs still backfill, fail-closed orphan creator stamping, expand proxy integration scenarios, add dbacl repair script, and rebuild local client data DBs after ACL cutover. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Capture a secure target topology (ALB → ECS proxy, Couch VPC-only), config/Conductor URL split, SG rules, and cutover order so the remaining infra follow-on from the proxy handover can be implemented safely. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Remote Pouch handles expose a URL as db.name; Conductor now enqueues DATA migrations under the logical data-{projectId} name, and performMigration passes that name through so _design/acl ensure cannot be skipped. Client ACL markers now bind to the advertised public base URL so a seal against open Couch is invalidated when COUCHDB_PUBLIC_URL flips to the proxy. Add operator cutover runbook and expand migration/deploy docs. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Resolve toctree conflict by keeping both Cutover and AwsCdk pages; cross-link the operator runbook with the CDK design brief on #20. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Point production proxy follow-up at CouchAuthProxyAwsCdk.md. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Add optional CouchAuthProxy ECS Fargate construct gated by couchAuthProxy.enabled. When enabled, public couch.* terminates on the proxy, Conductor gets distinct PUBLIC/INTERNAL Couch URLs, and Couch :5984 is limited to proxy + Conductor security groups. Legacy ALB→Couch behaviour is preserved when disabled for rollback. Do not enable in a live env until DATA v2 migration completes (see CouchAuthProxyCutover.md). Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Remove couchAuthProxy.enabled and attachToPublicAlb. The proxy is mandatory: ALB couch.* always terminates on ECS proxy, Couch is VPC-only, and Conductor always gets distinct PUBLIC/INTERNAL Couch URLs. Reject legacy enabled keys in config via Zod .strict(). Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Same-hostname AWS flips now bump COUCH_ACL_CLIENT_SCHEMA_VERSION so clients rebuild IndexedDB; fail-closed VDU requires creator/parent; restore and engine update paths re-stamp ACL; docs/tests aligned with always-on CDK. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Remove openLocalDataDbWithAclCutover, the _local/faims-acl-schema marker, and dataDb.acl_client_schema_version / COUCH_ACL_CLIENT_SCHEMA_VERSION. Keep public-URL changeover: re-point remotes after listing when Conductor advertises a new couch URL. Accept that pre-proxy leftover local records may linger until refresh / re-activate. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

…FAIMS3 into feat/proxy-auth-integration

…and other doc changes. Issue with auth proxy cold ACL cache on first sync resolved upstream v1.7 Signed-off-by: Peter Baker <peter.baker122@csiro.au>

…- all passing Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

stevecassidy

I've not done a full review but left a few comments. This is a big change. I wonder if we could break it into two parts:

Document changes needed to support the fine-grained ACL model
Support for the proxy

The first is likely to be relatively small and unproblematic. The proxy would ideally be optional and this split might make it easier to ensure that everything still works without it.

stevecassidy · 2026-07-27T01:16:21Z

+
+  # Live Couch + couch-auth-proxy ACL proof (guest isolation, VDU, dbacl).
+  # Fails hard if the proxy image/compose pin is unreachable (no skip in CI).
+  couch-auth-proxy:


Do we really want to deploy a proxy server during a build+lint check? Along with e2e tests this is becoming a very heavy process.

stevecassidy · 2026-07-27T01:16:48Z

@@ -0,0 +1,278 @@
+#!/usr/bin/env bash


Why is a CDK script here rather than in the AWS directory?

stevecassidy · 2026-07-27T01:19:05Z


+/**
+ * Stamp missing couch-auth-proxy ACL fields on restored data docs.
+ * Backups taken before DATA v1→v2 (or from open Couch) may lack `creator` /


We would need to do a migration in general from the db version that was backed up to now.

stevecassidy · 2026-07-27T01:21:56Z

    );
  }

+  // Proxy owns `_design/acl` map/VDU; warm it then patch FAIMS `dbacl`.


spell out VDU please, not a common acronym

stevecassidy · 2026-07-27T01:23:13Z

      delete project._rev;
      // add database connection details
-      if (project.dataDb) project.dataDb.base_url = config.couchdbPublicUrl;
+      if (project.dataDb) {


why braces now? This file has not changed.

stevecassidy · 2026-07-27T01:24:41Z

@@ -0,0 +1,189 @@
+/* eslint-disable n/no-process-exit */


This looks useful but why is it here?

stevecassidy · 2026-07-27T01:29:00Z

@@ -0,0 +1,137 @@
+/* eslint-disable n/no-process-exit */
+/**
+ * Idempotent ops tool: warm couch-auth-proxy `_design/acl` on every project


I don't know what 'warm' means here and I can't work it out from the script. Do you mean it will be replaced by an updated version or something about it will be changed? You used 'warm' elsewhere too. Can you use a more descriptive term?

stevecassidy · 2026-07-27T01:30:29Z

@@ -0,0 +1,230 @@
+#!/usr/bin/env bash


Ah ok, you have these in each project - maybe one script in the AWS folder would be better.

cursoragent and others added 30 commits June 30, 2026 04:45

docs: add Cursor Cloud dev environment setup instructions

bad233a

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Merge pull request #15 from PeterBaker0/feat/dev-environment-setup-fcd6

9233755

docs: add Cursor Cloud dev environment setup instructions

7549b22

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Merge branch 'main' of github.com:PeterBaker0/FAIMS3

a4815df

Merge upstream FAIMS/FAIMS3 main into PeterBaker0 fork main

4411bdc

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Merge upstream FAIMS/FAIMS3 main into PeterBaker0 fork main

8df5fc6

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Env builder script and couchDB changes in CDK to use discrete dashboa…

c48cc1f

…rd name Signed-off-by: Peter Baker <peter.baker122@csiro.au>

WIP

88aabf3

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Improved process guide

1f93de8

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Merge branch 'main' into feat/recovery-checks

77cafc4

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Minor improvements

5e75b73

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Merge branch 'main' into feat/recovery-checks

d494c3e

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Improvements to streamline

1b5904e

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Merge fix

2018744

Merge PR #19: couch-auth-proxy per-document sync ACL

b25866e

Bring feat/couch-auth-proxy-integration-413d onto feat/proxy-auth-integration as-is from main. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

docs: merge AWS CDK couch-auth-proxy design into proxy integration

e12278d

Resolve toctree conflict by keeping both Cutover and AwsCdk pages; cross-link the operator runbook with the CDK design brief on #20. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

docs: link cutover runbook to AWS CDK design

7f9ed91

Point production proxy follow-up at CouchAuthProxyAwsCdk.md. Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

docs: note always-on proxy has no enabled-flag rollback

66277e7

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

style: oxfmt after ACL swapover removal

5f58196

Co-authored-by: Peter Baker <PeterBaker0@users.noreply.github.com>

Upgrade version

215fd38

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

PeterBaker0 added 10 commits July 24, 2026 21:40

Improvements

25bd648

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Merge branch 'feat/recovery-checks' into feat/proxy-auth-integration

25ca426

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

IMprovements

483312d

Merge branch 'feat/proxy-auth-integration' of github.com:PeterBaker0/…

85c7d39

…FAIMS3 into feat/proxy-auth-integration

Missing files

26f7f6e

Improvements to how we do migrations to keep clean single step path, …

f071bc9

…and other doc changes. Issue with auth proxy cold ACL cache on first sync resolved upstream v1.7 Signed-off-by: Peter Baker <peter.baker122@csiro.au>

testing for integration after improvements to root access and v bump …

f118903

…- all passing Signed-off-by: Peter Baker <peter.baker122@csiro.au>

Formatting linting typing

79823c1

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

CI fix

7bff5af

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

env improvements to be able to pull env for debug apk builds

ce8fe3e

Signed-off-by: Peter Baker <peter.baker122@csiro.au>

stevecassidy reviewed Jul 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(major): new authentication layer between couch DB and app#2211

feat(major): new authentication layer between couch DB and app#2211
PeterBaker0 wants to merge 40 commits into
FAIMS:mainfrom
PeterBaker0:feat/proxy-auth-integration

PeterBaker0 commented Jul 26, 2026

Uh oh!

stevecassidy left a comment

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

stevecassidy Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

PeterBaker0 commented Jul 26, 2026

Summary

The problem

What is couch-auth-proxy?

Auth / ACL model (FAIMS + proxy)

What’s in this PR

Migration / cutover path

Risks vs rewards

Docs

Uh oh!

stevecassidy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants