Skip to content

feat: add distributed browser load-testing framework#2112

Open
PeterBaker0 wants to merge 24 commits into
mainfrom
feat/load-testing
Open

feat: add distributed browser load-testing framework#2112
PeterBaker0 wants to merge 24 commits into
mainfrom
feat/load-testing

Conversation

@PeterBaker0

@PeterBaker0 PeterBaker0 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Disclaimer

This load testing framework is experimental and should not be pointed at prod. For a couple of reasons

  • it was heavily AI implemented based on my architectural decisions, and has not been heavily vetted by me (nor I propose by this reviewer)
  • load testing against prod is risky - excessive load could cause unstable behaviour, data loss etc - we don't know yet
  • the security posture of this ephemeral load testing infrastructure is very poor - it is publicly visible, there is no authentication on any aspect of the infrastructure

With this being said, the purpose of this is a throwaway infra stack to scale up a distributed agent using the app - it is not on the critical app path so doesn't need to be as well engineered from a code quality perspective.

I have tested this up to 30 concurrent agents on AWS for 30 minutes and works - we see significant DB stress on our AWS dashboards + other key metrics in the grafana dashboard. The DASS plan will be to deploy a temporary prod clone with no real data, users or risk, which emulates the performance characteristics of our real prod - we can then not worry about security posture of this ephemeral infra.

With this in mind, I don't think this PR needs to be heavily scrutinised from a code quality perspective, however the integration points with the 'main' code (e.g. instrumentation), the overall architecture/approach and other aspects are worth considering. If this can't make it into the main codebase - I may just maintain a fork or branch with this code in it, as it is useful enough that I will use it for the DASS regardless of the outcome of this review, but would rather see it contribute to the open source repo.

AI generated PR below:


Summary

Adds a distributed, browser-based load-testing framework for the FAIMS3 collection app. A coordinator executes JSON sequence plans and assigns steps to Playwright agents; metrics flow to Prometheus/Grafana via Pushgateway. Includes local docker-compose workflow, AWS CDK infra for on-demand ECS Fargate runs, and supporting app instrumentation + test hooks.

Not for production — designed for ephemeral staging/dev environments with disposable test accounts.

What's included

Load-testing framework (load-testing/)

  • Coordinator — Hono server: agent registry, sticky account pool, sequence-plan engine (loop / split / phase steps), progress reporting, metrics push
  • Agents — Playwright workers polling GET /step; scenarios for onboarding, online/offline collection, patchy network, and export stress
  • Shared package (@faims3/load-testing-shared) — Zod schemas, coordinator client, bundled sequence plans and collection profiles (declarative record-fill workflows)
  • Observability — Prometheus, Pushgateway, Grafana dashboard, CouchDB exporter configs
  • AWS infra (CDK) — VPC, ECS cluster + Fargate task defs, S3 bucket for plan delivery (avoids ECS 8KB env limit), metrics EC2 with Route53
  • Scriptsrun-load-test.sh, account seeding, coordinator status polling, cost estimation, debug helpers

App & library support

  • @faims3/instrumentation — User Timing API wrapper with standardised dass.* measure names
  • Performance marks in survey activation, sync push/pull/reconnect/conflict, and record save UI
  • data-testid hooks on activation, add-record, sync status, form navigation, and finish-anyway dialog — for stable Playwright selectors

API tooling

  • seedLoadTestAccounts.ts — create/update load-test users (no DB migrations)
  • env-from-cdk-stack.sh — generate .env from a deployed Conductor ECS task definition

Other

  • Dockerfile: include new workspace packages; fix web service CMD typo
  • pnpm workspace entries for instrumentation and load-testing packages

Architecture note

Plans are loaded only by the coordinator (file, inline, or S3 URI). Agents receive individual steps over HTTP and never hold plan JSON. Collection profile filenames are inlined before S3 upload on AWS runs.

Known limitations

  • Web/PWA collection app only (not native iOS/Android)
  • Collection profiles skip photo/attachment fields today — requires future app-side test hooks
  • Requires pre-seeded users or local login; SSO-only envs need manual setup

Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
…twork smashing and timeouts

Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
Signed-off-by: Peter Baker <peter.baker122@csiro.au>
@PeterBaker0
PeterBaker0 requested a review from stevecassidy June 9, 2026 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant