Skip to content

05_TARGET_ARCHITECTURE.md

Latest commit

 

History

History
331 lines (259 loc) · 7.97 KB

05_TARGET_ARCHITECTURE.md

File metadata and controls

331 lines (259 loc) · 7.97 KB
Raw

05 — Target Architecture

Architecture goal

Evolve the current batch-scored adoption app into a dual-mode platform:

Mode A: Adoption Quest
- ongoing individual scoring
- system-table driven
- scheduled scoring
- good for workspace adoption

Mode B: GameDay Event
- time-boxed team gameplay
- manifest-driven content
- immediate validation attempts
- live scoring and host console
- good for enablement and sales events

Recommended architecture

React App
  ├─ Adoption dashboard
  ├─ Event lobby
  ├─ Team gameplay view
  ├─ Quest detail / submit flow
  ├─ Live leaderboard
  └─ Host console
        ↓
FastAPI
  ├─ Auth and role resolution
  ├─ Event lifecycle service
  ├─ Quest pack loader
  ├─ Validation orchestration service
  ├─ Scoring service
  ├─ Leaderboard service
  ├─ Admin/host service
  └─ Reporting service
        ↓
Lakebase PostgreSQL — operational state
  ├─ events, teams, participants
  ├─ quest packs, quest versions, tasks
  ├─ attempts and validation results
  ├─ scoring events and leaderboard materializations
  ├─ hints, announcements, manual adjustments
  └─ audit log
        ↓
Delta/Unity Catalog — analytics and audit
  ├─ append-only event log
  ├─ post-event analytics
  ├─ adoption scoring outputs
  ├─ workspace telemetry joins
  └─ reporting datasets
        ↑
Databricks Jobs / Serverless validation workers
  ├─ SQL assertions
  ├─ Databricks SDK validators
  ├─ notebook validators
  ├─ system-table validators
  ├─ Python code validators
  └─ resource bootstrap/reset jobs

Why Lakebase should become operational state

The current app reads from Lakebase, but the source of truth is effectively Delta tables generated by a scheduled notebook. That model is too slow for GameDay.

GameDay requires:

  • immediate attempt submission
  • near-real-time validation status
  • live leaderboard
  • host actions
  • transactional scoring
  • low-latency reads

Lakebase is a better operational store for these state transitions. Delta remains essential for audit, analytics, and joining with system tables.

Recommended data flow

Event content loading

Quest pack YAML/JSON
    ↓ import/lint
Quest pack tables in Lakebase
    ↓ optional sync
Delta content registry

Participant submission

Player clicks Submit
    ↓
POST /api/events/{event_id}/tasks/{task_id}/attempts
    ↓
Lakebase: create attempt record
    ↓
Validation service runs sync or async validator
    ↓
Lakebase: validation result + scoring event
    ↓
Leaderboard materialized view updates
    ↓
React polling/SSE updates player and host views
    ↓
Delta audit sync captures event facts

Background telemetry scoring

System tables
    ↓
Adoption scoring pipeline
    ↓
Delta adoption scoring tables
    ↓
Lakebase adoption read model

System-table GameDay validation

Task validator requests telemetry check
    ↓
SQL Warehouse/system table query
    ↓
Validation result in Lakebase
    ↓
Scoring event in Lakebase

Core services

QuestPackService

Responsibilities:

  • validate manifest schema
  • resolve quest pack version
  • load packs into Lakebase
  • expose quests/tasks/hints/assets
  • enforce immutable event pack versions

EventLifecycleService

Responsibilities:

  • create event
  • set active quest pack version
  • manage start/pause/freeze/end states
  • enforce event time windows
  • manage registrations and teams

ValidationEngine

Responsibilities:

  • parse validator definitions
  • enforce allowlists and safety rules
  • run validators
  • normalize result status
  • persist evidence
  • support sync and async validation

ScoringService

Responsibilities:

  • calculate points
  • apply bonus/penalty rules
  • enforce idempotency
  • write scoring events
  • update team and individual scores
  • handle manual adjustments

LeaderboardService

Responsibilities:

  • compute event leaderboard
  • compute quest-level leaderboard
  • freeze final leaderboard
  • expose time-series leaderboard snapshots

ResourceBootstrapService

Responsibilities:

  • create team schemas/catalog resources
  • seed datasets
  • create starter notebooks/files if needed
  • reset resources between dry-runs/events
  • clean up resources safely

ReportingService

Responsibilities:

  • event summary
  • participant/team export
  • skill coverage
  • validation failure patterns
  • feature adoption signals
  • sales follow-up artifact

Validation execution patterns

Synchronous validators

Use for quick checks:

  • SQL assertion
  • Lakebase state check
  • Databricks SDK object existence
  • simple REST check

Target response: under 5 seconds.

Asynchronous validators

Use for heavier checks:

  • notebook execution
  • Python test suite
  • job run validation
  • larger system-table scans
  • resource checks requiring retries

Pattern:

attempt.status = queued
worker picks up attempt
attempt.status = running
validator writes result
scoring service awards points
attempt.status = passed/failed/error

Event scale considerations

For large events:

  • use team-level validation submissions to reduce volume
  • pre-create resources
  • validate via SQL Warehouse/serverless compute
  • cache quest pack content
  • materialize leaderboard periodically or incrementally
  • use polling every 5–10 seconds instead of aggressive websocket requirements
  • store detailed evidence separately from leaderboard summaries

Deployment model

MVP deployment

Single workspace:

  • one Databricks App
  • one Lakebase database
  • one SQL Warehouse
  • one Unity Catalog catalog/schema for Quest state/audit
  • one or more event resource catalogs/schemas
  • scheduled adoption scoring job
  • validation worker job

Multi-workspace federation (ADR_006)

Large GameDay events provision one workspace per attendee. Federation is delivered by the shared-Lakebase model (see adr/ADR_006_SHARED_LAKEBASE_MULTI_WORKSPACE_FEDERATION.md), selected by a single QUEST_ROLE parameter on the same codebase/build:

  • standalone (default) — single workspace, local Lakebase. Unchanged.
  • master — owns the one shared Lakebase, host console, reporting, roster / identity map, and provisions the shared event-writer credential.
  • child — runs gameplay + validation locally but points its DB layer at the master's Lakebase (over Postgres :5432, not the Apps auth proxy) using the shared INSERT-only event-writer credential, stamping writes with QUEST_WORKSPACE_ID. Aggregation is automatic in the shared DB; the child UI shows the event-wide leaderboard and highlights its own team's rank.

Identity (labuser+{n}@awsbricks.com → real person/team) is reconciled centrally via a host-uploaded roster in participant_identity_map; the leaderboard view resolves federated rows through it while preserving the standalone team_id path.

Still future

  • account-level provisioning of the child workspaces themselves
  • central quest pack registry
  • partner-authored quest packs
  • per-child OAuth writer roles (stronger isolation than the shared credential)

Compatibility with existing repo

Preserve:

  • deploy.sh as one-shot installer
  • databricks.yml
  • current frontend build path to app/static
  • current /api/profile, /api/missions, /api/leaderboard
  • current adoption scoring notebook initially

Add:

  • migrations
  • validation_worker.py
  • event APIs
  • quest pack APIs
  • new frontend routes
  • Lakebase operational tables
  • Delta audit sync

Architecture decision summary

Decision Recommendation
Replace current app? No. Add event mode alongside adoption mode.
Content config YAML/JSON quest packs imported into tables.
Operational state Lakebase.
Analytics/audit state Delta/Unity Catalog.
Validation Pluggable engine, sync + async.
MVP deployment Single workspace, local resources.
Frontend Route-based app with player and host experiences.
Existing system-table scoring Keep as adoption mode and validator input.