Skip to content

feat: expand native review journeys and performance budgets - #5968

Open
jedwards27 wants to merge 9 commits into
block:mainfrom
jedwards27:feat/native-review-journeys
Open

feat: expand native review journeys and performance budgets#5968
jedwards27 wants to merge 9 commits into
block:mainfrom
jedwards27:feat/native-review-journeys

Conversation

@jedwards27

Copy link
Copy Markdown

Summary

  • expand the native macOS review harness with search-shortcut and multiline-composer journeys plus causal fail-loud mutations
  • persist named native action-to-observed-postcondition timings and sample app CPU/resident memory during recorded journeys
  • add repeatable performance cohorts, machine-compatible baseline/head comparison, checked-in absolute/relative budgets, and fail-closed receipts

Stack

Stacked on #5958 (feat/native-review-harness) so the journey/performance layer can be reviewed independently.

Validation

  • pre-push hooks passed at 7c856176d1e1f3282a7739ea2097025fdc373ea6: branch skew, Desktop check/typecheck/test, Rust tests, Tauri checks
  • native performance cohort: 3/3 clean exact-SHA tooltip journeys with finalized MP4s and cleanup
    • test-results/native-review/7c856176d1e1/tooltip_fresh_dwell/tooltip_fresh_dwell-20260815T090411-69d3d0/receipt.json
    • test-results/native-review/7c856176d1e1/tooltip_fresh_dwell/tooltip_fresh_dwell-20260815T090449-2619cd/receipt.json
    • test-results/native-review/7c856176d1e1/tooltip_fresh_dwell/tooltip_fresh_dwell-20260815T090522-b61840/receipt.json
  • self-comparison passed all absolute and relative budgets; medians: tooltip 31.96ms, app CPU 27.1%, peak RSS 195.16MB
  • Python harness suite: 20/20

Performance semantics

  • at least 3 clean samples per cohort; median drives verdict while raw samples/min/max remain in the report
  • baseline and candidate must share flow and machine/OS fingerprint
  • each cohort must bind to one source revision; app hashes remain recorded per run because debug bundle bytes are not deterministic
  • dirty runs, failed cleanup, missing metrics, mixed revisions, incompatible hosts, or breached budgets fail closed

Carl added 5 commits August 14, 2026 20:54
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
…urneys

* origin/main:
  fix(desktop): hide the offcanvas-collapsed sidebar so it stops painting over the community rail (block#5947)
  Polish mobile message threads and composer (block#5645)
  chore(release): release Buzz Desktop version 0.5.14 (block#5917)
  ci(release): remove desktop smoke gate (block#5914)

Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
@jedwards27
jedwards27 requested a review from a team as a code owner August 15, 2026 16:09
@jedwards27

Copy link
Copy Markdown
Author

Review as a logical stack on #5958. GitHub cannot target the fork-only base branch, so this PR is mechanically based on main; the first commit is exactly the open MVP PR head and the journey/performance delta begins after 8b3450af0ef952f5a0b5b8e593f29deb523e5d87.

wesbillman and others added 4 commits August 15, 2026 15:21
Fail closed on non-standard fixture ports, use fixed loopback database
coordinates, scrub repository-controlled subprocess environments, and
remove generated review keys when fixture seeding fails.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Apply absolute maximum budgets to the worst candidate sample while retaining
median-based relative regression comparisons. Cover the catastrophic-outlier
case directly.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Default harness commands to a credential-free environment and isolate cleanup HOME without reconstructing the host environment.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
(cherry picked from commit 3635687)

@themiguelamador themiguelamador left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found several correctness and evidence-integrity issues that need to be resolved before this can merge:

  • The launcher imports PyYAML without declaring or pinning it, so a clean checkout cannot run the documented workflow reproducibly.
  • Journey validation accepts boolean/non-finite numeric values and unbounded durations/deltas, and scroll actions do not require or move to their declared target.
  • The AVAssetWriter backpressure path does not advance the frame clock, causing a busy-spin, and capture errors do not stop the loop.
  • Performance comparisons accept malformed provenance, missing machine identity, non-finite metrics, duplicate samples, and the same receipts in both cohorts; those cases can produce a meaningless passing budget result.
  • The semantic snapshot receiver is unauthenticated and accepts unbounded payloads, while the renderer trusts an arbitrary probe destination and malformed fixture identity. That permits local cross-process writes and can exfiltrate the review DOM snapshot if the environment is influenced. Relay parsing also accepts ambiguous URL components.

I prepared a signed fix commit covering all findings: https://github.com/Complear/buzz/commit/69a2a8cb8

Verification on the fix:

  • 36 Python native-review tests pass with ResourceWarning promoted to an error
  • Swift native driver release build passes
  • native-review shell scripts parse cleanly
  • desktop Biome/file-size/text/pubkey checks pass (only existing warnings/info)
  • desktop TypeScript typecheck passes
  • semantic probe URL regression test passes
  • full desktop suite previously passed: 4,954 tests
  • git diff whitespace check passes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants