fix(benchmarks): demote the three bimodal onboarding metrics out of the gate - #45444
Open
MajorLift wants to merge 1 commit into
Open
fix(benchmarks): demote the three bimodal onboarding metrics out of the gate#45444MajorLift wants to merge 1 commit into
MajorLift wants to merge 1 commit into
Conversation
MajorLift
requested review from
a team,
HowardBraham and
itsyoboieltr
as code owners
August 12, 2026 09:20
Contributor
|
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
Contributor
✨ Files requiring CODEOWNER review ✨👨🔧 @MetaMask/extension-platform (2 files, +93 -3)
🧪 @MetaMask/qa (1 files, +19 -3)
👨🔧 @HowardBraham (1 files, +74 -0)
👨🔧 @itsyoboieltr (1 files, +74 -0)
|
metamask-ci
Bot
added
team-extension-platform
Contributor
Builds ready [2a5b089] [reused from 244ac8b]
⚡ Performance Benchmarks (Total: 🟢 12 pass · 🟡 6 warn · 🔴 4 fail)
Bundle size diffs
|
This was referenced Aug 12, 2026
Open
Both demotion triggers in the file's own graduation procedure fire on `onboardingNewWallet.total`: CV 41.8% against a >35% trigger, and an FP rate of 11/14 = 79% against a >10% trigger. Its `doneButtonToAssetList` step and the import flow's `doneButtonToHomeScreen` are the same ~37% per-iteration slow path, so all three are coin flips rather than measurements. Thresholds are left in place so the metrics keep reporting.
MajorLift
force-pushed
the
jongsun/ci/demote-bimodal-onboarding-metrics
branch
from
August 12, 2026 10:34
2a5b089 to
aad9a3e
Compare
Contributor
Builds ready [aad9a3e] [reused from cc5d691]
⚡ Performance Benchmarks (Total: 🟢 8 pass · 🟡 5 warn · 🔴 4 fail)
Bundle size diffs
|
5 tasks
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog
CHANGELOG entry: null
Description
Three gated metrics are downstream of the same ~37% per-iteration slow path (#45266). Against a bimodal null no threshold value is correct: a ceiling either sits in the gap between the two clusters and picks a side of a coin flip, or sits above the slow cluster and never fires. The metric is not noisy — it is not measuring one thing.
gated-metrics.tsalready documents the exit:Both triggers fire on
onboardingNewWallet.total, over the 13 runs sampled in #45266:This is that procedure running, not an exception to it.
Which metrics, and which stays
onboardingNewWallet.totalonboardingNewWallet.doneButtonToAssetListonboardingImportWallet.doneButtonToHomeScreenonboardingImportWallet.totalThe scope is wider than
onboardingNewWallet.totalalone because the coin flip is a step, and it is gated directly in both flows. Within-run sd is 3279ms foronboardingNewWallet.totalagainst 120ms foronboardingImportWallet.total; the import flow's total is a clean unimodal 8.2% CV and remains #45205's calibration target.Thresholds in
thresholds.tsare untouched, so all four keep reporting into the PR comment and Sentry. This suppresses enforcement, not measurement — #45266 needs the series to continue in order to confirm its fix.Not proposed: relaxing the ceilings to swallow the slow cluster. A ceiling above 10478ms passes every observed run including a real regression up to ~5×, converting an uninformative red into an uninformative green. Removal is the honest state and is visible in a way a widened threshold is not.
Verification
Both arms were run:
The three failures on base are exactly the three new cases:
A fourth test is the control —
onboardingImportWallet.totalstill blocks on a fail breach — and it passes in both arms, so the change is scoped to the three rather than disabling the gate wholesale. The pre-existing test asserting a 🔴 fordoneButtonToHomeScreenin the PR comment also still passes, which is the evidence that reporting survives.Restore condition
Each removal carries an inline reference to #45266. If that fix lands and works, these three go back.
Related issues
onboardingImportWallet.totaldoneButtonToHomeScreenadditionally measures boundary placement rather than duration, so it has two independent reasons not to gateManual testing steps
yarn jest development/metamaskbot-build-announce/compare-benchmarks.test.tsonboardingNewWallet.totallands in the slow cluster reports the breach in the PR comment without failing the quality gate.Pre-merge author checklist
Note
Medium Risk
Weakens CI enforcement for onboarding performance, so real regressions in those paths may no longer block PRs until #45266 is fixed. Scope is limited to three metrics and follows the documented demotion procedure.
Overview
Demotes three bimodal onboarding metrics out of
GATED_METRICSso threshold breaches warn instead of failing the PR quality gate. Thresholds are unchanged, so reporting to PR comments and Sentry continues.Removed from hard enforcement (restore when #45266 lands):
onboardingNewWallet.totalonboardingNewWallet.doneButtonToAssetListonboardingImportWallet.doneButtonToHomeScreenonboardingImportWallet.totalstays gated. Tests cover the three demotions plus that control.Reviewed by Cursor Bugbot for commit aad9a3e. Bugbot is set up for automated code reviews on this repo. Configure here.