[P1] Demote the three bimodal onboarding metrics out of the enforced set until #45266 lands

[MajorLift]

Parent Epic: #6944 Performance Quality Gates → #7184 Enforcement → #7211 Workflow, Overrides, Alerts
File: test/e2e/benchmarks/utils/gated-metrics.ts
Size: S | Hours: ~2-3

---

PR open: #45444. Filed after the fact to give the change a tracked home; the PR carries the full measurement.

Problem

gated-metrics.ts documents its own exit condition:

Demote (gated → warn) when CV > 35% OR FP rate > 10% for 2+ consecutive weeks. Cadence: monthly review. Output: PR updating the array below.

Both triggers fire on onboardingNewWallet.total, over the 13 runs sampled in #45266:

	value	trigger	fires
CV	41.8% (mean 7672ms, sd 3204ms)	> 35%	yes
FP rate	79% (11/14, no regression involved)	> 10%	yes

So this is not an override of the process — it is the process running, roughly two months late. The monthly review the procedure calls for has not been happening, which is the second thing this ticket records.

Three gated metrics are downstream of the same ~37% per-iteration coin flip documented in #45266. Against a bimodal null, no threshold value is correct — a ceiling either sits in the gap between the two clusters and picks a side of the coin, or sits above the slow cluster and never fires at all. The metric is not noisy; it is not measuring one thing.

onboardingNewWallet.total is the clearest case. Its p75 splits into a 2257–2300ms cluster and a 6858–10478ms cluster, and the 5460ms fail ceiling sits inside the 4558ms gap. Observed failure rate 11/14 = 79%, with no regression involved (#45205, #45266).

They stay enforced today because #45266 is the fix and #45205 is the recalibration, and both are open. Neither is close, and in the meantime the gate is producing a red that carries no information about the commit.

Which metrics, and which stay

gated metric	line	bimodal?	disposition
onboardingNewWallet.total	47	yes — spill is terminal, so it propagates	remove
onboardingNewWallet.doneButtonToAssetList	42	yes — this is the coin flip, 24/65 slow	remove
onboardingImportWallet.doneButtonToHomeScreen	41	yes — same coin, 30/80 slow	remove
onboardingImportWallet.total	46	no — a step follows the slow one, which absorbs the spill	keep

The scope is wider than onboardingNewWallet.total alone because the coin flip is a step, and it is gated directly in both flows. Within-run sd is 3279ms for onboardingNewWallet.total against 120ms for onboardingImportWallet.total; the import flow's total is a clean unimodal measurement and its 862ms between-run sd is a real calibration target, so it keeps its gate and stays with #45205.

onboardingImportWallet.doneButtonToHomeScreen additionally measures boundary placement rather than duration (planning#7531) — it has two independent reasons not to gate.

---

Solution

Remove the three entries from GATED_METRIC_VALUES in test/e2e/benchmarks/utils/gated-metrics.ts. Leave the threshold values in thresholds.ts untouched so the metrics keep reporting into the PR comment and Sentry — this suppresses enforcement, not measurement, and #45266 needs the series to continue in order to confirm its fix.

Add a comment at each removal naming #45266 as the condition for restoring it, so the removals are re-gated when the slow path is fixed rather than quietly becoming permanent.

Explicitly not proposed: relaxing the ceilings to swallow the slow cluster. A ceiling above 10478ms passes every observed run including any real regression up to ~5×, which converts an uninformative red into an uninformative green. Removal is the honest state, and it is visible in a way a widened threshold is not.

---

Acceptance Criteria

• The three metrics above are absent from GATED_METRIC_VALUES; onboardingImportWallet.total remains present
• All four still appear in the PR comment and in the Sentry series
• A run whose onboardingNewWallet.total lands in the slow cluster no longer fails the quality gate
• Each removal carries an inline reference to [P1] Onboarding benchmarks still take a ~7s slow path on 37% of iterations after #43961 #45266 as its restore condition
• [P1] Onboarding benchmarks still take a ~7s slow path on 37% of iterations after #43961 #45266 gains an acceptance criterion covering re-gating

---

Labels

team-extension-platform, area-testSuite, area-CI

---

Dependencies

Blocked by: nothing — this is the interim state while the two below are open
Related: #45266 (the slow path; restoring these gates is its exit condition), #45205 (recalibration, which covers onboardingImportWallet.total), planning#7531 (step boundaries)

[metamask-ci]

Triage Agent -- Classification

Severity: Sev2-normal (confidence: 0.90) ✅

The bimodal onboarding metrics are causing false positive gate failures (79% false positive rate) which blocks CI/CD pipelines unnecessarily. This doesn't affect end users directly but significantly impacts developer productivity and CI reliability. No actual regression exists - the metrics themselves are flawed due to a coin-flip slow path. A workaround exists (the fix is to demote the metrics), and the underlying issue is being addressed in #45266.

Size: size-XS (confidence: 0.95) ✅

The fix is explicitly scoped: remove three metric entries from the GATED_METRIC_VALUES array in gated-metrics.ts, leave thresholds.ts untouched, and add inline comments referencing #45266. This is a config-level change in a single file, likely a one-liner per metric (3 removals + 3 comments).

Team: team-extension-platform ✅

Matched via area fallback: area-performance (no CODEOWNERS match)

---

Root Cause Analysis

Affected component: E2E performance benchmarks gating system (gated-metrics.ts)

Three onboarding performance metrics (onboardingNewWallet.total, onboardingNewWallet.doneButtonToAssetList, onboardingImportWallet.doneButtonToHomeScreen) exhibit bimodal distributions because the underlying onboarding flow has a ~37% chance per iteration of hitting a slow code path. This creates two distinct timing clusters (fast and slow) that span the gate threshold, causing the statistical gate to flag false positives at a 79% rate. The gate treats these as regressions when they are actually expected variance from the bimodal distribution. The root cause is in the slow path itself (to be fixed by #45266), but the immediate issue is that these metrics should not be in the enforced gate set until the slow path is fixed.

Evidence:

• Issue explicitly states 79% false positive rate on onboardingNewWallet.total
• ~37% per-iteration coin flip in the slow path identified as root cause of bimodality
• Three specific metrics named: onboardingNewWallet.total, onboardingNewWallet.doneButtonToAssetList, onboardingImportWallet.doneButtonToHomeScreen
• Issue references [P1] Onboarding benchmarks still take a ~7s slow path on 37% of iterations after #43961 #45266 as the fix for the underlying slow path
• Metrics split into two distinct clusters with a gap between them

Suggested Fix

Confidence: medium: likely but needs investigation - exact file path for gated-metrics.ts not in KB context, but the approach is clearly described in the issue itself

Remove the three bimodal onboarding metrics from the GATED_METRIC_VALUES array in gated-metrics.ts so they are demoted from enforced to warning-only status. Add inline comments on each removal referencing #45266 as the condition for restoring them. Leave thresholds.ts unchanged so the metrics continue to be reported in PR comments and Sentry.

Target files: gated-metrics.ts
Target functions: GATED_METRIC_VALUES

Approach:

1. Open gated-metrics.ts and locate the GATED_METRIC_VALUES array.
2. Remove or comment out the entries for: onboardingNewWallet.total, onboardingNewWallet.doneButtonToAssetList, and onboardingImportWallet.doneButtonToHomeScreen.
3. Add an inline comment for each removal: '// Demoted: bimodal distribution causes false positives. Restore when [P1] Onboarding benchmarks still take a ~7s slow path on 37% of iterations after #43961 #45266 lands.'
4. Verify thresholds.ts still contains the threshold values for these metrics (do not modify thresholds.ts).
5. Confirm the metrics still appear in PR comments and Sentry reporting (they should, since only the gating enforcement is removed).

---

Confidence threshold: 0.85. Labels marked ✅ were auto-applied. Labels marked ⚠️ are advisory only (below threshold).
To re-triage, remove the ta-bot-triage label and re-apply ta-needs-triage.