[P2] Restore benchmark gate signal: publish a mocked series and re-derive ceilings from CI data

[MajorLift]

Parent Epic: #6944 Performance Quality Gates → #7287 Automated Threshold Adjustment & Self-Calibration
File: test/e2e/benchmarks/utils/thresholds.ts, .github/scripts/benchmark-stats-commit.sh, development/metamaskbot-build-announce/historical-comparison.ts
Size: L | Hours: ~16-24

State as of 2026-08-12. Both blockers now have open PRs, and the first half of this ticket is unblocked.

was blocking	now
#45451 — publishing rejected at GitHub's file-size limit	PR #45445 caps the series at 100 and removes both continue-on-error masks
#45266 — onboardingNewWallet.total bimodal, so uncalibratable	PR #45443 ports the missing Solana discovery mocks
the metric gating while uncalibratable	PR #45444 demotes it and two sibling steps (#45448)

That splits this ticket cleanly in two.

Publishing a mocked series is no longer blocked on the size ceiling, but it is blocked on #45204, which is still open. mockMode and the consumer-side selection it describes live on that branch, not on main — this line previously claimed the work was implementable today and named a PR number that belongs to an unrelated PR; both were wrong and are corrected here. The mocked-series change stacks on #45204.

Re-deriving the ceilings cannot start until that series has accumulated depth. Weeks of data, not a PR.

Calibration of onboardingImportWallet.total was never blocked on #45266 and still is not — its 862ms between-run sd is a clean, mixture-free measurement.

---

Problem

#45204 stopped the quality gate from failing on the live-network population, which ends the standing red on main (#45046). It deliberately left two things unfixed, and the gate is not a trustworthy signal until both are.

1. Mocked runs have no baseline. Only main/release/* publish to extension_benchmark_stats, and both run live. With populations kept apart, mocked PR runs resolve no baseline, so PR comments show no historical deltas. Reporting nothing is correct — the previous deltas compared a mocked measurement against a live baseline and so described upstream latency rather than the commit — but it is a real loss of signal.

2. The ceilings were never derived from the population they gate. #42291 set the four total values from Sentry 30-day production p75/p95. That is a third population: production is heterogeneous devices and networks with a heavy right tail; CI is one machine class with n≈10–30 reps. A quantile of one is not a quantile of the other. Two defects follow:

• Referent mismatch. Sentry p75 is over user sessions; the CI p75 is over repetitions within a single run; historical-comparison.ts then averages within-run p75s across commits (calculateMean(values.p75)). A mean of p75s is not the p75 of the pooled distribution — quantiles do not commute with averaging.
• Fixed constants over a drifting series end in permanent red. onboardingImportWallet.total breached on 100% of recorded main runs from 2026-06-09 onward, ~2 months before it was treated as a standing failure.

---

Measurement

n=30 PR runs, chrome-webpack, mocked population, 2026-08-05 16:21–20:21 UTC, from the benchmark-chrome-webpack-userJourneyOnboardingImport / -OnboardingNew artifacts with the runner image read off each producing job.

Where the current ceilings sit

onboardingImportWallet.total, empirical quantiles: Q0.50 = 10495, Q0.75 = 10778, Q0.90 = 11465, Q0.95 = 11665, Q0.99 = 11718ms.

ceiling	value	position	fires on
warn p75	9100ms	Q0.07	28/30 = 93% — carries no information
fail p75	11050ms	Q0.87	4/30 = 13% of clean PRs

Median 10495ms, MAD 171ms → σ_robust 254ms, against an actual sd of 862ms. MAD understates the scale 3.4×, so a median/MAD-derived ceiling lands ~11257 and fires on ~13% of clean runs. The distribution is unimodal with a dense core and heavy tails on both sides.

The two flows have different problems

	onboardingNewWallet.total	onboardingImportWallet.total
within-run sd	3279ms	120ms
bimodal	yes	no — absorbed by the container
between-run sd	dominated by the mixture	862ms (8.2% CV), mixture-free
fixed by #45266	the whole problem	nothing

Both flows end on the same Solana token wait, but the import flow has a step after it and the new-wallet flow does not. In the import flow the ~5.6s moves between two adjacent labels and total absorbs it either way. In the new-wallet flow doneButtonToAssetList is terminal, so the bimodality propagates into total.

Consequence: the import flow's 862ms is already a clean measurement of residual noise, and calibration is the only lever there. That number will not move when #45266 lands.

onboardingNewWallet.total is a binomial draw, not a measurement

cluster	n	median	range	condition
PASS	3	2265ms	2257–2300	nSlow ≤ 1
FAIL	11	9233ms	6858–10478	nSlow ≥ 2

Gap 4558ms, the 5460ms fail ceiling sits inside it. Separation by nSlow is perfect across the 13 runs where iteration counts were recoverable. Observed failure rate 11/14 = 79%.

Per iteration, doneButtonToAssetList is ~1.7–2.5s or ~8.4–10.2s with nothing between; p(slow) = 24/65 = 37%, indistinguishable from the import flow's 30/80 = 37.5%. Run-level p75 over 5 samples is the 4th order statistic, so it flips at 2 slow draws.

Outlier trimming makes the split discontinuous rather than merely noisy:

run	iterations (s)	trimmed	p75 total
31035768357	1.87, 1.80, 1.79, 1.91, 1.85	0	2265
31038082343	2.01, 9.30, 1.90, 1.73, 1.69	1	2300
31043191301	2.43, 8.94, 8.84, 1.65, 1.62	0	9233
31043480002	10.17, 1.67, 1.67, 10.08, 1.72	0	10478

One slow draw is trimmed away; two widen the outlier boundary until nothing is trimmed.

doneButtonToHomeScreen measures boundary placement, not duration

It and openAccountMenuToAccountListLoaded are two labels for one span. Whenever one is slow the other collapses to ~1.05s, and the sum is flat (sd ≈ 0.2s):

run	doneButtonToHomeScreen	openAccountMenuToAccountListLoaded	sum
31026037892	3.38, 3.48, 3.34, 9.72, 3.19	7.87, 7.51, 7.88, 1.05, 8.15	11.25, 10.99, 11.22, 10.77, 11.34
31042229025	3.35, 3.42, 8.94, 3.24, 8.77	6.82, 6.87, 1.05, 6.85, 1.04	10.17, 10.29, 9.99, 10.09, 9.81

Across all 30 runs, median within-run sd: doneButtonToHomeScreen 2662ms, openAccountMenuToAccountListLoaded 2674ms, total 120ms (< 400ms in 30/30). Near-identical spread in two components against a stable container is the signature of two variables trading a fixed quantity.

total is the per-run sum of the flow's non-unit step timers (runner.ts:242), so the steps do
partition it per iteration. An earlier revision of this issue claimed otherwise, on the basis that
p75(doneButtonToHomeScreen) + p75(openAccountMenuToAccountListLoaded) exceeds p75(total) in
16 of 30 runs. That is a percentile-addition artifact rather than an overlap: the two steps are
anti-correlated within a run, so each one's p75 falls on a different iteration. Percentiles are not
additive, which the computation's own comment notes.

Per iteration the arithmetic holds. In run 31042229025, done + acct is 10.17, 10.29, 9.99, 10.09
and 9.81s against a total spanning 10.27-10.75s, the remainder being the flow's five other steps.

Ruled out, and constraints on any future method

• Runner image is not a splitting variable — 20260720.247.2 on all 29 runs that resolved a label. It does not vary.
• More repetitions cannot help. Within-run sd is 120ms against between-run 862ms. σ for the gate is the between-run component; reps only shrink the within-run one. Only cross-run or cross-commit aggregation moves it.
• Minimum detectable effect is 20–32% of the median for onboardingImportWallet.total — 2144ms at α=0.05, 2731ms at α=0.01, 3390ms at α=0.001, all at power 0.8. Anything smaller is invisible at one run per commit whatever ceiling is chosen. An MDE against a bimodal null is not meaningful, so this figure does not transfer to onboardingNewWallet.total.

---

Solution

Publish a mocked series

Run the gated matrix mocked on main and publish it as the baseline PRs compare against, keeping the live matrix as a separate, non-gated series. mockMode is already recorded per entry by #45204 and aggregateHistoricalData already selects on it, so this is a publishing change rather than a consumer one. Blocked on #45451.

Derive ceilings from the CI null distribution

The level must come from the population being gated. Sentry keeps two legitimate roles, neither of them the level:

• Metric selection — production volume and p75 say which flows have user impact and deserve a gate at all.
• Defining Δ — "a regression worth blocking moves production p75 by X ms". Converting X into CI units needs a measured transfer function: regress the CI statistic against the production statistic across past releases. If that relationship is unstable — expected for anything network-bound — the instability is itself the finding, and the metric should not be ceiling-gated.

1. Collect the per-run statistic over K recent clean main commits. Use a robust location/scale, but validate the scale estimator against the empirical spread before trusting it.
2. Set ceiling = Q₁₋α of that null distribution against an explicit false-positive budget. Use the empirical quantile with a bootstrap CI, or a peaks-over-threshold GPD fit for the tail — not a normal-theory z-score, since these distributions are right-skewed and the normal approximation understates the tail.
3. Choose α from run volume. At ~25 main runs/day, α=0.01 is a false red every four days. Either lower α or add a confirmation rule — fail only on two consecutive breaching runs, which squares the FP rate at the cost of one commit of latency.
4. Check the gate can do its job before calibrating it: compute the MDE per metric. For some the honest answer is "not gateable at this sample size".
5. For drifting metrics, prefer EWMA/CUSUM changepoint detection over a fixed ceiling. A step-size alarm is scale-free and self-relocating, and answers "did a step land at commit X" — the question a merge gate actually needs.

Fix the metrics that are not measuring what they claim

doneButtonToHomeScreen should be merged with its neighbour, given a deterministic boundary, or dropped from the allowlist — not re-thresholded. Structural half tracked in #45452; mechanism half in #45266.

---

Acceptance Criteria

• A mocked benchmark series is published and PR runs resolve a mocked baseline
• PR-comment deltas are restored and compare mocked-to-mocked
• Every gated metric's ceiling is derived from CI data with its α, window, and scale estimator recorded — and the estimator validated against the empirical spread
• Minimum detectable effect computed per gated metric; metrics below the detection floor are removed from GATED_METRICS rather than gated with thresholds that cannot work
• The mean-of-p75s aggregation in historical-comparison.ts is replaced or explicitly justified
• Drift-prone metrics use changepoint detection instead of a fixed ceiling
• doneButtonToHomeScreen is merged with its neighbour, given a deterministic boundary, or removed from the allowlist
• onboardingNewWallet.total calibrated only after [P1] Onboarding benchmarks still take a ~7s slow path on 37% of iterations after #43961 #45266 lands and its residual noise floor is measured over ≥20 clean runs. The only mixture-free evidence today is one all-fast run (within-run sd 50ms) and three PASS-cluster runs (between-run sd 23ms) — not a distribution, and silent on the tails a ceiling must clear
• quality-gate verified green on main for a full day with all metrics re-gated
• M4 (promoting quality-gate to a required check) can be reconsidered

---

Labels

team-extension-platform, area-CI, area-testSuite

---

Dependencies

Depends on: #45204, #45451
Blocked on (for onboardingNewWallet.total only): #45266
Related: #45046, #45452, #42291, #39587, #43958, #43961

[MajorLift]

Retracting the reading in this comment.

I attributed onboardingNewWallet.total's two clusters to something switching between runs — a runner-image or environment difference. That was wrong, and anyone who acted on it would have gone looking in the wrong place.

The split is per-iteration, not per-run. doneButtonToAssetList lands at either ~1.7–2.5s or ~8.4–10.2s on each iteration independently, with nothing in between; p(slow) = 24/65 = 37%. Run-level p75 over 5 samples is the 4th order statistic, so it flips once two iterations happen to draw slow. Nothing about the environment differs between a run that lands in the fast cluster and one that lands in the slow cluster.

The measurement and the current explanation are in the issue body.

[MajorLift]

Superseded — folded into the issue body.

[MajorLift]

Superseded — folded into the issue body.