[Perf] Linux/arm64: 4 Regressions on 8/11/2026 9:14:21 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	azurelinux 3.0
Queue	CobaltAzureLinux
Baseline	643b8962835e549e8f6c88fd8b19f6f79685fba0
Compare	633ab1a41439ad2405ba2eb241295ba1842fcf5a
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Select - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	438.41 ns	504.23 ns	1.15	0.11	False
Select - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	439.32 ns	504.42 ns	1.15	0.20	False
Select - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	439.01 ns	503.81 ns	1.15	0.21	False
Concat_Once - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	792.66 ns	881.50 ns	1.11	0.16	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Details

System.Linq.Tests.Perf_Enumerable.Select(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Select(input: List)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Select(input: IList)

ETL Files

Histogram

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Concat_Once(input: IEnumerable)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: ⚠️ Likely NOT a functional regression. Bisection of Select(input: IEnumerable) found commit 0abc149 which is unrelated to the test area. This strongly suggests a code alignment/layout artifact. Consider marking as noise.

Finding Confidence: 2/5 (analysis accuracy)
Regression Confidence: 1/5 (likelihood of true regression)
⚠️ Assessment: Likely noise - recommend closing or monitoring for recurrence.

🤖 Proposed Next Actions: Likely noise — close & label — 1 done, 2 dry-run (rule R1_all_noise — see Full Analysis for the table).

📊 Full Analysis (click to expand)

Summary: All 4 reported LINQ Perf_Enumerable regressions (+11–15%) on Linux/arm64 are measurement noise, not a genuine runtime regression. Across 293–296 prior runs each test shows a multi-regime distribution (CV 11–14%), and every compare value (~504 ns / ~881 ns) falls within a cluster seen repeatedly in the before-series (z=0.88–0.95, all below adaptive thresholds of 22–28%). No commit attribution is made. Finding confidence 4/5 (plentiful history), regression confidence 1/5 (all deltas within ~1σ).

Issue Overview

• Issue: #78186 - [Perf] Linux/arm64: 4 Regressions on 8/11/2026 9:14:21 PM +00:00
• Date: 2026-08-11 | Queue: CobaltAzureLinux (AzureLinux.3.Cobalt.Arm64.Perf) | OS/Arch: azurelinux 3.0/arm64
• Commits: 643b8962 → 633ab1a4 (13 commits)
• Labels: untriaged, perf-regression, os-linux, branch-refs/heads/main, arch-arm64, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Runtime stack: Desktop + CoreCLR (arm64, tiered)
• Historical Data: Available (293–296 prior runs per test; 1,179 before-runs examined)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
Perf_Enumerable.Select(input: IEnumerable)	438.41 ns	504.23 ns	+15.01%	453.91	54.68	293	0.92	24.09%	Y	Within variance; multi-regime history
Perf_Enumerable.Select(input: List)	439.32 ns	504.42 ns	+14.82%	448.35	59.17	296	0.95	26.39%	Y	Within variance; compare matches prior regime
Perf_Enumerable.Select(input: IList)	439.01 ns	503.81 ns	+14.76%	445.28	62.44	296	0.94	28.04%	Y	Within variance; compare within prior ranges
Perf_Enumerable.Concat_Once(input: IEnumerable)	792.66 ns	881.50 ns	+11.21%	802.27	90.13	294	0.88	22.47%	Y	Within variance; ~881 regime seen extensively

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Plentiful, high-quality history (>290 runs per test, 1,303 total runs examined). Slight reduction from 5 because the multi-regime distribution and occasional infrastructure outlier spikes (~260–320 ns interspersed) inflate stddev and complicate interpretation.

Candidate Stack Relevance

No Likely Cause is assigned (all tests are noise). For completeness, the 13-commit range was scanned; none directly touches the managed LINQ Enumerable.Select/Concat code paths on CoreCLR arm64.

Commit	Touched Paths (summary)	Relevance	Notes
633ab1a4	src/coreclr/jit/ (wasm local address frame offsets)	LessLikely	JIT change scoped to wasm memargs; not exercised by arm64 CoreCLR LINQ.
4f6f3a51	src/coreclr/ (NativeAOT thread suspension GC)	Shared	CoreCLR runtime; no plausible mechanism for LINQ iterator overhead.
0abc1499	src/native/ (minipal thread ID TLS cache)	Shared	Native TLS dedup; broad infra, no targeted LINQ impact.
eef6936f	src/libraries/ (HTTP header logging)	LessLikely	Unrelated BCL area (System.Net.Http).
4b61662b	src/libraries/ (System.IO stream seekability)	LessLikely	Unrelated BCL area (System.IO).

Confirmed Regressions

None. All 4 reported tests classified as Noise by statistical analysis over 293–296 prior runs. No regression is confirmed and no commit attribution is warranted.

Open Questions

• Historical Data: Adequate quantity (>290 runs/test) but high variance — multi-regime distributions (CV 11–14%) with ~3 distinct clusters per test (e.g. ~420/~438/~505 ns for Select). Compare values land inside a previously-seen regime, so the "regression" is a regime shuffle rather than a step change.
• Confidence Limitations: Multi-modal history and interspersed infrastructure outlier spikes (~260–320 ns) inflate σ; adaptive thresholds (22–28%) are correspondingly wide.
• Verification Needed: None required — signal is within ~1σ for all tests. If a real underlying regime shift is suspected, tightening the baseline window to the most recent stable regime would improve future detection sensitivity.
• Bisection Status: Not performed (all tests noise; no culprit to isolate).

Recommended Actions

1. Close as noise / no action - All 4 deltas (+11–15%) are within historical variance (z=0.88–0.95, below adaptive thresholds of 22–28%); no runtime regression is present.
2. Investigate benchmark stability (infra follow-up) - These LINQ Perf_Enumerable benchmarks show multi-modal historical distributions; recurring environment/infrastructure changes on the CobaltAzureLinux arm64 queue may be inflating CV and masking real signals.
3. Tighten detection window (optional) - Consider restricting the historical baseline to the most recent stable regime (~438 ns Select / ~793 ns Concat) to reduce future false-positive autofilings on this test family.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)
Select(input: IEnumerable)	✅ Culprit found	0abc14991784	⚠️ UNCERTAIN	438.4→504.2 ns (+15%)	268.5→509.5 ns (+90%)
Select(input: List)	🔁 Did not reproduce	-	-	439.3→504.4 ns (+15%)	≈ (baseline ≈ head)
Select(input: IList)	✅ Culprit found	0abc14991784	UNCERTAIN	439.0→503.8 ns (+15%)	439.3→505.2 ns (+15%)
Concat_Once(input: IEnumerable)	🔎 Reproduced (culprit not localized)	-	-	792.7→881.5 ns (+11%)	↑ reproduced (culprit not localized)

Total bisection time: 48m 27s
Tests measured: 4/4
Culprit found: 2/4

⚠️ Alignment Warning

Bisection found commits that are unrelated to the regressed tests. This strongly suggests the performance changes are caused by code alignment, binary layout, or other non-functional factors rather than actual regressions. Consider marking as noise.

System.Linq.Tests.Perf_Enumerable.Select(input: IEnumerable)

• Culprit: 0abc14991784
• Relevance: UNCERTAIN
• Explanation: Last good commit 4f6f3a5 (268.45 ns, below baseline) is immediately followed by 0abc149 (509.51 ns, matching the ~504 ns compare/bad value), an adjacent single-commit good→bad transition. However, the just-measured commits 6597edb and 4f6f3a5 both reported ~268 ns, dramatically below both the good baseline (~438) and bad compare (~504) reference values, indicating a probable code-alignment/measurement anomaly; the culprit's relevance to LINQ Select cannot be confirmed from measurement data alone.
• ⚠️ This commit is UNRELATED to the test — likely a code alignment artifact

Bisection details

• Culprit: 0abc14991784
• Relevance: UNCERTAIN
• Baseline: 438.415 ns
• Culprit value: 509.512 ns

Per-commit measurements (oldest → newest) (ns)

#	Commit	Result	Measurement
0	643b8962835e	baseline/good	432.764 ns
2	6597edb0936d	good	433.043 ns
4	4f6f3a51111a	good	268.451 ns
5	0abc14991784	culprit/bad	509.512 ns
8	1fdccbc9c369	bad	504.463 ns
13	633ab1a41439	head/bad	508.738 ns

Helix jobs

Job ID
6b24d825-54b6-445d-a1fc-e7bd29f772f4
39cd0f3d-2498-405d-8cc8-17ee8c494e42

System.Linq.Tests.Perf_Enumerable.Select(input: List)

• Status: inconclusive

Bisection details

• Baseline: 439.324 ns

Per-commit measurements (oldest → newest) (ns)

#	Commit	Result	Measurement
0	643b8962835e	baseline	444.462 ns
5	0abc14991784	-	500.929 ns
8	1fdccbc9c369	-	500.438 ns
13	633ab1a41439	head	314.314 ns

Helix jobs

Job ID
1b817707-07f7-438b-aeae-20c19268f441

System.Linq.Tests.Perf_Enumerable.Select(input: IList)

• Culprit: 0abc14991784
• Relevance: UNCERTAIN
• Explanation: The good→bad transition is between 4f6f3a5 (439.29 ns, baseline) and 0abc149 (505.15 ns, regressed), which are adjacent in the window, isolating 0abc149 as the culprit. Commit contents were not inspected, so relevance to the Select(IList) regression cannot be confirmed.

Bisection details

• Culprit: 0abc14991784
• Relevance: UNCERTAIN
• Baseline: 439.006 ns
• Culprit value: 505.154 ns

Per-commit measurements (oldest → newest) (ns)

#	Commit	Result	Measurement
0	643b8962835e	baseline/good	432.935 ns
2	6597edb0936d	good	432.445 ns
3	5de80d3fe2cb	good	436.092 ns
4	4f6f3a51111a	good	439.286 ns
5	0abc14991784	culprit/bad	505.154 ns
8	1fdccbc9c369	bad	504.622 ns
13	633ab1a41439	head/bad	507.919 ns

Helix jobs

Job ID
aa1b42bf-5534-45e0-b88b-711f0d344423
3b6f0f48-e7f5-4d79-b1a8-3aecb734bf1e
9c917f65-c211-4363-b359-c6f9e0f174e9

System.Linq.Tests.Perf_Enumerable.Concat_Once(input: IEnumerable)

• Status: inconclusive

Bisection details

• Baseline: 792.663 ns

Per-commit measurements (oldest → newest) (ns)

#	Commit	Result	Measurement
0	643b8962835e	baseline/good	791.810 ns
5	0abc14991784	bad	887.647 ns
13	633ab1a41439	head/bad	882.543 ns

Helix jobs

Job ID
94f70b2d-b8b9-41e1-92d2-9c0e1e9372c6

🤖 Proposed Next Actions

Rule: R1_all_noise — Likely noise — close & label.

All reported regressions were classified as noise.

#	Action	Status	Details
1	Add label	✅ Done	noise
2	Remove label	⏭️ Skipped (dry-run)	untriaged
3	Close issue	⏭️ Skipped (dry-run)	reason not_planned

🦆 Pre-bisection Re-analysis (advisory)

⚠️ Disputes the pre-bisection plan — advisory only, not applied

Flags: possible-noise, wasted-bisection-slot

Rationale: Every finding is classified as noise with change confidence 1/5, no bimodal flag, and no likely commit, yet all four tests are still queued for bisection, which burns ~60+ Helix minutes each on signals the analysis itself believes are not real changes. Unless stronger per-finding evidence (z-score, sustained after-values) is supplied, none of these should consume a bisection slot.

Suggested drop from bisection: System.Linq.Tests.Perf_Enumerable.Select(input: IEnumerable), System.Linq.Tests.Perf_Enumerable.Select(input: List), System.Linq.Tests.Perf_Enumerable.Select(input: IList), System.Linq.Tests.Perf_Enumerable.Concat_Once(input: IEnumerable) (advisory — not applied)

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

✅ Agrees with the analysis — advisory only, not applied

Rationale: The analysis is appropriately conservative: change confidence is 1/5 with all findings called noise, and no culprit is claimed at the packet level even though bisection surfaced 0abc149 on two tests, because that hit had UNCERTAIN relevance plus an alignment warning and the other two bisections were inconclusive. The high per-finding (4/5) finding confidence paired with noise=True and low change confidence is the expected self-consistent pattern, not overconfidence.

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

---

Generated by PerfTriageAgent on 2026-08-20 10:05 UTC

[LoopedBard3]

Seems to be real and likely #131991 based on the bisection results. FYI @lateralusX.

Related Regressions:

• [Perf] Linux/arm64: 3 Regressions on 8/11/2026 11:24:58 PM +00:00 perf-autofiling-issues#78182

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-linq
See info in area-owners.md if you want to be subscribed.