slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1079922	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono	Log
1079602	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono	Log	dotnet/runtime#116167
1079344	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono	Log
1078701	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#116907
1078664	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono	Log
1077715	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#116907
1078394	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono	Log	dotnet/runtime#116167
1078252	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono	Log
1077912	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#116383
1078057	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#116167
1077977	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#117018
1077963	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#115335
1077902	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#117021
1077917	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#116923
1077908	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#117016
1077945	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#115766
1077898	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#117020
1077926	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#116978
2737101	dotnet-runtime	ios-arm64 release iOSMono	Log
2737092	dotnet-runtime	ios-arm64 release iOSMono	Log
2737091	dotnet-runtime	ios-arm64 release iOSNativeAOT	Log
2737087	dotnet-runtime	ios-arm64 release iOSMono	Log
2736738	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0	Log
2736573	dotnet-runtime	osx-x64 release	Log
2736549	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2736525	dotnet-runtime	ios-arm64 release iOSNativeAOT	Log
1076566	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2736160	dotnet-performance	Performance bepuphysics windows 20H1 arm64 Tiger main	Log
1076043	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2736011	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2735988	dotnet-runtime	ios-arm64 release iOSNativeAOT	Log
2735970	dotnet-runtime	ios-arm64 release iOSNativeAOT	Log
2735895	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1075072	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1075039	dotnet/runtime	osx-x64 Debug PALTests	Log
1075019	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#116899
1075016	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#116354
2735292	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2734379	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1073416	dotnet/arcade	Linux Build_Debug	Log	dotnet/arcade#15917
1073386	dotnet/arcade	Windows_NT Build_Release	Log	dotnet/arcade#15916
1073308	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
1073284	maestro-auth-test/helix-queue-insights-test	HelixTests	Log	maestro-auth-test/helix-queue-insights-test#1042
1071917	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2733023	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1071502	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#116788
2732512	dotnet-performance	Performance fsharpmicro ubuntu 2204 arm64 Ampere 9.0	Log
2732497	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2732323	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2732152	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1070132	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#62366
2732048	dotnet-performance	Performance scenarios ubuntu 2204 arm64 Ampere 9.0	Log
2732021	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2731756	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2731655	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2731603	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2731580	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2731427	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1069182	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#116699
2731261	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2731200	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere 9.0	Log
2731111	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2730959	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2730850	dotnet-performance	Performance roslyn windows 20H1 arm64 Tiger 9.0	Log
2730702	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2730659	dotnet-performance	Performance powershell windows 20H1 arm64 Tiger 9.0	Log
1068514	dotnet/runtime	osx-x64 Release CoreCLR_Release	Log
1068509	dotnet/runtime	osx-x64 Release CoreCLR_Release	Log
2730430	dotnet-performance	Performance imagesharp windows 20H1 arm64 Tiger 9.0	Log
2730417	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2730337	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2730225	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2730222	dotnet-performance	Performance mlnet windows 22H2 arm64 Ampere 9.0	Log
2730177	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log
2730105	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2729996	dotnet-sdk	TestBuild: macOS (x64)	Log
2729972	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2729959	dotnet-performance	Performance imagesharp windows 20H1 arm64 Tiger 9.0	Log
2729920	dotnet-sdk	TestBuild: macOS (x64)	Log
2729913	dotnet-dotnet	MacCatalyst_Shortstack_arm64	Log
2729755	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2729787	dotnet-sdk	TestBuild: macOS (x64)	Log
2729766	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2729689	dotnet-sdk	TestBuild: macOS (x64)	Log
2729626	dotnet-sdk	Darwin_AoT_Tests	Log	#50874
2729624	dotnet-sdk	Darwin_AoT_Tests	Log	#50872
2729625	dotnet-sdk	Darwin_AoT_Tests	Log	#50873
2729592	dotnet-sdk	AoT: macOS (x64)	Log
2729569	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2729553	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere 9.0	Log
2729494	dotnet-sdk	TestBuild: macOS (x64)	Log
2729401	dotnet-sdk	TestBuild: macOS (x64)	Log
2729396	dotnet-sdk	Darwin	Log	#50859
2729393	dotnet-sdk	Darwin_AoT_Tests	Log	#50858
2729361	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2729237	dotnet-sdk	AoT: macOS (x64)	Log	#50855
2729236	dotnet-sdk	AoT: macOS (x64)	Log	#50854
2729280	dotnet-dotnet-monitor	Test MacOS arm64 Release	Log
2729177	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2729184	dotnet-performance	Performance mlnet windows 22H2 arm64 Ampere 9.0	Log
Displaying 100 of 194 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	39	194

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.