Reduce test wall-clock time by increasing minThreads#15502
Merged
nohwnd merged 1 commit intomicrosoft:mainfrom Mar 18, 2026
Merged
Reduce test wall-clock time by increasing minThreads#15502nohwnd merged 1 commit intomicrosoft:mainfrom
nohwnd merged 1 commit intomicrosoft:mainfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce CI test wall-clock time by enabling more parallel execution, unblocking parallelism via test-class splits, reducing long waits in tests, and mitigating ThreadPool starvation during parallel runs.
Changes:
- Reduced wait/sleep/timeout values in multiple unit tests to shorten runtime.
- Enabled MSTest class-level parallelization at the assembly level for select unit test projects, with opt-out via
[DoNotParallelize]on specific test classes. - Split a large acceptance test class into multiple smaller classes/files and increased ThreadPool minimum threads for integration tests.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| test/vstest.console.UnitTests/Internal/ProgressIndicatorTests.cs | Reduces a Thread.Sleep duration used for progress indicator verification. |
| test/datacollector.UnitTests/DataCollectionAttachmentManagerTests.cs | Reduces cancellation timeout to shorten a parallel-access test. |
| test/TranslationLayer.UnitTests/VsTestConsoleRequestSenderTests.cs | Reduces a wait timeout constant used across request-sender tests. |
| test/TranslationLayer.UnitTests/AssemblyInfo.cs | Enables assembly-level MSTest class parallelization. |
| test/Microsoft.TestPlatform.CommunicationUtilities.UnitTests/TestRequestSenderTests.cs | Marks a test class as non-parallelizable under assembly parallelization. |
| test/Microsoft.TestPlatform.CommunicationUtilities.UnitTests/Serialization/TestResultSerializationTests.cs | Marks a test class as non-parallelizable under assembly parallelization. |
| test/Microsoft.TestPlatform.CommunicationUtilities.UnitTests/Serialization/TestCaseSerializationTests.cs | Marks a test class as non-parallelizable under assembly parallelization. |
| test/Microsoft.TestPlatform.CommunicationUtilities.UnitTests/JsonDataSerializerTests.cs | Marks a test class as non-parallelizable under assembly parallelization. |
| test/Microsoft.TestPlatform.CommunicationUtilities.UnitTests/DataCollectionRequestHandlerTests.cs | Marks a test class as non-parallelizable under assembly parallelization. |
| test/Microsoft.TestPlatform.CommunicationUtilities.UnitTests/AssemblyInitializer.cs | Enables assembly-level MSTest class parallelization. |
| test/Microsoft.TestPlatform.CommunicationUtilities.Platform.UnitTests/SocketCommunicationManagerTests.cs | Reduces a polling delay inside a socket communication test loop. |
| test/Microsoft.TestPlatform.Common.UnitTests/ExtensionFramework/ExtensionDecoratorTests.cs | Reduces sleep duration used in a concurrency-related unit test. |
| test/Microsoft.TestPlatform.Client.UnitTests/Execution/TestRunRequestTests.cs | Reduces configured test session timeout and a Thread.Sleep in a mock callback. |
| test/Microsoft.TestPlatform.Client.UnitTests/Discovery/DiscoveryRequestTests.cs | Marks a test class as non-parallelizable under assembly parallelization. |
| test/Microsoft.TestPlatform.Client.UnitTests/AssemblyInfo.cs | Enables assembly-level MSTest class parallelization. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RunsettingsTests.cs | Removes multiple tests from a large class (split into new files) to improve parallelism. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RunsettingsAdapterAndEnvTests.cs | New split-out acceptance tests for adapter path and environment variable runsettings behavior. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/LegacySettingsTests.cs | New split-out acceptance tests for legacy settings scenarios. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionTests.cs | Updates imports and adds a condition attribute impacting which tests execute. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionPlatformTests.cs | New split-out platform-related execution acceptance tests. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionFrameworkTests.cs | New split-out framework-related execution acceptance tests. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionDiagnosticsTests.cs | New split-out diagnostics-related execution acceptance tests. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Build.cs | Increases ThreadPool minimum threads during integration test assembly initialization. |
| .github/skills/optimize-test-project/SKILL.md | Adds guidance doc for systematically optimizing MSTest wall-clock time. |
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
16
to
18
| [TestClass] | ||
| [SkipIOutOfProcessTestOnNetFrameworkCondition] | ||
| public class ExecutionTests : AcceptanceTestBase |
Comment on lines
+49
to
+51
| // **\*test *.dll | ||
| // ! * *\*TestAdapter.dll | ||
| // ! * *\obj\** |
| SetTestEnvironment(_testEnvironment, runnerInfo); | ||
|
|
||
| var testAssemblyPath = _testEnvironment.GetTestAsset("XUTestProject.dll"); | ||
| var allDllsMatchingTestPattern = Directory.GetFiles(Path.GetDirectoryName(testAssemblyPath)!, "*test*.dll"); |
Comment on lines
+97
to
+106
| private static string GetRunsettingsFilePath(Dictionary<string, string>? runConfigurationDictionary, TempDirectory tempDirectory) | ||
| { | ||
| var runsettingsPath = Path.Combine(tempDirectory.Path, "test_" + Guid.NewGuid() + ".runsettings"); | ||
| if (runConfigurationDictionary != null) | ||
| { | ||
| CreateRunSettingsFile(runsettingsPath, runConfigurationDictionary); | ||
| } | ||
|
|
||
| return runsettingsPath; | ||
| } |
Comment on lines
+202
to
+211
| private static string GetRunsettingsFilePath(Dictionary<string, string>? runConfigurationDictionary, TempDirectory tempDirectory) | ||
| { | ||
| var runsettingsPath = Path.Combine(tempDirectory.Path, "test_" + Guid.NewGuid() + ".runsettings"); | ||
| if (runConfigurationDictionary != null) | ||
| { | ||
| CreateRunSettingsFile(runsettingsPath, runConfigurationDictionary); | ||
| } | ||
|
|
||
| return runsettingsPath; | ||
| } |
Comment on lines
296
to
300
| while (dataReceived < 2048 * 5) | ||
| { | ||
| dataReceived += server.ReceiveRawMessageAsync(CancellationToken.None).Result!.Length; | ||
| Task.Delay(1000).Wait(); | ||
| Task.Delay(100).Wait(); | ||
| } |
| private readonly ITranslationLayerRequestSender _requestSender; | ||
| private readonly Mock<ICommunicationManager> _mockCommunicationManager; | ||
| private readonly int _waitTimeout = 2000; | ||
| private readonly int _waitTimeout = 500; |
|
|
||
| _consoleHelper.Setup(c => c.CursorLeft).Returns(30); | ||
| System.Threading.Thread.Sleep(1500); | ||
| System.Threading.Thread.Sleep(1200); |
test/Microsoft.TestPlatform.Common.UnitTests/ExtensionFramework/ExtensionDecoratorTests.cs
Outdated
Show resolved
Hide resolved
…el integration tests Each integration test class spawns vstest.console and testhost processes. The async process management, I/O redirection callbacks, and TranslationLayer socket communication all require ThreadPool threads. With the default MinThreads (equal to processor count), parallel test execution causes ThreadPool starvation — threads queue up waiting for the pool to ramp, adding seconds per test. SetMinThreads(ProcessorCount * 4) pre-allocates enough threads to avoid the ramp-up delay. Measured ~10-14% wall-clock improvement on both acceptance and library integration tests (back-to-back runs on idle machine): Acceptance: 323s -> 290s (-33s) Library: 214s -> 183s (-31s)
drognanar
force-pushed
the
slow_tests_analysis
branch
from
76c756c to
95ceace
Compare
Member
Author
|
In the latest I'm only implementing the threadpool changes. Makes it easier to investigate this fix independently of others (if they're still needed). In testing the threadpool minsize has had a 15% performance impact based on latest main branch. |
Member
|
Run in CI shows basically no difference maybe it was just unlucky run: after change 16:33, before change 16:34- 17:02 minutes. Not enough data, merging will not hurt. :) Potentially reporting the result of setting the threads (by writing to console) would be useful, so we know we are not expecting speedups when the maxThreads is capped to lower value and does not allow the minThreads to be set. |
Member
|
Stuck in macos queue again. |
nohwnd
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR reduces wall-clock time for several test projects by addressing three root causes of slow CI test runs: oversized test classes blocking parallelism, ThreadPool starvation under parallel execution, and unnecessarily long Thread.Sleep/Task.Delay values.
Changes