Migrate test infrastructure from xUnit v2 to xUnit v3

[MichaelSimons]

Related to #52914.

Switch all ~55 test projects from xUnit v2 (2.9.3) to xUnit v3 (3.2.2) using the arcade SDK's built-in XUnitV3.targets support. MTP enablement is deferred to a follow-up (#52914).

Key changes

• Set TestRunnerName=XUnitV3 and OutputType=Exe in test/Directory.Build.targets
• Switch package references: xunit → xunit.v3, XUnitExtensions → XUnitV3Extensions, Verify.Xunit → Verify.XunitV3, TemplateEngine test packages to XunitV3 variants
• Remove custom Program.cs entry point (xUnit v3 generates its own)
• Add CallerFilePath/CallerLineNumber to custom Fact/Theory attributes (xUnit3003)
• Add chmod +x for POSIX Helix work items (AppHost loses execute bit in zip)
• Thread RuntimeIdentifier through test publish for cross-arch Helix queues
• Fix Ctrl+C test crash: launch child processes in a new process group to isolate from xUnit v3's out-of-process test host

[tmat]

Can this be addressed in Arcade instead?

BTW, the test targets in Arcade SDK were created a long time ago, when the product test runners lacked necessary features. Can we now use dotnet test directly?

[MichaelSimons]

Yes, I would like to do this in a follow-up PR.

[MichaelSimons]

@dotnet/dotnet-analyzers - After upgrading the NetAnalyzers to XUnit V3, I am seeing fairly consistent failures in the NetAnalyzers legs. Could you take a look at the crash dump. The following is a /ci-crash-dump analysis from Copilot. TIA.

NetAnalyzers SIGSEGV Crash on macOS ARM64

Summary

Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll.161 crashes with exit code 139 (SIGSEGV) on the macOS ARM64 Helix leg. The test process segfaults ~60 seconds in with zero tests completed. A core dump and crash report were collected automatically.

What's happening

The crash occurs on Thread 23 inside the xUnit v3 test runner pipeline while executing DoNotUseDSATests.TestCreateObjectOfDSADerivedClassWithCngKeyParameterDiagnosticAsync. The stack shows the test entering the Roslyn CodeFixTest.RunImplAsync() → SolutionState.WithProcessedMarkup() → TestFileMarkupParser.Parse() path, then segfaulting in native code while creating ImmutableArray structures.

The main thread (Thread 0) is blocked in XunitAutoGeneratedEntryPoint.Main() — the new xUnit v3 auto-generated entry point, which replaced the xUnit v2 hosting model.

Helix work item test filter

The dotnet test command used in the Helix work item:

dotnet test Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll \
  --filter 'Microsoft.NetCore.Analyzers.Security.UnitTests.DoNotUseDSATests.|Microsoft.NetCore.Analyzers.Security.UnitTests.DoNotUseInsecureCryptographicAlgorithmsTests.' \
  --blame-hang --blame-hang-timeout 60m \
  --logger trx --logger 'console;verbosity=detailed'

The console log shows "No test matches the given testcase filter" but this message appeared after the process crash — the long-running test warnings confirm the tests were discovered and actively running before the SIGSEGV. The "no match" message is likely a non-issue (class-level filter that didn't match additional tests beyond what was already running).

Contributing factors

1. xUnit v3 process model change — the test now runs through XunitAutoGeneratedEntryPoint.Main (v3) instead of the v2 host. The full xUnit v3 runner pipeline (XunitTestRunner, XunitTestCaseRunner, XunitRunnerHelper, ExceptionAggregator, etc.) is on the stack.
2. Assembly binding failure — Microsoft.Bcl.AsyncInterfaces, Version=9.0.0.8 fails to load at test platform startup. This may contribute to instability or indicate a packaging/dependency gap in the xUnit v3 migration.
3. Two long-running tests at crash time — CA5353TripleDESCryptoServiceProviderInGetDeclarationAsync and TestCreateObjectOfDSADerivedClassWithCngKeyParameterDiagnosticAsync were both running for 41+ seconds before the crash.

Environment

• Queue: osx.15.arm64.open
• Machine: DNCHELIXMAC063
• SDK: 11.0.100-ci
• Test duration before crash: ~1 minute

Artifacts

Artifact	URL
Core dump	https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-52930-merge-1698e2a9c9454761a8/Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll.161/1/coredump.65593?helixlogtype=result
Crash report JSON	https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-52930-merge-1698e2a9c9454761a8/Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll.161/1/coredump.65593.crashreport.json?helixlogtype=result
Console log	https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-52930-merge-1698e2a9c9454761a8/Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll.161/1/console.5b2c1c17.log?helixlogtype=result
Test host log	https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-52930-merge-1698e2a9c9454761a8/Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll.161/1/dotnetTestLog.host.26-03-27_10-12-52_21578_5.log?helixlogtype=result
Test diagnostic log	https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-52930-merge-1698e2a9c9454761a8/Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll.161/1/dotnetTestLog.log?helixlogtype=result
AzDO job	https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1354916&view=logs&j=32d0f345-9bdd-572a-2118-4d6c9a54e829
Helix work item	https://helix.dot.net/api/2019-06-17/jobs/1698e2a9-c945-4761-a8cc-2db99693e03d/workitems/Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll.161/console
PR	#52930

Crashed thread stack (abbreviated)

DoNotUseDSATests.TestCreateObjectOfDSADerivedClassWithCngKeyParameterDiagnosticAsync()
  → AnalyzerTest`1.RunAsync()
    → CodeFixTest`1.RunImplAsync()
      → SolutionState.WithProcessedMarkup()
        → TestFileMarkupParser.Parse()
          → ImmutableArray.CreateBuilder()
            → [native frames → SIGSEGV]

Note: The core dump is a macOS Mach-O format file — it requires lldb on a macOS machine to debug (not dotnet-dump).

[MichaelSimons]

Current status

1. Several dotnet new integration tests are failing waiting for Fix Verify snapshot naming for xunit v3 compatibility templating#10052 to flow in. Currently blocked by [main] Source code updates from dotnet/dotnet #53613.
2. There are intermittent failures in the NetAnalyzer tests as reported in Migrate test infrastructure from xUnit v2 to xUnit v3 #52930 (comment). These are happening too frequently to merge.

[MichaelSimons]

@jeffhandley - I was given your name to help investigate a crash in the NetAnalyzer tests after upgrading to xunit v3 -- see #52930 (comment) for details. TIA.

[Youssef1313]

The only thing I can see that is really special with the specific class that's failing for NetAnalyzers tests is that it setup the tests with ReferenceAssemblies = ReferenceAssemblies.NetFramework.Net462.Default. I don't see how it can be a problem, but I thought it's worth noting.