Build Analysis misclassifies known Helix test failures after telemetry category removal

[steveisok]

Build Analysis misclassifies known Helix test failures after telemetry category removal

Description

Build Analysis can remain red when every failing Helix test is matched to a known issue. The Helix SDK reports each failed test through CheckAzurePipelinesTestResults using Log.LogError(FailureCategory.Test, ...), but these errors now reach the Azure DevOps timeline as plain MSBuild errors:

Microsoft.DotNet.Helix.Sdk/.../AzurePipelines.MultiQueue.targets(44,5): error : Test <test> has failed.

Build Analysis correctly matches the corresponding test results to known issues, but also treats the unclassified timeline messages as independent build failures. The check therefore remains red.

This appears related to:

• Remove the Arcade msbuild logger #16814, which removed Microsoft.DotNet.ArcadeLogging and the (NETCORE_ENGINEERING_TELEMETRY=<Category>) decoration.
• Restore Arcade msbuild logger as a minimal error/warning-to-timeline translator #17110, which restored a minimal logger that forwards MSBuild errors and warnings to the Azure DevOps timeline, but did not restore failure-category propagation.

The Helix task still supplies FailureCategory.Test; that classification is no longer represented in the timeline message consumed by Build Analysis.

Example

• PR: Remove redundant fAcquireLock from binder SetupBindingPaths runtime#132290
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1552976
• Helix SDK: 11.0.0-beta.26381.1 from Arcade commit 93eebf1a31a5eaafd44326f1a81ca107913e098c

The build had two failed tests:

1. Loader/ContextualReflection/ContextualReflection/ContextualReflection.cmd, matched to Assert failure: !"OBJECTREF being accessed while thread is in preemptive GC mode." runtime#131925.
2. Loader\classloader\StaticVirtualMethods\Regression\GitHub_130545\GitHub_130545.dll, matched to [ci-scan] Test failure: Loader.classloader.StaticVirtualMethods.Regression.GitHub_130545.DelegateOverStaticAbstractThroughGshare [Content truncated due to length] runtime#132030.

Build Analysis displayed both under Known test errors, but also displayed the AzurePipelines.MultiQueue.targets(44,5) messages under Build Failures. Those duplicate build failures kept the check red.

Expected behavior

Once every failed test is matched to a known issue, the corresponding Helix task-level errors should not independently fail Build Analysis.

Actual behavior

The known test failures are matched, but their plain MSBuild timeline errors remain unmatched and cause Build Analysis to fail.

Possible fixes

Either:

• preserve FailureCategory.Test in a form Build Analysis can consume when forwarding MSBuild events to the Azure DevOps timeline, or
• update Build Analysis to recognize and de-duplicate CheckAzurePipelinesTestResults errors against the corresponding failed test results.

[steveisok]

@mmitche is this the right place for this kind of issue?

[mmitche]

@steveisok Yep. @ViktorHofer Likely fallout from the category removal. No good deed goes unpunished.

[akoeplinger]

we didn't notice this because right after the change landed in dotnet/runtime we enabled the Job Monitor which doesn't hit this issue, but now that it is turned off again we're seeing it

[ViktorHofer]

Is the BuildAnalysis integration different with Job Monitor and doesn't rely on these telemetry categories?