Known Test Repository Issue Sometimes Misses Test Failures

[nagilson]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1196220

Build leg reported

Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.BasicTest

Pull Request

dotnet/sdk#51591

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The SSL connection",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

So, I believe there is an issue with the known issue flagger, where it is failing to mark certain failures as known, even if they are a known issue. Please let me know if I am mistaken.

Here's the issue:
Looking at this image, this test was not considered a known issue:
https://github.com/dotnet/sdk/pull/51591/checks?check_run_id=54740778719

I believe that, because it's showing as a test failure, and other similar test failures are counted in the above section:

Furthermore, it appeared to be blocking build-analysis from being green.

Here's such an example of 1 of several 'known issues' that flag this issue:
#5015
dotnet/sdk#51574
#6410

As you can see in, #51574 looks for the string "The SSL connection could not be established" as a ErrorMessage which is supposed to use String.Contains. The test, however, that I think was not flagged as a 'known issue', did indeed have that error message string. It looks like some were flagged, while others were not. Here's the output of that 'unknown' test failure:

Expected command to exit with 0 but it did not.\nFile Name: /Users/runner/work/1/s/artifacts/bin/redist/Release/dotnet/dotnet\nArguments: new console --search --debug:custom-hive /Users/runner/work/1/s/artifacts/tmp/Release/dotnet-new.IntegrationTests/SharedHomeDirectory/20251106211951401\nExit Code: 103\nStdOut:\nWarning: use of 'dotnet new --search' is deprecated. Use 'dotnet new search' instead.\nFor more information, run: \n   dotnet new search -h\n\nSearching for the templates...\nStdErr:\nMatches from template source: NuGet.org\nSearch failed: Failed to update search cache. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)\n\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#103\n [[Full log]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220&view=ms.vss-test-web.build-test-results-tab&runId=33106418&resultId=100166&paneView=debug)

While there may be a confounding variable in that there are now several known issues which are tracking substrings of each other, or may overlap, the whole reason there are multiple in the first place is because the first existing known test repository issue did not flag all of the tests which failed with the known issue.

Could this alternatively be a limitation of:

1. A limit of how many tests can be flagged as a known issue for the same known issue
2. A different 'failing configuration' which causes issues to be missed?
3. A misunderstanding of which strings get searched? Note that I believe the error is also in the 'helix logs'.
4. A failure of my own accord to file the known issue? @dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL Connection C]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/6/2025 10:39:11 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL connection]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/6/2025 10:42:09 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1196423	dotnet/machinelearning	Build	Log	dotnet/machinelearning#7543
1196398	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#64262
1196235	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121356
1196186	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121417
1196168	dotnet/runtime	Send tests to Helix (Unix)	Log

Build	Definition	Test	Pull Request
1196358	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51558
1196220	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51591

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	7	7

Report

Build	Definition	Step Name	Console log	Pull Request
1358474	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#64562
1357796	dotnet/maui	Controls ListView	Log
2939384	dotnet-maui	Controls (vlatest) Border,BoxView,Brush,Button	Log
2939246	dotnet-maui	Controls (vlatest) Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1353885	dotnet/maui	Controls CollectionView	Log
2936433	dotnet-maui	Controls (vlatest) TabbedPage,TableView,TimePicker,TitleView,ToolbarItem,Triggers	Log
2936391	dotnet-maui	Publish iOS snapshot diffs	Log
2936387	dotnet-maui	Controls (vlatest) Shell	Log
2936371	dotnet-performance	Run performance job script	Log
1352607	dotnet/maui	Controls Layout	Log	dotnet/maui#34540
2935701	dotnet-maui	Controls (v18.5) Entry	Log
2935693	dotnet-maui	Controls (v18.5) Entry	Log
1352407	dotnet/maui	Controls (vlatest) Label	Log
2935616	dotnet-maui	Pack	Log
2935587	dotnet-maui	Controls (vlatest) SearchBar,Shape,Slider	Log
2934004	dotnet-maui	Controls (vlatest) CarouselView	Log
2935475	dotnet-maui	Controls (vlatest) Layout	Log
2934665	dotnet-maui	Controls ListView	Log
1350457	dotnet/runtime	Build product	Log	dotnet/runtime#125610
1350141	dotnet/maui	Controls (vlatest) ListView	Log
2934547	dotnet-maui	Controls (vlatest) Page,Performance,Picker,ProgressBar	Log
2934538	dotnet-maui	Controls (vlatest) Page,Performance,Picker,ProgressBar	Log
2934529	dotnet-maui	Controls (v18.5) Layout	Log
2934048	dotnet-performance	Run performance job script	Log
2934008	dotnet-maui	Controls (vlatest) Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
1348963	dotnet/maui	Controls (v18.5) ListView	Log
2933887	dotnet-maui	Controls (v18.5) WebView	Log
2933779	dotnet-maui	Run Integration Tests - RunOniOS_BlazorRelease	Log
2933799	dotnet-maui	Controls (v18.5) ListView	Log
2933780	dotnet-maui	Controls (vlatest) SearchBar,Shape,Slider	Log
2933765	dotnet-maui	Controls (v18.5) Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
2932447	dotnet-maui	Controls (vlatest) Page,Performance,Picker,ProgressBar	Log
1348442	dotnet/maui	Controls (v18.5) Label	Log
2933718	dotnet-maui	Controls (v18.5) SoftInput,Stepper,Switch,SwipeView	Log
2933664	dotnet-maui	Controls (v18.5) ScrollView	Log
2932406	dotnet-maui	Controls (v18.5) Layout	Log
1347891	dotnet/runtime	Build product	Log	dotnet/runtime#125610
1347498	dotnet/runtime	Send to Helix	Log	dotnet/runtime#125875
2931046	dotnet-maui	Controls (vlatest) Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
2931685	dotnet-maui	Controls (v18.5) TabbedPage,TableView,TimePicker,TitleView,ToolbarItem,Triggers	Log
1345333	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#65890
1344924	dotnet/maui	Controls ListView	Log
2931502	dotnet-maui	Controls (vlatest) Border,BoxView,Brush,Button	Log
2931515	dotnet-runtime	Build product	Log
2931493	dotnet-maui	Controls (v18.5) ListView	Log
2931483	dotnet-maui	Controls (v18.5) SafeAreaEdges,Shadow	Log
1344201	dotnet/maui	Controls Entry	Log
2931083	dotnet-maui	Controls (vlatest) Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
2930728	dotnet-maui	Controls (v18.5)	Log
2930583	dotnet-performance	Run performance job script	Log
1343541	dotnet/runtime	Build product	Log
2930647	dotnet-maui	Controls (vlatest) Navigation	Log
1343078	dotnet/runtime	Build product	Log	dotnet/runtime#121223
1343022	dotnet/runtime	Build product	Log	dotnet/runtime#125615
1342934	dotnet/runtime	Build product	Log	dotnet/runtime#125441
1342906	dotnet/runtime	Build product	Log	dotnet/runtime#124266
1341609	dotnet/maui	Controls (v18.4) WebView	Log
1341425	dotnet/runtime	Build product	Log	dotnet/runtime#124645
1341246	dotnet/maui	Controls (API 30) Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
2929168	dotnet-performance	Run performance job script	Log
2929045	dotnet-performance	Run performance job script	Log
1339388	dotnet/runtime	Build product	Log	dotnet/runtime#124544
2928225	dotnet-performance	Run performance job script	Log
1331289	dotnet/runtime	Build product	Log	dotnet/runtime#125075
2924652	dotnet-runtime	Restore and Build Product	Log	#58444
1330272	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1330626	dotnet/maui	🛠️ Build Microsoft.Maui.sln	Log	dotnet/maui#34429
1328072	dotnet/maui	Controls Page,Performance,Picker,ProgressBar	Log	dotnet/maui#33450
2922489	dotnet-performance	Run performance job script	Log
1323428	dotnet/maui	Publish the mac_ui_tests_controls test results	Log
1319664	dotnet/maui	Controls (v18.4) Entry	Log	dotnet/maui#33834
2916155	dotnet-runtime	Restore and Build Product	Log	#58444

Build	Definition	Test	Pull Request
1352660	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1352271	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1351548	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1350680	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.4.WorkItemExecution	dotnet/sdk#53350
1349831	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1349307	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1348766	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1346458	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.X509FilesystemTests.CrlDiskCacheRecovers	dotnet/runtime#125824
1344071	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
1342208	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1339813	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1338859	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1337805	dotnet/sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	dotnet/sdk#53377
1332305	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1319685	dotnet/sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	dotnet/sdk#53245
1319267	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1317758	dotnet/sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	dotnet/sdk#53155
1316056	dotnet/runtime	System.Net.Http.Functional.Tests.NtAuthTests.Http2_FakeServer_SessionAuthChallenge_AutomaticallyDowngradesToHttp11	dotnet/runtime#123827
1316114	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	dotnet/runtime#125056
1315961	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	30	92

[nagilson]

I'm still seeing this issue where the string contains match is identical, yet the known issue labeler doesn't think it's a known issue.