Creating lists of intermittently failing tests

[adamfarley]

This change lists known unreliable tests in a consistent format.

Resolves #6471

[adamfarley]

Requesting reviews from @ShelleyLambert and @sophia-guo please.

[smlambert]

thanks @adamfarley - a heads up that my active Github tag is @smlambert ... but I did see this review request in any event.

[smlambert]

Thank you for compiling this list @adamfarley. Only openjdk testcase failures are handled by jtreg exclude Problemlist format. For the other test groups that are part of AQAvit, they get excluded via playlist exclusions.

I would have expected the perf and system targets to be listed via playlist files (which do not use a different file for each version as the problem lists do).

I guess it depends on what we intend to do with these lists. If we are adding a feature to exclude unreliable targets on the fly, or post-process failures against a list, we have to consider what format to use, and how to organize them within the directory (as different vendors may eventually have different lists).

[adamfarley]

Thank you for compiling this list @adamfarley. Only openjdk testcase failures are handled by jtreg exclude Problemlist format. For the other test groups that are part of AQAvit, they get excluded via playlist exclusions.

I noticed that. For Perf and Systemtest, I used the jtreg format for consistency, rather than explicit usability (though a consistent format does make any future parsing easier)

I would have expected the perf and system targets to be listed via playlist files (which do not use a different file for each version as the problem lists do).

I considered that, but opted not to because I'm unaware of a mechanism for disabling targets via the playlist files sometimes. As far as I know, for a given vendor, platform, and version, a target is either enabled or disabled. An openjdk test can include an extra exclude file dynamically, so we can choose whether to exclude the test in some runs or not.

I guess it depends on what we intend to do with these lists. If we are adding a feature to exclude unreliable targets on the fly, or post-process failures against a list, we have to consider what format to use, and how to organize them within the directory (as different vendors may eventually have different lists).

My first priority with this task was to provide a unified list of all intermittently failing tests.

• The lists should use a single format to allow parsing and easy-reading.
• The lists should be stored in the repository to allow use at runtime.

Specific uses were a secondary priority. My first thought was that they could be used as exclude files during releases (or during reruns, or reruns during a release), but I didn't want to sidetrack myself looking too much into applications. I figured that if we did pick an application for this data, the best thing I can do right now is to use a consistent format across the board, to make conversion/parsing easier later on.

I'll add a note to the General Retrospective to start discussion into some of our options here.

[smlambert]

openjdk jtreg problemlists are the 'odd man out'... no other test group uses that format

I do not want to store non-openjdk test material in problemlist format as it might lead people to think that is the commonly used format, which it is not.

Agree that we need to have a requirements gathering session and design discussion on this (in terms of formats, and how to structure and where to locate such files). If this is merely a form of documentation, and not a plan for using for debug / temp exclude, then likely all could be in a doc folder.

[adamfarley]

openjdk jtreg problemlists are the 'odd man out'... no other test group uses that format

I do not want to store non-openjdk test material in problemlist format as it might lead people to think that is the commonly used format, which it is not.

Fair. Perhaps I was over-focused on the OpenJDK ProblemList format due to the majority of the intermittent failures being OpenJDK tests.

Agree that we need to have a requirements gathering session and design discussion on this (in terms of formats, and how to structure and where to locate such files). If this is merely a form of documentation, and not a plan for using for debug / temp exclude, then likely all could be in a doc folder.

In my mind, this is a debug/exclude tool first. Mind if I get the ball rolling on Slack?