Run all external #5785
Run all external #5785
Conversation
judovana
force-pushed
the
runAllExternal
branch
4 times, most recently
from
1b0b47e to
22dd1d4
Compare
Tuned default excludes to excllude run wrapper and criu Only resolve RESULTS_DIR if it already exists Set absolute path to results dir after its creation Enable skip of finished tests Including variables and make call to main log made non found nonfatal
judovana
force-pushed
the
runAllExternal
branch
from
8ea2981 to
517ff8f
Compare
|
@adamfarley @sxa hi! Afaic this PR is not for merge, but I had finally composed the matrix of pass/fail for external tests: https://github.com/judovana/aqa-tests/tree/runAllExternal/external#passfail-matrix-112025 I think I wills tart to fix it one by one... It will be very slow run :) Any other ideas what next? Maybe we should merge only the table? IDK.... |
Good work @judovana! I suggest merging the table, excluding the failing external tests (so it runs green, or at least yellow), and fixing the individual tests in separate PRs to enable maximum coverage in the short/medium term. What do you think? |
|
The table is generated by java class from branch, which crawls data in results pushed in that branch and links those data.. the data structure is dictated by the script in that branch. So pushing the table as it is is not exactly straightforward :) I'm not fan of excluding tests without proper investigations, so I would rather keep state as it is, but to fix it all, is a lot of work. Even to gather results is a lot of work. On contrary I belive that once I fix soem first ten, it will become faster. So maybe really, contribute the table as it is, and then big fixing/excluding can start, and somwhen in future, the table will be regenerted. I do not have strong opinion on this, except my determination to make it much more green:) |
|
We will not store test results in this repository. So there will be no plan to merge this PR here. For such a change, I would have expected a design discussion or at least an issue outlining the direction and plan, to have saved you the time on this draft. If we are to utilize the work you have done, we will have to move the reporting aspect under a new/separate repository. Before that, an overall outline of the design and plan would be good. I appreciate the effort you have gone to, but would like to have discussed direction first (as per the initial comment on the topic, #5575 (comment)) |
I agree. And I was clear on it in previous comment, wasn't I? Please? As for publishing the table itself is different topic. It have sense, because from time to time I hear people running it and it is having failures, so asking what is the expected pass/fail matrix. But I do not insist, really. Afaict it is perfectly ok if it is in fork, or in other branch. Also we already had this discussion? #5575 I would like to make the table as green as possible, which is subject of another discussion, because it seems there is other work on external tests in progress, and I do not want to muddle below anybody's hands. I have a topic to discuss it with adoptium for half a year, and those results were laying on my machine for two months, and to fix the individual test, I needed the matrix, and htmls/github is good place. And if it is useful, then why not to share it. I know you are bussy with other stuff, and so am I, so it is rolling extremely slowly. Still I have great interest in external tests, so I found few cycles to at least compose the matrix for wider audience. |
My message (re: no test results) was for Adam's awareness |
This is script designed to verify/test/fix all external tests on local-host. It is designed to help with: #5575 (comment)
It also includes results from the January run
Whether to integrate it at the end is disputable.