no support for web-platform-tests format logs

[jmaher]

If there is a push which contains many failures:
https://treeherder.mozilla.org/jobs?repo=try&revision=c2ed18f748046d19578705aeed91f40033800273

in this push, all jobs and failures are part of the test suite web-platform-tests. These have a different relative path in the source tree and some different logic than other tests.

Currently if run:
citools push failures try c2ed18f748046d19578705aeed91f40033800273

there will be hundreds of output lines showing:
WARNING | citools.dump_failures:63 - Invalid test: /upgrade-insecure-requests/gen/worker-module-inherit.http-rp/upgrade/websocket.https.html

and of course many other test names.

We should support web-platform-tests in here, or at the very least reduce the amount of warnings output printed in order to reduce confusion.

[chichi012]

Hi @jmaher , I will like to work on this.

What I understand so far,

We have manifests, each manifest contains a set of tests. It will be good to have a formal understanding of the mozilla test suites. What do manifests represents and how are they relate with tasks?

While exploring this issue locally with the example you provided, I noticed that dump_failures builds a map of tests to their manifests based on the tests listed in the log line with action suite_start.

mozci-tools/citools/dump_failures.py

Lines 35 to 38 in 3d40276

	if data["action"] == "suite_start":
	for manifest, tests in data["tests"].items():
	for t in tests:
	manifests[t] = manifest

The reason why we have warning lines you described above is because, some test that ended with action crash or test_end were not included in the list when the map above was generated. dump_failures() assumes that all test log lines with action crash or test_end will have been included in the suite_start action above.

To fix this problem for web-platform-tests, I also noticed the test names are very similar and of the format prefix/{more specific name} and multiple tests share where prefix is/is/a/path/.

To reduce the noise, we can log just one warning line for each group of test and tests will be grouped by their prefix /{first}/second/{third}/ i.e we will log just one member of a group.

A better fix can be to fix the source of the problem, i.e we make sure all tests are included in the suite_start log line.

[jmaher]

first off thanks for working on this. This issue will be a bit more involved as this is less well defined than the previous issues you have completed- I have full confidence you can figure it out!

You have provided a great set of questions and your understanding to date.

let me explain a bit more about manifests- most of the manifests support .ini format- this is done primarily through manifestparser:
https://searchfox.org/mozilla-central/source/testing/mozbase/docs/manifestparser.rst

We read .ini files and parse some of the key:values from the sections. manifestparser .ini files are usually 1 per directory and include all the tests in that given directory.

We have a different test suite called web-platform-tests or we abbreviate WPT. this is a great test suite where we many sure that the web-platform (html, css, etc.) is common between all browsers- and all browser vendors contribute and run these tests. Since this is shared, we vendor in the tools and tests from github:
https://github.com/web-platform-tests/wpt/

we put these in our firefox tree. These do not have .ini files, instead they read each directory and find test* files to run (I am simplifying this a little bit). WPT harness supports manifests to adjust test expectations (i.e. expected-fail, expected-crash, etc.). These expectations are stored in .ini files in a different directory than the tests. Take a look here:
https://searchfox.org/mozilla-central/search?q=send-file-form-controls.html&path=&case=false&regexp=false

notice the .ini file is inside the testing/web-platform/meta directory, not the testing/web-platform/tests directory.

this .ini file has a single test in it (there are some exceptions) which is much different than the manifestparser format.

For this bug you should make sure that we understand there are manifests and tests that have different paths than tests and manifests from the manifestparser based tests.

for a recent failure try:
citools push failures mozilla-central 91860dc631992d2a607aba71784fefe1c27cacad

I assume there will be more questions- please ask more questions.

[chichi012]

@jmaher Thank you for the explanation. I think I understand things better now. What I've noticed, when I run the command
citools push failures mozilla-central 91860dc631992d2a607aba71784fefe1c27cacad

For web-platform-tests test suites, the manifests listed inside tests are not .ini files.
This seems to be a problem from the "raw.log" file as the "raw.log" does not list the correct manifest names for such test suites (html, css, etc).

How do we update "raw.log" to include the correct manifests names for these type of tests?

PS: my interpretation of the problem might be incorrect though.

Some things that is not clear to me:

• Can we fix dump_failure.py to support manifests/tests for tests like web-platform-tests?
• If we can't do that at this time. Will reducing the warnings in the logs (maybe by skipping such tests) be an acceptable band-aid solution for now and the issue can stay open till we find a better fix.

[jmaher]

thanks for looking into this and you have come back with great questions, specifically I like how your questions are trying to get to a resolution right away.

Here is an example from that referenced mozilla-central revision of wpt failures raw data:
https://firefoxci.taskcluster-artifacts.net/Tfqzou2rRVCZVI3SyNtMTQ/0/public/test_info/wpt_errorsummary.log

{"test": "/html/infrastructure/urls/resolving-urls/query-encoding/windows-1251.html?include=workers", "subtest": "SharedWorker constructor", "group": "/html/infrastructure/urls", "status": "PASS", "expected": "TIMEOUT", "message": null, "stack": null, "known_intermittent": ["PASS"], "action": "test_result", "line": 39220}

The group is not <relative_path>/mochitest.ini, instead it is <relative_wpt_path>.

Here we need to adjust things to edit the manifest dynamically. Typically mozci reads the group field, but in this case we need the test field. specifically we would want: /html/infrastructure/urls/resolving-urls/query-encoding/windows-1251.html. and then transform that to testing/web-platform/meta/<test>.ini which would look like:
testing/web-platform/meta/html/infrastructure/urls/resolving-urls/query-encoding/windows-1251.html.ini

There is a larger issue of how does this different format affect current and future designs of dump_failures and mozci- I think for now it is best to get something that works with comments indicating the assumptions being made.

[chichi012]

@jmaher thank you, its becoming clearer. From the screenshot below, gotten from running the citools ... command
For the test /FileAPI/file/send-file-form-x-user-defined.html, the corresponding .ini file will be
testing/web-platform/meta/FileAPI/file/send-file-form-x-user-defined.html.ini. Is this correct?

P.S I noticed in raw.log that there is no group. We have just mapping of manifests to list of tests. Does this affect anything?

[jmaher]

you are correct, the .ini for that test would be testing/web-platform/meta/FileAPI/file/send-file-form-x-user-defined.html.ini.

As for the raw log not having a group, I am not sure if that will be a problem or not. I don't think so since we are not really using a group for web-platform-tests.

I think you have enough information to go on here. Do let me know if you have more questions.