[Feature]: Ability to merge Playwright reports from the main run and --last-failed flag

[OSnuszka]

🚀 Feature Request

In my CI/CD structure, I first run a certain scope of tests, and then I have the option to rerun failed tests using the --last-failed flag.

I would like to have the ability to merge the report from the full scope with the one generated by the --last-failed run.

Ideally, the tests from the --last-failed report would be treated similarly to retries in the merged report. This would mean having a new tab or section in the summary report, showing the failed tests as if they had been retried in a regular run.

Example

No response

Motivation

Currently, when using the --last-failed flag, Playwright generates a separate report for the rerun tests, which makes it harder to get a complete picture of the test outcomes. Combining the main report with the --last-failed results would provide a unified view, similar to how retries are displayed, improving test analysis and making it easier to track the progress of failed tests.

This feature would be especially useful in CI/CD pipelines, where developers often rely on reports to quickly assess the overall health of their tests and identify issues. Merging these reports would streamline the process, reduce confusion, and save time.

[dgozman]

@OSnuszka I'd recommend the retries feature instead of --last-failed. The latter is aimed for the local development workflow. You can also look into merging reports, but it won't be the "same as retries" treatment.

[OSnuszka]

Thanks for the recommendation! I’m already merging reports (my automation setup splits them into shards/blobs), and I use --last-failed after a full test run to rerun tests at a later time, which helps mitigate issues related to environment stability, etc. I also use retries, but --last-failed is crucial in my automation structure and really supports continuous test runs.

On that note, it would be great if you could also consider the feature request I submitted regarding merging the --last-failed report with the main report. This would allow a clearer view of test outcomes, similar to how retries are handled in reports, and would really streamline things for CI/CD pipelines. It would provide a unified view, making it easier to assess test health and track the progress of failed tests.

[dgozman]

@OSnuszka Thank you for explaining your workflow, it sounds very interesting! What happens if you merge --last-failed report with a regular one today? I'd think everything should work, but last-failed tests will not be treated as retries, but rather as separate tests. Is there anything else not working in this scenario? I'd like to better understand the shortcomings to frame this feature request.

[OSnuszka]

@dgozman I've just tested it, and it looks like a test that failed in the first run but passed with --last-failed creates two separate records in the report after merging the blobs. I see one record for the failed test and another for the passed one in the report. Is there something I can do to work around this, or would it require a feature request to address?

[dgozman]

@OSnuszka Thank you for confirming. That's the expected behavior, there is no workaround today.

[RemigiuszDudek]

@OSnuszka as a kind of a work-around you can use allure-reporter - with this reporter, when running tests with --last-failed flag, the newest tests replace previous runs. So the information about the failures is lost (if you want to retain it, you can always produce both reports), but there are no duplicated tests as in playwright report.

Though I agree that it would be nice to have a flag for merge-reports command specifying what should be done when "duplicated" test is encountered during merging process:

• overwrite - newer test replaces the older execution
• as-retry - keep newer execution as if they were simple retries
• separate - keep two runs as separate test execution (current behaviour)

Here is how the comparison of both reports looks like:

[OSnuszka]

@RemigiuszDudek Thank you very much for the suggestion and the detailed expansion of ideas for report merging parameters – it’s a really practical approach! I’ve used Allure before with other frameworks, and it certainly offers a lot of options, but for Playwright, I prefer to stick with the default reporter. Currently, we're at a stage in our projects where the Playwright reporter fits best, especially considering the complexity of our CI/CD setup and the code itself. From my experience, Allure can sometimes introduce complications, especially when updating the framework – I’ve had mixed results with it in Cypress.

Thanks again for the idea!

[sattukishore]

I too have this use case where I want to run all failed tests at the end on my CI machine. last run reports(failed tests) are replacing main run reports even if I want to merge them. @OSnuszka how did you merge main run and failed run reports?

[Chr3is]

@OSnuszka We have the same requirement:

1. Run all tests in a gitlab pipeline and upload the reports & metadata (./test-results/.last-run.json) to artifactory
2. If some tests failed and job is retriggered:
   2.1 download metadata & previous blob report for this job
   2.2 if we have metadata for this job, we had a previous run -> run command with --last-failed
   2.3 merge previous blob with the retry blob-report
   2.4 upload metadata & merged report to artifactory (which replaces the previous report)

Expected:
The merged report does not duplicate the tests, which where invoked with --last-failed. It should be possible to

1. overwrite
2. extend (as retries)
3. keep old behaviour