Experimental Rerun Optimization docs#6565
Conversation
cypress-documentation
 |
||||||||||||||||||||||||||||||||||||||||
| Project |
cypress-documentation
|
| Branch Review |
marktnoonan/rerun-optimization
|
| Run status |
|
| Run duration | 07m 34s |
| Commit |
|
| Committer | marktnoonan |
| View all properties for this run ↗︎ | |
| Test results | |
|---|---|
 |
0
|
 |
0
|
 |
0
|
 |
0
|
 |
326
|
| View all changes introduced in this branch ↗︎ | |
UI Coverage
43.3%
|
|
|---|---|
|
|
406
|
|
|
3
|
Accessibility
96.89%
|
|
|---|---|
|
|
1 critical
3 serious
4 moderate
1 minor
|
|
|
38
|
| <Tabs groupId="rerun-group-id"> | ||
|
|
||
| <TabItem value="Jenkins"> | ||
|
|
||
| ```groovy | ||
| environment { | ||
| CYPRESS_RERUN_GROUP_ID = "${env.BUILD_TAG}" | ||
| } | ||
| ``` | ||
|
|
||
| </TabItem> | ||
|
|
||
| <TabItem value="GitHub Actions" active> | ||
|
|
||
| ```yaml | ||
| env: | ||
| CYPRESS_RERUN_GROUP_ID: ${{ github.run_id }} | ||
| ``` | ||
|
|
||
| </TabItem> | ||
|
|
||
| <TabItem value="Azure"> | ||
|
|
||
| ```yaml | ||
| steps: | ||
| - script: npx cypress run --record | ||
| env: | ||
| CYPRESS_RERUN_GROUP_ID: $(System.PlanId) | ||
| ``` | ||
|
|
||
| </TabItem> | ||
|
|
||
| <TabItem value="CircleCI"> | ||
|
|
||
| ```yaml | ||
| environment: | ||
| CYPRESS_RERUN_GROUP_ID: ${CIRCLE_PIPELINE_ID} | ||
| ``` | ||
|
|
||
| </TabItem> | ||
|
|
||
| <TabItem value="Bitbucket"> | ||
|
|
||
| ```yaml | ||
| pipelines: | ||
| default: | ||
| - step: | ||
| script: | ||
| - export CYPRESS_RERUN_GROUP_ID=$BITBUCKET_BUILD_NUMBER | ||
| ``` | ||
|
|
||
| </TabItem> | ||
|
|
||
| </Tabs> |
There was a problem hiding this comment.
I worry users would take these examples literally and paste them into their configs, even though they're not actually valid. I don't know why we'd include these for supported CI providers, since the auto-detection we do is better than these examples. For Jenkins, it's a bit more involved. My inclination is that we shouldn't document CYPRESS_RERUN_GROUP_ID for now. I suggest we do more research into how users can best construct these for providers like Jenkins, GitLab, etc. and then provide recipes.
There was a problem hiding this comment.
Hmm. I think we'll run into people needing to use it in the very first cohort, so I'll like to explain how it works at least, but we can skip these detailed examples and just add them over time when needed.
| - **CLI messaging is limited.** The CLI prints a notice for the specs being | ||
| re-run, but the skipped specs are not listed in the output. | ||
| - **Analytics may be incomplete.** Analytics features such as App Quality won't be fully accurate for rerun-optimized runs, because those runs contain only a subset of specs. | ||
| - **[Cypress Accessibility](/accessibility/get-started/introduction) and [UI Coverage](/ui-coverage/get-started/introduction) reports may be incomplete.** Rerun-optimized runs reflect only the specs that were re-executed, not the full suite from the anchor run. |
There was a problem hiding this comment.
May be worth noting that reruns will only filter out passing specs up to a depth of 5 runs. So a 6th rerun will re-execute the passing specs from the 1st run. That's new behavior that will land with https://github.com/cypress-io/cypress-services/pull/13685. Currently the depth is only 1 run, so a 2nd rerun re-executes all the passing specs from the 1st run.
I updated the Rerun Optimization Guide noting this limitation and added a table to demonstrate it.
Resolve conflicts in cloud FAQ and run-cancellation docs by keeping both Rerun Optimization and Spec Prioritization FAQ sections. Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
||
| ::: | ||
|
|
||
| ## Setting the rerun group ID manually |
There was a problem hiding this comment.
We should note this requires Cypress 15.15.0+
|
|
||
| ::: | ||
|
|
||
| ## Opting out in CI |
There was a problem hiding this comment.
We should note this requires Cypress 15.15.0+
3 participants
This describes the current implementation and experimental status of Rerun Optimizations
Note
Low Risk
Documentation-only changes with no application or infrastructure code; low risk aside from ensuring links and nav discoverability for the new page.
Overview
Adds documentation for Cypress Cloud’s experimental Re-run optimization Smart Orchestration feature: CI reruns can execute only specs that failed in the anchor run instead of the full suite.
A new guide (
rerun-optimization.mdx) covers behavior, Business/Enterprise availability, org/project settings, automatic vs manualCYPRESS_RERUN_GROUP_IDsetup (with CI examples),CYPRESS_RERUN_ALL_TESTSopt-out, anchor-run semantics, and known limitations (reporter counts, analytics, accessibility/UI coverage). The Smart Orchestration overview is updated from “four” features to include Re-run optimization, and Cloud FAQ gains a dedicated Q&A section (vs test retries, recording requirements, Re-run tag, etc.).Docusaurus enables Groovy syntax highlighting for the Jenkins snippet in the new page.
Reviewed by Cursor Bugbot for commit 53b80bb. Bugbot is set up for automated code reviews on this repo. Configure here.