EDU-18161: make documentation page sampling deterministic#1249
Open
PedroAntunesCosta wants to merge 17 commits intomainfrom
Open
EDU-18161: make documentation page sampling deterministic#1249PedroAntunesCosta wants to merge 17 commits intomainfrom
PedroAntunesCosta wants to merge 17 commits intomainfrom
Conversation
✅ Deploy Preview for elated-hoover-5c29bf ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Performance ResultsPage /docs/api-reference/buyer-organizations#post-/api/b2b/import/buyer-orgs/validate/-importId-
Complete results here Page /docs/api-reference/search-api#get-/api/catalog_system/pub/facets/search/-term-
Complete results here Page /docs/troubleshooting/the-price-in-the-shelf-or-in-the-search-results-is-different-from-the-product-page
Complete results here Page /updates/release-notes/sum-of-installments-without-interest-on-search-results-page
Complete results here Page /updates/release-notes/2024-12-19-new-feature-available-for-anti-fraud-providers-gift-card-transaction-analysis
Complete results here Overall Performance
|
End-to-end testsContent regressions2 tests failed: api-guides2 failing tests:
Preview infrastructureNo infrastructure failures. OtherNone. For more information, open the cypress action summary page. |
Each failed test now emits a JSON line to cypress-failures.jsonl with spec, title, attempt, type, and message fields instead of a plain-text line in cypress.log. All callers updated accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
summary-report.js now reads cypress-failures.jsonl, groups records by spec and title, and reports the number of unique failing tests regardless of how many times each was retried. A (retried Nx) suffix is shown when a test was attempted more than once. Workflow files updated to preserve, merge, and clean up cypress-failures.jsonl instead of the old cypress.log. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…spec Replace the expect() status assertion with a conditional that sets a failureType flag (http / load_timeout / dom) before the potentially failing command. afterEach reads the flag so writeLog emits the correct type for each failure rather than always emitting 'dom'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
summary-report.js now renders three sections: - Content regressions (type=dom) — drives the headline count - Preview infrastructure (type=http/load_timeout) — visible but not counted in the headline, each entry shows the status code or timeout - Other — defensive bucket for unrecognised types The headline "N tests failed" now reflects only actionable content failures. Infrastructure noise is surfaced separately so reviewers can distinguish flaky preview from real broken pages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
load_timeout entries had no explanation in the Preview infrastructure section. Now falls back to the type name (e.g. "load timeout") so reviewers always know why a failure was classified as infrastructure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the Cypress component tests crash (not just test failures), no cypress-failures.jsonl is written. Previously the report ran with an empty input and published "All tests were successful!" which is wrong. Now the report only runs when failure records exist OR the component test step actually completed successfully. A Cypress crash instead publishes a "failed to run" message. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without a timeout, the integration tests can run for GitHub's full 6-hour job limit (e.g. when the sampled pages include many slow API reference pages), preventing the summary report step from ever executing.
…pages section - Surface the sample seed in the top-level H1 so it's immediately visible - Content regressions / Preview infrastructure / Other sections are always rendered; each falls back to a "none" line when the bucket is empty - Sampled pages list remains at the bottom of the comment - Remove the now-inlined printSampleMetadata helper
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the documentation page status sampling deterministic, show reviewers exactly which pages were tested, and reduce avoidable Cypress runtime in the sampled docs check.
Changes
Why
The tests-summary PR comment was hard to trust because identical inputs could sample different pages and the workflows were wired to two different navigation sources. On top of that, the documentation pages status spec could spend hours retrying content failures and sleeping on every visited page. These changes make reruns reproducible for the same seed, show the sampled page list directly in the comment, and reduce avoidable runtime in the longest Cypress step.
Related Task
EDU-18161