Skip to content

feat(ci): redesign quarantine report for readability and error visibility#1030

Merged
mergify[bot] merged 1 commit intomainfrom
devs/JulianMaurin/feat/ci-quarantine-report-redesign/Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06
Mar 16, 2026
Merged

feat(ci): redesign quarantine report for readability and error visibility#1030
mergify[bot] merged 1 commit intomainfrom
devs/JulianMaurin/feat/ci-quarantine-report-redesign/Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06

Conversation

@JulianMaurin
Copy link
Copy Markdown
Contributor

The CLI quarantine report was hard to scan in web-based CI consoles
(GitHub Actions). Test failures only showed names without error details,
forcing developers to scroll through test runner output to find
stacktraces. The report lacked visual hierarchy and mixed concerns.

Report improvements:

  • Add intro section explaining what CI Insights does and linking to docs
  • Display error stacktraces for unquarantined failures using box-drawing
    characters (┌ │ └─) for clear visual separation
  • Use ══ heavy separators to frame the header and verdict sections
  • Show "(no error details in JUnit report)" when failure has no exception
    attributes, so users understand the absence is expected
  • Condense verdict into a framed block with consistent X/Y format
    (e.g. "1/2 failures quarantined", "all tests passed, no quarantine needed")
  • Skip quarantine section entirely when no failures are detected

Error handling improvements:

  • Display quarantine API errors inline in the report (not hidden in stderr)
    with ⚠️ section, explanation, and detail box
  • Show all test failure stacktraces when quarantine check fails, treating
    all failures as unquarantined so users can still see what went wrong
  • Display upload errors inline with clear messaging about impact
    ("Mergify CI Insights won't process these test results.
    Quarantine status and CI outcome are unaffected.")
  • Remove redundant/confusing error messages (QUARANTINE_INFO_ERROR_MSG)

Architecture improvements:

  • Separate concerns: move all display logic from quarantine.py to cli.py
  • quarantine.check_and_update_failing_spans now returns a QuarantineResult
    dataclass instead of an int, carrying categorized span lists
  • upload.upload() no longer prints — just uploads and raises on error
  • junit.files_to_spans() returns (run_id, spans) tuple so cli.py controls
    the run ID display
  • Extract _run_process test helper to eliminate ~20 lines of boilerplate
    per test case
  • Move XML fixtures to tests/ci/fixtures/ directory

Test coverage:

  • 15 test scenarios covering all report paths with full output assertions
  • Add tests for: invalid XML, empty testsuites, no exception attributes,
    exception type only, None attributes, upload+quarantine combined errors,
    multiple report files, pluralization
  • Consolidate duplicated tests from test_cli.py into
    junit_processing/test_cli.py
  • Use realistic error messages (HTTP 422, httpx.ConnectError)

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

@mergify mergify bot had a problem deploying to Mergify Merge Protections March 15, 2026 14:48 Failure
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 15, 2026

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 🤖 Continuous Integration

Wonderful, this rule succeeded.
  • all of:
    • check-success=ci-gate

🟢 👀 Review Requirements

Wonderful, this rule succeeded.
  • any of:
    • #approved-reviews-by>=2
    • author = dependabot[bot]
    • author = mergify-ci-bot
    • author = renovate[bot]

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?:

🟢 🔎 Reviews

Wonderful, this rule succeeded.
  • #changes-requested-reviews-by = 0
  • #review-requested = 0
  • #review-threads-unresolved = 0

🟢 📕 PR description

Wonderful, this rule succeeded.
  • body ~= (?ms:.{48,})

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 15, 2026

🧪 CI Insights

Here's what we observed from your CI run for 37e10e6.

🟢 All jobs passed!

But CI Insights is watching 👀

@JulianMaurin JulianMaurin force-pushed the devs/JulianMaurin/feat/ci-quarantine-report-redesign/Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06 branch from b30386d to 3226522 Compare March 15, 2026 14:55
@mergify mergify bot had a problem deploying to Mergify Merge Protections March 15, 2026 14:55 Failure
@JulianMaurin JulianMaurin force-pushed the devs/JulianMaurin/feat/ci-quarantine-report-redesign/Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06 branch from 3226522 to 72c4a90 Compare March 15, 2026 14:59
@mergify mergify bot had a problem deploying to Mergify Merge Protections March 15, 2026 14:59 Failure
@JulianMaurin JulianMaurin force-pushed the devs/JulianMaurin/feat/ci-quarantine-report-redesign/Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06 branch from 72c4a90 to b0f3ed2 Compare March 15, 2026 15:01
@mergify mergify bot had a problem deploying to Mergify Merge Protections March 15, 2026 15:01 Failure
…lity

The CLI quarantine report was hard to scan in web-based CI consoles
(GitHub Actions). Test failures only showed names without error details,
forcing developers to scroll through test runner output to find
stacktraces. The report lacked visual hierarchy and mixed concerns.

Report improvements:
- Add intro section explaining what CI Insights does and linking to docs
- Display error stacktraces for unquarantined failures using box-drawing
  characters (┌ │ └─) for clear visual separation
- Use ══ heavy separators to frame the header and verdict sections
- Show "(no error details in JUnit report)" when failure has no exception
  attributes, so users understand the absence is expected
- Condense verdict into a framed block with consistent X/Y format
  (e.g. "1/2 failures quarantined", "all tests passed, no quarantine needed")
- Skip quarantine section entirely when no failures are detected

Error handling improvements:
- Display quarantine API errors inline in the report (not hidden in stderr)
  with ⚠️ section, explanation, and detail box
- Show all test failure stacktraces when quarantine check fails, treating
  all failures as unquarantined so users can still see what went wrong
- Display upload errors inline with clear messaging about impact
  ("Mergify CI Insights won't process these test results.
  Quarantine status and CI outcome are unaffected.")
- Remove redundant/confusing error messages (QUARANTINE_INFO_ERROR_MSG)

Architecture improvements:
- Separate concerns: move all display logic from quarantine.py to cli.py
- quarantine.check_and_update_failing_spans now returns a QuarantineResult
  dataclass instead of an int, carrying categorized span lists
- upload.upload() no longer prints — just uploads and raises on error
- junit.files_to_spans() returns (run_id, spans) tuple so cli.py controls
  the run ID display
- Extract _run_process test helper to eliminate ~20 lines of boilerplate
  per test case
- Move XML fixtures to tests/ci/fixtures/ directory

Test coverage:
- 15 test scenarios covering all report paths with full output assertions
- Add tests for: invalid XML, empty testsuites, no exception attributes,
  exception type only, None attributes, upload+quarantine combined errors,
  multiple report files, pluralization
- Consolidate duplicated tests from test_cli.py into
  junit_processing/test_cli.py
- Use realistic error messages (HTTP 422, httpx.ConnectError)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Change-Id: Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06
@JulianMaurin JulianMaurin force-pushed the devs/JulianMaurin/feat/ci-quarantine-report-redesign/Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06 branch from b0f3ed2 to 37e10e6 Compare March 15, 2026 15:12
@JulianMaurin JulianMaurin marked this pull request as ready for review March 15, 2026 15:12
@mergify mergify bot deployed to Mergify Merge Protections March 15, 2026 15:12 Active
@mergify mergify bot requested a review from a team March 15, 2026 15:14
@mergify mergify bot requested a review from a team March 16, 2026 07:05
mergify bot added a commit that referenced this pull request Mar 16, 2026
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 16, 2026

Merge Queue Status

This pull request spent 2 minutes 31 seconds in the queue, including 2 minutes 17 seconds running CI.

Required conditions to merge

@mergify mergify bot added the queued label Mar 16, 2026
@mergify mergify bot merged commit edeb5c1 into main Mar 16, 2026
19 checks passed
@mergify mergify bot deleted the devs/JulianMaurin/feat/ci-quarantine-report-redesign/Ib03c1d49ec4e740b021e4bd4a05758ee3b428d06 branch March 16, 2026 10:51
@mergify mergify bot removed the queued label Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants