Skip to content

[pull] master from mattermost:master #713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pull merged 2 commits into from
May 19, 2026
Merged

[pull] master from mattermost:master #713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d1580c
chore: update reusable workflows to specific commit sha (#36600)
saturninoabril May 19, 2026
9d318dc
refactor: speed up E2E test workflows and eliminate npm cache-restore…
saturninoabril May 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 103 additions & 9 deletions .github/e2e-tests-workflows.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,23 +16,117 @@ All pipelines follow the **smoke-then-full** pattern: smoke tests run first, ful

```
.github/workflows/
├── e2e-tests-ci.yml # PR orchestrator
├── e2e-tests-on-merge.yml # Merge orchestrator (master/release branches)
├── e2e-tests-on-release.yml # Release cut orchestrator
├── e2e-tests-cypress.yml # Shared wrapper: cypress smoke -> full
├── e2e-tests-playwright.yml # Shared wrapper: playwright smoke -> full
├── e2e-tests-cypress-template.yml # Template: actual cypress test execution
└── e2e-tests-playwright-template.yml # Template: actual playwright test execution
├── e2e-tests-ci.yml # PR orchestrator
├── e2e-tests-on-merge.yml # Merge orchestrator (master/release branches)
├── e2e-tests-on-release.yml # Release cut orchestrator
├── e2e-tests-cypress.yml # Shared wrapper: routes to v1 or v2 template
├── e2e-tests-playwright.yml # Shared wrapper: routes to v1 or v2 template
├── e2e-tests-cypress-template-v2.yml # Active: cypress + test-system-io dispatch
├── e2e-tests-playwright-template-v2.yml # Active: playwright + test-system-io dispatch
├── e2e-tests-cypress-template.yml # Deprecated v1 (legacy in-job execution)
└── e2e-tests-playwright-template.yml # Deprecated v1 (legacy in-job execution)
```

> **v1 templates are deprecated.** They remain available behind a feature flag during cutover but receive no further changes. New work targets the v2 templates exclusively. The wrappers route by `vars.E2E_USE_TEST_IO_DISPATCH` — `'true'` selects v2, anything else falls back to v1.

### Call hierarchy

```
e2e-tests-ci.yml ─────────────────┐
e2e-tests-on-merge.yml ───────────┤──► e2e-tests-cypress.yml ──► e2e-tests-cypress-template.yml
e2e-tests-on-release.yml ─────────┘ e2e-tests-playwright.yml ──► e2e-tests-playwright-template.yml
e2e-tests-on-merge.yml ───────────┤──► e2e-tests-cypress.yml ─────┐
e2e-tests-on-release.yml ─────────┘ e2e-tests-playwright.yml ──┤
│
┌──────────────────────────┘
│ routes on E2E_USE_TEST_IO_DISPATCH
▼
v2 (active) ──► e2e-tests-{cypress,playwright}-template-v2.yml
v1 (legacy) ──► e2e-tests-{cypress,playwright}-template.yml
```

---

## Workflow Architecture (v2)

v2 splits the template into five jobs — `prepare-run`, `prep-deps`, `dispatch-begin`, `workers` (matrix), and `report` — and pushes spec-level execution to [Test System IO](https://github.com/mattermost/mattermost-test-system-io) so workers stay thin and identical.

```
┌──────────────────────────────────────────────────────────────────────────┐
│ Template v2: e2e-tests-{cypress,playwright}-template-v2.yml │
│ │
│ ┌───────────────────┐ ┌──────────────────────────────┐ │
│ │ prepare-run │ │ prep-deps │ │
│ │ (1 runner) │ parallel │ (1 runner) │ │
│ │ │ ◄────────────► │ │ │
│ │ • build workers │ │ Cypress: │ │
│ │ matrix [1..N] │ │ • cypress/node_modules │ │
│ │ • compute commit │ │ • ~/.cache/Cypress (binary)│ │
│ │ status context │ │ │ │
│ │ • emit composite │ │ Playwright: │ │
│ │ identity │ │ • webapp/platform/{client, │ │
│ │ │ │ types}/{lib,node_mod} │ │
│ │ │ │ • playwright/node_modules │ │
│ │ │ │ • playwright/lib/dist │ │
│ │ │ │ • ~/.cache/ms-playwright │ │
│ │ │ │ (chromium only) │ │
│ │ │ │ │ │
│ │ │ │ → saved to actions/cache │ │
│ └─────────┬─────────┘ └───────────────┬──────────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌──────────────────────────────┐ │
│ │ │ dispatch-begin │ │
│ │ │ • register run with │ │
│ │ │ Test System IO │ │
│ │ │ • runs immediately before │ │
│ │ │ workers to minimise the │ │
│ │ │ inactivity-timeout window │ │
│ │ └───────────────┬──────────────┘ │
│ │ │ │
│ └────────────────────┬─────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ workers (matrix, fail-fast: false) │ │
│ │ Cypress full: N=40 | Playwright full: N=10 │ │
│ │ │ │
│ │ each worker, in parallel: │ │
│ │ 1. sparse-checkout actions + full checkout-repo │ │
│ │ 2. setup-node │ │
│ │ 3. restore caches ◄─── actions/cache (from prep-deps) │ │
│ │ (fail-on-cache-miss: true) │ │
│ │ 4. cloud-init + start-server (docker compose stack) │ │
│ │ 5. prepare-cypress | prepare-playwright (run setup project) │ │
│ │ 6. dispatch-run ──────────────────────────────────┐ │ │
│ │ (pulls specs from Test System IO, runs locally, │ │ │
│ │ posts result, loops until queue is empty) │ │ │
│ │ 7. cloud-teardown │ │ │
│ └────────────────────┬────────────────────────────────────┼───────┘ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────┐ │ │
│ │ report │ │ │
│ │ • pull aggregated results from Test System IO │ ◄────┘ │
│ │ • post commit status │ │
│ │ • send webhook notification │ │
│ └─────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┬─┘
│
┌───────────────────────────▼───┐
│ Test System IO (external) │
│ • spec-level dispatch │
│ • result aggregation │
│ • retry orchestration │
└───────────────────────────────┘
```

### Key properties

- **Spec-level vs. job-level parallelism.** The matrix sizes the runner pool; Test System IO does the spec assignment. Slow specs don't block a worker — fast workers keep pulling the next spec from the queue.
- **Cache-only workers.** `prep-deps` installs once per workflow run and saves to `actions/cache`. Every worker restores with `fail-on-cache-miss: true` and runs zero `npm ci`. Eliminates the 40-way `EEXIST/ENOENT` race in npm's shared cacache writer.
- **dispatch-begin runs late.** It depends on `prep-deps` so the gap between Test System IO run registration and the first worker calling `dispatch-run` is just per-worker setup (~3–5 min). Registering earlier risks the run timing out before any worker checks in, bulk-failing every spec.
- **Playwright slim slice.** Playwright only consumes `@mattermost/client` and `@mattermost/types` from webapp, so prep-deps caches just those two packages' built `lib/` and `node_modules` (~10–30 MB) instead of the full `webapp/node_modules` tree (~1–2 GB).
- **Browser/binary caches.** Cypress caches `~/.cache/Cypress` (cypress binary lives outside node_modules); playwright caches `~/.cache/ms-playwright` (chromium only). Both keyed on the framework's lockfile so they invalidate on version bumps.
- **No retry plumbing in the template.** Test System IO handles per-spec retries; the workflow only sees aggregated results.

---

## Pipeline 1: PR (`e2e-tests-ci.yml`)
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/e2e-tests-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ jobs:
cache-dependency-path: |
e2e-tests/cypress/package-lock.json
e2e-tests/playwright/package-lock.json
webapp/package-lock.json
- name: ci/npm-cache-verify
# Heal any partial/dangling entries left in the restored ~/.npm cache
# before running `npm ci`. Avoids the intermittent EEXIST/ENOENT
# failures in npm's cacache writer.
run: npm cache verify

# Cypress check
- name: ci/cypress/npm-install
Expand Down
90 changes: 68 additions & 22 deletions .github/workflows/e2e-tests-cypress-template-v2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ env:
SERVER_IMAGE: "${{ inputs.server_image_repo }}/${{ inputs.server_edition == 'fips' && 'mattermost-enterprise-fips-edition' || inputs.server_edition == 'team' && 'mattermost-team-edition' || 'mattermost-enterprise-edition' }}:${{ inputs.server_image_tag }}"

jobs:
dispatch-begin:
prepare-run:
runs-on: ubuntu-24.04
permissions:
contents: read
Expand Down Expand Up @@ -193,13 +193,62 @@ jobs:
run: |
echo "workers=$(jq -nc --argjson n ${{ inputs.workers }} '[range(1; $n+1)]')" >> $GITHUB_OUTPUT
echo "start_time=$(date +%s)" >> $GITHUB_OUTPUT

# Install cypress node_modules once, then workers restore from cache.
prep-deps:
name: prep-deps
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
steps:
- name: ci/checkout-repo
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ inputs.commit_sha }}
fetch-depth: 1
- name: ci/setup-node
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
with:
node-version-file: ".nvmrc"
- name: ci/cache-cypress-deps
# node_modules + the cypress binary (downloaded to ~/.cache/Cypress by
# cypress's postinstall, not into node_modules). Both must be cached;
# otherwise workers see "cypress npm package installed but binary missing".
id: cache-cypress
uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: |
e2e-tests/cypress/node_modules
~/.cache/Cypress
key: e2e-cypress-deps-${{ runner.os }}-${{ hashFiles('e2e-tests/cypress/package-lock.json') }}
- name: ci/install-cypress-deps
if: steps.cache-cypress.outputs.cache-hit != 'true'
working-directory: e2e-tests/cypress
run: npm ci

# Register the Test System IO run AFTER prep-deps so workers reach
# dispatch-run within Test System IO's inactivity window.
dispatch-begin:
runs-on: ubuntu-24.04
needs: [prepare-run, prep-deps]
permissions:
contents: read
id-token: write
statuses: write
steps:
- name: ci/checkout-repo
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ inputs.commit_sha }}
fetch-depth: 1
- name: ci/dispatch-begin
uses: mattermost/mattermost-test-system-io/.github/actions/test-system-io-dispatch-begin@main
uses: mattermost/mattermost-test-system-io/.github/actions/test-system-io-dispatch-begin@a2ea7f005484c28fedf51e16645f6d3bd683fd63 # 2026-05-16
with:
use-staging: ${{ vars.E2E_USE_STAGING_TEST_IO_URL != 'false' }}
framework: cypress
repo-dir: ${{ github.workspace }}
composite-identity: ${{ steps.composite-identity.outputs.composite-identity-json }}
composite-identity: ${{ needs.prepare-run.outputs.composite-identity-json }}
total-reports-expected: ${{ inputs.workers }}
retest-on-fail: ${{ inputs.retest_on_fail }}
cypress-stage: ${{ inputs.cypress_stage }}
Expand All @@ -217,16 +266,16 @@ jobs:
name: dispatch-run-${{ matrix.worker_index }}
runs-on: ubuntu-24.04
timeout-minutes: 30
needs: dispatch-begin
needs: [prepare-run, dispatch-begin]
permissions:
contents: read
id-token: write
strategy:
fail-fast: false
matrix:
worker_index: ${{ fromJSON(needs.dispatch-begin.outputs.workers-matrix) }}
worker_index: ${{ fromJSON(needs.prepare-run.outputs.workers-matrix) }}
env:
COMPOSITE_IDENTITY: ${{ needs.dispatch-begin.outputs.composite-identity-json }}
COMPOSITE_IDENTITY: ${{ needs.prepare-run.outputs.composite-identity-json }}
SERVER: "${{ inputs.server }}"
MM_LICENSE: "${{ secrets.MM_LICENSE }}"
ENABLED_DOCKER_SERVICES: "${{ inputs.enabled_docker_services }}"
Expand Down Expand Up @@ -260,29 +309,26 @@ jobs:
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
with:
node-version-file: ".nvmrc"
cache: npm
cache-dependency-path: "e2e-tests/cypress/package-lock.json"
- name: ci/get-webapp-node-modules
working-directory: webapp
run: make node_modules
- name: ci/restore-cypress-deps
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: |
e2e-tests/cypress/node_modules
~/.cache/Cypress
key: e2e-cypress-deps-${{ runner.os }}-${{ hashFiles('e2e-tests/cypress/package-lock.json') }}
fail-on-cache-miss: true
- name: ci/cloud-init
working-directory: e2e-tests
run: make cloud-init
- name: ci/start-server
working-directory: e2e-tests
run: make start-server
# `npm ci` in the host context replaces the container-built native
# binaries with host-built ones, since the dispatch adapter spawns
# `npx cypress run` on the host.
- name: ci/prepare-cypress
working-directory: e2e-tests/cypress
run: npm ci
- name: ci/dispatch-run
uses: mattermost/mattermost-test-system-io/.github/actions/test-system-io-dispatch-run@main
uses: mattermost/mattermost-test-system-io/.github/actions/test-system-io-dispatch-run@a2ea7f005484c28fedf51e16645f6d3bd683fd63 # 2026-05-16
with:
use-staging: ${{ vars.E2E_USE_STAGING_TEST_IO_URL != 'false' }}
framework: cypress
composite-identity: ${{ needs.dispatch-begin.outputs.composite-identity-json }}
composite-identity: ${{ needs.prepare-run.outputs.composite-identity-json }}
repo-dir: ${{ github.workspace }}
artifacts-root: ${{ github.workspace }}/worker-artifacts
github-token: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -306,7 +352,7 @@ jobs:

report:
runs-on: ubuntu-24.04
needs: [dispatch-begin, workers]
needs: [prepare-run, dispatch-begin, workers]
if: always()
permissions:
contents: read
Expand All @@ -321,10 +367,10 @@ jobs:
- name: ci/run-summary
id: summary
continue-on-error: true
uses: mattermost/mattermost-test-system-io/.github/actions/test-system-io-summary@main
uses: mattermost/mattermost-test-system-io/.github/actions/test-system-io-summary@a2ea7f005484c28fedf51e16645f6d3bd683fd63 # 2026-05-16
with:
use-staging: ${{ vars.E2E_USE_STAGING_TEST_IO_URL != 'false' }}
composite-identity: ${{ needs.dispatch-begin.outputs.composite-identity-json }}
composite-identity: ${{ needs.prepare-run.outputs.composite-identity-json }}
framework: cypress
report-type: ${{ inputs.report_type }}
image-tag: ${{ inputs.server_image_tag }}
Expand Down
Loading
Loading