Install/Update test matrix (P1: blocking VSCode 3-OS lane + board)

[dev-punia-altimate]

What

Adds a telemetry-weighted install + update test matrix for the extension. Each cell installs the VSIX (fresh or upgrade-from-baseline), launches the editor headless against the hermetic `dbt-core-sample-duckdb` fixture, asserts activation + dbt-project init, and emits a uniform RESULT_JSON. A Python aggregator renders an install matrix and update matrix, posts them to the PR (check + sticky comment) and Slack, and gates the release.

This is P1: the blocking stock VSCode lane on Linux/Windows/macOS (fresh + upgrade), Insiders (non-blocking, Linux), and the existing code-server smoke wrapped into the same board. Forks (Cursor/Windsurf/Kiro) are P2; the `ms-python.python` portability check + Antigravity are P3.

Why (data)

30-day extension telemetry (App Insights): ~48k active installs — macOS 44% / Windows 35% / Linux 21%; VSCode 79% / Cursor 13.5%. Only ~40% on latest; ~19% on pre-0.61. Today the VSIX install/upgrade smoke runs only on code-server/Linux (1.3% of installs) while Cursor (13.5%) and Windows (35%) of the VSIX path are untested. Upgrade baselines (`0.61.4`, `0.60.7`, `0.59.5`, `0.55.5`) come straight from the version distribution.

How it gates

• Stock VSCode is the only blocking lane → a failed VSCode cell fails the `Install/Update Matrix` check (PR) and blocks marketplace publish (tag) via the new `matrix-gate` job in `ci.yml`.
• Insiders / code-server are non-blocking (render ⚠️, never veto).
• Runs on every PR (check + comment + Slack) and daily 06:30 UTC (Slack only). Free GitHub-hosted runners (public repo).

Files

• `test-matrix/aggregate.py` (+ pytest) — renders matrices + Slack payload + blocking-gate exit code
• `test-matrix/vscode-cell.mjs` — VSCode/Insiders cell via `@vscode/test-electron` (fresh + upgrade)
• `test-matrix/setup-dbt-env.sh` — hermetic dbt-duckdb env
• `src/test/matrix/{index,activation.test}.ts` — in-host activation + dbt-init assertion
• `test-matrix/codeserver-cell.sh` — wraps existing `docker-setup/vsix-smoke.sh`
• `.github/workflows/install-update-matrix.yml` + `ci.yml` release gate

Pending

• `MATRIX_SLACK_WEBHOOK` secret (channel TBD) — Slack step degrades gracefully (skips) until set.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b887775f-09ab-4b3f-82f9-b59d2d63cd82

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch matrix/install-update-p1

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

VSIX Install + Update Matrix — target pr-build (pull_request)

Install matrix (fresh install of target)

Runtime	linux	windows	macos
vscode	✅	✅	✅
vscode-insiders	✅	—	—
code-server	✅	—	—

Update matrix (upgrade baseline → target)

Runtime / OS	from 0.55.5	from 0.59.5	from 0.60.7	from 0.61.4
vscode (linux)	✅	✅	✅	✅
vscode (windows)	—	—	—	✅
vscode (macos)	—	—	—	✅
code-server (linux)	—	—	—	✅

[dev-punia-altimate]

✅ CI validated green — the install/update matrix runs and passes end-to-end on real GitHub-hosted runners (run 26679047912).

Install matrix (fresh)

Runtime	linux	windows	macos
vscode	✅	✅	✅
vscode-insiders	✅	—	—
code-server	✅	—	—

Update matrix (upgrade → pr-build)

Runtime / OS	from 0.55.5	from 0.59.5	from 0.60.7	from 0.61.4
vscode (linux)	✅	✅	✅	✅
vscode (windows)	—	—	—	✅
vscode (macos)	—	—	—	✅
code-server (linux)	—	—	—	✅

It already paid for itself: the first run caught a real Windows-only install failure (Node can't execFileSync code.cmd without a shell) — fixed in this PR. Without this matrix that ships to the 35% of users on Windows.

Next (separate PRs): P2 = Cursor/Windsurf/Kiro fork lanes (non-blocking, Linux); P3 = the ms-python.python Open-VSX portability check + Antigravity best-effort. MATRIX_SLACK_WEBHOOK secret still pending (Slack step skips gracefully until set).

[dev-punia-altimate]

✅ Tests — All Passed

cc @dev-punia-altimate