feat(qa): vsix-smoke — verify published VSIX install + activation - #1979
Open
dev-punia-altimate wants to merge 8 commits into
Open
feat(qa): vsix-smoke — verify published VSIX install + activation#1979dev-punia-altimate wants to merge 8 commits into
dev-punia-altimate wants to merge 8 commits into
Conversation
Adds a smoke test that the published dbt Power User VSIX installs and activates cleanly in a fresh code-server container. Catches publish- breakage that `tests.yml` cannot (because that workflow only exercises the source tree via @vscode/test-electron, never the published artifact). Single bash script — docker-setup/vsix-smoke.sh — runs in CI and locally on Anas/Sai's laptop so there's no drift between automation and manual testing. Reuses docker-setup/Dockerfile so Python + dbt-duckdb + the three dependency extensions are already baked in. Triggers: - schedule: daily 06:00 UTC sanity check - workflow_dispatch: manual --version / --from-version inputs - release: post-publish verification (with 5-min marketplace propagation sleep) On scheduled failure the workflow opens (or comments on) a vsix-smoke- labelled GitHub issue. No Slack secret required. Verified locally: 55s end-to-end on warm image, all 6 expected extensions listed, no activation errors in the code-server boot log. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR adds post-release smoke-test automation for the dbt Power User VSIX. A new Bash script builds a Docker image, launches an isolated code-server container, installs and activates the extension (either from marketplace or local file), validates installation and version, optionally tests upgrade paths, and scans logs for activation failures. A new GitHub Actions workflow ( ChangesPublished VSIX smoke testing
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bundle Size Reportdarwin-arm64: 74.1 MB
linux-x64: 75.8 MB
win32-x64: 76.7 MB
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/vsix-smoke.yml:
- Line 46: The workflow uses tag-pinned actions (e.g., the step with "uses:
actions/checkout@v4" and the other two "uses:" entries flagged) which must be
pinned to commit SHAs; find every "uses:" line in the workflow (including the
occurrences currently using tags like actions/checkout@v4) and replace the tag
with the corresponding full commit SHA from the action's repository (e.g.,
actions/checkout@<full-commit-sha>), ensuring each action reference is
commit-pinned rather than tag-pinned.
- Around line 105-114: The current issues.listForRepo call returns any open
vsix-smoke issues regardless of age; update the github.rest.issues.listForRepo
call to include a seven-day window by passing a since parameter (e.g. const
since = new Date(Date.now() - 7*24*60*60*1000).toISOString()) so the query only
returns issues updated/created in the last 7 days; modify the call that assigns
{ data: issues } from github.rest.issues.listForRepo to include since before
checking if (issues.length > 0).
In `@docker-setup/vsix-smoke.sh`:
- Around line 35-42: The option handlers for --version and --from-version assume
a following argument exists and will fail under set -u; update the case branch
logic for --version and --from-version to validate that "$2" is present and is
not another flag (e.g., empty or begins with '-'), call usage and exit with a
non-zero status on invalid/missing value, and only then assign to VERSION and
FROM_VERSION and shift 2; reference the option names (--version,
--from-version), variables (VERSION, FROM_VERSION) and the usage function when
making the change.
- Around line 118-123: The grep pattern uses the unescaped $version so regex
metacharacters in the version can cause false matches; escape $version before
using it in the regex (e.g., create version_escaped=$(printf '%s' "$version" |
sed 's/[][\.^$*+?{}|()]/\\&/g') or use grep -F -x with a fixed string) and
replace the pattern use of ${version} with the escaped variable (e.g.,
"${version_escaped}") in the check that compares "${EXTENSION_ID}" and
"$version" against "$listed".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e79efba7-f831-4630-a2e7-2166f7042831
📒 Files selected for processing (3)
.github/workflows/vsix-smoke.ymldocker-setup/README.mddocker-setup/vsix-smoke.sh
| runs-on: ubuntu-latest | ||
| timeout-minutes: 25 | ||
| steps: | ||
| - uses: actions/checkout@v4 |
Contributor
There was a problem hiding this comment.
Pin GitHub Actions to commit SHAs.
These uses: entries are tag-pinned, not commit-pinned. That violates strict workflow supply-chain policy and allows silent upstream drift.
Also applies to: 88-88, 98-98
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 46-46: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 46-46: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/vsix-smoke.yml at line 46, The workflow uses tag-pinned
actions (e.g., the step with "uses: actions/checkout@v4" and the other two
"uses:" entries flagged) which must be pinned to commit SHAs; find every "uses:"
line in the workflow (including the occurrences currently using tags like
actions/checkout@v4) and replace the tag with the corresponding full commit SHA
from the action's repository (e.g., actions/checkout@<full-commit-sha>),
ensuring each action reference is commit-pinned rather than tag-pinned.
Comment on lines
+105
to
+114
| // Find any open vsix-smoke issue from the last 7 days so we don't spam | ||
| const { data: issues } = await github.rest.issues.listForRepo({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| state: 'open', | ||
| labels: 'vsix-smoke', | ||
| per_page: 5, | ||
| }); | ||
|
|
||
| if (issues.length > 0) { |
Contributor
There was a problem hiding this comment.
Implement the promised 7-day issue window.
The comment says “last 7 days,” but the query currently selects any open vsix-smoke issue. This can keep appending to very old incidents.
Suggested fix
const today = new Date().toISOString().split('T')[0];
+ const sevenDaysAgo = Date.now() - 7 * 24 * 60 * 60 * 1000;
const runUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const body = `Scheduled \`vsix-smoke\` workflow failed on ${today}.\n\nRun: ${runUrl}`;
@@
- if (issues.length > 0) {
+ const recentIssue = issues.find(i => new Date(i.created_at).getTime() >= sevenDaysAgo);
+ if (recentIssue) {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
- issue_number: issues[0].number,
+ issue_number: recentIssue.number,
body,
});
- core.info(`Commented on existing issue #${issues[0].number}`);
+ core.info(`Commented on existing issue #${recentIssue.number}`);
} else {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/vsix-smoke.yml around lines 105 - 114, The current
issues.listForRepo call returns any open vsix-smoke issues regardless of age;
update the github.rest.issues.listForRepo call to include a seven-day window by
passing a since parameter (e.g. const since = new Date(Date.now() -
7*24*60*60*1000).toISOString()) so the query only returns issues updated/created
in the last 7 days; modify the call that assigns { data: issues } from
github.rest.issues.listForRepo to include since before checking if
(issues.length > 0).
Comment on lines
+118
to
+123
| if [ "$version" != "latest" ]; then | ||
| if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${version}$"; then | ||
| red "FAIL: expected version $version but installed list shows:" | ||
| echo "$listed" | grep "^${EXTENSION_ID}" || true | ||
| return 1 | ||
| fi |
Contributor
There was a problem hiding this comment.
Escape version before regex match.
grep -E treats . as wildcard, so version checks can match unintended strings.
Suggested fix
if [ "$version" != "latest" ]; then
- if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${version}$"; then
+ local escaped_version="${version//./\\.}"
+ if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${escaped_version}$"; then
red "FAIL: expected version $version but installed list shows:"
echo "$listed" | grep "^${EXTENSION_ID}" || true
return 1
fi
fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if [ "$version" != "latest" ]; then | |
| if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${version}$"; then | |
| red "FAIL: expected version $version but installed list shows:" | |
| echo "$listed" | grep "^${EXTENSION_ID}" || true | |
| return 1 | |
| fi | |
| if [ "$version" != "latest" ]; then | |
| local escaped_version="${version//./\\.}" | |
| if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${escaped_version}$"; then | |
| red "FAIL: expected version $version but installed list shows:" | |
| echo "$listed" | grep "^${EXTENSION_ID}" || true | |
| return 1 | |
| fi | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docker-setup/vsix-smoke.sh` around lines 118 - 123, The grep pattern uses the
unescaped $version so regex metacharacters in the version can cause false
matches; escape $version before using it in the regex (e.g., create
version_escaped=$(printf '%s' "$version" | sed 's/[][\.^$*+?{}|()]/\\&/g') or
use grep -F -x with a fixed string) and replace the pattern use of ${version}
with the escaped variable (e.g., "${version_escaped}") in the check that
compares "${EXTENSION_ID}" and "$version" against "$listed".
Two additions on top of the first commit's daily marketplace check: 1. Pre-release gate (ci.yml) New release-smoke job runs between build and release-vsstudio-marketplace. On every tag push it: packages the linux-x64 VSIX, resolves the currently- published version from OpenVSX as the baseline, then runs the smoke against the local VSIX file with --from-version BASELINE. Publish to all 5 platform targets is blocked if the smoke fails because release-vsstudio-marketplace now `needs: [build, release-smoke]`. Stops bad releases at the door instead of finding them post-publish. 2. Upgrade test always-on (vsix-smoke.yml + script) Scheduled and dispatched runs now auto-resolve the previous published version from OpenVSX and exercise the upgrade scenario (install N-1 from marketplace, then install the target on top). User can override the baseline via workflow_dispatch input, or pass --skip-upgrade for fresh- install-only. Catches migration bugs that fresh-install cannot. 3. Script extension (docker-setup/vsix-smoke.sh) New --vsix-file PATH mode that docker-cp's a local .vsix file into the container and installs it with code-server --install-extension /path. Used by the pre-release gate. Existing --version / --from-version flags unchanged; --vsix-file composes with --from-version for the full "marketplace baseline → local file" upgrade scenario. Verified locally end-to-end: marketplace fresh-install (--version 0.61.5): 58s PASS local-file fresh-install (--vsix-file ...0.61.5.vsix): 27s PASS upgrade (--from-version 0.61.4 --vsix-file ...0.61.5.vsix): 82s PASS All three modes confirm the extension lands at the expected version per --list-extensions and produces no activation errors in the code-server boot log. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removed the appended "Smoke test the PUBLISHED VSIX" section from docker-setup/README.md. The script itself documents its flags via --help, and the PR description has the local-use snippet. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (4)
docker-setup/vsix-smoke.sh (2)
50-52:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winGuard option values before reading
$2.
--version,--vsix-file, and--from-versionstill assume a value exists; withset -u, this exits abruptly instead of returning a clean CLI error.Proposed fix
- --version) VERSION="$2"; shift 2 ;; - --vsix-file) VSIX_FILE="$2"; shift 2 ;; - --from-version) FROM_VERSION="$2"; shift 2 ;; + --version) + [ $# -ge 2 ] && [ -n "${2:-}" ] && [[ "${2:-}" != -* ]] || { echo "Missing/invalid value for --version" >&2; usage >&2; exit 2; } + VERSION="$2"; shift 2 ;; + --vsix-file) + [ $# -ge 2 ] && [ -n "${2:-}" ] && [[ "${2:-}" != -* ]] || { echo "Missing/invalid value for --vsix-file" >&2; usage >&2; exit 2; } + VSIX_FILE="$2"; shift 2 ;; + --from-version) + [ $# -ge 2 ] && [ -n "${2:-}" ] && [[ "${2:-}" != -* ]] || { echo "Missing/invalid value for --from-version" >&2; usage >&2; exit 2; } + FROM_VERSION="$2"; shift 2 ;;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docker-setup/vsix-smoke.sh` around lines 50 - 52, The option handlers for --version, --vsix-file, and --from-version assume $2 exists and blow up under set -u; update the case branches that set VERSION, VSIX_FILE, and FROM_VERSION to first guard for a following argument (e.g. test [ -n "${2:-}" ] or check $# -lt 2) and on missing value print a clear CLI error/usage and exit non‑zero instead of dereferencing $2, then only assign and shift when the value is present.
157-159:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winEscape expected version before regex match.
grep -Etreats.and other metacharacters specially, so version comparison can match unintended values.Proposed fix
if [ -n "$expected_version" ]; then - if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${expected_version}$"; then + expected_version_escaped=$(printf '%s' "$expected_version" | sed 's/[][\.^$*+?{}|()]/\\&/g') + if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${expected_version_escaped}$"; then red "FAIL: expected version $expected_version but installed list shows:"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docker-setup/vsix-smoke.sh` around lines 157 - 159, The current grep check uses grep -qE with a regex built from EXTENSION_ID and expected_version, so metacharacters in expected_version (like dots) can be interpreted as regex; update the check to match the literal version by either escaping expected_version before embedding it in the regex or switching to a fixed-string, exact match grep invocation (e.g., replace the grep -qE usage with a fixed/exact match grep such as grep -qxF against "${EXTENSION_ID}@${expected_version}" or escape metacharacters in expected_version) so the comparison against listed is reliable; modify the line that references EXTENSION_ID, expected_version, listed, and grep -qE accordingly..github/workflows/vsix-smoke.yml (2)
55-58:⚠️ Potential issue | 🟠 Major | ⚡ Quick winPin Actions to commit SHAs.
These
uses:references are tag-pinned and can drift silently. Pin to immutable commit SHAs.Also applies to: 135-135, 144-144
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/vsix-smoke.yml around lines 55 - 58, The workflow uses floating tags for actions (uses: actions/checkout@v4 and uses: docker/setup-buildx-action@v3) which can drift; update those two occurrences (and the other occurrences indicated at lines 135 and 144) to pin to immutable commit SHAs instead of version tags—replace actions/checkout@v4 and docker/setup-buildx-action@v3 with their specific commit SHA refs (e.g., actions/checkout@<commit-sha> and docker/setup-buildx-action@<commit-sha>) so the workflow references stable commits.
151-157:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winApply the documented 7-day window when reusing issues.
The query currently pulls any open
vsix-smokeissue, so very old incidents can keep accumulating comments.Proposed fix
+ const since = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString(); const { data: issues } = await github.rest.issues.listForRepo({ owner: context.repo.owner, repo: context.repo.repo, state: 'open', labels: 'vsix-smoke', + since, per_page: 5, });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/vsix-smoke.yml around lines 151 - 157, The issue query pulls all open vsix-smoke issues regardless of age; update the github.rest.issues.listForRepo call (the const { data: issues } = await github.rest.issues.listForRepo({...}) block) to include a since parameter set to 7 days ago (e.g. new Date(Date.now() - 7*24*60*60*1000).toISOString()) so only issues updated within the last 7 days are returned; keep the existing owner/repo/state/labels/per_page parameters unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 184-189: Replace the fragile VSIX discovery line that uses
`VSIX_FILE=$(ls *.vsix | head -1)` with a robust glob check: iterate over the
shell glob (for f in *.vsix; do if [ -e "$f" ]; then VSIX_FILE="$f"; break; fi;
done) so the script won't fail under strict shell settings, then keep the
existing `[ -z "$VSIX_FILE" ]` check and error/exit behavior; update references
to the `VSIX_FILE` variable accordingly.
- Line 156: Replace the lightweight action tags and brittle vsix discovery: pin
actions/checkout@v4 and actions/upload-artifact@v4 to their full commit SHAs
(replace the tag usages with the corresponding immutable SHA commits) and add
persist-credentials: false to the checkout step to avoid leaking repo
credentials; for the VSIX discovery step, stop using an unguarded `ls *.vsix`
and instead use a nullglob-safe glob check (enable shell nullglob or populate an
array of *.vsix then test its length) so the workflow emits the intended "no
vsix" error rather than failing earlier.
In @.github/workflows/vsix-smoke.yml:
- Around line 63-68: The step-level environment override for GITHUB_OUTPUT is
shadowing the runner-provided outputs file path; remove the env entry that sets
GITHUB_OUTPUT in the vsix-smoke.yml step so the Python code that reads
os.environ["GITHUB_OUTPUT"] (the Python step around lines where outputs are
written) can write to the correct runner file. Locate the step that declares
env: GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }} and delete that env key (or move
any needed values to a different variable), leaving other env entries intact.
---
Duplicate comments:
In @.github/workflows/vsix-smoke.yml:
- Around line 55-58: The workflow uses floating tags for actions (uses:
actions/checkout@v4 and uses: docker/setup-buildx-action@v3) which can drift;
update those two occurrences (and the other occurrences indicated at lines 135
and 144) to pin to immutable commit SHAs instead of version tags—replace
actions/checkout@v4 and docker/setup-buildx-action@v3 with their specific commit
SHA refs (e.g., actions/checkout@<commit-sha> and
docker/setup-buildx-action@<commit-sha>) so the workflow references stable
commits.
- Around line 151-157: The issue query pulls all open vsix-smoke issues
regardless of age; update the github.rest.issues.listForRepo call (the const {
data: issues } = await github.rest.issues.listForRepo({...}) block) to include a
since parameter set to 7 days ago (e.g. new Date(Date.now() -
7*24*60*60*1000).toISOString()) so only issues updated within the last 7 days
are returned; keep the existing owner/repo/state/labels/per_page parameters
unchanged.
In `@docker-setup/vsix-smoke.sh`:
- Around line 50-52: The option handlers for --version, --vsix-file, and
--from-version assume $2 exists and blow up under set -u; update the case
branches that set VERSION, VSIX_FILE, and FROM_VERSION to first guard for a
following argument (e.g. test [ -n "${2:-}" ] or check $# -lt 2) and on missing
value print a clear CLI error/usage and exit non‑zero instead of dereferencing
$2, then only assign and shift when the value is present.
- Around line 157-159: The current grep check uses grep -qE with a regex built
from EXTENSION_ID and expected_version, so metacharacters in expected_version
(like dots) can be interpreted as regex; update the check to match the literal
version by either escaping expected_version before embedding it in the regex or
switching to a fixed-string, exact match grep invocation (e.g., replace the grep
-qE usage with a fixed/exact match grep such as grep -qxF against
"${EXTENSION_ID}@${expected_version}" or escape metacharacters in
expected_version) so the comparison against listed is reliable; modify the line
that references EXTENSION_ID, expected_version, listed, and grep -qE
accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ac5056be-88f1-4576-b486-26f9aaeeb364
📒 Files selected for processing (3)
.github/workflows/ci.yml.github/workflows/vsix-smoke.ymldocker-setup/vsix-smoke.sh
| timeout-minutes: 25 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 |
Contributor
There was a problem hiding this comment.
Pin GitHub Actions to immutable SHAs and harden checkout
actions/checkout@v4andactions/upload-artifact@v4are only version-tag pinned; pin to the full commit SHA (also at the other checkout/upload-artifact occurrence around line 197).- Add
persist-credentials: falseto the checkout step (not currently set), unless the workflow truly needs push/auth creds. - The VSIX discovery uses
ls *.vsixbefore the “empty” guard; switch to a safer glob/nullglob pattern so the step fails with the intended error message rather than breaking earlier.
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 155-156: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 156-156: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml at line 156, Replace the lightweight action tags
and brittle vsix discovery: pin actions/checkout@v4 and
actions/upload-artifact@v4 to their full commit SHAs (replace the tag usages
with the corresponding immutable SHA commits) and add persist-credentials: false
to the checkout step to avoid leaking repo credentials; for the VSIX discovery
step, stop using an unguarded `ls *.vsix` and instead use a nullglob-safe glob
check (enable shell nullglob or populate an array of *.vsix then test its
length) so the workflow emits the intended "no vsix" error rather than failing
earlier.
Comment on lines
+184
to
+189
| VSIX_FILE=$(ls *.vsix | head -1) | ||
| echo "Testing local VSIX: $VSIX_FILE (upgrade from baseline: ${BASELINE:-none})" | ||
| if [ -z "$VSIX_FILE" ]; then | ||
| echo "::error::No .vsix file produced by vsce package" | ||
| exit 1 | ||
| fi |
Contributor
There was a problem hiding this comment.
Make VSIX discovery robust under shell error settings.
VSIX_FILE=$(ls *.vsix | head -1) can fail before your explicit [ -z "$VSIX_FILE" ] check runs.
Proposed fix
- VSIX_FILE=$(ls *.vsix | head -1)
+ shopt -s nullglob
+ files=( *.vsix )
+ VSIX_FILE="${files[0]:-}"
echo "Testing local VSIX: $VSIX_FILE (upgrade from baseline: ${BASELINE:-none})"
if [ -z "$VSIX_FILE" ]; then
echo "::error::No .vsix file produced by vsce package"
exit 1
fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| VSIX_FILE=$(ls *.vsix | head -1) | |
| echo "Testing local VSIX: $VSIX_FILE (upgrade from baseline: ${BASELINE:-none})" | |
| if [ -z "$VSIX_FILE" ]; then | |
| echo "::error::No .vsix file produced by vsce package" | |
| exit 1 | |
| fi | |
| shopt -s nullglob | |
| files=( *.vsix ) | |
| VSIX_FILE="${files[0]:-}" | |
| echo "Testing local VSIX: $VSIX_FILE (upgrade from baseline: ${BASELINE:-none})" | |
| if [ -z "$VSIX_FILE" ]; then | |
| echo "::error::No .vsix file produced by vsce package" | |
| exit 1 | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 184 - 189, Replace the fragile VSIX
discovery line that uses `VSIX_FILE=$(ls *.vsix | head -1)` with a robust glob
check: iterate over the shell glob (for f in *.vsix; do if [ -e "$f" ]; then
VSIX_FILE="$f"; break; fi; done) so the script won't fail under strict shell
settings, then keep the existing `[ -z "$VSIX_FILE" ]` check and error/exit
behavior; update references to the `VSIX_FILE` variable accordingly.
Comment on lines
+63
to
+68
| RELEASE_TAG: ${{ github.event.release.tag_name }} | ||
| DISPATCH_VERSION: ${{ inputs.version }} | ||
| DISPATCH_FROM: ${{ inputs.from_version }} | ||
| SKIP_UPGRADE: ${{ inputs.skip_upgrade }} | ||
| GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }} | ||
| run: | |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/vsix-smoke.yml"
echo "== File: $FILE =="
if [ ! -f "$FILE" ]; then
echo "Missing file: $FILE" >&2
exit 1
fi
echo "== Show lines 40-120 =="
nl -ba "$FILE" | sed -n '40,120p'
echo "== Search for GITHUB_OUTPUT occurrences =="
rg -n "GITHUB_OUTPUT" "$FILE" || true
echo "== Search for any uses of outputs file or echo >> $GITHUB_OUTPUT =="
rg -n "(\\$GITHUB_OUTPUT|GITHUB_OUTPUT\\b|::set-output|output file|GITHUB_ENV)" "$FILE" || true
echo "== Check the referenced step block around run: | (lines ~55-80) =="
nl -ba "$FILE" | sed -n '55,90p'
echo "== Find actions usage to evaluate pinning/supply chain claims quickly =="
rg -n "uses:\s*" "$FILE" || trueRepository: AltimateAI/vscode-dbt-power-user
Length of output: 207
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/vsix-smoke.yml"
echo "== File: $FILE =="
if [ ! -f "$FILE" ]; then
echo "Missing file: $FILE" >&2
exit 1
fi
echo "== Show lines 40-120 =="
awk 'NR>=40 && NR<=120 {printf "%d:%s\n", NR, $0}' "$FILE"
echo "== Search for GITHUB_OUTPUT occurrences in workflow =="
rg -n "GITHUB_OUTPUT" "$FILE" || true
echo "== Search for output-file writing patterns in workflow =="
rg -n "(\$GITHUB_OUTPUT|::set-output|output file|GITHUB_ENV)" "$FILE" || true
echo "== Show around the snippet (lines 55-90) =="
awk 'NR>=55 && NR<=90 {printf "%d:%s\n", NR, $0}' "$FILE"
echo "== List uses: lines in the workflow =="
rg -n "uses:\s*" "$FILE" || trueRepository: AltimateAI/vscode-dbt-power-user
Length of output: 5257
🌐 Web query:
GitHub Actions set env GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }} break outputs file path
💡 Result:
Setting an environment variable named GITHUB_OUTPUT (e.g., via env: GITHUB_OUTPUT: ... in your workflow YAML) will likely break the ability to set step outputs correctly [1]. In GitHub Actions, GITHUB_OUTPUT is a predefined environment variable that holds the path to a temporary file used for capturing step outputs [2]. When you define a step, the GitHub Actions runner uses the value of the GITHUB_OUTPUT environment variable to know where to write output commands [2]. If you override this environment variable by setting it yourself, the runner may no longer be able to locate the correct file or may write data to an invalid location, causing your outputs to fail to register [1]. To correctly set step outputs, you must append key-value pairs to the file specified by the existing GITHUB_OUTPUT variable using the following syntax: Bash: echo "MY_OUTPUT_NAME=my_value" >> $GITHUB_OUTPUT PowerShell: echo "MY_OUTPUT_NAME=my_value" >> $env:GITHUB_OUTPUT Key Requirements for Outputs: 1. Step ID: The step setting the output must have an id assigned to it [3][4][5]. 2. Referencing: You can reference the output in subsequent steps using the syntax ${{ steps.<step_id>.outputs.<output_name> }} [3][4][5]. 3. Job Outputs: If you need to pass this output to a different job, you must explicitly declare the output in your job's outputs block, mapping it from the step output [3][4]. Avoid manually defining GITHUB_OUTPUT in your env block, as it is a reserved system variable intended for internal runner communication [2].
Citations:
- 1: GITHUB_OUTPUT variables work from workflow echo but not by core API call actions/toolkit#1906
- 2: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands
- 3: https://fixdevs.com/blog/github-actions-env-var-between-steps/
- 4: https://matthewrich.com/2022/10/13/github-actions-job-output/
- 5: https://stackoverflow.com/questions/59191913/how-do-i-get-the-output-of-a-specific-step-in-github-actions
🌐 Web query:
``${{ env.GITHUB_OUTPUT }} step env override breaks outputs file path
💡 Result:
In GitHub Actions, GITHUB_OUTPUT is a runner-provided default environment variable that contains the path to the temp “outputs” file for the current step (the file you append name=value lines to). (docs.github.com)
When you add a step-level env like:
- name: my step
env:
GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }}you are overriding GITHUB_OUTPUT for that step (step env overrides job/workflow env with the same name). (docs.github.com)
And because ${{ env.* }} is the workflow env context (not the runner’s default env vars), ${{ env.GITHUB_OUTPUT }} is typically empty/undefined, so you replace the real temp-file path with an empty/wrong value—then >> $GITHUB_OUTPUT writes to the wrong place and outputs “break.” (docs.github.com)
Fix
- Don’t set/override
GITHUB_OUTPUTinenv:. - In scripts, reference the runner-provided path directly:
- id: s
run: echo "foo=bar" >> "$GITHUB_OUTPUT"(If you need your own variable, use a different name, e.g. OUTFILE, but don’t reuse GITHUB_OUTPUT.) (docs.github.com)
Citations:
- 1: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions?ref=scrum-bug&utm_source=openai
- 2: https://docs.github.com/actions/reference/accessing-contextual-information-about-workflow-runs?utm_source=openai
- 3: https://docs.github.com/actions/reference/accessing-contextual-information-about-workflow-runs?utm_source=openai
- 4: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions?ref=scrum-bug&utm_source=openai
Remove the GITHUB_OUTPUT step env override in vsix-smoke
The workflow redefines GITHUB_OUTPUT at the step level (env: GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }}), which can replace the runner-provided outputs file path; the Python step then writes outputs using os.environ["GITHUB_OUTPUT"] (lines ~105-108), so step outputs may not be captured correctly. Remove the step env entry.
Proposed fix
env:
RELEASE_TAG: ${{ github.event.release.tag_name }}
DISPATCH_VERSION: ${{ inputs.version }}
DISPATCH_FROM: ${{ inputs.from_version }}
SKIP_UPGRADE: ${{ inputs.skip_upgrade }}
- GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| RELEASE_TAG: ${{ github.event.release.tag_name }} | |
| DISPATCH_VERSION: ${{ inputs.version }} | |
| DISPATCH_FROM: ${{ inputs.from_version }} | |
| SKIP_UPGRADE: ${{ inputs.skip_upgrade }} | |
| GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }} | |
| run: | | |
| RELEASE_TAG: ${{ github.event.release.tag_name }} | |
| DISPATCH_VERSION: ${{ inputs.version }} | |
| DISPATCH_FROM: ${{ inputs.from_version }} | |
| SKIP_UPGRADE: ${{ inputs.skip_upgrade }} | |
| run: | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/vsix-smoke.yml around lines 63 - 68, The step-level
environment override for GITHUB_OUTPUT is shadowing the runner-provided outputs
file path; remove the env entry that sets GITHUB_OUTPUT in the vsix-smoke.yml
step so the Python code that reads os.environ["GITHUB_OUTPUT"] (the Python step
around lines where outputs are written) can write to the correct runner file.
Locate the step that declares env: GITHUB_OUTPUT: ${{ env.GITHUB_OUTPUT }} and
delete that env key (or move any needed values to a different variable), leaving
other env entries intact.
- Pin docker/setup-buildx-action to commit SHA per CodeQL supply-chain guidance (third-party action; relevant risk per tj-actions Mar 2025 incident). first-party actions/* keep @vn tag refs to match repo convention in tests.yml, ci.yml, deploy-docs-to-s3.yaml. - Arg parser now validates --version / --vsix-file / --from-version each got a value before assigning. Previously a bare --version with no follow-up would fail at runtime under set -u with an unbound-$2 message. Now exits with "Missing value for --X" + usage and exit 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets the workflow fire pre-merge whenever a PR touches docker-setup/** or the workflow file itself. Catches script/workflow regressions before they land on master, without waiting for the daily cron. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The PR-triggered run on 26518345615 surfaced a silent no-op in the
upgrade scenario:
==> Phase 2/2: upgrade to latest
==> Installing innoverio.vscode-dbt-power-user
Extension 'innoverio.vscode-dbt-power-user' v0.61.4 is already installed.
==> Verifying with --list-extensions
innoverio.vscode-dbt-power-user@0.61.4 ← still old, no upgrade
install + activation OK ← false pass
When --version=latest, the install command was passing a bare
extension id without a version qualifier. code-server treats that as
"any version present is fine" and no-ops when an older version is
already installed. The verifier then skipped the exact-version check
for the 'latest' sentinel, letting the false pass through.
Two changes:
- Up front, resolve 'latest' via OpenVSX /api/.../latest into a
concrete semver. From now on VERSION is always a real version string
before install_and_verify ever sees it.
- install_and_verify now refuses to install marketplace with
'latest' or empty arg, and always asserts exact-version match.
Verified locally with --from-version 0.61.4 (default --version):
Phase 1: 0.61.4 installed -> list shows 0.61.4
Phase 2: 'Updating the extension ... to the version 0.61.5'
Phase 2 list shows 0.61.5
83s end-to-end PASS
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
docker-setup/vsix-smoke.sh (1)
197-199:⚠️ Potential issue | 🟠 Major | ⚡ Quick winEscape
expected_versionbeforegrep -Eexact-match check.Line 198 uses unescaped
expected_versionin an ERE pattern, so version metacharacters can cause false positives and weaken the release gate assertion.Proposed fix
if [ -n "$expected_version" ]; then - if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${expected_version}$"; then + local expected_version_escaped + expected_version_escaped=$(printf '%s' "$expected_version" | sed 's/[][(){}.^$*+?|\\-]/\\&/g') + if ! echo "$listed" | grep -qE "^${EXTENSION_ID//./\\.}@${expected_version_escaped}$"; then red "FAIL: expected version $expected_version but installed list shows:" echo "$listed" | grep "^${EXTENSION_ID}" || true return 1 fi fi#!/bin/bash # Quick verification of regex-safety behavior (no repo access needed): set -euo pipefail EXT='innoverio.vscode-dbt-power-user' LISTED="${EXT}`@1.2.3`+meta" # Current-style unescaped check (can misbehave depending on metacharacters) if echo "$LISTED" | grep -qE "^${EXT//./\\.}`@1.2.3`+meta$"; then echo "Unescaped pattern matched" else echo "Unescaped pattern did not match" fi # Escaped check (deterministic literal match) VER_ESCAPED=$(printf '%s' '1.2.3+meta' | sed 's/[][(){}.^$*+?|\\-]/\\&/g') echo "$LISTED" | grep -qE "^${EXT//./\\.}@${VER_ESCAPED}$" && echo "Escaped pattern matched correctly"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docker-setup/vsix-smoke.sh` around lines 197 - 199, The grep -E pattern uses the unescaped variable expected_version which allows ERE metacharacters to change matching; before constructing the pattern used in grep -qE "^${EXTENSION_ID//./\\.}@${expected_version}$" escape regex metacharacters in expected_version (e.g. assign a new VER_ESCAPED by escaping characters like [](){}.^$*+?|\\- using sed or similar) and use VER_ESCAPED in the grep call, keeping the EXTENSION_ID//./\\. substitution as-is.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@docker-setup/vsix-smoke.sh`:
- Around line 197-199: The grep -E pattern uses the unescaped variable
expected_version which allows ERE metacharacters to change matching; before
constructing the pattern used in grep -qE
"^${EXTENSION_ID//./\\.}@${expected_version}$" escape regex metacharacters in
expected_version (e.g. assign a new VER_ESCAPED by escaping characters like
[](){}.^$*+?|\\- using sed or similar) and use VER_ESCAPED in the grep call,
keeping the EXTENSION_ID//./\\. substitution as-is.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7fd14043-b2cd-4767-b774-a3cd5181bec6
📒 Files selected for processing (1)
docker-setup/vsix-smoke.sh
Make the proof scannable. Script now writes a structured markdown report at $VSIX_SMOKE_REPORT (default /tmp/vsix-smoke-summary.md) that captures, for every phase: - Exact code-server command used - Raw install stdout (showing "Installing"/"Updating" lines) - Full --list-extensions --show-versions output after install - Exact-version match assertion - Activation-error scan result Workflow appends the report to GITHUB_STEP_SUMMARY (renders at the top of the run page) AND uploads it as a 30-day artifact, plus source-under-test metadata: trigger, ref, commit, PR number, and permalinks to the workflow and script files at the tested SHA. Anyone reviewing a run can now answer "what was tested and how" without scrolling raw logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…gate ci.yml's release-smoke job only fires on tag push, so the unreleased- VSIX install codepath had no observable CI evidence on a PR. Add pr-build-smoke to vsix-smoke.yml that: - Only runs on pull_request - Builds linux-x64 VSIX from PR head (= unreleased build) - Resolves the current marketplace baseline from OpenVSX - Runs the smoke with --vsix-file + --from-version BASELINE (same flags release-smoke uses on tag push) - Emits the standard evidence report to GITHUB_STEP_SUMMARY - Uploads the PR-built VSIX + report as 30-day / 7-day artifacts After this lands, every PR touching docker-setup or vsix-smoke.yml will show TWO CI checks: - "VSIX install + activation" (marketplace upgrade scenario) - "PR-built VSIX install + activation" (unreleased upgrade scenario) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
✅ Tests — All Passed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an automated install + activation smoke test for the dbt Power User VSIX, in two layers:
ci.yml— blocksvsce publishif a fresh install or the upgrade-from-previous-marketplace-version path fails on the locally-built VSIX. Stops bad releases at the door.vsix-smoke.yml— runs every morning at 06:00 UTC against the marketplace, plus post-publish, plus on-demand. Auto-resolves the previous version and exercises both fresh-install AND upgrade scenarios.Single bash script (
docker-setup/vsix-smoke.sh) is the source of truth — runs in both workflows and on Anas/Sai's laptops with the same flags. No drift between automation and manual testing.Files
docker-setup/vsix-smoke.sh--version), local-file mode (--vsix-file), upgrade mode (--from-version),--keep. Reusesdocker-setup/Dockerfileso Python + dbt-duckdb + the three dependency extensions are baked in..github/workflows/vsix-smoke.ymlworkflow_dispatch+release: published. Auto-resolves previous published version from OpenVSX so the upgrade scenario runs without manual input. Opens/comments avsix-smoke-labelled issue on scheduled failure..github/workflows/ci.ymlrelease-smokejob betweenbuildandrelease-vsstudio-marketplace. Packages linux-x64 VSIX, resolves marketplace baseline from OpenVSX, runs the script with--from-version BASELINE --vsix-file ./*.vsix. All 5 platform publishes are blocked if smoke fails.docker-setup/README.mdWhy this exists
tests.ymlruns against the source tree.ci.ymlpackages and publishes the VSIX but never installs it back to verify. Once a VSIX goes to the marketplace, nothing catches a regression where install or activation fails — until App Insights telemetry from real customers reports it. This closes that gap with two layers:Local use (for Anas/Sai)
Verified locally end-to-end
All three modes against the live OpenVSX marketplace and a real packaged VSIX:
--version 0.61.5(marketplace fresh install)--vsix-file vscode-dbt-power-user-linux-x64-0.61.5.vsix--from-version 0.61.4 --vsix-file ...0.61.5.vsix(upgrade)All three confirm correct version per
code-server --list-extensions --show-versionsand no activation errors in the boot log.What this does NOT cover (intentional v1 scope)
Test plan
--vsix-fileinstall — PASS in 27sbash -nandpython3 -c 'import yaml; yaml.safe_load(...)'syntax clean0 6 * * *run is green and includes the upgrade phaserelease-smokeruns before publish and blocks if brokenvscode-altimate-mcp-server🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Improvements