Skip to content

Remove various npm hacks, switch to yarn for package management - #24

Merged
lornajane merged 10 commits into
OAI:mainfrom
handrews:yarn-package-management
Aug 20, 2026
Merged

Remove various npm hacks, switch to yarn for package management#24
lornajane merged 10 commits into
OAI:mainfrom
handrews:yarn-package-management

Conversation

@handrews

Copy link
Copy Markdown
Member

This new system was rapidly becoming crufty with npm-specific workarounds (PRs #15 and #19), which led to all current dependabot PRs failing. GPT 5.5 "High" was suggesting yet another even more rickety workaround, so i had GPT Sol 5.6 "Extra High" review the whole history and propose a simplification. Three stacked workarounds is at least two too many; this PR is built entirely with supported yarn functionality instead.

GPT Sol initially proposed pnpm, but I asked it to assess all available package management systems and it changed that suggestion to yarn as it is more actively supported by GitHub (pnpm would have to be pinned to an older version, and other package managers were more change than was needed- yarn is long-established and well-supported).

I also had it test the fix in OpenAPI-Specification with the branch for PR OAI/OpenAPI-Specification#5499 which caught an issue specific to schema testing. The fix is in this PR; no changes to 5499 are required.


GPT's initial assessment (including pnpm); the "proposed workflow" was the latest hack proposed by GPT 5.5 that I stopped it from finishing:

Yes. The lockfile snapshot subsystem is fighting npm’s package model, and the proposed Dependabot workflow would compound that mistake. I recommend replacing it rather than automating it further.

What Happened

The history shows a clear progression:

  • 7c57c8f manually repaired an npm lockfile that macOS generated incompletely.
  • 90ec2ee correctly stopped asserting exact incidental transitive versions after Dependabot changed them.
  • e467f7a added a 5,894-line copy of package-lock.json because npm omits a dependency’s lockfile.
  • f1940b8 added the copy command.
  • 435db89 made synchronization a test prerequisite.
  • The proposed workflow would now modify Dependabot branches to maintain that copy.

The original problematic trees have since disappeared: Vitest’s Rolldown update removed the WASI dependency requiring @emnapi/*, and publishing updates removed the proxy-agent tree.

More importantly, oai-spec-sync-lockfile is not structurally safe. It copies root-level package paths from build-infra’s lockfile into the consumer lockfile. Those paths describe build-infra as a root project, not as a dependency, and could overwrite versions selected for a consumer’s own dependencies.

npm deliberately ignores package-lock.json outside the root project. npm-shrinkwrap.json would once have been the native alternative, but npm 12 now ignores shipped shrinkwraps and recommends bundled dependencies instead. Bundling the entire dependency tree would be bulky and particularly awkward for platform-specific packages. npm lockfile documentation

Recommended Direction

Migrate build-infra and its consumers to pnpm 10, while continuing to install build-infra directly from Git.

This provides:

  • One authoritative pnpm-lock.yaml per repository.
  • Normal Dependabot PRs with no follow-up commits.
  • Git dependencies still locked to an exact commit.
  • Cross-platform optional dependencies represented correctly.
  • Frozen CI installs with pnpm install --frozen-lockfile.
  • No package publication or registry authentication.

GitHub currently supports pnpm lockfiles through pnpm 10, using the existing package-ecosystem: npm Dependabot setting. GitHub Dependabot support

I tested this locally with pnpm 10.28.0:

  • The historical Rolldown 1.1.4 tree recorded the WASI package and both @emnapi/core and @emnapi/runtime.
  • A Git-installed build-infra dependency resolved to an exact commit.
  • Its public test/schema exports imported correctly.
  • All 29 build-infra tests passed after importing the current npm lockfile into pnpm.

pnpm also has an explicit supportedArchitectures facility if the project later needs to install optional artifacts for more than the current platform. pnpm architecture settings

Proposed Migration

On a new build-infra branch:

  1. Import package-lock.json into a pnpm 10 lockfile.
  2. Pin the pnpm version and Node 24 toolchain.
  3. Pin build-infra’s direct runtime dependencies exactly. This package is a toolchain, so consumers should receive the versions tested in build-infra.
  4. Change CI to frozen pnpm installs.
  5. Remove oai-spec-sync-lockfile, the copied lockfile, synchronization scripts, checks, tests, and documentation.
  6. Do not commit the currently untracked Dependabot snapshot workflow.
  7. Add a real integration test that installs build-infra as a Git dependency, rather than constructing node_modules by hand.
  8. Migrate consumers individually and run their existing build/test workflows before merging.

The lower-change alternative is to remain on npm, remove the snapshot system, and require consumer lockfile updates to be generated on Linux. That leaves a platform-specific update procedure, so I consider pnpm the cleaner long-term choice for a volunteer-maintained system.


GPT's report of what it did, after switching the plan to use yarn instead of pnpm:

Implemented the Yarn migration across the requested repositories.

Branches

  • build-infra/yarn-package-management at efada0c
  • OpenAPI-Specification/yarn-package-management at 12510211
  • sig-lifecycle/yarn-package-management at e08cfd4
  • sig-security/shared-infra at 0e4e905

The migration uses Yarn 4.18 with nodeLinker: node-modules, immutable CI installs, exact shared-tool versions, approved Git repositories, and an explicit Puppeteer build allowlist. The npm lockfile snapshot, merge command, tests, and draft Dependabot workaround are gone. Documentation is updated in README.md and CONTRIBUTING.md.

During integration, I also fixed two issues exposed by real consumers:

  • Yarn needed Node launchers around Bash-backed package commands.
  • Hyperjump registration and coverage required a shared runtime instance.

The self-contained Git consumer test now covers Git installation, exact commit locking, yarn up -R, immutable reinstalls, hardened-mode CI, unmerged provider commits, package exports, and Bash commands.

Verification

  • build-infra: 10 test files, 28 tests passed.
  • OpenAPI-Specification main: 8 tests, Markdown/link validation, and published HTML build passed.
  • OpenAPI dev plus fix/schema-test-build-infra: 28 tests passed with BASE=dev.
  • sig-lifecycle: schema tests and full source build passed.
  • sig-security: Markdown/link validation and source build passed.
  • Final immutable installs passed in every repository.
  • Every new commit has the GitHub co-author trailer.
  • All consumer locks resolve build-infra commit efada0c867950970e88a8c6c8dd6d03f69bd1f38.

Nothing was pushed. Push the build-infra branch first so efada0c is reachable, then push the consumer branches. If the build-infra PR is squash- or rebase-merged, refresh the consumer locks with yarn up -R @oai/build-infra; a normal merge preserves the currently locked commit. Arazzo was left unchanged in this batch.

handrews and others added 10 commits August 18, 2026 11:15
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Comment thread .github/workflows/test.yml
Comment thread .github/workflows/test.yml
@lornajane
lornajane requested review from baywet and lornajane August 20, 2026 16:19

@lornajane lornajane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable, let's merge

@lornajane
lornajane merged commit b9b8777 into OAI:main Aug 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants