RFC: Env flag for npm packages to conditionally load WASM binaries instead of native binaries

[EricSimons]

UPDATE

This proposal has changed and instead relies on import/export maps, and has officially landed in Node.js core.

I have kept the original (now outdated) proposal below for future reference:

view original (outdated) proposal

The problem

Web development toolchains are increasingly including binaries for compilers/bundlers as they provide substantially faster performance than JS equivalents. The ecosystem is also increasingly adopting WebAssembly for distributing & executing binaries as it provides far greater security guarantees, are directly inspectable, and "build once and run everywhere".

Today esbuild and swc are the most popular of these new tools, and both release a WASM version in a separate package in lockstep with their native binaries (esbuild-wasm and @swc/wasm). As such, consumers of these packages need to determine at runtime whether they should use the native binaries or the WASM binary.

Parcel v2 currently uses a custom flag process.env.PARCEL_SWC_WASM that indicates it should require the WASM binary and not the platform specific variant. Other toolchains that have adopted swc/esbuild (Vite, Next.js, Snowpack, etc) need a similar way of determining this as well, and custom flags for each package is not ideal.

Proposed Solution: process.env.REQUIRES_WASM

I propose we standardize on a new flag, process.env.REQUIRES_WASM, that instructs npm packages to use WASM binaries instead of unsandboxed platform specific binaries.

This is important for environments that require high security guarantees (i.e. targets of supply chain attacks like enterprises and OSS devs), WebAssembly based containers like WebContainer or kubelet, in-browser playgrounds, and other cases where native binaries cannot/should not be executed.

Indicates WASM is required, not just preferred.

Importantly, this flag does not indicate that the host environment prefers WASM. Instead it is an instruction that the host requires WASM binaries and cannot (or does not) allow execution of unsandboxed native binaries.

Intended functionality

If process.env.REQUIRES_WASM is true at runtime, npm packages should import WASM binaries (or a JS file generated via emscripten, etc) instead of corresponding platform specific binaries.

Should a package not have a WASM binary available (and/or can't polyfill the functionality otherwise), the program should throw an error if process.env.REQUIRES_WASM is present.

[sokra]

I would propose to use an exports/imports field condition instead of a (runtime) environment variable.

An exports field condition has the advantage that it's statically known instead of at runtime. This allows to avoid using import() and instead import ... from "..." can be used.

see also

• https://webpack.js.org/guides/package-exports/
• https://nodejs.org/api/cli.html#cli_c_condition_conditions_condition

[matthewp]

Great idea!

Another angle to this problem is: many of these tools rely on the postinstall hook of npm. One common problem they encounter is that if you run npm install --ignore-scripts, they postinstall hook won't be called and the native binaries won't be installed. Some times users have this on without knowing it (such as in an .npmrc) and file issues about it.

If we could convince npm to make this option the default it would push these types of tools to prioritize wasm and make it the primary way to use their tools.

[evanw]

(I'm commenting here because I was pinged on Twitter as the author of esbuild, but I don't have any stake in this PR)

If process.env.REQUIRES_WASM is true at runtime, npm packages should import WASM binaries (or a JS file generated via emscripten, etc) instead of corresponding platform specific binaries.

Each binary executable is many megabytes. Solutions which require shipping binary executables for multiple platforms in the same package are very undesirable because then the weight of the package (installation time and disk size) would be huge. The reason esbuild's WebAssembly package is separate from the native package is to reduce the weight of the package. Having that be a run-time switch instead of an install-time switch would increase the weight of the package by a lot, and is likely not something I would ever consider doing for esbuild.

If we could convince npm to make this option the default it would push these types of tools to prioritize wasm and make it the primary way to use their tools.

This isn't ever going to happen for esbuild. WASM has a number of major drawbacks and I would never want it to be the primary way to interact with esbuild. If there were no drawbacks, there wouldn't be a reason to ship native executables. Some of the drawbacks:

• AFAIK there's still no way to get node to cache the translated WASM assembly code so every time you run it, it's recompiled from scratch. This can be a huge performance penalty over native.

• Node has a pretty bad bug where the WASM compilation must finish before node is allowed to exit: tiered WebAssembly compilation broken nodejs/node#36616. This causes short runs of large WASM programs (e.g. CLI scenarios) to take way, way longer than they should (up to 40x longer).

• Depending on the scenario, WASM's 2gb address size limit can be prohibitive and lead to crashes or large slowdowns as available memory gets low. Native binaries do not have this restriction and generally have a better chance of working correctly.

In esbuild's case WASM can easily be 10x slower than native. WASM will never be the default implementation.

Another angle to this problem is: many of these tools rely on the postinstall hook of npm. One common problem they encounter is that if you run npm install --ignore-scripts, they postinstall hook won't be called and the native binaries won't be installed. Some times users have this on without knowing it (such as in an .npmrc) and file issues about it.

I'm currently experimenting with an alternative installation strategy for esbuild that uses optionalDependencies instead of a postinstall script. If it works, that would solve this particular problem without needing any new environment flags and without relying on WASM. See this post for more information: evanw/esbuild#789 (comment). The only drawback is that optionalDependencies doesn't provide a way to have a default "fallback" dependency if none of the optional dependencies apply. There's another proposal from Yarn that attempts to address that need: yarnpkg/berry#2751.

[matthewp]

Thanks @evanw, the optionalDependencies trick sounds interesting. Am I reading this right that it would install the wasm version for webcontainer (which needs to run in the browser)?

[matthewp]

Another idea, probably for the Node.js security team is, if there was a way to disable 3rd party packages from launching subprocesses that would help discourage the install-binary patterns. Deno has this with --allow-run (this applies to all code, not just 3rd party), and the result is that you see mostly wasm used there.

[matthewp]

@EricSimons Given the discussion here and elsewhere, I think standardizing on an export map condition could be the best immediate-impact solution here. stackblitz could run install with --condition=requires_wasm (bikeshed on name) and everything should just work.

[EricSimons]

@sokra @matthewp I think export maps definitely could be a good solution here. It does diverge from how folks are currently handling this issue atm, as both Next and Parcel load the appropriate binary using runtime conditions. I'd be curious to see if these teams would have any reservations on this approach (I imagine Next would be on board as Tobias works on it, but also cc'ing @devongovett for the Parcel team's take).

[evanw]

Thanks @evanw, the optionalDependencies trick sounds interesting. Am I reading this right that it would install the wasm version for webcontainer (which needs to run in the browser)?

No, it has nothing to do with WASM. I just mentioned it since it's a way of avoiding problems with --ignore-scripts and postinstall scripts. If you need to run esbuild in the browser, then you'll need to explicitly install the WASM version of esbuild instead of the native version.

[EricSimons]

@evanw thanks for taking the time to respond here- some extra perspective/info that might be relevant:

Solutions which require shipping binary executables for multiple platforms in the same package are very undesirable because then the weight of the package (installation time and disk size) would be huge

AFAIK many npm packages are starting to ship included binaries because it removes the CPU time (and often errors) that happen on install and during CI jobs. It also eliminates the need for postinstall, which is ideal from a security POV. I do understand why you don't want to go this route for esbuild though.

Depending on the scenario, WASM's 2gb address size limit can be prohibitive and lead to crashes or large slowdowns as available memory gets low. Native binaries do not have this restriction and generally have a better chance of working correctly.

FYI that this is now 4gb as of May 2020, and wasm64 just hit stage 3 in June which will remove this limit. v8 has already begun work on implementation.

In esbuild's case WASM can easily be 10x slower than native. WASM will never be the default implementation.

As you pointed out in a previous issue, this is in part because esbuild-wasm can currently only run single threaded whereas the native variant has full multithreading. WebAssembly Threads are now available in Chrome/Edge/Firefox and enables high performance multithreading. We use this for WebContainers and the performance has been able to exceed local execution speeds. This might be worth exploring for esbuild-wasm.

Big fan of esbuild and would love to see/help it adopt the latest improvements for the webassembly version!

[devongovett]

Isn't this what the "browser" field in package.json is for? It's widely supported by pretty much every bundler, and allows substituting the whole module or specific files for the browser. We already use it for some of our native packages, and I think that was the plan for the rest as well. The environment variable we currently use for other packages is just a temporary hack I believe.

[EricSimons]

@devongovett for bundling use cases definitely, but for environments that provide full Node runtimes requiring strong security guarantees (i.e. targets of supply chain attacks like enterprises and OSS devs, WebAssembly based containers like WebContainer or kubelet, etc) having the browser field affect Node's real resolution algorithm could be problematic for a few reasons:

• Since the browser field is mostly used for bundling use cases, they often specify overrides that patch Node.js code in a way where it would run in a browser but not in Node
• Ignoring the above, the host env would have to modify Node's resolution algorithm to account for this, which isn't ideal

So IMO given the advent and growth of WASM adoption in npm packages for local envs, It seems there's a bit of a middle ground we need to accomodate that the browser doesn't seem well suited for.

[devongovett]

Hmm interesting. Unfortunately, I think WASM has a couple drawbacks over native packages, which is why we distribute native binaries by default.

• Like @evanw said, WASM is significantly slower in our experience. For our case, this is not due to multi-threading. In Parcel, threading is handled in JS using Node's worker_threads, and the native module runs within those workers (so, single threaded). We've seen WASM run up to 30x slower than native in some cases. In particular, the inability to pass buffers between JS and WASM without copying is a huge problem for us.
• Some features currently don't work in WASM, e.g. anything involving the filesystem. For our browser REPL, we've reimplemented some of these in JS using a Memory FS layer, but that wouldn't really be the expected way to use Parcel in Node.

For those reasons, I don't think WASM would be the default way to use Parcel, but perhaps could be opt-in. The first issue is fine if you accept the perf hit, but the second I'm not sure how to solve.