experiment: --override flag to support multiple package versions

[rvanasa]

Work in progress!

In order to release the new Motoko base library under the same package name, it's important that we provide a way for Mops and other package managers to be able to use more than one version of a package within a project.

This PR implements a command-line flag (currently --override, subject to change) which makes it possible to redirect imports within a specific directory. This way, package managers can use previous base library (or other package) versions of transitive dependencies with breaking changes that would otherwise result in compilation errors inside of imported packages.

Due to of a large number of tradeoffs for different approaches (e.g. npm-style dependency solving, Deno- and Go-style explicit imports, PureScript-style package sets), this design makes it so that Motoko package managers can fully decide how to resolve dependencies rather than building an opinionated solution into the compiler.

Here is an example using Mops:

[dependencies]
base = "1.0.0" # New base library
vector = "0.4.1" # Depends on original base library (0.13.0)

import Queue "mo:base/Queue"; // From new base library
import Vector "mo:vector"; // Uses original base library within the imported package

actor {
  ...
}

Because the new base library includes breaking changes, importing mo:vector would normally cause unfixable compiler errors in this situation. This PR makes it so that Mops can pass a compiler flag via the mops sources command to rewrite the mo:base imports within the vector package:

--package base .mops/[email protected]/src
--package [email protected] .mops/[email protected]/src
--package vector .mops/[email protected]/src

--override .mops/[email protected] base [email protected]

Another benefit of this approach is that it becomes possible to protect packages from accessing dependencies which are not specified in the corresponding mops.toml file. For example:

--package base/v0 .mops/[email protected]/src
--package base/v1 .mops/[email protected]/src
--package vector/v0 .mops/[email protected]/src

# Top-level dependencies
--override . base base/v1
--override . vector vector/v1

# `vector` dependencies
--override .mops/[email protected] base base/v0

When importing a package which isn't specified in the project's mops.toml file, this convention would produce a "package not defined" compilation error to prevent accidentally importing a transitive dependency.

Note that package names containing the / character are inaccessible using Motoko's import syntax. We can choose whether it's possible to directly access the package versions using the @ symbol instead of / as a convention. This gives package managers more flexibility around whether explicit package versions are globally available.

Below is a hypothetical example for Go-style imports, configuring defaults with --override:

--package base@0 .mops/[email protected]/src
--package base@1 .mops/[email protected]/src
--package vector@0 .mops/[email protected]/src

# Default versions for project
--override . base base@1
--override . vector vector@1

# Default versions for `vector` package
--override .mops/[email protected] base base@0

import Queue "mo:base@1/Queue"; // Import an explicit major version (1.x)

The idea is that --override can support all of the above patterns in an unopinionated way depending on how Mops and other package managers choose to pass the flags in the sources command.

Progress:

• Set up command-line flag
• Use package overrides
• Include a way to override imports outside of a package (currently using .)
• Add error/warning for unknown package in --override
• Add moc.js configuration logic
• Add tests

[github-actions]

Comparing from a70cee8 to 7780d91:
The produced WebAssembly code seems to be completely unchanged.

[Gekctek]

Is this still something that is being evaluated?
Not sure since base went to core, but as a Motoko library developer, something like this is desperately needed
With MOPS, having a growing tree of dependencies with all different versions is breaking all of my code. Mainly dealing with major version/breaking changes. Version pinning only handles a subset of the issues because I don't have full control over others' libraries and it would require everyone to pin every dependency they have
I don't really care what the solution is, if this or something else, but I am hitting a wall