src: add an option to make compile cache path relative #58797
Conversation
nodejs-github-bot
added
c++
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #58797 +/- ##
==========================================
- Coverage 90.06% 90.03% -0.03%
==========================================
Files 645 648 +3
Lines 189930 191043 +1113
Branches 37222 37450 +228
==========================================
+ Hits 171062 172011 +949
- Misses 11571 11660 +89
- Partials 7297 7372 +75
🚀 New features to boost your workflow:
|
doc/api/cli.md
Outdated
| @@ -3181,6 +3181,10 @@ added: v22.1.0 | |||
| Enable the [module compile cache][] for the Node.js instance. See the documentation of | |||
| [module compile cache][] for details. | |||
|
|
|||
| ### `NODE_COMPILE_CACHE_RELATIVE_PATH=0` | |||
There was a problem hiding this comment.
I wonder if "relative" is more of an implementation detail, what we really want to allow is "portable" compile cache (i.e. you can move the code and the compile cache to a different base directory, the cache will still be hit as long as they maintain the same directory structure) So maybe NODE_COMPILE_CACHE_PORTABLE=0 would be more intuitive and we can explain about the directory structure with an example in the docs.
There was a problem hiding this comment.
Also, please add some documentation to the Module compile cache section in module.md as well.
There was a problem hiding this comment.
I concur.
Also, why is this not the default?
There was a problem hiding this comment.
I think it would be nice to eventually make it the default, though it would be more cautious to make it opt-in first - there are some points that might need some battle testing, for example, we are keying or computing by filenames but in practice not all "filenames" are filenames (source mapped files, synthetic IDs, and of course...monkey patchers that can use whatever for this), and we need to see whether this just leads to more cache misses or can lead to a stability issue.
src/compile_cache.cc
Outdated
| if (use_relative_ && IsAbsoluteFilePath(file_path)) { | ||
| // Normalise the paths to ensure they are consistent. | ||
| std::string normalised_file_path = NormalisePath(file_path); | ||
| std::string normalised_cache_dir = |
There was a problem hiding this comment.
This is something we can just compute once and store in the CompileCacheHandler instance. This is currently being repeatedly computed for every path match even though it's the same every time.
| // This tests NODE_COMPILE_CACHE works with the NODE_COMPILE_CACHE_RELATIVE_PATH | ||
| // environment variable. | ||
|
|
||
| require('../common'); |
There was a problem hiding this comment.
Can you add some tests for module.getCompileCacheDir() as well? I think it should return a relative path (from process.cwd()?) in that case.
mcollina
added
the
semver-minor
Adds an option (NODE_COMPILE_CACHE_PORTABLE) for the built-in compile cache to encode the hashes with relative file paths. On enabling the option, the source directory along with cache directory can be bundled and moved, and the cache continues to work. When enabled, paths encoded in hash are relative to compile cache directory.
Aditi-1400
force-pushed
the
compile-cache
branch
from
0d4fb87 to
b6eb597
Compare
| ```js | ||
| module.enableCompileCache({ path: '...', portable: true }); | ||
| ``` |
There was a problem hiding this comment.
It's likely worth expanding the example here a bit more to illustrate the difference between false and true. As it is, someone who isn't that familiar with how this is implemented likely wouldn't be able to determine exactly when to use this.
There was a problem hiding this comment.
Hmm, is this better? -
By default, cache keys are computed using the absolute paths of modules. This means the cache is not reusable if the project directory is moved or copied elsewhere.
To make the cache portable, relative path computation can be enabled for compile cache. This allows previously compiled modules to be reused across different directory locations as long as the relative layout remains the same.
There are two ways to enable the portable mode:
Using the portable option in module.enableCompileCache():
// Absolute paths (default): cache breaks if project is moved
module.enableCompileCache({ path: '.cache' });
// Relative paths (portable): cache works after moving project
module.enableCompileCache({ path: '.cache', portable: true });Or by setting the environment variable: NODE_COMPILE_CACHE_PORTABLE=1
If a module's absolute path cannot be made relative to the cache directory, Node.js will fall back to using the absolute path.
Adds an option (NODE_COMPILE_CACHE_RELATIVE_PATH) for the built-in compile cache to encode the hashes with relative file paths. On enabling the option,
the source directory along with cache directory can be bundled and moved, and the cache continues to work.
When enabled, paths encoded in hash are relative to compile cache directory.
Fixes: #58755
Refs: #52696
Thanks @joyeecheung for all the help :)