Skip to content

src: add an option to make compile cache path relative #58797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/api/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -3253,6 +3253,10 @@ added: v22.1.0
Enable the [module compile cache][] for the Node.js instance. See the documentation of
[module compile cache][] for details.

### `NODE_COMPILE_CACHE_PORTABLE=1`

When set to 1, the path for [module compile cache][] is considered relative.

### `NODE_DEBUG=module[,…]`

<!-- YAML
Expand Down
13 changes: 13 additions & 0 deletions doc/api/module.md
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,18 @@ the [`NODE_COMPILE_CACHE=dir`][] environment variable if it's set, or defaults
to `path.join(os.tmpdir(), 'node-compile-cache')` otherwise. To locate the compile cache
directory used by a running Node.js instance, use [`module.getCompileCacheDir()`][].

When portable mode is enabled, cache keys use paths relative to the compile cache directory.
This allows the cache to be reused after moving the project across directories etc.

It can be enabled via:

```js
module.enableCompileCache({ path: '...', portable: true });
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's likely worth expanding the example here a bit more to illustrate the difference between false and true. As it is, someone who isn't that familiar with how this is implemented likely wouldn't be able to determine exactly when to use this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, is this better? -

By default, cache keys are computed using the absolute paths of modules. This means the cache is not reusable if the project directory is moved or copied elsewhere.
To make the cache portable, relative path computation can be enabled for compile cache. This allows previously compiled modules to be reused across different directory locations as long as the relative layout remains the same.
There are two ways to enable the portable mode:

Using the portable option in module.enableCompileCache():

// Absolute paths (default): cache breaks if project is moved
module.enableCompileCache({ path: '.cache' });

// Relative paths (portable): cache works after moving project
module.enableCompileCache({ path: '.cache', portable: true });

Or by setting the environment variable: NODE_COMPILE_CACHE_PORTABLE=1

If a module's absolute path cannot be made relative to the cache directory, Node.js will fall back to using the absolute path.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, much better :-)


or [`NODE_COMPILE_CACHE_PORTABLE=1`][] environment variable.
If a relative path can't be computed, Node.js falls back to using the absolute path.

Currently when using the compile cache with [V8 JavaScript code coverage][], the
coverage being collected by V8 may be less precise in functions that are
deserialized from the code cache. It's recommended to turn this off when
Expand Down Expand Up @@ -1789,6 +1801,7 @@ returned object contains the following keys:
[`--import`]: cli.md#--importmodule
[`--require`]: cli.md#-r---require-module
[`NODE_COMPILE_CACHE=dir`]: cli.md#node_compile_cachedir
[`NODE_COMPILE_CACHE_PORTABLE=1`]: cli.md#node_compile_cache_portable1
[`NODE_DISABLE_COMPILE_CACHE=1`]: cli.md#node_disable_compile_cache1
[`NODE_V8_COVERAGE=dir`]: cli.md#node_v8_coveragedir
[`SourceMap`]: #class-modulesourcemap
Expand Down
8 changes: 8 additions & 0 deletions doc/node.1
Original file line number Diff line number Diff line change
Expand Up @@ -701,6 +701,14 @@ Enable the
.Sy module compile cache
for the Node.js instance.
.
.It Ev NODE_COMPILE_CACHE_PORTABLE
When set to '1' or 'true', the
.Sy module compile cache
uses relative paths when computing cache keys. This makes the cache
portable across directories etc.
This can be used in conjunction with .Ev NODE_COMPILE_CACHE
to enable on-disk caching with relative path resolution.
.
.It Ev NODE_DEBUG Ar modules...
Comma-separated list of core modules that should print debug information.
.
Expand Down
25 changes: 19 additions & 6 deletions lib/internal/modules/helpers.js
Original file line number Diff line number Diff line change
Expand Up @@ -371,18 +371,31 @@ function stringify(body) {
}

/**
* Enable on-disk compiled cache for all user modules being complied in the current Node.js instance
* Enable on-disk compiled cache for all user modules being compiled in the current Node.js instance
* after this method is called.
* If cacheDir is undefined, defaults to the NODE_MODULE_CACHE environment variable.
* If NODE_MODULE_CACHE isn't set, default to path.join(os.tmpdir(), 'node-compile-cache').
* @param {string|undefined} cacheDir
* This method accepts either:
* - A string `cacheDir`: the path to the cache directory.
* - An options object `{path?: string, portable?: boolean}`:
* - `path`: A string path to the cache directory.
* - `portable`: If `portable` is true, the cache directory will be considered relative. Defaults to false.
* If cache path is undefined, it defaults to the NODE_MODULE_CACHE environment variable.
* If `NODE_MODULE_CACHE` isn't set, it defaults to `path.join(os.tmpdir(), 'node-compile-cache')`.
* @param {string | { path?: string, portable?: boolean } | undefined} options
* @returns {{status: number, message?: string, directory?: string}}
*/
function enableCompileCache(cacheDir) {
function enableCompileCache(options) {
let cacheDir;
let portable = false;

if (typeof options === 'object' && options !== null) {
({ path: cacheDir, portable = false } = options);
} else {
cacheDir = options;
}
if (cacheDir === undefined) {
cacheDir = join(lazyTmpdir(), 'node-compile-cache');
}
const nativeResult = _enableCompileCache(cacheDir);
const nativeResult = _enableCompileCache(cacheDir, portable);
const result = { status: nativeResult[0] };
if (nativeResult[1]) {
result.message = nativeResult[1];
Expand Down
114 changes: 108 additions & 6 deletions src/compile_cache.cc
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,14 @@
#include <unistd.h> // getuid
#endif

#ifdef _WIN32
#include <windows.h>
#endif
namespace node {

#ifdef _WIN32
using fs::ConvertWideToUTF8;
#endif
using v8::Function;
using v8::Local;
using v8::Module;
Expand Down Expand Up @@ -223,13 +229,102 @@ void CompileCacheHandler::ReadCacheFile(CompileCacheEntry* entry) {
Debug(" success, size=%d\n", total_read);
}

#ifdef _WIN32
constexpr bool IsWindowsDeviceRoot(const char c) noexcept {
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
}
#endif

static std::string NormalisePath(std::string_view path) {
std::string normalised_string(path);
constexpr std::string_view file_scheme = "file://";
if (normalised_string.rfind(file_scheme, 0) == 0) {
normalised_string.erase(0, file_scheme.size());
}

#ifdef _WIN32
if (normalised_string.size() > 2 &&
IsWindowsDeviceRoot(normalised_string[0]) &&
normalised_string[1] == ':' &&
(normalised_string[2] == '/' || normalised_string[2] == '\\')) {
normalised_string[0] = ToLower(normalised_string[0]);
}
#endif
for (char& c : normalised_string) {
if (c == '\\') {
c = '/';
}
}

normalised_string = NormalizeString(normalised_string, false, "/");
return normalised_string;
}

// Check if a path looks like an absolute path or file URL.
static bool IsAbsoluteFilePath(std::string_view path) {
if (path.rfind("file://", 0) == 0) {
return true;
}
#ifdef _WIN32
if (path.size() > 2 && IsWindowsDeviceRoot(path[0]) &&
(path[1] == ':' && (path[2] == '/' || path[2] == '\\')))
return true;
if (path.size() > 1 && path[0] == '\\' && path[1] == '\\') return true;
#else
if (path.size() > 0 && path[0] == '/') return true;
#endif
return false;
}

static std::string GetRelativePath(std::string_view path,
std::string_view base) {
// On Windows, the native encoding is UTF-16, so we need to convert
// the paths to wide strings before using std::filesystem::path.
// On other platforms, std::filesystem::path can handle UTF-8 directly.
#ifdef _WIN32
std::wstring wpath = ConvertToWideString(std::string(path), CP_UTF8);
std::wstring wbase = ConvertToWideString(std::string(base), CP_UTF8);
std::filesystem::path relative =
std::filesystem::path(wpath).lexically_relative(
std::filesystem::path(wbase));
if (relative.empty()) {
return std::string();
}
std::string relative_path = ConvertWideToUTF8(relative.wstring());
return relative_path;
#else
std::filesystem::path relative =
std::filesystem::path(path).lexically_relative(
std::filesystem::path(base));
if (relative.empty()) {
return std::string();
}
return relative.generic_string();
#endif
}

CompileCacheEntry* CompileCacheHandler::GetOrInsert(Local<String> code,
Local<String> filename,
CachedCodeType type) {
DCHECK(!compile_cache_dir_.empty());

Utf8Value filename_utf8(isolate_, filename);
uint32_t key = GetCacheKey(filename_utf8.ToStringView(), type);
std::string file_path = filename_utf8.ToString();
// If the relative path is enabled, we try to use a relative path
// from the compile cache directory to the file path
if (portable_ && IsAbsoluteFilePath(file_path)) {
// Normalise the path to ensure it is consistent.
std::string normalised_file_path = NormalisePath(file_path);
std::string relative_path =
GetRelativePath(normalised_file_path, normalised_compile_cache_dir_);
if (!relative_path.empty()) {
file_path = relative_path;
Debug("[compile cache] using relative path %s from %s\n",
file_path.c_str(),
absolute_compile_cache_dir_.c_str());
}
}
uint32_t key = GetCacheKey(file_path, type);

// TODO(joyeecheung): don't encode this again into UTF8. If we read the
// UTF8 content on disk as raw buffer (from the JS layer, while watching out
Expand Down Expand Up @@ -500,11 +595,15 @@ CompileCacheHandler::CompileCacheHandler(Environment* env)
// - $NODE_VERSION-$ARCH-$CACHE_DATA_VERSION_TAG-$UID
// - $FILENAME_AND_MODULE_TYPE_HASH.cache: a hash of filename + module type
CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
const std::string& dir) {
const std::string& dir,
bool portable) {
std::string cache_tag = GetCacheVersionTag();
std::string absolute_cache_dir_base = PathResolve(env, {dir});
std::string cache_dir_with_tag =
absolute_cache_dir_base + kPathSeparator + cache_tag;
std::string base_dir = dir;
if (!portable) {
base_dir = PathResolve(env, {dir});
}

std::string cache_dir_with_tag = base_dir + kPathSeparator + cache_tag;
CompileCacheEnableResult result;
Debug("[compile cache] resolved path %s + %s -> %s\n",
dir,
Expand Down Expand Up @@ -546,8 +645,11 @@ CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
return result;
}

result.cache_directory = absolute_cache_dir_base;
result.cache_directory = base_dir;
compile_cache_dir_ = cache_dir_with_tag;
absolute_compile_cache_dir_ = PathResolve(env, {compile_cache_dir_});
portable_ = portable;
normalised_compile_cache_dir_ = NormalisePath(absolute_compile_cache_dir_);
result.status = CompileCacheEnableStatus::ENABLED;
return result;
}
Expand Down
7 changes: 6 additions & 1 deletion src/compile_cache.h
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ struct CompileCacheEnableResult {
class CompileCacheHandler {
public:
explicit CompileCacheHandler(Environment* env);
CompileCacheEnableResult Enable(Environment* env, const std::string& dir);
CompileCacheEnableResult Enable(Environment* env,
const std::string& dir,
bool portable);

void Persist();

Expand Down Expand Up @@ -103,6 +105,9 @@ class CompileCacheHandler {
bool is_debug_ = false;

std::string compile_cache_dir_;
std::string absolute_compile_cache_dir_;
std::string normalised_compile_cache_dir_;
bool portable_ = false;
std::unordered_map<uint32_t, std::unique_ptr<CompileCacheEntry>>
compiler_cache_store_;
};
Expand Down
16 changes: 13 additions & 3 deletions src/env.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1122,11 +1122,21 @@ void Environment::InitializeCompileCache() {
dir_from_env.empty()) {
return;
}
EnableCompileCache(dir_from_env);
std::string portable_env;
bool portable = credentials::SafeGetenv(
"NODE_COMPILE_CACHE_PORTABLE", &portable_env, this) &&
!portable_env.empty() &&
(portable_env == "1" || portable_env == "true");
if (portable) {
Debug(this,
DebugCategory::COMPILE_CACHE,
"[compile cache] using relative path\n");
}
EnableCompileCache(dir_from_env, portable);
}

CompileCacheEnableResult Environment::EnableCompileCache(
const std::string& cache_dir) {
const std::string& cache_dir, bool portable) {
CompileCacheEnableResult result;
std::string disable_env;
if (credentials::SafeGetenv(
Expand All @@ -1143,7 +1153,7 @@ CompileCacheEnableResult Environment::EnableCompileCache(
if (!compile_cache_handler_) {
std::unique_ptr<CompileCacheHandler> handler =
std::make_unique<CompileCacheHandler>(this);
result = handler->Enable(this, cache_dir);
result = handler->Enable(this, cache_dir, portable);
if (result.status == CompileCacheEnableStatus::ENABLED) {
compile_cache_handler_ = std::move(handler);
AtExit(
Expand Down
3 changes: 2 additions & 1 deletion src/env.h
Original file line number Diff line number Diff line change
Expand Up @@ -1022,7 +1022,8 @@ class Environment final : public MemoryRetainer {
void InitializeCompileCache();
// Enable built-in compile cache if it has not yet been enabled.
// The cache will be persisted to disk on exit.
CompileCacheEnableResult EnableCompileCache(const std::string& cache_dir);
CompileCacheEnableResult EnableCompileCache(const std::string& cache_dir,
bool portable);
void FlushCompileCache();

void RunAndClearNativeImmediates(bool only_refed = false);
Expand Down
2 changes: 2 additions & 0 deletions src/node_file.h
Original file line number Diff line number Diff line change
Expand Up @@ -531,6 +531,8 @@ int SyncCallAndThrowOnError(Environment* env,
FSReqWrapSync* req_wrap,
Func fn,
Args... args);

std::string ConvertWideToUTF8(const std::wstring& wstr);
} // namespace fs

} // namespace node
Expand Down
8 changes: 7 additions & 1 deletion src/node_modules.cc
Original file line number Diff line number Diff line change
Expand Up @@ -513,8 +513,14 @@ void EnableCompileCache(const FunctionCallbackInfo<Value>& args) {
THROW_ERR_INVALID_ARG_TYPE(env, "cacheDir should be a string");
return;
}

bool portable = false;
if (args.Length() > 1 && args[1]->IsTrue()) {
portable = true;
}

Utf8Value value(isolate, args[0]);
CompileCacheEnableResult result = env->EnableCompileCache(*value);
CompileCacheEnableResult result = env->EnableCompileCache(*value, portable);
Local<Value> values[3];
values[0] = v8::Integer::New(isolate, static_cast<uint8_t>(result.status));
if (ToV8Value(context, result.message).ToLocal(&values[1]) &&
Expand Down
2 changes: 1 addition & 1 deletion test/parallel/test-compile-cache-api-error.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ require('../common');
const { enableCompileCache } = require('module');
const assert = require('assert');

for (const invalid of [0, null, false, () => {}, {}, []]) {
for (const invalid of [0, null, false, 1, NaN, true, Symbol(0)]) {
assert.throws(() => enableCompileCache(invalid), { code: 'ERR_INVALID_ARG_TYPE' });
}
Loading
Loading