Skip to content

Fix lore-revision vfs feature: forward-port projfs/serve.rs to the public crate APIs#83

Open
kelvincai522 wants to merge 1 commit into
EpicGames:mainfrom
kelvincai522:vfs-fix-main
Open

Fix lore-revision vfs feature: forward-port projfs/serve.rs to the public crate APIs#83
kelvincai522 wants to merge 1 commit into
EpicGames:mainfrom
kelvincai522:vfs-fix-main

Conversation

@kelvincai522

Copy link
Copy Markdown

Problem

lore-revision/src/projfs/serve.rs (the Windows ProjFS provider behind lore clone --virtual) was imported as an unmodified "Initial code copy from private repo" and has never compiled in the open-source tree. Building with the vfs feature fails on Windows:

cargo build --features lore-revision/vfs

with 8 errors, because the surrounding public crates moved on while serve.rs stayed frozen.

Fix

Adapt serve.rs to the current public APIs (no behavior change):

  • From<GUID> for Context violated the orphan rule (both types are foreign now that Context lives in lore_storage) → inline the existing transmute_copy into context_from_guid().
  • VirtualLayer.module.path is Option<PathBuf>.as_deref().expect() (×4).
  • serve() returns () → handle repository.require_path() inline instead of ?.
  • require_path()? in the i32/HRESULT ProjFS callback → map_err to a named E_INVALIDARG.
  • private immutable::ReadOptionslore_storage::options::ReadOptions.
  • drop the now-unused DOT_URC import.

Verification

Built --release --features lore-revision/vfs on Windows (MSVC). lore clone --virtual then serves a ProjFS mount that hydrates lazily: a ~15.7 GB logical tree materialized at ~0.4 MB on disk, and opening a single file pulled down only that file (placeholder → hydrated, ~110 ms).

Scope: Windows/ProjFS only (#[cfg(all(target_family = "windows", feature = "vfs"))]).

The `vfs` feature (lore-revision/vfs) does not compile in the open-source
tree: lore-revision/src/projfs/serve.rs was imported as an unmodified copy
from the private repo ("Initial code copy from private repo") and never built
against the public crate APIs, so `--features lore-revision/vfs` fails on
Windows with 8 errors. Adapt serve.rs to the current public APIs:

- From<GUID> for Context violated the orphan rule (both types are foreign now
  that Context lives in lore_storage) -> inline the transmute_copy into the
  existing context_from_guid() helper.
- VirtualLayer module.path is now Option<PathBuf> -> .as_deref().expect() (x4).
- serve() returns () -> handle repository.require_path() inline instead of `?`.
- require_path()? inside the i32/HRESULT ProjFS callback -> map_err to a named
  E_INVALIDARG (no implicit InvalidArguments -> i32 conversion exists).
- immutable::ReadOptions re-export is private -> lore_storage::options::ReadOptions.
- Drop the now-unused crate::repository::DOT_URC import.

With this, `cargo build --release --features lore-revision/vfs` succeeds and
`lore clone --virtual` serves a ProjFS mount that hydrates files lazily
(verified: a 15.7 GB logical tree materialized at ~0.4 MB on disk; opening a
file pulls down only that file).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Kelvin Cai <kelvin.cai@impossible.place>
@kelvincai522

Copy link
Copy Markdown
Author

Sharing some follow-on VFS work that builds on this PR — for context only, not to expand this PR (it should stay the focused compile-fix).

On top of this forward-port I've prototyped a FUSE backend for lore clone --virtual on Linux (the counterpart to the ProjFS provider). It's on the fuse-vfs branch of my fork; design notes + findings are in docs/fuse-vfs.md there.

FUSE provider (lore-revision/src/fuse/serve.rs): overlay-style — a backing fd opened before mount2, backing-first passthrough with virtual-tree fallback + hydrate-on-read, non-blocking async-spawn callbacks, and write-through materialization. Lazy reads + write-through verified on both Linux/FUSE and Windows/ProjFS (≈15 GB logical tree served at a few MB on disk; opening a file pulls only that file, and a read-back md5 is identical across platforms). One structural note: because FUSE replaces the directory (vs ProjFS overlaying it), the in-mount .lore store gets shadowed and lore's synchronous store reads re-enter the mount; serving the immutable store from outside the mount (--use-shared-store) avoids it — which also matches the roadmap's "serve from the shared store" intent. Details in the doc.

clone --virtual base fix (a behavior change — flagging for your view): the virtual path returns before store_current_anchor, so the local branch stays at revision 0; editing in the mount + stage/commit then diverges from the remote and push is refused. Setting the anchor to the served revision before serving makes stop → stage <file> → commit → push work as a fast-forward, with no sync. If revision-0 was intentional pending the VFS/locking work, happy to drop or rework this.

Two things that look like core-VFS / roadmap territory rather than provider bugs (same on ProjFS and FUSE):

  1. Commit/push while serving blocks on the exclusive repository flock — the serving process is both the file provider and the lock-holder, so a concurrent stage/commit can't acquire it. Seems to map to the "scalable file locking" roadmap item.
  2. No in-place re-serve — re-running clone --virtual on an existing clone errors ("Repository already exist"), and there's no mount/serve command, so serving is effectively one-shot per clone (a mount-lifecycle gap).

Once this PR lands I can rebase the FUSE work onto main (so the projfs forward-port drops out of its diff) and open it as a separate PR. Mainly wanted to surface the direction and get your read on the avoid-sync behavior question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant