feat: Full VSMB support for Windows guest VMs

[Vigilans]

Background

NanaBox first attempted VSMB support in ab355ec (Add EnableHostDriverStore option for GPU-PV settings), which added a VirtualSmb share to expose the host's DriverStore to the guest. Shortly after, 3d0d857 removed it:

Remove Virtual SMB mode for EnableHostDriverStore option because there is no proper way to initialize Virtual SMB shares without Windows Container.

The core issue: a full Windows guest (unlike a container UVM) has no guest agent to initialize the SMB-over-VMBus redirector.

We've now solved this and every downstream problem. This issue documents the full technical journey and proposes integrating VSMB as a first-class NanaBox feature.

Solution summary

Guest-side VSMB initialization — Reverse-engineered from hcsshim's gcs-sidecar/vsmb.go: two FSCTLs (FSCTL_LMR_START_INSTANCE + FSCTL_LMR_BIND_TO_TRANSPORT) on the LanmanRedirector create a \Device\vmsmb instance and bind it to the VMBus transport. A keepalive handle must be held to prevent mrxsmb.sys BSOD.

Mounting via mklink — mklink /d C:\Shared "\\?\GLOBALROOT\Device\vmsmb\VSMB-{guid}\ShareName" makes shares accessible at local paths. Persists across reboots; resolves automatically once VSMB is initialized.

HostDriverStore + GPU-PV — GPU-PV drivers load at kernel boot, before VSMB is available. Solved by syncing HostDriverStore from VSMB to local disk post-boot, then restarting GPU-PV devices on change to reload drivers.

UAC elevation fix — mklink symlinks resolve to \Device\vmsmb\..., which has no volume mapping. consent.exe calls GetFinalPathNameByHandle(VOLUME_NAME_DOS) → ERROR_INVALID_FUNCTION → "Incorrect function" dialog. Reverse-engineering Windows Sandbox revealed it uses Bind Filter (BfSetupFilter with flag 0x20) instead of symlinks, making VSMB files appear as local volume paths. Verified working for UAC elevation.

Performance

Compared to Plan9Shares (the current file sharing mechanism): ~900 MB/s (VSMB) vs ~100 MB/s (Plan9), with excellent small-file performance as well.

Guest-side init service

Runs four stages at boot:

1. VSMB transport init — FSCTL_LMR_START_INSTANCE + FSCTL_LMR_BIND_TO_TRANSPORT (hardcoded VMBus constants, no config needed)
2. Bind Filter mount — read mount config from a VSMB share (now accessible after stage 1), call BfSetupFilter per share to create local mount points
3. HostDriverStore sync + GPU-PV restart — sync driver files to local disk via mounted share, restart GPU-PV devices on change
4. Keepalive — hold a VSMB share handle open to prevent VMBus channel teardown

Roadmap

Post	Topic
2	Guest-side VSMB initialization and mklink mounting
3	HostDriverStore synchronization and GPU-PV integration
4	UAC limitation, Sandbox reverse-engineering, and Bind Filter solution
5	Proposed feature design for NanaBox

[Vigilans]

Guest-side VSMB initialization and mklink mounting

Why a full Windows VM can't use VSMB out of the box

HCS creates the VirtualSmb VMBus channel when the VM starts — the host side is ready. But inside a full Windows guest, the SMB redirector doesn't know about this VMBus transport. In container UVMs, gcs-sidecar handles this initialization automatically. A full Windows VM has no such agent.

The initialization method

Reverse-engineered from hcsshim's gcs-sidecar/vsmb.go:VsmbMain(). The guest-side procedure is:

1. Ensure LanmanWorkstation service is running
2. Open \\?\GLOBALROOT\Device\LanmanRedirector
3. NtFsControlFile(FSCTL_LMR_START_INSTANCE) — creates a new redirector instance \Device\vmsmb with flags:
   • LMR_INSTANCE_FLAG_REGISTER_FILESYSTEM (0x2)
   • LMR_INSTANCE_FLAG_USE_CUSTOM_TRANSPORTS (0x4)
   • LMR_INSTANCE_FLAG_ALLOW_GUEST_AUTH (0x8)
   • LMR_INSTANCE_FLAG_SUPPORTS_DIRECTMAPPED_IO (0x10)
4. Open \\?\GLOBALROOT\Device\vmsmb
5. NtFsControlFile(FSCTL_LMR_BIND_TO_TRANSPORT) — binds to VMBus transport \Device\VMBus\{4d12e519-17a0-4ae4-8eaa-5270fc6abdb7}-{dcc079ae-60ba-4d07-847c-3493609c0870}-0000

After this, VSMB shares are accessible at \\?\GLOBALROOT\Device\vmsmb\VSMB-{dcc079ae-60ba-4d07-847c-3493609c0870}\<ShareName>.

Keepalive requirement

Without an open handle to any VSMB share, the VMBus channel's reference count drops to zero. The kernel tears down the channel, but mrxsmb.sys has pending async receives — they complete with STATUS_PIPE_BROKEN, hitting an assertion in VmbusRecvAsyncCompletion → BSOD SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (0x7E).

The fix mirrors gcs-sidecar's behavior: open a handle to any share and hold it for the process lifetime. This is why the init service must be a long-running process, not a one-shot script.

DirectMapped I/O

The LMR_INSTANCE_FLAG_SUPPORTS_DIRECTMAPPED_IO (0x10) flag enables zero-copy file access — the hypervisor maps host file pages directly into guest physical memory, bypassing VMBus round-trips. This is the primary reason VSMB achieves ~900 MB/s throughput.

Controlled by VirtualSmb.DirectFileMappingInMB in HCS schema (NanaBox defaults to 2048 MB). Per-share NoDirectmap option is available for incompatible backing filesystems (e.g. Azure Files).

Mounting via mklink

Once VSMB is initialized, shares need to be mapped to local paths. The simplest method:

mklink /d C:\Shared "\\?\GLOBALROOT\Device\vmsmb\VSMB-{dcc079ae-60ba-4d07-847c-3493609c0870}\Shared"

Properties:

• Persistent — symlink is stored in NTFS metadata, survives reboots
• Lazy resolution — symlink can be created before VSMB is initialized; it becomes functional once the init service starts
• Transparent — applications see C:\Shared as a regular directory

This makes VSMB shares usable for most applications. The limitation discovered later (UAC elevation failure) is covered in Post 4.

[Vigilans]

HostDriverStore synchronization and GPU-PV integration

The timing problem

GPU-PV devices appear in the guest as PCI\VEN_1414&DEV_008E (Microsoft Hyper-V paravirtualized GPU). The host GPU driver files must be present in C:\Windows\System32\HostDriverStore at kernel boot — Windows loads the driver during PnP enumeration, long before any user-mode service starts.

VSMB is only available after the init service runs in user-mode. So the HostDriverStore share cannot be used directly as the driver source at boot time.

Sync-on-boot approach

1. NanaBox exposes C:\Windows\System32\DriverStore as a read-only VSMB share (HostDriverStore)
2. After VSMB init + mount, the init service compares the VSMB share contents against the local C:\Windows\System32\HostDriverStore
3. Changed files are copied to the local directory
4. If driver files changed, GPU-PV devices are restarted (Disable-PnpDevice → Enable-PnpDevice) to reload drivers

Ignore rules prevent unnecessary restarts: .log, .etl, .tmp, .dmp files and directories like EeuDumps/Logs are excluded from change detection (but still synced).

TermService crash chain (known issue)

A separate issue when GPU-PV devices are in Error state (before driver sync). RDP's TermService uses Media Foundation for H.264 encoding, which enumerates D3D11 devices at startup. If the GPU-PV device is in Error state, loading the NVIDIA UMD (nvldumdx.dll) causes an access violation in ucrtbase.dll.

With AVC444ModePreferred=1 (from BetterRDP or similar), this is a hard requirement for GPU hardware encoding — no CPU fallback. TermService crashes on boot, auto-restarts, crashes again, potentially hundreds of times, and can escalate to BSOD.

Workaround: Set AVC444ModePreferred=0 (explicitly disable, not delete).

[Vigilans]

UAC limitation, Sandbox reverse-engineering, and Bind Filter solution

The problem

Running an exe that requires elevation (e.g. requestedExecutionLevel level="requireAdministrator" in manifest) from a VSMB share via mklink fails with "Incorrect function" (ERROR_INVALID_FUNCTION, error code 1). No consent dialog appears — the error fires before consent.exe gets to show anything.

The same exe works fine from local disk, from a regular SMB network share, and even from Windows Sandbox's VSMB-backed MappedFolders.

Root cause

Traced by calling GetFinalPathNameByHandle with different flags on VSMB files:

Flag	VSMB (via mklink)	Local file
VOLUME_NAME_DOS	❌ error=1	✅ \\?\C:\...
VOLUME_NAME_GUID	❌ error=3	✅ \\?\Volume{...}\...
VOLUME_NAME_NT	✅ \Device\vmsmb\VSMB-{...}\...	✅ \Device\HarddiskVolume3\...

\Device\vmsmb is not a volume — it has no mount point in the DOS namespace. When the UAC flow (ShellExecuteEx(runas) → AppInfo → consent.exe) calls GetFinalPathNameByHandle(VOLUME_NAME_DOS) to canonicalize the exe path, the call fails and AppInfo surfaces the error directly.

Regular SMB works because \Device\Mup\server\share resolves to \\?\UNC\server\share — a valid DOS path format.

How Windows Sandbox does it

Investigating a running Sandbox instance revealed the answer. VSMB MappedFolders in Sandbox resolve to \Device\HarddiskVolume2\... (the C: volume), not \Device\vmsmb\...:

C:\Shared\SomeInstaller.exe
  VOLUME_NAME_DOS  = \\?\C:\Shared\SomeInstaller.exe          ✅
  VOLUME_NAME_NT   = \Device\HarddiskVolume2\Shared\SomeInstaller.exe

Key observations from Sandbox:

• C:\Shared is not a symlink or reparse point — it's a plain directory
• Its volume serial number differs from C:\ (host filesystem serial vs local NTFS serial)
• fltmc instances shows bindflt (Bind Filter, altitude 409800) attached to both C: and \Device\vmsmb
• BfGetMappings on C:\ reveals the exact mappings:

flags=0x20  root: \Device\HarddiskVolume2\Shared  target: \Device\vmsmb\VSMB-{...}\c0562ff2...
flags=0x21  root: \Device\HarddiskVolume2\Scripts  target: \Device\vmsmb\VSMB-{...}\a83b1f09...

Sandbox uses BfSetupFilter (from bindfltapi.dll) with flag 0x20 for read-write and 0x21 for read-only mounts.

Flag identification

From the bindflt header definitions:

Value	Name	Effect
0x01	BINDFLT_FLAG_READ_ONLY_MAPPING	Read-only mount
0x08	BINDFLT_FLAG_REPARSE_ON_FILES	File-level reparse (does NOT rewrite device path)
0x20	BINDFLT_FLAG_CLOUD_FILES_ECPS	Full path virtualization — GetFinalPathNameByHandle reports host volume path

The critical difference: 0x08 (which hcsshim's go-winio uses as NO_MULTIPLE_TARGETS) only does I/O redirection. 0x20 makes the kernel report the bind target's volume path instead of the backing device path, which is exactly what consent.exe needs.

Verification on full VM

BfSetupFilter(0, 0x20, "C:\\BoundShared",
    "\\\\?\\GLOBALROOT\\Device\\vmsmb\\VSMB-{dcc079ae-...}\\Shared", NULL, 0)

Result:

C:\BoundShared\SomeInstaller.exe
  VOLUME_NAME_DOS  = \\?\C:\BoundShared\SomeInstaller.exe       ✅
  VOLUME_NAME_NT   = \Device\HarddiskVolume3\BoundShared\SomeInstaller.exe  ✅ (no longer \Device\vmsmb!)

UAC elevation on C:\BoundShared\SomeInstaller.exe shows a normal consent dialog. Problem solved.

bindflt vs mklink

	mklink /d	BfSetupFilter (0x20)
GetFinalPathNameByHandle(DOS)	❌ error=1	✅ \\?\C:\...
UAC elevation	❌ "Incorrect function"	✅ normal consent dialog
Persistence	NTFS metadata, survives reboot	Kernel memory, lost on reboot
Compatibility	Works with symlink targets	❌ target must be a plain directory, not a symlink

bindflt mounts must be recreated at every boot by the init service, but this is inherent — the VSMB transport itself also needs re-initialization at every boot.

[Vigilans]

Proposed feature design for NanaBox

Current state

All technical pieces are working as standalone scripts:

Script	Role
vsmb_init.py	Guest-side VSMB transport init + bindflt mount + HostDriverStore sync + keepalive
vsmb_service_install.ps1	NSSM service registration for vsmb_init

The goal is to integrate this into NanaBox as a proper feature.

Host-side: configuration changes

Proposed as incremental PRs:

Step 1 — Add VirtualSmbShares to the .7b config, matching HCS schema (Name + Path + ReadOnly):

{
  "VirtualSmbShares": [
    { "Name": "Shared", "Path": "C:\\HostShared" },
    { "Name": "Projects", "Path": "D:\\Projects", "ReadOnly": true }
  ]
}

Step 2 — Add Mounts array (parallel to VirtualSmbShares) and NanaBoxStateDirectory:

{
  "Mounts": [
    { "Type": "VirtualSmb", "Share": "Shared", "Target": "C:\\Shared" },
    { "Type": "VirtualSmb", "Share": "Projects", "Target": "C:\\Projects" }
  ],
  "NanaBoxStateDirectory": ".\\NanaBox State\\Windows"
}

NanaBoxStateDirectory is a host-side path (like GuestStateFile / RuntimeStateFile). NanaBox writes mounts.json (the Mounts array content) into this directory, then exposes it as a read-only VSMB share. Inside the guest, the init service mounts it to a fixed location (C:\ProgramData\NanaBox\State) and reads the config from there.

Guest-side: init service

Four-stage boot service as described in Post 1:

1. VSMB transport init — hardcoded constants, no config dependency
2. Bind Filter mount — read mounts.json from state directory share, BfSetupFilter(0, 0x20, ...) per VirtualSmb entry, create target directories if needed
3. HostDriverStore sync + GPU-PV — compare and copy, restart devices on change
4. Keepalive — hold share handle, run indefinitely

Current implementation is Python (vsmb_init.py). For production, a compiled binary (C/C++/Rust) would eliminate the Python dependency in the guest. The core logic is straightforward Win32 API calls (NtFsControlFile, BfSetupFilter, CreateFileW).

First-time deployment

The init tool must be present on the guest's local disk before it can initialize VSMB. Options:

• One-line installer: download and run a setup script from a NanaBox-provided URL (GitHub release / project domain), which downloads the init binary to C:\ProgramData\NanaBox and registers the service
• Manual install: user copies the tool into the guest and registers the service once

vsmb_init python example

vsmb_init.py

[MouriNaruto]

Thanks, but I still need to talk about something additional.

Actually, performance for 9p over Hyper-V Socket provided by HCS is not as bad as you think. Maybe the issue comes from Microsoft guys' WSL implementations.

Also, I have a demo video to show that performance, based on 9p client created by me. (https://github.com/ProjectMile/Mile.Cirno)

https://x.com/MouriNaruto/status/1958468737985589724

Kenji Mouri

[MouriNaruto]

Although hcsshim provide the code, we still need to know the minimum supported Windows build.

According to my reverse engineering experience, it seems Microsoft frequently changes some structures.

Kenji Mouri

[Vigilans]

Plan9:

VirtualSmb:

I'm surprised that I can get nearly native performance with VSMB using CrystalDiskMark. Maybe it's the 2048MB VSMB's DirectFileMapping come into effect.

The 100MB/s vs 900MB/s report in the first post simply derives from copying LLM's model weights.

[Vigilans]

Here's what I've been able to verify with Claude Code digging through hcsshim source and public docs:

Component	Minimum Build	Source
HCS API / VirtualSmb schema	17763 (Win10 1809 / Server 2019)	MS Docs HCS Overview: "Applies to: Windows Server 2019, Windows 10 October 2018 Update"
bindfltapi.dll (BfSetupFilter)	18362 (Win10 1903)	Nukem9/BindFltAPI: "expects Windows 10 RS6 or newer. Older editions have minor API changes"
LMR FSCTL structures (START_INSTANCE / BIND_TO_TRANSPORT)	❓ Unknown	gcs-sidecar code only added Nov 2025, no version guards in hcsshim
DirectFileMappingInMB	❓ Unknown	No version documentation in hcsshim or public docs

I've only verified the full flow (VSMB init + bindflt mount) on a Windows 11 25H2 (Build 26200.8037) guest with a Windows 11 24H2 (Build 26100.4652) host. Testing on older builds (especially 1809/1903) would help establish the actual minimum.

Worth noting: the gcs-sidecar VSMB init code (8a611dfa, 2025-11-11) was added after NanaBox removed VSMB support (3d0d857, 2025-07-05). The reference implementation we reverse-engineered from simply didn't exist at the time.

[MouriNaruto]

At least 19041 has DirectFileMappingInMB.

But I recall some LMR FSCTL structures, which may have changed a lot between 19041 and 22xxx.

But I think I can test some information about the guest initialization you provided.

Kenji Mouri

[MouriNaruto]

Also, I forgot to say that I don't think it's good for posting LLM-generated content directly.

According to https://github.com/M2Team/NanaZip/blob/main/CONTRIBUTING.md, people will know my opinion about the LLM.

It's OK to use AI/LLM tools to help you write code, but you must ensure that the final commit is written and reviewed by yourself, following our coding style and conventions.

Kenji Mouri

[Vigilans]

Also, I forgot to say that I don't think it's good for posting LLM-generated content directly.

According to https://github.com/M2Team/NanaZip/blob/main/CONTRIBUTING.md, people will know my opinion about the LLM.

It's OK to use AI/LLM tools to help you write code, but you must ensure that the final commit is written and reviewed by yourself, following our coding style and conventions.

Kenji Mouri

While it can be reference to demonstrate your opinion towards LLM generated content. It should be noted that:

1. This is issue comment, not code/final commit, this contributing guide does not directly apply.
2. I explicitly state that this is LLM generated research output, since LLM is suitable here to extract information from web and codebase.
3. I reviewed each reference of the component. BindFltAPI repository is provided by me during LLM's search for windows build.
4. PR Add VirtualSmbShares configuration support #50 is reviewed, organized and committed into 4 commits by myself.

[Vigilans]

Resolved by 86786da

[MouriNaruto]

Continue for some unfinished things.

I think for the small Virtual SMB configuration tool, I will make a standalone project which will need to be set co-authored-by you at least several times. Because I don‘t think we should limited that only to NanaBox, maybe for Hyper-V researchers will be much happy for playing Windows Sandbox, lol.

I won't close this issue until I have done that tool.

Kenji Mouri