Add Xbox 360 linker support (fake mspdb PDB interface) by freeqaz · Pull Request #110 · decompals/wibo

freeqaz · 2026-02-11T20:17:17Z

Motivation

The Dance Central 3 decompilation project needs to link decomp .obj files with original object code using the MSVC Xbox 360 linker (link.exe 16.00.11886.00). Previously this required wine, which is a heavy dependency and harder to integrate into CI. It's also slow af. This PR lets the linker run natively under wibo.

The linker hard-depends on mspdb80.dll for PDB/XDB output — if the PDB interface returns failure, the linker emits LNK1207 and deletes the output file. We need to satisfy the interface, not produce a real PDB. (at least today!)

Approach

Fake PDB vtables — We provide 8 COM-style objects (PDB, DBI, Mod, TPI, GSI, Dbg, NameMap, Stream) with vtables full of tiny x86 __thiscall stubs generated at runtime into mmap'd executable memory. Each stub does the minimum to keep the linker happy: return success codes, write output pointers to sub-objects, or return empty query results. The vtable slot assignments and calling conventions were determined empirically against the linker binary, cross-referenced with microsoft-pdb.

64-bit safe — All fake objects, vtable arrays, and sentinel values are allocated from the MAP_32BIT code page (not static globals in host BSS). This ensures addr32() never truncates pointers on a 64-bit host where BSS can be at >4GB addresses.

Supporting kernel32 changes — The linker also exercises file mapping and file I/O paths that wibo didn't previously cover:

MapViewOfFile with FILE_MAP_ALL_ACCESS (needs MAP_SHARED, not MAP_PRIVATE)
MAP_FIXED fallback when MAP_FIXED_NOREPLACE fails
Inode-based tracking of mapped files to defer truncation (matches Windows semantics)
SetEndOfFile, GetFileType, SetFilePointerEx
flushAllFileViews() on SIGSEGV to preserve partial PE output

Confidence

Byte-identical output: The linked PE produced by wibo matches wine's output exactly, minus 2 timestamps and 17 bytes of CodeView GUID/PDB path (all expected differences).
Integration test: test_mspdb.c exercises the full LoadLibrary → GetProcAddress → function call path for all C-style mspdb exports, plus vtable dispatch through PDB and DBI objects (the exact code path the linker uses).
All 36 existing tests pass with zero regressions.
Cleanup pass: The second commit replaces asserts with production-safe error handling, deduplicates the PDB open functions, and makes the crash handler async-signal-safe.
64-bit tested: Built and tested as a 64-bit ELF binary. Vtable dispatch test validates that guest pointers returned by addr32() are dereferenceable and that stubs execute correctly.

Test plan

ctest --output-on-failure — 36/36 pass (including new test_mspdb)
End-to-end: X360 linker produces valid 19.5MB PE under wibo
Output matches wine baseline (20 bytes differ: timestamps + GUID)
Vtable dispatch test: calls QueryInterfaceVersion() and OpenDBI() through PDB vtable, then dispatches through the returned DBI object's vtable

Note: I wrote a significant amount of this with Claude Code so if I got anything wrong, that's why. It does work though!

encounter · 2026-02-11T23:30:46Z

dll/kernel32/memoryapi.cpp


 	errno = 0;
 	void *mapBase = mmap(requestedBase, mapLength, prot, mapFlags, mmapFd, alignedOffset);
+#ifdef MAP_FIXED_NOREPLACE


It's possible we could just MAP_FIXED in the first place, as long as it properly checks against the allocated ranges properly

encounter · 2026-02-11T23:30:55Z

dll/kernel32/memoryapi.cpp

 		wibo::heap::registerViewRange(mapBase, mapLength, protect, view.protect);
+	} else if (baseAddress) {
+		// Caller-specified base: register with the heap manager so VirtualAlloc
+		// won't allocate overlapping regions (matching Windows address space behavior).


llm comment

encounter · 2026-02-11T23:31:29Z

dll/kernel32/internal.h

 	int fd = -1;
 	std::filesystem::path canonicalPath;
 	uint32_t shareAccess = FILE_SHARE_READ | FILE_SHARE_WRITE;
+	uint32_t accessCategory = 0; // FILE_SHARE_READ/WRITE/DELETE bits indicating what this handle does


llm comment

encounter · 2026-02-11T23:31:58Z

dll/kernel32/fileapi.cpp

+		setLastError(ERROR_PATH_NOT_FOUND);
+		return INVALID_HANDLE_VALUE;
+	}
+	if (fInfoLevelId != FindExInfoStandard) {


need debug logs for all these branches to indicate unimplemented behavior

encounter · 2026-02-11T23:32:11Z

dll/kernel32/fileapi.cpp

 	}

 	std::string narrow = wideStringToString(lpFileName);
+	DEBUG_LOG("GetFullPathNameW input: %s\n", narrow.c_str());


feels unnecessary

encounter · 2026-02-11T23:33:49Z

src/main.cpp

 				optionDebug = true;
 				continue;
 			}
+			if (strncmp(arg, "--path-alias=", 13) == 0) {


what is this used for?

encounter · 2026-02-11T23:34:32Z

src/main.cpp

+	{
+		struct sigaction sa = {};
+		sa.sa_sigaction = [](int sig, siginfo_t *, void *) {
+			kernel32::flushAllFileViews();


there are probably other things to call before exiting

encounter · 2026-02-11T23:35:13Z

dll/rpcrt4.h

 										   RPC_SECURITY_QOS *securityQos);
 RPC_STATUS WINAPI RpcBindingFree(GUEST_PTR *binding);
 RPC_STATUS WINAPI RpcStringFreeW(GUEST_PTR *string);
-#ifndef __x86_64__ // TODO


should check the generated x64 stubs to see what this actually generates. it's probably very broken

encounter · 2026-02-11T23:36:11Z

dll/kernel32/fileapi.cpp

+
+	std::filesystem::path checkPath = files::canonicalPath(hostPath);
+	DEBUG_LOG("  share check: path=%s accessCat=%u shareMode=%u\n", checkPath.c_str(), accessCategory, dwShareMode);
+	if (files::checkShareViolation(checkPath, accessCategory, dwShareMode)) {


was this functionally necessary for some reason?

encounter · 2026-02-11T23:42:00Z

dll/mspdb.cpp

This implementation is technically impressive but incredibly scary. It's probably better to build as a separate DLL and include it (similar to our msvcrt), rather than manually writing x86 instructions to a buffer

how would you package this up such that it could be included as a DLL? Do you have an example repo?

Also thanks for the in depth review. Much appreciated.

Implement fake PDB COM vtable objects so the MSVC X360 link.exe (16.00.11886.00) can run under wibo without wine. The linker requires a working PDB interface to write PE output; returning failure causes LNK1207 and output deletion. Key changes: - dll/mspdb.cpp: Runtime x86 __thiscall stub generator with 8 fake COM object types (PDB, DBI, Mod, TPI, GSI, Dbg, NameMap, Stream). Stubs are NULL-safe and handle linker's internal calling conventions. - Memory mapping: inode tracking, MAP_FIXED fallback, MAP_SHARED for ALL_ACCESS, heap registration, flushAllFileViews for crash safety. - File I/O: SetEndOfFile, GetFileType, SetFilePointerEx, additional CreateFile flags, FlushFileBuffers improvements. - Crash handler: SIGSEGV/SIGTRAP handler flushes memory-mapped files. - Module loading: builtin mspdb80.dll resolution. Output is byte-identical to wine (minus timestamps and PDB GUID).

- Replace magic vtable array sizes with named constants (kPdbVtableSlots etc.) - Replace assert() with DEBUG_LOG + abort() in codeAlloc and mmap fallback - Factor out duplicated PDB open logic into openPDB() helper - Simplify resolveByName() by delegating C-style names to mspdbThunkByName() - Fix async-signal-safe crash handler (write() instead of fprintf()) - Remove unused g_fakeNameMap_legacy - Add test_mspdb fixture test covering LoadLibrary, GetProcAddress, and calling PDB/Stream exports end-to-end

…page In a 64-bit wibo build, static globals live in host BSS which can be at addresses >4GB. The addr32() helper silently truncates these to garbage 32-bit guest pointers, breaking all vtable dispatch. Fix: allocate all fake PDB objects, vtable arrays, and the legacy stream sentinel from the MAP_32BIT code page (already RWX, guaranteed <4GB) instead of using static globals. Uses ~1.2KB extra from the 16KB page. Also adds vtable dispatch coverage to test_mspdb: calls QueryInterfaceVersion and OpenDBI through the PDB vtable, then dispatches through the returned DBI object's vtable. This exercises the exact code path that breaks when objects aren't 32-bit addressable.

…piled DLL Replace 900 lines of runtime x86 machine code generation with a real 32-bit PE DLL cross-compiled by MinGW. The DLL implements PDB COM interfaces as C++ classes with compiler-generated vtables, eliminating MAP_32BIT code pages and manual stub assembly. Review comment fixes: - Simplify MapViewOfFileEx to use MAP_FIXED directly - Remove LLM-style comments (memoryapi, internal.h, memoryapi.h) - Add DEBUG_LOG to FindFirstFileExW validation branches - Remove unnecessary debug logs from GetFullPathNameW - Restore #ifdef __x86_64__ guard on NdrClientCall2 - Remove unused --path-alias feature - Remove speculative share violation tracking

encounter reviewed Feb 11, 2026

View reviewed changes

freeqaz added 4 commits February 12, 2026 03:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Xbox 360 linker support (fake mspdb PDB interface)#110

Add Xbox 360 linker support (fake mspdb PDB interface)#110
freeqaz wants to merge 4 commits intodecompals:mainfrom
freeqaz:x360-linker-support

freeqaz commented Feb 11, 2026 •

edited

Loading

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

encounter Feb 11, 2026

Uh oh!

freeqaz Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

freeqaz commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Approach

Confidence

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

freeqaz commented Feb 11, 2026 •

edited

Loading