Invalid Asyncify stack when using dlopen() with SIDE_MODULE and asyncify imports

[jamiebird-arm]

I'm seeing what appears to be an invalid Asyncify call stack when using dlopen to dynamically load a SIDE_MODULE which uses a JavaScript library with asyncify imports. The runtime errors seen are similar to those I see when the Asyncify stack is not large enough (see JS console output below) but I have tried a variety of large stack sizes.

This issue is only present when building as a SIDE_MODULE/MAIN_MODULE and using dlopen. Building the library as a regular callable module and invoking using ccall works as expected. See working example below.

Minimal reproducer

dummy_if.cpp (SIDE_MODULE)

#include <stdio.h>
#include <emscripten.h>
 
extern "C"
{
    extern void dummyAsyncImport();
}

extern "C" EMSCRIPTEN_KEEPALIVE
int Dummy_Open(int *handle, void* extra)
{
    // Causes RuntimeError: unreachable, on Asyncify.handleSleep wakeUp()
    // Removing this, dummyAsyncImport() works asynchronously as expected
    // See JS console below
    printf("pre dummyAsyncImport\n");
 
    dummyAsyncImport();
 
    // Causes RuntimeError: function signature mismatch, *before* Asyncify.handleSleep wakeUp()
    // See JS console below
    printf("post dummyAsyncImport\n");
 
    return 0;
}

$ em++ -o dummy_if.wasm -s SIDE_MODULE=1 -s EXPORT_ALL=1 dummy_if.cpp
$ mv dummy_if.wasm dummy_if.so

library-dummy.js (JS library required by SIDE_MODULE)

mergeInto(LibraryManager.library, {
    dummyAsyncImport: function() {
        return Asyncify.handleSleep(wakeUp => {
            setTimeout(wakeUp, 3000);
        });
    }
});

dlopen_test.cpp (MAIN_MODULE)

#include <stdio.h>
#include <dlfcn.h>
 
int main()
{
    printf("Hello dlopen test\n");
 
    // dlopen
    void* handle = dlopen("/dummy_if.so", RTLD_NOW | RTLD_LOCAL);
    if (!handle) {
        printf("Cannot open library: %s\n", dlerror());
        return 1;
    }
 
    printf("dlopen'd\n");
 
    // Load symbols
    typedef int (*Dummy_Open_t)(int *pHandle, const void *pDetails);
    Dummy_Open_t Dummy_Open = (Dummy_Open_t) dlsym(handle, "Dummy_Open");
    char *dlsym_error = dlerror();
    if (dlsym_error) {
        printf("Cannot load symbol Dummy_Open: %s\n", dlsym_error);
        dlclose(handle);
        return 1;
    }
 
    int ifHandle = 0;
    int err = Dummy_Open(&ifHandle, NULL);
    printf("Dummy_Open err: 0x%08X\n", err);
 
    dlclose(handle);
 
    return 0;
}

$ em++ -o dlopen_test.html -s MAIN_MODULE=1 --preload-file dummy_if.so --use-preload-plugins -s EXPORT_ALL=1 --js-library ./library-dummy.js -s ASYNCIFY=1 -s 'ASYNCIFY_IMPORTS=["dummyAsyncImport"]' -s ASSERTIONS=1 -s ASYNCIFY_STACK_SIZE=40960 dlopen_test.cpp

RuntimeErros

Working example without SIDE_MODULE/dlopen

Build as a entry point library:
$ em++ -o dummy_if.html -s EXPORT_ALL=1 --js-library ./library-dummy.js -s ASYNCIFY=1 -s 'ASYNCIFY_IMPORTS=["dummyAsyncImport"]' -s ASSERTIONS=1 -s ASYNCIFY_STACK_SIZE=40960 dummy_if.cpp

Use ccall to invoke from JS console:
> await ccall("Dummy_Open", 'number', ['number', 'number'], { async : true });

Enviroment

$ em++ -v
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 2.0.10
clang version 12.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project 445289aa63e1b82b9eea6497fb2d0443813a9d4e)
Target: x86_64-unknown-linux-gnu
Thread model: posix

[donalffons]

...this seems to be related to #13026. 👍

[sbc100]

When printf -> __fwritex fails like that it means that the "vtable" of of the stdout object (which has functions pointer for write and other methods) is either:
(a) corrupted in memory and therefore pointing the wrong table slot, or (b) corrupted in the table and therefor no longer pointing the right function.

See

emscripten/system/lib/libc/musl/src/stdio/stdout.c

Line 26 in 097cfb8

.write = __stdio_write,

Most likely this is (b) since dlopen and libray loading both modify the table.

Sorry I don't have a solution, only pointers.

[jamiebird-arm]

Thanks for the pointers @sbc100.

I would be hesitant to blame the corruption on dlopen or library loading becuas if I make Asyncify.handleSleep completely synchronous by moving the wakeUp() call outside of a setTimeout callback, everything works as expected. Which is why I think it is the Asyncify call stack that is corrupted.

mergeInto(LibraryManager.library, {
    dummyAsyncImport: function() {
        return Asyncify.handleSleep(wakeUp => {
            wakeUp();
        });
    }
});

Using Asyncify.handleAsync also always causes this issue, as this is always asynchronous.

[jamiebird-arm]

Some further investigation into this issue. I’ve been able to remove dlopen(), dlsym(), and the filesystem from the equation by using the general dynamic linking approach with:
Module.dynamicLibraries = [“dummy_if.so”];

The issue persists - which isn’t too surpassing as both approaches use loadDynamicLibrary in the generated runtime JS code.

The Firefox dev tools do give a slightly more detailed exception with:
exception thrown: RuntimeError: indirect call to null

This is the indirect call from __fwritex -> __stdio_write which supports @sbc100 suggestion of the vtable being invalid.

I’m struggling to debug this further due to the dev tools truncating the wasm disassembly due to size, hiding the __fwritex disassembly.
;; .... text is truncated due to size

Next step is to rebuild the dev tools to increase this limit, although I’ve seen mention the the chrome debug engine may struggle with over 1M lines of disassembly…