Performance

[mariotoffia]

Hi, I'm new to extism but find your project very attractive! Thanks!

I wonder what to expect in terms of performance, and is there a way to improve that?

I'm using tinygo to compile the was file using tinygo build -o <outfile> -target wasi <gofile>

I've done two benchmarks:

1. Noop Function

Noop function:

//export hello_world_noop
func helloWorldNoop() int32 {
	return 0
}

Host benchmark:

func BenchmarkNoopFunction(t *testing.B) {
	wasm, err := cbwasm.CompileGoWasmToBytes("testdata/plugins/hello_world/main.go")
	require.NoError(t, err)

	manifest := extism.Manifest{
		Wasm: []extism.Wasm{
			extism.WasmData{Name: "hello_world_noop", Data: wasm}},
	}

	ctx := context.Background()
	config := extism.PluginConfig{
		EnableWasi: true,
	}

	plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{})
	require.NoError(t, err)

	t.ResetTimer()

	for i := 0; i < t.N; i++ {
		_, _, err = plugin.CallWithContext(ctx, "hello_world_noop", nil)
		require.NoError(t, err)
	}
}

Output:

goos: darwin
goarch: arm64
cpu: Apple M1 Pro
BenchmarkNoopFunction-10    	  170604	      7429 ns/op	   70157 B/op	      23 allocs/op
PASS

2. Passing large JSON array

plug-in:

type Person struct {
	Name string `json:"name"`
	Age  int    `json:"age"`
}

//export arr_json_benchmark
func arrJsonBenchmark() int32 {
	var input []Person

	if err := pdk.InputJSON(&input); err != nil {
		pdk.SetError(err)
		return -1
	}

	if err := pdk.OutputJSON(input); err != nil {
		pdk.SetError(err)
		return -1
	}

	return 0
}

Host benchmark:

func BenchmarkSerializeMuchJsonData(t *testing.B) {
	wasm, err := cbwasm.CompileGoWasmToBytes("testdata/plugins/json_test/main.go")
	require.NoError(t, err)

	manifest := extism.Manifest{
		Wasm: []extism.Wasm{extism.WasmData{Name: "arr_json_benchmark", Data: wasm}},
	}

	ctx := context.Background()
	config := extism.PluginConfig{EnableWasi: true}

	plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{})
	require.NoError(t, err)

	var (
		data    []byte
		count   int = 1000
		compare []byte
	)

	data = append(data, []byte("[")...)
	for i := 0; i < count; i++ {
		data = append(data, []byte(fmt.Sprintf(`{"name":"John Doe","age":%d},`, count))...)
	}

	data = append(data[:len(data)-1], []byte("]")...)

	compare = data

	t.ResetTimer()

	for i := 0; i < t.N; i++ {
		exit, out, err := plugin.Call("arr_json_benchmark", data)
		require.NoError(t, err)
		require.Equal(t, uint32(0), exit)
		require.Equal(t, compare, out)
	}
}

Output (Native is just go std json.Unmarshal/Marshal for comparison):

goos: darwin
goarch: arm64
cpu: Apple M1 Pro
BenchmarkSerializeMuchJsonDataNative-10    	    3133	    377033 ns/op	  101108 B/op	    1019 allocs/op
BenchmarkSerializeMuchJsonData-10          	     222	   5015298 ns/op	  151928 B/op	      30 allocs/op

[nilslice]

Hi, thanks for the benchmarking! Curious if you found anything interesting about this or if it aligns with your expectation?

[mariotoffia]

Hi @nilslice - I was hoping for say 20% - 50% decrease I performance in comparison to native. But I do not know the underlying wazero performance and what to expect. I'm going to try to write a super simple test case using wazero from scratch to see if it is possible to get better performance.

As I understand it, wazero, compiles (AOT) to machine code and hence should in theory have quite good performance. Maybe marshal data between the host and the runtime is quite expensive.

Cheers,
Mario :)

[nilslice]

Yea I would expect to see a little better performance here too - and I think it could partially be a symptom of only really comparing JSON serialization vs other code execution. In any case, tagging @evacchi for awareness.

[evacchi]

I don't think in general you get that kind of perf on any Wasm runtime, but specifically, you may want to tune TinyGo. Consider that the LLVM output for native is kind of different than the one for Wasm; take a look here https://tinygo.org/docs/guides/optimizing-binaries/

[mariotoffia]

Thanks @evacchi I'll have a look and do some more experiments.

Cheers,
Mario :)