Benchmarks strongly favor wasmer runtime characteristics about wasmer-go HOT 5 CLOSED

In addition to those remarks, I personally would be interested in the Cgo penalty that wasmer incurs, to give an idea of the "worst case" compared to pure Go implementations. For instance, calling from wasmer in a loop, a Go function that increments a number and returns it.

from wasmer-go.

That's a good point. @tiborvass also wrote me about improving the benchmarks to add more insight into the Cgo penalty.

Life AOT (Polymerase)

For benchmarking, we tried to use the Life with Polymerase (the one that is AOT) and we were unable to get it working.

At the same time, Polymerase transpiles the wasm code to C to then compile it with clang as a shared library (source here: https://github.com/perlin-network/life/blob/3010eb89946c347a3d158749b49aafe9bcd26d65/platform/aot_unix.go#L206), which is not ideal since requires clang to be present.

Because of that requirement we thought not a lot of people will be able to use it on their environments... and that's why we chose to not put more effort into get it working for the benchmark after our first trial.

Further analysis

In order to provide good benchmarks that are not biased we need to measure things like:

• How fast compiles the modules: since it relies on the Wasmer code generation, we can trust the numbers provided here
• How fast is the code at runtime: provided in the benchmarks
• How fast is able to cross boundaries:
  • From Wasm to host functions in go
  • From go to Wasm functions

The last part is the one that we haven't measured yet, that I think it will be quite valuable for everyone.

What are your thoughts?
Are there any other things that will be useful to measure?

from wasmer-go.

I would like to complete the answer about Life Polymerase. Cranelift is a JIT engine and is the default backend we use since it offers the best tradeoff between compilation-time and execution-time. If we want to get close to native, as Life with Polymerase does by transforming Wasm to C and then compiling it with clang, then we can use the LLVM Wasmer backend. It brings a slower compilation-time, but an extremely fast execution-time (as fast as native). In the current benchmarks, we wanted to compare apples to apples; hence comparing Life “raw” to Wasmer with Cranelift: Both do not require any third-party softwares to be present. With Polymerase, Life requires clang to be present, and with LLVM, Wasmer requires… LLVM to be present, … or not. It is likely we will generate a dynamic library for Wasmer that already includes LLVM: It largely increases the dylib size, but it is possible.

I wished that, just like Cargo/Rust has a feature compilation flag, Go could have something similar, but I didn't found a way to do that yet (any clue?). If a compilation flag can be used, then the user could decide what backend Wasmer must use: Singlepass, Cranelift or LLVM. I think it's the best option until we complete backend tiering.

from wasmer-go.

@Hywan I think the differences you're describing are an issue with choosing benchmarks to highlight performance. Apples to apples would be "here are benchmarks between N pure-go wasm runtimes".

JIT is great, but if you're looking for performance and reduced runtime dependencies maybe it would be better to compile your wasm with Life/clang and ship the resulting binary with your Go program: https://github.com/perlin-network/life/blob/57f3819/platform/aot_unix.go#L234-L237

As far as I can tell, this isn't an option with Wasmer (maybe pre-populate the cache?).

The critique (which I think has been heard) is that "go-ext-wasm is much faster than everything else" is an incomplete picture. The use cases for building wasm into Go code are surely varied and there are a variety of pros and cons to each approach.

from wasmer-go.

@Hywan also Cranelift is young, I'm sure it'll be as fast as llvm in no time! 😉

from wasmer-go.