Using sightglass to benchmark shootout about lucet HOT 5 CLOSED

Hi,

Sightglass compares different implementations of the same functions.

It works by loading shared libraries containing overlapping sets of functions, and will report how the function named fx in library ly compares to fx defined in library ly'.

It's not specifically designed for Lucet, and works at least on Linux, macOS and BSD. You can use it to compare gcc vs clang, or to compare different optimization levels for a given platform.

But in this container, we produce two shared libraries with the same benchmarks: one using clang compiling directly to native code, and another one compiling to WebAssembly before using Lucetc to produce the native code.

Sightglass has its own repository: https://github.com/fastly/sightglass which includes some updates over the version currently shipped with Lucet. Maybe this is a better place to discuss Sightglass-specific issues.

It can run in the Docker container. The error you are seeing may be caused by the filesystem allocated to Docker running out of space.

However, if you installed Lucet (using . devenv_setenv.sh), you already have Sightglass installed. The command is simply called sightglass in the host, and in the Docker guest, it will also be in your PATH after sourcing /opt/lucet/bin/devenv_setenv.sh.

If you compile it independently, using release builds is highly recommended (cargo build --release or cargo install ...). The overhead of non-optimized code in debug builds may be significant compared to what you are actually trying to measure.

from lucet.

Also, in order to run the Lucet benchmark, type:

cd /lucet
make bench

in the container.

Alternatively, type ./devenv_run.sh make bench in the host.

Compilation with clang can take quite some time due to the switch benchmark.

from lucet.

By the way, we will probably publish a blog post on Sightglass soon.

That will be a good opportunity to also improve its documentation.

from lucet.

Ok thanks a lot. So this is a way to directly compare C->WASM->Native with C->Native. This is nice.

So I did a make bench but ended up with an build error:

error: Found argument '--reserved-size' which wasn't expected, or isn't valid in this context
Did you mean --min-reserved-size?

I submitted a patch but see it has already been patched. I also saw an error:

No such file or directory (os error 2)
Makefile:37: recipe for target 'run' failed
make[1]: *** [run] Error 1
make[1]: Leaving directory '/lucet/benchmarks/shootout'
Makefile:32: recipe for target 'bench' failed
make: *** [bench] Error 2

Looks like the make bench command actually runs as well (I see many shootout results), but the above error looks unexpected.

from lucet.

I'm not able to reproduce the second error you reported above. From #107 it looks like you got it to work. If not, please reopen and we'll track it down.

from lucet.