From #138:

Ah, and one more thought: have we considered any statistical analysis that would look for multi-modal distributions (and warn, at least)? If we see that e.g. half of all runs of a benchmark run in 0.3s and half in 0.5s, and the distribution looks like the sum of two Gaussians, it may be better to warn the user "please check settings X, Y, Z; you seem to be alternating between two different configurations randomly" than to just present a mean of 0.4s with some wide variance, while the latter makes more sense if we just have a single Gaussian with truly random noise.

We can use the Shapiro-Wilk test to determine normality: https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test

Yes ... it is the same for other VMs though and not specific to enabling wamr. I was working on a separate cleanup patch to do this across all git downloads consistently while also cleaning up a bunch of other things.

Originally posted by @jlb6740 in #28 (comment)

Most modern CPUs scale their clock frequency according to demand, and this CPU frequency scaling is always a headache when running benchmarks. There are two main dimensions in which this variance could cause trouble:

• Varying frequency across time: if the CPU load of benchmarking causes the CPU to ramp up its frequency, then different benchmark runs could observe different results based on different CPU frequency.
• Varying frequency across space: if different CPU cores are running at different frequencies, then benchmark runs might intermittently experience very different performance if they are not pinned to specific cores.

I've been seeing some puzzling results lately and I suspect at least part of the trouble has to do with the above. I've set my CPU cores to the Linux kernel's performance governor, but even then, on my 12-core Ryzen CPU, I see clock speeds between 3.6GHz and 4.2GHz, likely due to best-effort frequency boost (which is regulated by thermal bounds and so unpredictable).

Note that measuring only cycles does not completely remove the effects of clock speed, because parts of performance are pinned to other clocks -- e.g., memory latency depends on the DDR clock, not the core clock, and L3 cache latency depends on the uncore clock.

The best ways I know to avoid noise from varying CPU performance are:

• Have longer benchmarks. Some of the benchmarks in this suite are only a few milliseconds long; this is not enough time to reach a steady state.
• Interleave benchmark runs appropriately. Right now, it looks like the top-level runner does a batch of runs with one engine, then a batch of runs with another. If the runs for different engines/configurations were interleaved at the innermost loop, then system effects that vary over time would at least impact all configurations roughly equally.
• Pin to a particular CPU core. For single-threaded benchmarks, this is probably the most robust way to have accurate A/B comparisons: if cores have slightly different clock frequencies, just pick one of them. Even better would be to do many runs and average across them all, but in high-core-count systems, removing this noise would take a lot of runs (hundreds of processes); with a few (5-10) process starts, it's entirely possible for the variance in mean core speed to be significant.
• Observe CPU governor settings when on a known platform (Linux: /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor text file, will usually be ondemand, we wantperformance) and warn if scaling is turned on

Thoughts? Other ideas?

The average isn't a stable statistical measurement. (Whenever Bill Gates enters a very crowded bar, everyone in this bar becomes a billionaire on average.)

It would be nice to add the median value to the set of benchmarking results, since it tends to be more stable overall.

I spent some time looking into why my benchmark runs seemed to be "choppy", sitting doing ~nothing between each (very fast) Wasm benchmark run. It seems that we have pauses of up to a few seconds between each run.

It turns out that this is because the wall_cycle measurement provider in the recorder crate invokes precision::Precision::new, building a new precision measurement object, for each run: link

The precision crate notes in its docs that "Note that on Linux system, this will perform calibration before returning" and recommends against running it more than necessary.

It seems we could get a significant speedup by saving one Precision object, maybe in a lazy-init'd global or somesuch, and reusing it.

@sunfishcode @jedisct1 Is it fair game to add BSD licensed code to this project or to cranestation in general. Specifically I was looking at the potential of adding some benchmarks for testing SIMD https://github.com/ngzhian/simd-benchmarks and http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/index.html that use a BSD license.

For the future: I really wish this were running in CI; it's very difficult to know what effect these changes will have without downloading the branch and running the thing on my system. There's no reason that GitHub actions couldn't do this, right?

Originally posted by @abrown in #26 (comment)

We should report the code size of each Wasm benchmark when it is compiled to native code.

For example

cargo run -- --stop-after compilation -- benchmarks-next/pulldown-cmark/benchmark.wasm

starts instantiating and executing the benchmark instead of just compiling it.

cc @abrown

In order to display tables of significant performance changes, we discussed creating a Markdown table from the analyzed data. Here is a proposed table:

| Benchmark      | Phase       | Event                | `main` (SHA: `0123456`) | PR #42 (SHA `abcdef2`) | Significant Change  |
|----------------|-------------|----------------------|-------------------------|------------------------|---------------------|
| shootout-fib2  | Execution   | cycles               | 100,000,000             | 110,000,000            | -10%                |
| shootout-gimli | Compilation | instructions-retired | ...                     | ...                    | +2%                 |
| ...            | ...         | ...                  | ...                     | ...                    | ...                 |

While watching the output of benchmarks-next/build.sh scroll by, I noticed that it seems that every individual benchmark build is reconstructing a Docker image from scratch, including a download of the 48MB WASI SDK tarball as well as a number of Ubuntu packages. This is somewhat wasteful (bandwidth is not free) and also means the build is slower than it could be.

Could we consider building a base Docker image once and then using it to build each benchmark in turn?

One useful feature of the original sightglass code was the ability to run the benchmarks as native machine code in order to form a baseline for comparison. If we migrate this functionality from webui_runner to benchmarks-next (e.g.), we can then fully replace the old sightglass runner with the new one.

This involves some investigation to determine how to hook into the bench_start() and bench_end() calls with perf.

👍 Ahh, yes, maybe. Unless there is something about these preprocessor directives that won't allow this. If there is no issue this is probably a good idea. Like some of the other comments, this will apply generally so we can test this out in a future patch that applies consistently in all wrappers.

Originally posted by @jlb6740 in #28 (comment)

@jameysharp mentioned in #202 (comment) that it would be nice if measurement results were emitted as soon as they were collected. Currently this is not the case: all the measurements for all of the benchmark runs are collected and emitted at the end. This issue could be fixable by refactoring how the measurements are serialized by serde.

We can measure multiple perf counters and also wall time at the same time, no need to run the benchmarks multiple times, once for each measurement (which balloons experiment time).

We talked about adding an independence test to check that the measurements of multiple runs aren't correlated (@fitzgen, is that right?). We discussed also implementing this in Rust and it would seem to fit in the sightglass-analysis crate.

From #138:

Observe CPU governor settings when on a known platform (Linux: /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor text file, will usually be ondemand, we want performance) and warn if scaling is turned on

If anyone knows how to detect the equivalent thing for other OSes, please share :)

Implement a statistically sound way (we discussed doing this in Rust) to determine the confidence intervals of the speedup/slowdown observed between two engines.

In #46, we introduced a new test runner implementation and conventions but there are many remaining tasks for improving that PR. I am copying the issues listed in #46 (comment) here and will check them off as they are completed:

• move and rename webui/sg-history to crates/result-server
• add the ability to run multiple iterations (e.g. -n 10) to sightglass-recorder
• fix the perf-counting in sightglass-recorder, probably by using perf-event or, if not, by resolving gz/rust-perfcnt#19
• add the ability to choose the output format (e.g. CSV vs JSON); this can probably be lifted from what sightglass currently does
• add a sightglass-analysis crate to aggregate and synthesize results
• add more benchmarks! (Initially, port the existing benchmarks in shootout and polybench to the new style)
• add a CLI command (build-engine?) for building engine libraries from name@commit strings in addition to passing paths to pre-built engines (the current approach)
• eventually, integrate duplicated parts (e.g. sightglass's test runner, webui_runner's Docker infrastructure) and remove unused code
• re-enable the test (and run it in CI) verifying that sightglass-artifact can build Docker images into Wasm benchmarks despite differences in Docker CLI output (i.e. potentially switch to using a crate exposing the Docker API)--perhaps we should just regenerate all the benchmark Wasm files with benchmarks-next/build.sh and check for changes to ensure that stuff is deterministic
• add a test (and run it in CI) to run sightglass-recorder on a compiled Wasm benchmark--perhaps we just run all the benchmark Wasm files with benchmarks-next/run.sh to make sure they are all runnable

This will allow us to clean up after ourselves, avoid wasting disk space, and let users double check that they are using the latest wasmtime.

A few things might lead us to measuring the wrong thing / not measuring what we want:

• A buggy benchmark program that does the wrong computation
• A codegen bug in a compiler PR we are testing
• Not writing the benchmark program's result through I/O, allowing the compiler to optimize it away

I propose we solve these issues by:

• requiring that benchmark programs write the result of their computation to stdout
• the bench API's WASI context redirects the benchmark program's stdout to a known log file
• the known log file is passed into the bench API through a config struct (that will also have the WASI working dir, so that won't be a standalone argument anymore)
• benchmark programs have default.expected and (if it supports small workloads) small.expected files containing the hash of the expected stdout
• the benchmark runner hashes the stdout log file and compares it to the expected hash
  • if they match, the runner keeps doing its thing
  • if not, then it exits with an error

As noted in a few places (e.g, here and here), there is duplication in the code that enables each of the VMs to be run inside sightglass. This would be resolved by factoring out the duplicated parts in webui_runner/plugs into one common implementation.

From #138:

Interleave benchmark runs appropriately. Right now, it looks like the top-level runner does a batch of runs with one engine, then a batch of runs with another. If the runs for different engines/configurations were interleaved at the innermost loop, then system effects that vary over time would at least impact all configurations roughly equally.

The sightglass-cli benchmark command produces output like the following:

  baseline.so is 1.00x to 1.03x FASTER than target/release/libwasmtime_bench_api.so!
  target/release/libwasmtime_bench_api.so is 0.97x to 1.00x SLOWER than baseline.so!

The intent to show the ratio in both directions is nice; however, these two statements actually contradict, I think. if A is n times faster than B, then B is n times slower than A. However, this summary says that B is 1/n times slower than A.

In other words, either both words above should be "FASTER", or the second ratio should also be 1.00x to 1.03x.

This was producing some confusion for me earlier and I had to read the source to figure out which direction is intended.

On Windows, the benchmark_effect_size test has started intermittently failing since #170. It is unclear to me what could have modified the execution of the code that copies the built engine to a location provided by tempfile::NamedTempFile. I added an assert in hopes that it would trigger the error sooner but the OS always thinks the file exists:

One example of this failure is here:

thread 'benchmark::benchmark_effect_size' panicked at 'Unexpected failure.
code--1073741819
...
command=`"D:\\a\\sightglass\\sightglass\\target\\debug\\sightglass-cli.exe" "benchmark" "--engine" "\\\\?\\D:\\a\\sightglass\\sightglass\\engines\\wasmtime\\engine.dll" "--engine" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\.tmpaNVVum" "--processes" "1" "--iterations-per-process" "3" "../../benchmarks-next/noop/benchmark.wasm"`

If this random link is to be believed, the code referenced above is an access violation that might indicate referencing a null pointer, e.g. But why? The original engine library seems to run fine according to the logs; only the copied engine library has the problem.

The two known users of the original Sightglass code (see src) are:

• lucet, see the Git submodule and the benchmarks directory
• webui_runner, a component of Sightglass (see webui_runner)

If lucet migrates to sightglass-next, then it is significantly easier to refactor this repository to remove the old Sightglass code.

I am proposing in this issue that we add a way for sightglass-cli to benchmark a specific set of files together.

Why

• I have noticed that certain users only run the larger benchmarks (spidermonkey, bz2, pulldown-cmark); in the past I have wanted to only run the shootout-* micro-benchmarks or "all the benchmarks that use SIMD."
• New benchmarks, like the wasi-nn one submitted in #201, would require special setup and would not run successfully with sightglass-cli benchmark ... benchmarks/*/benchmark.wasm (as the README currently advises the user). These special benchmarks (e.g., due to environment setup, compile-time or runtime flags, etc.) could be separated in to their own suite.
• As described in the RFCs on benchmarking, candidate benchmarks should be "real, widely used programs, or at least extracted kernels of such programs." Though this is good criteria, it can be subjective (e.g., though some shootout benchmarks are "extracted kernels" some may consider them too small to be representative); organizing the benchmarks into suites would recognize and document these differences.
• #71 requests a default set of benchmarks to run automatically if no benchmark files are specified — one could imagine a default suite for that.

How

The core idea would be to create a set of *.suite files. *.suite files would contain a newline-delimited list of benchmark paths relative to the Sightglass project directory and would accept # line comments. E.g., foo.suite could look like:

# This suite contains benchmarks that call the `foo` function.
benchmarks/foo/benchmark.wasm
benchmarks/bar/benchmark.wasm
benchmarks/baz/specially-crafted-benchmark.wasm

sightglass-cli would accept a new flag — --suite — with the path to the *.suite file to use. It would be an error to specify both a suite and a benchmark file; other than that, running benchmarks by path (the current way) would remain unchanged. Users could create their own *.suite files and manage these outside the repository but the Sightglass repository would contain a few *.suite files. There are many ways to slice this, but I could see the following initial set:

• default.suite
• shootout.suite
• simd.suite

If no benchmark paths or suites are specified, sightglass-cli would run the benchmarks contained in the default.suite.

I would appreciate feedback on this proposal before implementing it. Any thoughts one way or another are appreciated!

I was unable to work out how to collect JSON output into a machine-readable file when running a sequence of slightglass-cli in-process-benchmark commands. For now I am grepping through and manually cleaning up a transcript of stdout/stderr -- but this is far from ideal. Would it be possible to either add an option to write output to a particular file, or document this if there is already a way?

At some point, the code in src will no longer be used and can be removed. When this happens, we should be able to remove that directory as well as the benchmarks directory (and rename benchmarks-next to benchmarks). There are several issues to close before this can happen:

• modify lucet to use new runner, #92
• integrate the output of the new runner with the webui, #94
• extract native baseline benchmarking from webui_runner, #96

This would allow us to get more samples in that much less time (could get, say, ten instantiation and execution samples per compilation) but would also let us stress test things like Wasmtime's pooling allocator.

(We can't allow multiple executions per instantiation because of WASI's contract for commands that _start is only called once).

In the long term we discussed building a new UI for historical graphs, but in the short term we can implement some conversions (perhaps in sightglass-data?) that will transform the output from the sightglass-next runner into output that the current webui accepts.

When measuring more than one benchmark, it would be nice to be able to aggregate the results into a single score. One common way to do this is to take the geometric mean of a set of results. This issue proposes adding a --aggregate-benchmarks flag to do exactly this. When enabled, sightglass-cli summarize --aggregate-benchmarks would emit a single, "aggregated by geomean" result for each of the phases using <all benchmarks>, e.g., for the benchmark column.

After chatting with @abrown we came up with the following extension to the benchmark program protocol to support multiple input workloads (eg a "default" workload and a "small" workload that is used for quickly testing an idea):

• the runner has a --small-workload flag
• this sets a WASM_BENCH_USE_SMALL_WORKLOAD=1 environment variable for benchmark programs
• the directory where the wasm benchmark program lives is preopened for its execution
• the wasm benchmark program may read that env var and use that to decide which input files to read and process as its workload

Example:

fn main() {
    let workload = if std::env::var("WASM_BENCH_USE_SMALL_WORKLOAD").is_ok() {
        read_small()
    } else {
        read_default()
    };
    bench_start();
    process(workload);
    bench_end();
}

Things to note:

• benchmarks can ignore the env var if they have only a single workload and it Just Works (eg micro benchmarks)
• easier to implement in runner than a stdin protocol, work is pushed to benchmark programs themselves, and it is easier for them then the runner
• flexible for situations where a workload is made up of multiple files, where as stdin would require some sort of JSON wrapping of them or something
• easy to add new env vars in the future for new extension points, if needed

So we can clear up disk usage or ensure that we're using the latest and greatest wasmtime.

As a policy, the Bytecode Alliance is changing the default branch names in all repositories. We would like for all projects to change the default to main by June 26. (We mention June 26th because there is some suggestion that GitHub may be adding the ability to make this process more seamless. Feel free to wait for that, but only up to June 26. We'll provide further support and documentation before that date.)

Please consider this a tracking issue. It is not intended for public debate.

• Change branch name
• Update CI
• Update build scripts
• Update documentation

@jedisct1 @pchickey @sunfishcode

Hi,

Having had some interest in surveying the performance of various standalone WASM VMs, I've made several updates to sightglass that I am thinking should be useful as a contribution. In my updates are here:

• https://github.com/jlb6740/sightglass/tree/sightglass_runner
• https://github.com/jlb6740/sightglass/tree/sightglass_webui

I've explicitly split the runner and the UI in master into different projects. One of the reasons for this is that I've also containerized each component so that they can be installed and used independently. In addition to the logical separations and container feature, there are several updates that should be useful:

1. The runner portion is now plugin based and automated for the combination of VMs and workloads. What this means is that you add a script that defines how to check out and compile a VM and the framework will automatically call this script before running a test suite. It also means that there is a separate script that defines how to build for a particular workload suite and the framework will automatically call this script as well when needed.

2. I've added script support for lucet and wasmtime but other VMs including (node, wamr, etc) can be plugged in as well. So instead of adding sightglass to lucet, lucet is added to sightglass and updated (and the same for all the other VMs)

3. Also added polybench http://web.cse.ohio-state.edu/~pouchet.2/software/polybench/ which currently runs for wasmtime_app and lucet_app

4. Added a driver to make running everything simple. In particular it is possible to send results from the sightglass_runner to the server running the sightglass_viewer.

5. Added a viewer for sightglass_runner to see results immediately without needing to send to server running sightglass_viewer which is really intended to track history.

To get started with sightglass_runner, build the container and run a command through the driver, i.e.:

./sg_container_build.sh
./sg_container_runner.sh --help
or
./sg_container_runner.sh -r lucet_app -p shootout

To get started with sightglass_ui, build the container and launch the containerized web-server

./sg_webui_build.sh
./sg_webui_run.sh

I'd like some feedback on this, but I have found sightglass to be useful, I've found these changes to be useful and I am thinking this contributions could either be contributed back to sightglass as separate branches or, more preferred ... folded into the cranestation project https://github.com/CraneStation since there is more visibility here and this would be useful for all projects that depend on cranestation and wasi.

Not sure what happened, but the CLI tests are hanging for me on my local machine after I pulled the latest main branch. They used to take 10s to complete, which I thought was a little bit longer than they should, but I was hopeful that after #135 they would speed up, so I wanted to measure them again. Now they are just hanging. So I did a git bisect to find where they started hanging, and now they are hanging in the very first commit that introduced the CLI tests: 4345122. This is perplexing, because they definitely didn't used to hang when I introduced that commit!

I've also verified that manually running the commands that the tests run does not hang locally.

I'm going to try profiling the tests to see where time is being spent now.

Right now we allow either

• the path to a libwasmtime_bench_api.{so,dll,dylib} shared library
• a URL to a hosted git repo that we can git clone

to be passed as engines to sightglass-cli benchmark.

We should also allow paths to wasmtime repo checkouts on the local file system.

The second option above would normally work for this, except that the git clone from inside a docker container, so we don't have access to that part of the filesystem. Also, re-cloning locally is kinda ridiculous since it would uselessly copy a bunch of files and also force a clean build when we could otherwise do a nice incremental build.

It might be nice to have an option for prettifying the JSON output for those who don't have jq installed, e.g.

Implement (or re-use) some Rust code that will be able to test whether the performance difference between two runs (i.e., the aggregate runs of two different engines) is statistically significant or not. Probably involves some additions to the sightglass-analysis crate and potentially some refactoring.

If none are passed, then we should default to ./benchmarks-next/*/benchmark.wasm to run every available benchmark.

To minimize the number of benchmarks needed to get a representative result, we could run PCA on the executed benchmarks. This would likely be implemented in R. There is a lot more discussion needed for this but this issue serves as a placeholder for that.

At some point in time CSV headers stopped being printed. We should fix that.

This feature request needs some background: I'm currently working on the new regalloc, and am wanting to use Sightglass to measure the impact of each possible feature or heuristic tweak as I make it. This is fairly fine-grained work that ideally benefits from a tight feedback loop; unfortunately, the current best practice for Sightglass has quite a lot more overhead than a "build wasmtime locally and measure bz2 runtime 5 times with time" approximate test I'm using now:

• Sightglass can build an engine based on a git repo and a branch/commit for wasmtime;
• wasmtime depends on regalloc.rs, and regalloc.rs provides a shim for the new regalloc and then pulls in regalloc2;
• The main-branch version of wasmtime refers to deps on crates.io, but for local development, I hack Cargo.toml files so that my wasmtime checkout refers to regalloc.rs refers to regalloc2 all in sibling directories;
• So to get Sightglass to build an engine with the new configuration, I would need to: (i) commit my regalloc2 experiment (possibly just a tweak to a constant in some heuristic) and push to GitHub; (ii) adjust Cargo.toml in regalloc.rs to refer to this new commit hash, commit that, and push that to GitHub; (iii) adjust Cargo.toml in cranelift-codegen to refer to the new regalloc.rs commit, commit that, and push that to GitHub; (iv) use that commit hash to ask Sightglass to build a new engine file.

The above gets even more fun if I try to use Cranelift settings to control experiment knobs: I would need to alter the defaults in cranelift-codegen, and commit that. If I have n regalloc2 variants and m different knob settings, I need to make n*m separate commits and push them all to GitHub.

The config-knob issue is separate (#103) so let's focus on the separate-versions-of-engine-code problem below.

IMHO, there is a possibly much easier way: we could make the tool fit into the same workflow that someone doing local testing of any other sort would do, and (i) build an artifact from the code on disk, and (ii) test that artifact.

Concretely, this means that Sightglass would (in this mode at least) no longer manage its own cache of engine .sos. Instead, it would have some command (for example):

$ # (make some tweaks to wasmtime/regalloc2/regalloc.rs source in local checkouts)
$ sightglass-cli compile-engine wasmtime ../../path/to/wasmtime -o wasmtime-variant1.so

Then just test using that file explicitly:

$ sightglass-cli run -f wasmtime-variant1.so ...

This sidesteps all of the questions of syntax for repository paths, what unique name (the "slug" as the source calls it) to use for the engine, etc., and fits much more nicely into the usual Unix tool ecosystem, IMHO; there is no hidden state to worry about.

Then, the infrastructure to build a canonical version of an engine from a clean git checkout with a Dockerized hermetic build could be built on top of this, and used by whatever CI infrastructure we have.

Thoughts?

It would be nice to benchmark these configurations (both one off and continuously over time):

• builds w/ vs w/out debug info
• optimized(speed) vs optimized(speed + size) vs not optimized
• bounds checking strategy (explicit vs virtual memory)
• with or without fuel

All of these things require dynamically setting flags on cranelift and/or wasmtime.

The point of the rebuild CI task is to ensure that we can reliably reproduce the benchmarks included with Sightglass. IIRC, there was a time when that task checked if the benchmark output had changed, e.g., with git diff --exit-code. That check is no longer in place but I believe it should be re-added so we can more confidently claim that we have reproducible benchmark builds.

It appears that docker build for docker version Docker version 20.10.5, build 55c4c88 (macOS) outputs this as the last line:

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

This breaks sightglass such that it parses them as the image id for what was just built, causing this error message:

Unable to find image 'them:latest' locally
Error response from daemon: pull access denied for them, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
 ERROR sightglass_artifact::docker > Failed: Child { stdin: None, stdout: None, stderr: None }
Error: failed to execute docker command:

Right now, Sightglass uses a single threshold based on a confidence interval computed by Behrens-Fisher to determine whether a sampled statistic shifted between configurations.

The result of this is that we get either "changed" (i.e., benchmark got 5% faster) or "not changed". However, the latter answer can also appear if we simply don't have enough data points to prove statistical significance, or if the system is too noisy.

This "false negative" is somewhat dangerous: we could make a change, see that it is performance-neutral according to Sightglass, and accept it, but actually we just didn't turn the knobs up high enough.

Ideally, Sightglass should provide a third output of "unsure" if the measurements aren't precise enough to prove either "changed" or "not changed" to the desired confidence.

In order to report performance results based on PRs, we talked about implementing an HTTP server (e.g. in crates/server) that would:

• listen for incoming POST requests that contain JSON with the PR URL, commit SHA, etc. necessary for doing a "master vs PR" comparison
• to avoid DoS, verify that the request is an authorized one (not exactly sure how to do this but the GitHub action will need some form of token)
• kick off some benchmark running, like sightglass-cli benchmark ... but we could call the same APIs from inside the web service
• upon success, push a Markdown table of the significant performance differences as a comment to the GitHub PR (implies that sightglass-server has a GitHub token, like one from bytecodealliance-highfive)
• upon failure, push the error message as a comment to the GitHub PR (implies that we maintain error logs somewhere)

There is a lot more to be done here but that should be a workable start.

They should only be emitted once regardless how many processes we use when collecting samples.