Save fuzzer/benchmark version in database, show them in the report about fuzzbench HOT 7 OPEN

Yes! We are planning to indicate the versions of the fuzzers, and the benchmarks too.

from fuzzbench.

@inferno-chromium @mbarbella-chromium
(@lszekeres @lmrs2 also please share thoughts)

I can think of two ways to determine version.

1. Make users set an environment variable (e.g. FUZZBENCH_FUZZER_VERSION) at build time that gets saved.
2. Use the method OSS-Fuzz uses for finding out git repo versions automagically (I think it searches directories in SRC iirc).

I lean towards option 2 because it doesn't make integration harder but option 1 is nice for cases like entropic where we apply a patch to a repo.

from fuzzbench.

I think we should automatically save the repo, branch, and the hash of the checked out version (e.g., git rev-parse HEAD). We should avoid relying on users manually setting versions also because we might want to track and benchmark some fuzzers at the HEAD of their master branch.

from fuzzbench.

I think we should automatically save the repo, branch, and the hash of the checked out version (e.g., git rev-parse HEAD). We should avoid relying on users manually setting versions also because we might want to track and benchmark some fuzzers at the HEAD of their master branch.

I forgot that forcing users to set the version means that the version is pinned and must be updated.
This seems like it might be a good thing though since results can be easily reproduced if we aren't simply building from master. WDYT?

from fuzzbench.

I think less stable or less maintained fuzzers are better pinned to a specific working version. However for stable and actively maintained fuzzers (like libfuzzer, honggfuzz, afl, afl++ ...) it's probably easier to use the head of master by default, as it'd be good to automatically track regressions/improvements in these, and that way we don't need to manually bump versions all the time. (With the versions saved, reproducing an earlier experiment shouldn't be an issue.)

from fuzzbench.

I dont think we can autoroll new versions/hashes as fuzzers will break with benchmark over time. We do this in OSS-Fuzz where project author have a responsibility to fix their build. We don't have that responsibility here, so manually bumping might be the way to go, for atleast next six months or so.

But i do agree that fuzzer repo hash is better rather than with some weirdly explicitly specified / hardcoded version.

from fuzzbench.

Yes, it's a good idea to have fuzzer maintainers explicitly opt in for fuzzbench auto tracking, with responsibility to fix broken builds.

from fuzzbench.