Different results in AFLGo and AFLFast papers about aflgo HOT 5 CLOSED

@mboehme I just noticed that in some other issues, some people are confused with the distance calculation on binutils.

from aflgo.

Thanks for your interest in our greybox fuzzing research! Correct, there seems to be a disagreement between the results for AFL in the AFLFast and AFLGo papers. However, the sections on experimental setup in both papers should well explain this difference.

First, we are using two different versions of AFL (the most recent in each case). The version of AFL used in the AFLGo paper (FidgetyAFL) already incorporates the explore-schedule which makes recent versions of AFL substantially faster than earlier versions (i.e., before AFLFast).

Second, in the AFLFast paper, we executed both AFL and AFLFast with the deterministic stage (w/o "-d"). However, as Michal Zalewski pointed out in a discussion on AFLFast, he suggested future experiments be conducted with "-d". Hence, we opted to executed both AFL and AFLGo without the deterministic stage (w/ "-d").

Third, we made sure that there is no disadvantage in the comparison of AFLGo with AFL or AFLFast with AFL. In both papers, all fuzzers are started on the same seed corpus, using the same command line parameters, and given the same time budget. In the case of AFL, AFLFast, AFLGo on binutils, the "seed corpus" was the empty file:

mkdir in
echo "" > in/in

Fourth, in order to allow other researchers (including you) to reproduce our results, we made our tools publicly available. The experimental infrastructure is discussed in the paper. I hope this has answered your questions.

Since there is no issue with the tool per-se, I am closing this issue.

from aflgo.

@mboehme I tried AFL 1.94b (I think it was the version AFLFast paper) with c++filt version before your fix. CVE-2016-4487/CVE-2016-4488 took a little long time to be found, but it reported all 46 unique crashes within 1 hour. I checked gdb+valgrind backtrace and some are relevant to CVE-2016-4492/CVE-2016-4493, CVE-2016-4490 and some other crashes. But I don't know how to use AFLGo to get the distance for binutils; can you open source the scripts? (I had some questions in #19 and hope you answer)

Also, I'm also interesting in that said by @karl-fuzznoob why AFLFast can be better than AFLGo sometimes?

from aflgo.

@fuseproj Interesting observations. @mboehme Can you share the details?

from aflgo.

Also, 4487/4488 and 4492/4493 are similar crashes, but in terms of time fuzzing results are different.

from aflgo.