google / fuzztest Goto Github PK

License: Other

Starlark 4.28% C++ 91.80% Shell 2.40% CMake 1.12% C 0.14% ANTLR 0.05% Jinja 0.02% Python 0.20%

fuzztest's Introduction

FuzzTest

What is FuzzTest?

FuzzTest is a C++ testing framework for writing and executing fuzz tests, which are property-based tests executed using coverage-guided fuzzing under the hood. Fuzz tests are like regular unit tests, but more generic and more powerful. Instead of saying: "for this specific input, we expect this specific output", we can say: "for these types of input, we expect this generic property to be true". For example:

void MyApiAlwaysSucceedsOnPositiveIntegers(int i) {
  bool success = MyApi(i);
  EXPECT_TRUE(success);
}
FUZZ_TEST(MyApiTest, MyApiAlwaysSucceedsOnPositiveIntegers)
    .WithDomains(/*i:*/fuzztest::Positive<int>());

It is our latest fuzz testing technology and the successor of previously used fuzzing tools, such as libFuzzer. It allows you to write powerful fuzz tests more easily than with previously used fuzz targets. You can use it together with GoogleTest, or other unit testing frameworks, allowing you to write fuzz test side by side with regular unit tests, and just as easily.

It is a first-of-its-kind tool that bridges the gap between fuzzing and property-based testing, as it is both:

a testing framework with a rich API (akin to property-based testing libraries), and
a coverage-guided fuzzing engine (akin to AFL or libFuzzer).

Who is it for?

FuzzTest is for everyone who writes C++ code. (Currently, only C++ is supported.) Fuzz testing is a proven testing technique that has found tens of thousands of bugs. With the FuzzTest framework writing these tests becomes a breeze. Because fuzz tests are more generic, they are more powerful than regular unit tests. They can find tricky edge cases automatically for us, edge cases that most likely we would never think of.

You can write fuzz tests as easily as you write unit tests using GoogleTest for example. Simply use the FUZZ_TEST macro like you would use GoogleTest's TEST macro.

Who uses it?

At Google, FuzzTest is widely used and software engineers love it. It has replaced the old style of writing fuzz targets.

How do I use it?

To get started, read the Quickstart with Bazel or Quickstart with CMake, then take a look at the Overview and the Codelab.

Once you have a high level understanding about fuzz tests, consider reading the rest of the documentation, including the:

I need help!

If you have a question or encounter a bug, please file an issue on GitHub.

fuzztest's People

Contributors

Stargazers

Watchers

fuzztest's Issues

Broken link for codelab.in README

The link for codelab is not available.

fuzztest/README.md

Line 62 in 6d21a37

[Codelab](doc/tutorial.md).

I don't see doc/tutorial.md is ever uploaded. Do you want to link it to codelab/ instead?

Wrong cmake version required in CMakeLists.txt

In the quickstart-cmake.md you ask the reader to begin base CMakeLists.txt with:

cmake_minimum_required(VERSION 3.14)

Unfortunately some CMakeLists.txt requires higher cmake versions than that. At least 3.19 according to the cmake documentation covering:
add_library( INTERFACE)

An example is: fuzztest/grammar_codegen/CMakeLists.txt

The error is:
CMake Error at fuzztest/grammar_codegen/CMakeLists.txt:6 (add_library):
add_library INTERFACE library requires no source arguments.

Byte Array Mutator failed on empty input

Byte mutation functions FlipBit, SwapBytes, ChangeByte will fail on empty input &data.

Is it as expected? iow, will Centipede take empty vector<uint8_t> as valid input? If this is not the case, perhaps the following should be added:

bool ByteArrayMutator::SwapBytes(ByteArray& data) {
+  if (!data.size())
+    return false;
  size_t idx1 = rng_() % data.size();
  size_t idx2 = rng_() % data.size();
  std::swap(data[idx1], data[idx2]);
  return true;
}

Latest commits have issues with absl/strings:string_view

Hello. When running with the commit referenced in the documentation (here: https://github.com/google/fuzztest/blob/main/doc/quickstart-bazel.md#set-up-a-bazel-workspace), fuzztest seems to work just fine:

bazel test //fuzz:my_fuzztest
Starting local Bazel server and connecting to it...
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
DEBUG: Rule 'com_google_fuzztest' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "231a36c2d6cdfdfec3ef3b259be46d69eea2d8dc9a1a582b1ac3a9aa68412b70"
INFO: Analyzed target //fuzz:my_fuzztest (62 packages loaded, 927 targets configured).
INFO: Found 1 test target...
(etc...)

When updating to the most recent commit, as recommended:

bazel test //fuzz:my_fuzztest
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
DEBUG: Rule 'com_google_fuzztest' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "3a01226622d2f13afd62b0cb093e175cedc8604c912277115aa3fccccd59a755"
...
ERROR: /home/.cache/bazel/_bazel/2ace5843c8a77dcebc05650431be8550/external/com_google_fuzztest/fuzztest/BUILD:354:11: no such target '@com_google_absl//absl/strings:string_view': target 'string_view' not declared in package 'absl/strings' defined by /home/.cache/bazel/_bazel/2ace5843c8a77dcebc05650431be8550/external/com_google_absl/absl/strings/BUILD.bazel (did you mean 'string_view.h'? Tip: use `query "@com_google_absl//absl/strings:*"` to see all the targets in that package) and referenced by '@com_google_fuzztest//fuzztest:io'
ERROR: Analysis of target '//fuzz:my_fuzztest' failed; build aborted: 
INFO: Elapsed time: 2.188s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (53 packages loaded, 325 targets configured)
    currently loading: @com_google_absl//absl
ERROR: Couldn't start the build. Unable to run tests

It appears there may be a typo in the abseil string_view reference?

Errors on compilation

I'm following the Quickstart with CMake, and when I try to compile I get the following error.

[ 96%] Building CXX object fuzztest/fuzztest/CMakeFiles/fuzztest_logging.dir/internal/logging.cc.o
c++: error: unrecognized command line option '-fsanitize-coverage=0'; did you mean '-fsanitize-recover=leak'?
gmake[2]: *** [fuzztest/fuzztest/CMakeFiles/fuzztest_logging.dir/build.make:76: fuzztest/fuzztest/CMakeFiles/fuzztest_logging.dir/internal/logging.cc.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:6653: fuzztest/fuzztest/CMakeFiles/fuzztest_logging.dir/all] Error 2

Also, if I disable that flag on the CMake (fuzztest/CMakeLists.txt:29 and fuzztest/CMakeLists.txt:30) I get the following error:

[ 96%] Building CXX object fuzztest/fuzztest/CMakeFiles/fuzztest_type_support.dir/internal/type_support.cc.o
In file included from xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h:36:0,
                 from xxx/first_fuzz_project/fuzztest/fuzztest/internal/type_support.cc:15:
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/meta.h: In lambda function:
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/meta.h:67:39: error: parameter packs not expanded with '...':
         [](F& f) { return std::move(f)(decltype(I){}); }...};
                           ~~~~~~~~~~~~^~~~~~~~~~~~~~~
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/meta.h:67:39: note:         'I'
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/meta.h: In lambda function:
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/meta.h:67:57: error: expansion pattern '<lambda>' contains no argument packs
         [](F& f) { return std::move(f)(decltype(I){}); }...};
                                                         ^~~
In file included from xxx/first_fuzz_project/fuzztest/fuzztest/internal/type_support.cc:15:0:
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h: In function 'void fuzztest::internal::PrintValue(const Domain&, fuzztest::internal::corpus_type_t<Domain>&, fuzztest::internal::RawSink, fuzztest::internal::PrintMode)':
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h:101:66: error: expression '<lambda>' is not a constant expression
                                     corpus_value, out, mode)) {})) {
                                                                  ^
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h: In member function 'void fuzztest::internal::OptionalPrinter<Domain, Inner>::PrintCorpusValue(fuzztest::internal::corpus_type_t<Domain>&, fuzztest::internal::RawSink, fuzztest::internal::PrintMode) const':
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h:255:35: error: the value of 'is_pointer' is not usable in a constant expression
           if constexpr (is_pointer) {
                                   ^
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h:242:20: note: 'is_pointer' used in its own initializer
     constexpr bool is_pointer = Requires<value_type>(
                    ^~~~~~~~~~
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h: In member function 'void fuzztest::internal::ProtobufPrinter::PrintUserValue(const T&, fuzztest::internal::RawSink, fuzztest::internal::PrintMode) const':
xxx/first_fuzz_project/fuzztest/./fuzztest/internal/type_support.h:291:61: error: expression '<lambda>' is not a constant expression
     if constexpr (Requires<T>([](auto v) -> decltype(*v) {})) {
                                                             ^
gmake[2]: *** [fuzztest/fuzztest/CMakeFiles/fuzztest_type_support.dir/build.make:76: fuzztest/fuzztest/CMakeFiles/fuzztest_type_support.dir/internal/type_support.cc.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:5941: fuzztest/fuzztest/CMakeFiles/fuzztest_type_support.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

I'm using clang-13

How can I build and run the vp8_encoder_fuzz_test

Hi all.
I'm trying to understand how to build vp8_encoder_fuzz_test using gn with no luck. I can build chrome but not that specific fuzzer.

Any suggestion on how to do it and run it as well?

Thanks

Build Error: `always_inline` function might not be inlinable

I get the following error

Compiling fuzztest/internal/coverage.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 35 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/fuzztest/fuzztest/internal/coverage.cc:274:5: error: 'always_inline' function might not be inlinable [-Werror=attributes]
  274 |     TraceCmp(uint64_t arg1, uint64_t arg2, uint8_t argsize_bit,
      |     ^~~~~~~~
external/fuzztest/fuzztest/internal/coverage.cc:274:5: error: 'always_inline' function might not be inlinable [-Werror=attributes]
external/fuzztest/fuzztest/internal/coverage.cc:274:5: error: 'always_inline' function might not be inlinable [-Werror=attributes]
external/fuzztest/fuzztest/internal/coverage.cc:274:5: error: 'always_inline' function might not be inlinable [-Werror=attributes]

when attempting to build a project that uses google/fuzztest as a dependency using bazel with the following WORKSPACE snippet:

# Fuzztest
http_archive(
    name = "fuzztest",
    strip_prefix = "fuzztest-" + fuzztest_commit_hash,
    urls = ["https://github.com/google/fuzztest/archive/" + fuzztest_commit_hash + ".zip"],
)

On my machine, the result of ldd --version is ldd (Debian GLIBC 2.34-8) 2.34 and the result of g++ --version is g++ (Debian 12.2.0-3) 12.2.0

Warning that Apple support is disabled

I went to try out this library and got it linked and compiling, but couldn't figure out why my tests weren't registered or running. It took me a while of digging before I found the following.

It would have been really nice to have gotten a warning during compilation, rather than silently dropping the test, to know that what I was trying wasn't supported.

// Temporarily disable fuzz tests under MSVC/iOS/MacOS.
// They might not support all the C++17 features we are using right now.
// Disables all registration and disables running the domain expressions by
// using a ternary expression. The tail code (eg .WithDomains(...)) will not be
// executed.
#if defined(__APPLE__) || defined(_MSC_VER)
#undef FUZZ_TEST
...

MBErr: Builder group name "chromium.fuzz" not found in "/opt/chromium/src/tools/mb/mb_config.pyl"

HI all,
I'm following this tutorial and I have noticed that chrome.fuzz isn't usable anymore.

Which group name should we use now? At least, which options were set by it?

I would also like to understand how to build and run the fuzzer. I have with

# Build the fuzz target.
autoninja -C chrome/browser/ash:create_fnmatch_query_fuzzer
# Run the fuzz target.
./out/libfuzzer/create_fnmatch_query_fuzzer

But with no luck.

➜  src git:(ccbeb246d2) autoninja -C chrome/browser/ash:create_fnmatch_query_fuzzer
ninja: Entering directory `chrome/browser/ash:create_fnmatch_query_fuzzer'
ninja: fatal: chdir to 'chrome/browser/ash:create_fnmatch_query_fuzzer' - No such file or directory
➜  src git:(ccbeb246d2)

Thanks

Using gcc for instrumentation

Many projects do not support clang. It seems that fuzztest uses -fsanitize=address,-fsanitize-coverage=inline-8bit-counters,-fsanitize-coverage=trace-cmp to compile the target program with coverage information.

Would it be possible to use gcc-based instrumentation? For example, in the same fashion as AFL++ instrumentators do?

Samples of public projects?

I'm doing some efforts on integrating this into OSS-Fuzz google/oss-fuzz#8784 and have it working with the escaping_test sample. In this context I'm looking for public projects that use this to try out how the integration will work -- do you know of any open source projects that use Fuzztest?

DFUZZTEST_COMPATIBILITY_MODE=libfuzzer failing on linking

I am using Ubuntu 22.04.2 LTS and the default clang version (14)

I am following exactly the documentation for the cmake for first_fuzz_test.cc file, but it fails in the linking:

../lib/libfuzztest_compatibility_mode.a(compatibility_mode.cc.o): in function fuzztest::internal::FuzzTestExternalEngineAdaptor::RunInFuzzingMode(int*, char***, fuzztest::internal::Configuration const&)': compatibility_mode.cc:(.text+0x113d): undefined reference to LLVMFuzzerRunDriver'
clang: error: linker command failed with exit code 1 (use -v to see invocation

Is this compatibility mode really working with clang 14?

support for timeout

There is no support for timeout in gtest, and the simplest way is to use async task timeout. However, the thread running the async task will block the fuzzer from knowing the task has failed, and not print the failing test case.

My current workaround is to use pthread kill to kill the async task thread, but this is only safe in some cases and platform specific. It would be better if this library can provide a way of handling timeout.

CMake FetchContent and grammar_domain_code_generator: not found error

When using FetchContent

include(FetchContent)

FetchContent_Declare(
  fuzztest
  GIT_REPOSITORY https://github.com/google/fuzztest.git
  GIT_TAG main
)

FetchContent_MakeAvailable(fuzztest)

to include fuzztest i receive the following error

[ 95%] Generating json_grammar.h
/bin/sh: 1: ../../../tools/grammar_domain_code_generator: not found
gmake[2]: *** [_deps/fuzztest-build/grammars/CMakeFiles/json_grammar.dir/build.make:71: _deps/fuzztest-build/grammars/json_grammar.h] Error 127
gmake[1]: *** [CMakeFiles/Makefile2:5564: _deps/fuzztest-build/grammars/CMakeFiles/json_grammar.dir/all] Error 2
gmake: *** [Makefile:156: all] Error 2

the work around is to manually copy the "tools" folder in the build directory but i was wondering whether there could exist a better way

FUZZ_TEST_F Should Respect GTEST_SKIP() in GoogleTest Base Fixture

GoogleTest allows calling GTEST_SKIP() in the SetUp() method of a fixture to skip all tests that use that fixture: https://google.github.io/googletest/advanced.html#skipping-test-execution.

FUZZ_TEST_F on the other hand continues to run the test. The test output is also confusing in this scenario. The logs and command line output indicate the test failed, but the status of each test appears to be skipped.

Minimal Example:

#include "testing/base/public/gunit.h"
#include "testing/fuzzing/fuzztest.h"
#include "third_party/googlefuzztest/googletest_fixture_adapter.h"

namespace {

class SkipFixture : public testing::Test {
 protected:
  void SetUp() override { GTEST_SKIP(); }
};

class FuzzTestFixture
    : public ::fuzztest::PerFuzzTestFixtureAdapter<SkipFixture> {
 public:
  void FailingTest() { FAIL(); /*Not Skipped*/ }

  void FailingTestIsSkipped() {
    GTEST_SKIP();
    FAIL(); /*Skipped*/
  }
};

TEST_F(SkipFixture, FailingTestIsSkipped) { FAIL(); /*Skipped*/ }

FUZZ_TEST_F(FuzzTestFixture, FailingTest);  // Test Fails!
FUZZ_TEST_F(FuzzTestFixture, FailingTestIsSkipped);

}  // namespace

CMake Integration

Hello,
Is there a plan for a future CMake integration ?
FetchContent support would also be helpful.
Thanks !

Save the corpus and use later as seed

Hi there!

A. Is it possible to save or dump the corpus that's been found so far? E.g. when I terminate the fuzzing run, it should save the corpus that's been discovered so far. Presumably the corpus path would be a command line flag.
B. When I fuzz the same target again later, using the same Domains and all that, can I reuse a previously saved corpus?

Obviously, this is not a new idea. For example, Chromium fuzzing talks about it.

A on its own is useful, even if B isn't done. I think it would be very useful to take the corpus from A, and create a unit test for every corpus element, and add that to continuous Integration and pre-commit testing.

Internal error in container_of_impl.h when MinSize > MaxSize

void TestVector(const std::vector<int>& unit)
{
    // ...
}

FUZZ_TEST(Vector, TestVector).WithDomains(VectorOf(Arbitrary<int>()).WithMinSize(1001));

This test results in

external/com_google_fuzztest/./fuzztest/internal/domains/container_of_impl.h:107: Internal error! Check (val.size() >= this->min_size_ && val.size() <= this->max_size_) failed: size 1001 is not between [1001; 1000]

from

fuzztest/fuzztest/internal/domains/container_of_impl.h

Lines 102 to 107 in c5fde4b

 void Mutate(corpus_type& val, absl::BitGenRef prng, bool only_shrink) { 

 permanent_dict_candidate_ = std::nullopt; 

 FUZZTEST_INTERNAL_CHECK( 

 val.size() >= this->min_size_ && val.size() <= this->max_size_, "size ", 

 val.size(), " is not between [", this->min_size_, "; ", this->max_size_, 

 "]");

Since it says "Internal error!", I guess that this should be considered a bug?

I can fix the test by specifying a "MaxSize" as well, but the error message is at least not ideal, in my opinion. Ideally it would be a precondition somewhere, not an "Internal error!" right?

Furthermore this behavior is not expected after reading the Custom Container Size section of the Domain Reference.

Backslashes are wrongly escaped in grammar_codegen

When I change fuzztest/grammars/JSON.g4 to support some escape sequences

STRING : '"' (CHARACTER | '\\' ESCAPED )* '"' ;

ESCAPED : '"' | '\\' | '/' | 'b' | 'f' | 'n' | 'r' | 't' ;

then the grammar tests fail because the nlohmann/json library fails to parse the generated json:

$ bazel test //domain_tests:in_grammar_domain_test
...
Executing tests from //domain_tests:in_grammar_domain_test
-----------------------------------------------------------------------------
Running main() from gmock_main.cc
[==========] Running 6 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from InJsonGrammar
[ RUN      ] InJsonGrammar.InitGeneratesDifferentValidJson
domain_tests/in_grammar_domain_test.cc:41: Failure
Value of: parsed_json.is_discarded()
  Actual: true
Expected: false
"{\"\\\\\"\":[80.0E3,false]}"
...

That's because backslashes are escaped when they shouldn't be. E.g. "\"" (quote, escaped quote, quote) becomes "\\"" (quote, escaped backslash, quote, quote), which is not correct json.

The linked PR fixes the issue.

How to implement custom stateful domains

Hi,

I'd like to use FuzzTest to test my event-driven state machines by feeding them random events. However, the events are stateful, meaning that by contract a state machine could assume that e.g. once an EVENT_OPEN is sent to it, EVENT_OPEN must not be re-sent before an EVENT_CLOSE is sent first.
Is there any way to achieve this with fuzztest?

Thanks!

Macro conflict in `fuzztest/regexp.cc` on `NO_THREAD_SAFETY_ANALYSIS`

While attempting to build fuzztest, I encountered the following (warning, upgraded to) error:

'NO_THREAD_SAFETY_ANALYSIS' macro redefined [-Werror,-Wmacro-redefined]
#define NO_THREAD_SAFETY_ANALYSIS \
         ^
external/com_googlesource_code_re2/util/util.h:39:9: note: previous definition is here
#define NO_THREAD_SAFETY_ANALYSIS
         ^
 1 error generated.

It looks like re2 and absl both enable a macro called NO_THREAD_SAFETY_ANALYSIS with different definitions. re2 is doing it with a #ifndef guard, while absl is unconditionally dropping it in.

It looks like this could possibly be solved by just making the re2 headers be included before the absl headers, though that would be a non-standard include order. The real fix may be to ask one or both of the upstream repos to change the name of their macro or at least #undef it when they're done with it. Alternatively, (though this would be ugly) they could be explicitly #undefed between the inclusions in this file.

#include <string> in fuzztest.h not found

Hello,

I am sorry, but I am running into a problem when building a project with cmake and fuzztest.
My project is a make c project but I have changed it to a simple cmake project using the provided guide about cmake projects.
I basically have a *.c and a *.h file and would like to build them with fuzztest.

CMakeLists.txt

cmake_minimum_required(VERSION 3.19)
project(fuzztest)

set(CMAKE_CXX_STANDARD 17)

add_subdirectory(fuzztest)

enable_testing()

include(GoogleTest)

fuzztest_setup_fuzzing_flags()

add_compile_definitions(-DMJS_MAIN -DMJS_EXPOSE_PRIVATE -DCS_ENABLE_STDIO -DMJS_ENABLE_DEBUG -DCS_MMAP -DMJS_MODULE_LINES)

add_executable(
  mjs_fuzztest
  mjs_fuzz/mjs.c
  mjs_fuzz/mjs.h
)

link_fuzztest(mjs_fuzztest)
gtest_discover_tests(mjs_fuzztest)

However when I run cmake --build . as described I get the following error:

[ 40%] Building C object CMakeFiles/mjs_fuzztest.dir/mjs.c.o
In file included from src/mjs_main.c:13:
/home/kali/SUT/mjsFuzzTest/fuzztest/fuzztest/fuzztest.h:18:10: fatal error: 'string' file not found
#include <string>

However when I run the example from the cmake file, the first_fuzz_test.cc, everything works like a charme.

How is it possible, that clang can not find the string header?

I hope to get some suggestion, what I might be doing wrong.

InRange "min must be smaller than max!" awkward when used with FlatMap

I am trying to write a fuzztest for something like the following:

void VectorBoolFlip(std::pair<std::vector<bool>, std::size_t> input)
{
    input.first[input.second].flip();
    // check result ...
}

For which I need input.second < input.first.size() to hold, so I define a new domain:

auto ArbitraryVectorBoolAndIndex()
{
    return FlatMap(
        [](std::size_t size) {
            return PairOf(VectorOf(Arbitrary<bool>()).WithSize(size),
                          InRange(0UL, size - 1));
        },
        InRange(1UL, 1024UL));
}

This, however triggers the following assertion:

fuzztest/fuzztest/internal/domains/in_range_impl.h

Lines 49 to 51 in c5fde4b

 explicit InRangeImpl(T min, T max) : min_(min), max_(max) { 

 FUZZTEST_INTERNAL_CHECK_PRECONDITION(min < max, 

 "min must be smaller than max!");

Since InRange yields values from a closed interval, InRange(c, c) should be semantically equivalent to Just(c). I understand that, when literals are passed, then e.g. InRange(0, 0) should be replace with Just(0). But when variables are used, as in FlatMap, then I see no reason why InRange should be more strict than necessary.

I realize that a workaround is possible:

auto ArbitraryBitSetAndIndex()
{
    return FlatMap(
        [](const std::size_t index) {
            return PairOf(VectorOf(Arbitrary<bool>()).WithMinSize(index + 1),
                          Just(index));
        },
        InRange(0UL, 1024UL));
}

Which led me to #172 ...

Target based fuzztest_setup_fuzzing_flags

#Feature

This would require

fuzztest_setup_fuzzing_flags( )
:= PRIVATE|PUBLIC|INTERFACE

Fuzztest uses internal re2 headers

Hello,

The re2/prog.h header as used by regex.* is not part of the public API.

See:
https://github.com/google/re2/blob/954656f47fe8fb505d4818da1e128417a79ea500/CMakeLists.txt#L149-L154

Could you please work-out a solution with the re2 authors?

Thanks,
Gregor

Add bzlmod support for Bazel

Bazel builds are moving towards a new modules system instead of the workspace files:
https://bazel.build/external/module

Filing this to track any work needed to add a module file for this project. I'll link to it from the request to add this project to the Bazel central registry.

[Question] how to run test for a limited time

Hi,
I'd like to use the framework in open-telemetry/opentelemetry-cpp#2246. I was wondering how I can run tests in limited time e.g.

./sdk/fuzz/atomic_unique_ptr_fuzz --fuzz=atomicIntTest --jobs=8 -workers=8 -runs=100

I had few more questions which I couldn't find in the documentation.

Is there a plan for release?
How can I run my fuzz tests with ctest without knowing the name of the tests.

Thanks

cc_fuzztest_grammar_library bazel macro cannot be used outside of com_google_fuzztest repo

# some/package/BUILD
load("@com_google_fuzztest//build_defs:cc_fuzztest_grammar_library.bzl", "cc_fuzztest_grammar_library")


cc_fuzztest_grammar_library(
    name = "some_grammar",                                                    
    srcs = ["some_grammar.g4"],                                                       
    top_level_rule = "some_rule",                                                      
)

results in

ERROR: no such package 'fuzztest': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /path/to/my/workspace/fuzztest
ERROR: /path/to/my/workspace/some/package/BUILD:122:28: no such package 'fuzztest': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /path/to/my/workspace/fuzztest and referenced by '//some/package:some_grammar'
ERROR: Analysis of target '//some/package:some_grammar' failed; build aborted: Analysis failed

The linked PR fixes the issue.

How to add a new fuzzer for libvpx

Hi all,
I'm trying to understand how to add a new fuzzer to libvpx. Currently I have added a new file called newfuzz.cc inside libvpx/tests/fuzzer and updated the gni file to someting like the following

if (fuzztest_supported) {
  test("vp8_encoder_fuzz_test") {
    sources = [ "tests/fuzzer/vp8_encoder_fuzz_test.cc" ]

    enable_fuzztest = true

    deps = [
      ":libvpx",
      "//third_party/fuzztest:fuzztest_gtest_main",
    ]
  }

  ...
test("newfuzz") {
    sources = [ "tests/fuzzer/new_fuzz.cc" ]

    enable_fuzztest = true

    deps = [
      ":libvpx",
      "//third_party/fuzztest:fuzztest_gtest_main",
    ]
  }
}

The .cc only calls get_vpx_encoder_by_name[1] and the necessary include was added. Unfortunately the reference to the function isn't found. Any idea on what I might have missed?

Thanks

[1] get_vpx_encoder_by_name

Stack overflow due to excessive recursion when using DomainBuilder

I was trying to define a domain of somewhat simple regexes:

// E -> E '|' E | T
// T -> T T | F
// F -> F* | F? | G
// G -> (E) | \\H | [characters]
// H -> \\ | '|' | * | ( | )
auto ArbitraryRegex()
{
    DomainBuilder builder;
    builder.Set<std::string>("E",
                             OneOf(Map([](const std::string& e1, const std::string& e2) { return e1 + '|' + e2; },
                                       builder.Get<std::string>("E"),
                                       builder.Get<std::string>("E")),
                                   builder.Get<std::string>("T")));
    builder.Set<std::string>("T",
                             OneOf(Map([](const std::string& e1, const std::string& e2) { return e1 + e2; },
                                       builder.Get<std::string>("T"),
                                       builder.Get<std::string>("T")),
                                   builder.Get<std::string>("F")));
    builder.Set<std::string>("F",
                             OneOf(Map([](const std::string& f) { return f + '*'; }, builder.Get<std::string>("F")),
                                   Map([](const std::string& f) { return f + '?'; }, builder.Get<std::string>("F")),
                                   builder.Get<std::string>("G")));
    builder.Set<std::string>(
        "G",
        OneOf(Map([](const std::string& e) { return '(' + e + ')'; }, builder.Get<std::string>("E")),
              Map([](const char h) { return std::string{"\\"} + h; }, builder.Get<char>("H")),
              StringOf(AlphaNumericChar()).WithSize(1)));
    builder.Set<char>("H", ElementOf({'\\', '|', '*', '(', ')'}));
    return std::move(builder).Finalize<std::string>("E");
}

The recursion in this domain blows up even 100MB of stack (ulimit -s 100000) almost immediately after starting fuzzing runs. Here's the bottom of the stack trace:

...
#355197 0x00005555557ed20e in fuzztest::internal::DomainBase<fuzztest::internal::AggregateOfImpl<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, (fuzztest::internal::RequireCustomCorpusType)0, fuzztest::Domain<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::UntypedInit (
    this=0x6020000030f0, ref=...) at external/com_google_fuzztest/./fuzztest/internal/domains/domain_base.h:133
#355198 0x0000555555866022 in fuzztest::internal::FuzzTestFuzzerImpl::InitializeCorpus (this=0x613000000040, prng=...)
    at external/com_google_fuzztest/fuzztest/internal/runtime.cc:555
#355199 0x0000555555867290 in fuzztest::internal::FuzzTestFuzzerImpl::RunInFuzzingMode(int*, char***)::$_5::operator()() const (this=0x7fffffffd210)
    at external/com_google_fuzztest/fuzztest/internal/runtime.cc:743
#355200 0x0000555555866fe3 in fuzztest::internal::FuzzTestFuzzerImpl::RunInFuzzingMode (this=0x613000000040)
    at external/com_google_fuzztest/fuzztest/internal/runtime.cc:709
#355201 0x0000555555856ee9 in fuzztest::internal::GTest_TestAdaptor::TestBody (this=0x604000000c50)
    at external/com_google_fuzztest/./fuzztest/googletest_adaptor.h:60
#355202 0x0000555555bb394b in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x604000000c50, 
    method=&virtual testing::Test::TestBody(), location=0x555555c0bf20 "the test body") at external/com_google_googletest/googletest/src/gtest.cc:2599
#355203 0x0000555555b96cad in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x604000000c50, 
    method=&virtual testing::Test::TestBody(), location=0x555555c0bf20 "the test body") at external/com_google_googletest/googletest/src/gtest.cc:2635
#355204 0x0000555555b78f63 in testing::Test::Run (this=0x604000000c50) at external/com_google_googletest/googletest/src/gtest.cc:2674
#355205 0x0000555555b79bad in testing::TestInfo::Run (this=0x612000000640) at external/com_google_googletest/googletest/src/gtest.cc:2853
#355206 0x0000555555b7a44a in testing::TestSuite::Run (this=0x6120000007c0) at external/com_google_googletest/googletest/src/gtest.cc:3012
#355207 0x0000555555b8c6ff in testing::internal::UnitTestImpl::RunAllTests (this=0x616000005780)
    at external/com_google_googletest/googletest/src/gtest.cc:5870
#355208 0x0000555555bb422b in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x616000005780, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555b8c280 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x555555c0c884 "auxiliary test code (environments or event listeners)") at external/com_google_googletest/googletest/src/gtest.cc:2599
#355209 0x0000555555b99633 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x616000005780, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555b8c280 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x555555c0c884 "auxiliary test code (environments or event listeners)") at external/com_google_googletest/googletest/src/gtest.cc:2635
#355210 0x0000555555b8c218 in testing::UnitTest::Run (this=0x5555566bdcb8 <testing::UnitTest::GetInstance()::instance>)
    at external/com_google_googletest/googletest/src/gtest.cc:5444
#355211 0x0000555555854861 in RUN_ALL_TESTS () at external/com_google_googletest/googletest/include/gtest/gtest.h:2293
#355212 0x0000555555851bae in main (argc=2, argv=0x7fffffffdba8) at external/com_google_fuzztest/fuzztest/fuzztest_gtest_main.cc:76

I tried to see if simplifying the domain definition helps, and it does. The following takes a few seconds more to blow up:

auto ArbitraryRecursive()
{
    DomainBuilder builder;
    builder.Set<std::string>("E",
                             OneOf(Map([](const std::string& e1, const std::string& e2) { return e1 + '|' + e2; },
                                       builder.Get<std::string>("E"),
                                       builder.Get<std::string>("E")),
                                   Just(std::string{})));
    return std::move(builder).Finalize<std::string>("E");
}

And the following doesn't run into stack overflows at all:

auto ArbitraryRecursive()
{
    DomainBuilder builder;
    builder.Set<std::string>("E",
                             OneOf(Map([](const std::string& inner) { return inner + '*'; },
                                       builder.Get<std::string>("E")),
                                   Just(std::string{})));
    return std::move(builder).Finalize<std::string>("E");
}

I'm thinking about how I could limit the recursion by changing the domain definition (e.g. introduce a "max depth" parameter. But I can't see how, because I can't pass additional parameters to DomainBuilder::Get.

If Get took additional parameters and Set could take a function (instead of the domain) which in turn takes these parameters and returns a domain, then I could consciously limit recursion, I think 🤔

error: static assertion failed due to requirement 'always_false<unsigned char *>': => Type not supported yet

Hi all,
I'm trying to port the example woff2 fuzzer to bazel. I have prepared the project so I can run simple tests but when I try to mimic the original LibFuzzer entry point I get an error.

This is the code

#include <stddef.h>
#include <stdint.h>
#include "fuzztest/fuzztest.h"
#include "gtest/gtest.h"

#include <woff2/decode.h>

void ConvertWOFF2ToTTF(uint8_t* data, int size) {
}
FUZZ_TEST(AlbertoFuzzTest, ConvertWOFF2ToTTF);

// Entry point for LibFuzzer.
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  std::string buf;
  woff2::WOFF2StringOut out(&buf);
  out.SetMaxSize(30 * 1024 * 1024);
  woff2::ConvertWOFF2ToTTF(data, size, &out);
  return 0;
}

This is the output when I run bazel run --config=fuzztest --copt=-I./include :alberto -- --fuzz=AlbertoFuzzTest.ConvertWOFF2ToTTF

In file included from external/com_google_fuzztest/fuzztest/fuzztest.h:24:
In file included from external/com_google_fuzztest/./fuzztest/domain.h:18:
In file included from external/com_google_fuzztest/./fuzztest/domain_core.h:48:
external/com_google_fuzztest/./fuzztest/internal/domains/arbitrary_impl.h:60:17: error: static assertion failed due to requirement 'always_false<unsigned char *>': => Type not supported yet. Consider filing an issue.
   60 |   static_assert(always_false<T>,
      |                 ^~~~~~~~~~~~~~~
external/com_google_fuzztest/./fuzztest/domain_core.h:398:10: note: in instantiation of template class 'fuzztest::internal::ArbitraryImpl<unsigned char *>' requested here
  398 |   return internal::ArbitraryImpl<T>{};
      |          ^
external/com_google_fuzztest/./fuzztest/internal/registration.h:65:20: note: in instantiation of function template specialization 'fuzztest::internal_no_adl::Arbitrary<unsigned char *>' requested here
   65 |     return TupleOf(Arbitrary<Args>()...);
      |                    ^
external/com_google_fuzztest/./fuzztest/internal/registry.h:84:66: note: in instantiation of member function 'fuzztest::internal::DefaultRegistrationBase<unsigned char *, int>::GetDomains' requested here
   84 |     return [target_function = reg.target_function_, domain = reg.GetDomains(),
      |                                                                  ^
external/com_google_fuzztest/./fuzztest/internal/registry.h:65:18: note: in instantiation of function template specialization 'fuzztest::internal::RegistrationToken::GetFuzzTestFuzzerFactory<fuzztest::internal::DefaultRegistrationBase<unsigned char *, int>, fuzztest::internal::NoFixture, void (*)(unsigned char *, int), void *>' requested here
   65 |                  GetFuzzTestFuzzerFactory(std::move(reg)));
      |                  ^
src/alberto.cc:11:1: note: in instantiation of function template specialization 'fuzztest::internal::RegistrationToken::operator=<fuzztest::internal::DefaultRegistrationBase<unsigned char *, int>, fuzztest::internal::NoFixture, void (*)(unsigned char *, int), void *>' requested here
   11 | FUZZ_TEST(AlbertoFuzzTest, ConvertWOFF2ToTTF);
      | ^
external/com_google_fuzztest/fuzztest/fuzztest.h:66:37: note: expanded from macro 'FUZZ_TEST'
   66 | #define FUZZ_TEST(suite_name, func) INTERNAL_FUZZ_TEST(suite_name, func)
      |                                     ^
external/com_google_fuzztest/./fuzztest/internal/registry.h:107:53: note: expanded from macro 'INTERNAL_FUZZ_TEST'
  107 |           ::fuzztest::internal::RegistrationToken{} =             \
      |                                                     ^
1 error generated.
Target //:alberto failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 7.115s, Critical Path: 6.92s
INFO: 66 processes: 26 internal, 40 linux-sandbox.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target
➜  woff2 git:(ccbeb246d2)

Which type should I use instead? Any other way to fix it?

how FUZZ_TEST enable multithreading

[question] how FUZZ_TEST enable multithreading

Cannot test a pointer type

google/fuzztest - Not working:
static assertion failed due to requirement 'always_false<char *>': => Type not supported yet. Consider filing an issue.

#include <cstdint>

#include <fuzztest/fuzztest.h>
#include <request.hpp>

void ParseRequestFuzz(const uint8_t *raw) {
  parse_request(raw);
}
FUZZ_TEST(Requests, ParseRequestFuzz);

LLVM libfuzzer

#include <cstdint>

#include <request.hpp>

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, std::size_t Size) {
  parse_request(Data);
  return 0;
}

Attempt to turn off `-Wpsabi` causes unknown warning error.

I'm currently trying to include fuzztest as a third party repository via http_repository in my project. I'm encountering the following error:

Step #3 - "presubmit": external/fuzztest/fuzztest/internal/coverage.cc:120:34: error: unknown warning group '-Wpsabi', ignored [-Werror,-Wunknown-warning-option]
Step #3 - "presubmit": #pragma clang diagnostic ignored "-Wpsabi"

It seems that the attempt to turn off the irrelevant -Wpsabi warning in coverage.cc is causing an "unknown warning error" in my compiler. I'm going to turn off the -Wunknown-warning-option to work around this, but thought I'd let y'all know in case there's something you can do to suppress this issue.

Fuzz test not recognized with quickstart with cmake

Hello! I am following the Quickstart with CMake guide provided here: https://github.com/google/fuzztest/blob/main/doc/quickstart-cmake.md and am having some trouble getting the simple fuzz test to be recognized.
My CMakeLists.txt looks as follows (pretty much copy paste from the quickstart guide, except for some minor directory changes since the quickstart guide specifies to create a directory called “first_fuzz”):
cmake_minimum_required(VERSION 3.19)
project(first_fuzz_project)

# GoogleTest requires at least C++17
set(CMAKE_CXX_STANDARD 17)

add_subdirectory(fuzztest)

enable_testing()

include(GoogleTest)

fuzztest_setup_fuzzing_flags()

add_executable(
 first_fuzz_test
 first_fuzz/first_fuzz_test.cc
)

# If you have other dependencies than FuzzTest, link them:
# target_link_libraries(first_fuzz_test PRIVATE other_deps)

link_fuzztest(first_fuzz_test)
gtest_discover_tests(first_fuzz_test)

In the new directory (first_fuzz) that I created, I added the test first_fuzz_test.cc file with the content specified in the quickstart guide:

 #include "fuzztest/fuzztest.h"
#include "gtest/gtest.h"

TEST(MyTestSuite, OnePlustTwoIsTwoPlusOne) {
 EXPECT_EQ(1 + 2, 2 + 1);
}

void IntegerAdditionCommutes(int a, int b) {
 EXPECT_EQ(a + b, b + a);
}
FUZZ_TEST(MyTestSuite, IntegerAdditionCommutes);

I then built the program with the specified cmake flags:

$ CC=clang CXX=clang++ cmake -DCMAKE_BUILD_TYPE=RelWithDebug -DFUZZTEST_FUZZING_MODE=on ..
$ cmake --build .

The build succeeds without any errors and I am able to run the unit tests without any issue. When trying to run the fuzz test, the binary reports containing coverage but there are no tests that are recognized:

$ ./first_fuzz_test --fuzz=MyTestSuite.IntegerAdditionCommutes

[.] Sanitizer coverage enabled. Counter map size: 35469, Cmp map size: 262144

No FUZZ_TEST matches the name: MyTestSuite.IntegerAdditionCommutes

Valid tests:
Just for documentation purposes, I am using Apple clang++ version 14.0.3 on a 2019 Intel Macbook Pro with cmake version 3.26.4.
Is there something missing in the quickstart cmake guide that I am forgetting? Any additional guidance would be helpful, thanks in advance!

Fuzzing a C code leads to undefined reference for fuzztest::internal::IntegralPrinter::PrintUserValue

Hello,
i'm trying to learn how to perform fuzzing on a c library.
My current setup is using a cmake fork (although i don't think that's an issue).
My code is the following:

extern "C" {
  #include "./lib.h"
}

#include "gtest/gtest.h"
#include "fuzztest/fuzztest.h"

void test0API(double var0, double var1){
	_Bool result = test_sum_double(var0, var1);
	EXPECT_TRUE(true);
}

FUZZ_TEST(AutomatedFuzzTesting, test0API)
  .WithDomains(/*var0:*/fuzztest::Arbitrary<double>(), 
               /*var1:*/fuzztest::Arbitrary<double>());

void test1API(int var0, int var1){
	_Bool result = test_div(var0, var1);
	EXPECT_TRUE(true);
}

FUZZ_TEST(AutomatedFuzzTesting, test1API)
  .WithDomains(/*var0:*/fuzztest::Positive<int>(), 
               /*var1:*/fuzztest::Positive<int>());

void test2API(int var0, int var1){
	int result = test_sum_int(var0, var1);
	EXPECT_TRUE(result==0);
}

FUZZ_TEST(AutomatedFuzzTesting, test2API)
  .WithDomains(/*var0:*/fuzztest::Positive<int>(), 
               /*var1:*/fuzztest::Positive<int>());

and the problem i'm facing is the following

/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25
CMakeFiles/test.dir/test.cpp.o: 
	in function `void fuzztest::internal::PrintValue<fuzztest::internal::ArbitraryImpl<double, void> >(fuzztest::internal::ArbitraryImpl<double, void> const&, fuzztest::internal::DomainTraitsImpl<fuzztest::internal::ArbitraryImpl<double, void>, void>::corpus_type const&, absl::FormatRawSink, fuzztest::internal::PrintMode)':
		test.cpp:(.text._ZN8fuzztest8internal10PrintValueINS0_13ArbitraryImplIdvEEEEvRKT_RKNS0_16DomainTraitsImplIS4_vE11corpus_typeEN4absl13FormatRawSinkENS0_9PrintModeE[_ZN8fuzztest8internal10PrintValueINS0_13ArbitraryImplIdvEEEEvRKT_RKNS0_16DomainTraitsImplIS4_vE11corpus_typeEN4absl13FormatRawSinkENS0_9PrintModeE]+0x23f):
			undefined reference to `fuzztest::internal::FloatingPrinter::PrintUserValue(double, absl::FormatRawSink, fuzztest::internal::PrintMode) const'

/usr/bin/ld: CMakeFiles/test.dir/test.cpp.o:
	in function `void fuzztest::internal::IntegralPrinter::PrintUserValue<int>(int, absl::FormatRawSink, fuzztest::internal::PrintMode) const':
		test.cpp:(.text._ZNK8fuzztest8internal15IntegralPrinter14PrintUserValueIiEEvT_N4absl13FormatRawSinkENS0_9PrintModeE[_ZNK8fuzztest8internal15IntegralPrinter14PrintUserValueIiEEvT_N4absl13FormatRawSinkENS0_9PrintModeE]+0x213):
			undefined reference to `fuzztest::internal::IntegralPrinter::PrintUserValue(long, absl::FormatRawSink, fuzztest::internal::PrintMode) const'

clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [SWUT/CMakeFiles/test.dir/build.make:156: SWUT/test] Error 1
make[1]: *** [CMakeFiles/Makefile2:5835: SWUT/CMakeFiles/test.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

In the IntegralPrinter struct i don't see in fact a PrintUserValue(long, ...) but i do see a PrintUserValue(double, ...) so i can't really figure out what's the problem here

how to use fuzztest on Android(aarch64)

Hi, I'm trying to follow the tutorial to build fuzztest for Android(aarch64), but I failed, here is my CMakeLists.txt

cmake_minimum_required(VERSION 3.19)
project(first_fuzz_project)

set(CMAKE_CXX_STANDARD 17)

add_subdirectory(fuzztest)

enable_testing()

include(GoogleTest)

add_executable(
  first_fuzz_test
  first_fuzz_test.cc
)

set(ANDROID_API_VER 29)
set(ABARCH1 arm64)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
set(NDK /home/muhe/Android/Sdk/ndk/21.1.6352462/)
set(CROSS_SYSROOT "${NDK}/platforms/android-${ANDROID_API_VER}/arch-${ABARCH1}")
set(BUILD_ARCH linux-x86_64)

set(TC_PATH ${NDK}/toolchains/llvm/prebuilt/${BUILD_ARCH})
set(TC_BASE ${TC_PATH}/bin/${CMAKE_SYSTEM_PROCESSOR}-linux-android)
set(PLATFORM android)
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_C_COMPILER "${TC_BASE}${ANDROID_API_VER}-clang")
set(CMAKE_CXX_COMPILER "${TC_BASE}${ANDROID_API_VER}-clang++")
set(CMAKE_STAGING_PREFIX "${CROSS_SYSROOT}")

set(CMAKE_FIND_ROOT_PATH "${CROSS_SYSROOT}")
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

link_fuzztest(first_fuzz_test)
gtest_discover_tests(first_fuzz_test)

It seems fuzztest now only support x86:

_deps will build for x86_64, not aarch64 as I specified, so I will get error as following:

_deps/re2-build/libre2.a(compile.cc.o) is incompatible with aarch64linux

fuzztest will link -lrt, so I just deleted all the -lrt to continuing the building process

So does fuzztest currently support aarch64? Or will aarch64 be supported in the future?

Oldest version of clang++ that is needed to efficiently instrument and build target code?

Hi,

there are some industry projects that have to use very old compilers.
I'm stuck in one with clang 8. Will this old version be a drawback when building FuzzTest or instrumenting target software?
Is there any clang version you would recommend to get the most out of FuzzTest?

Thanks!
/Olle

Documenting the 'magic' environment variables

There are some environment variables that influence the behavior, but they are not easily discoverable without browsing / searching the code.

f.e.:

FUZZTEST_MAX_FUZZING_RUNS
FUZZTEST_FUZZ_FOR
FUZZTEST_TESTSUITE_IN_DIR
FUZZTEST_MINIMIZE_TESTSUITE_DIR

etc.

How do we get shrunken test cases?

First of all, great project, I love it! It's super useful. Thanks for sharing it with the world.

I notice that FuzzTest has a concept of shrinking (as many property-based testing libraries and fuzzing libraries do). How do I use this? I've created a fuzztest. I've created Domains for my code under test, and fuzztest has found bugs in my code, but it won't shrink the failing stimulus. It's obviously harder to root cause my errors when fuzztest found the error using a complex stimulus, so how do I get fuzztest to shrink when an error is found?

libfuzzer compatibility mode spams `Unexpected file format` without making any progress (СMake)

Trying to follow quickstart https://github.com/google/fuzztest/blob/f939172/doc/quickstart-cmake.md

Libfuzzer compatibility mode currently is non-functional. Can be reproduced with Clang 16 on Ubuntu 23.10.

$ ./fuzz_test --fuzz=MyTestSuite.Issue
Note: Google Test filter = MyTestSuite.Issue
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MyTestSuite
[ RUN      ] MyTestSuite.Issue
FUZZTEST_PRNG_SEED=2-C28c09q3Dxdz_n7_-gOEsZE7SZ9VbpfJKhdXqOvnw
INFO: found LLVMFuzzerCustomMutator (0x55fea61dade0). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3953037668
INFO: Loaded 1 modules   (45036 inline 8-bit counters): 45036 [0x55fea6b59730, 0x55fea6b6471c), 
INFO: Loaded 1 PC tables (45036 PCs): 45036 [0x55fea6b64720,0x55fea6c145e0), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
[!] Unexpected file format.
INFO: A corpus is not provided, starting from an empty corpus
[!] Unexpected file format.
#2      INITED cov: 98 ft: 99 corp: 1/1b exec/s: 0 rss: 47Mb
[!] Unexpected file format.
        NEW_FUNC[1/275]: 0x55fea61363b0 in Issue(int) /home/alabazov/playground/fuzz_project/fuzz_test.cpp:4
        NEW_FUNC[2/275]: 0x55fea61368a0 in testing::AssertionResult testing::internal::CmpHelperNE<int, int>(char const*, char const*, int const&, int const&) /home/alabazov/playground/fuzz_project/build/_deps/googletest-src/googletest/include/gtest/gtest.h:1456
#3      NEW    cov: 471 ft: 513 corp: 2/33b lim: 4096 exec/s: 0 rss: 47Mb L: 32/32 MS: 1 Custom-
[!] Unexpected file format.
#8      NEW    cov: 471 ft: 515 corp: 3/76b lim: 4096 exec/s: 0 rss: 48Mb L: 43/43 MS: 5 Custom-Custom-Custom-Custom-Custom-
[!] Unexpected file format.
#14     NEW    cov: 471 ft: 517 corp: 4/100b lim: 4096 exec/s: 0 rss: 48Mb L: 24/43 MS: 1 Custom-
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
#140    NEW    cov: 471 ft: 519 corp: 5/130b lim: 4096 exec/s: 0 rss: 51Mb L: 30/43 MS: 1 Custom-
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
#276    REDUCE cov: 471 ft: 519 corp: 5/129b lim: 4096 exec/s: 0 rss: 52Mb L: 31/43 MS: 1 Custom-
#292    NEW    cov: 471 ft: 521 corp: 6/154b lim: 4096 exec/s: 0 rss: 52Mb L: 25/43 MS: 1 Custom-
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.
[!] Unexpected file format.

Test snippet with an obvious failure condition:

#include "fuzztest/fuzztest.h"
#include "gtest/gtest.h"

void Issue(int x) {
  EXPECT_NE(x, 42738947);
}
FUZZ_TEST(MyTestSuite, Issue);

Reading in Corpus

While trying to set an corpus file as the seed for my fuzz test it threw the error

 error: no matching member function for call to 'WithSeeds'
FUZZ_TEST(fuzzCompSuite, fuzzyFloatCompareTest).WithSeeds(fuzztest::ReadFilesFromDirectory(corpusPath));
. . .

/fuzztest/./fuzztest/internal/registration.h:173:8: note: candidate function not viable: no known conversion from 'std::vector<std::tuple<std::string>>' (aka 'vector<tuple<basic_string<char>>>') to 'absl::Span<const SeedT>' (aka 'Span<const tuple<float, float>>') for 1st argument
  auto WithSeeds(absl::Span<const SeedT> seeds) && {

I have also tried it using the format listed in the Macro Doc

FUZZ_TEST(fuzzCompSuite,fuzzyFloatCompareTest).WithSeeds(fuzztest::ReadFilesFromDirectory(absl::StrCat(std::getenv("/fuzzyCompare"), corpusPath)));

How would I go about resolving this issue?
Also where could I find documentation on formatting/generating corpus files

Have `WithSeeds` accept the `GoogleTest` `ParameterGenerator`s (such as `ValuesIn`, `Combine`, etc).

I'm currently in the process of pitching my team on using FUZZ_TEST instead of regular old GoogleTest TEST_P. Most of our tests currently use TEST_P, and my goal is to incrementally convert them from TEST_P to FUZZ_TEST keeping the behavior the same other than fuzzing. This means keeping the deterministically-executed values the same, which I plan to do with WithSeeds.

Unfortunately, WithSeeds is less flexible than the GoogleTest parameter generators in a number of important ways:

In the case where a TestWithParam is parameterized on a single parameter, TestWithParam does not insist that that single parameter is wrapped in a std::tuple, but WithSeeds does.
GoogleTest provides Combine, which allows taking lists of parameter generators and performing the cartesian product of them, producing all possible combinations of the parameter generator's outputs. To replicate this, it looks like I'll either have to manually generate each possible combination or whittle down the inputs and argue to my team that it's ok.

Ideally, I'd like to do neither of these things. I'd like to just continue passing the same parameter generators I sent into INSTANTIATE_TEST_SUITE_WITH_PARAMS into WithSeeds and have it figure it out. Doing so would make convincing others that I am not introducing regressions much simpler.

Memory leak in GetGlobalDomainDefaultInstance function

Hello, I have seen a memory leak in GetGlobalDomainDefaultInstance method in domain.h.

In that function it allocates Domain in heap by calling new, and never delete that. maybe it can be solved by using smart pointers.

Is there documentation on how to use this using git instead of bazel

We are yet to embark on using Bazel. Is Bazel a must to use this? If not, can we get an example of usage?

Indicate Domains that are cheap vs. expensive to vary, and vary the cheap ones first

Is there some way we can indicate that some Domains are expensive to change, while others are cheap and should be varied first? For example, imagine a situation where the code under test can execute much faster if the expensive Domains are unchanged while the cheap Domains are varied arbitrarily. Once the cheap Domains have been 'exhausted' of interesting things by varying them, the expensive Domains are allowed to arbitrarily change exactly once. Then we go back to varying the cheap Domains, rinse and repeat.

A prime example of this would be testing a compiler and the generated program. Assume compilation is expensive while executing the generated program on a specific program input is cheap. The Domains that change the compiler's behavior should be varied less frequently, and only when the generated program has been thoroughly tested by varying all the Domains that are inputs to the generated program. The code under test would, of course, only recompile the program when needed (i.e. when the inputs defining the program to compile have changed), effectively caching the compiled program.

Provide samples of well-known, complexer domains

An example for a well-known domain (for example, xml) would be very helpful.
The idea of xml is fairly simple, but the domain is actually pretty complex.

Without understanding everything, I gobbled together something like this:

auto XmlOpenTag() {
    return InRegexp("<[^(/><.)]+>");
}

auto XmlCloseTag() {
    return InRegexp("</[^(/><.)]+>");
}

auto XmlEmptyTag() {
    return InRegexp("<[^(/><.)]+/>");
}

FUZZ_TEST(XMLTest, XMLReadFromStringHasNoUndefinedBehavior)
    .WithDomains(OneOf(XmlOpenTag(), XmlCloseTag(), XmlEmptyTag(), Arbitrary<std::string>()))
    .WithSeeds([]() -> std::vector<std::tuple<std::string>> {
        return {
            { "<tag></tag>" },
            { "<tag><tag></tag></tag>" },
            { "<tag><tag>garbage</tag></tag>" },
            { "<tag><tag stuff=\"xxxxx\">garbage</tag></tag>" },
            { "<tag><tag><tag /></tag></tag>" },
        };
    });

But this is far from a decent implementation.

	void Mutate(corpus_type& val, absl::BitGenRef prng, bool only_shrink) {
	permanent_dict_candidate_ = std::nullopt;
	FUZZTEST_INTERNAL_CHECK(
	val.size() >= this->min_size_ && val.size() <= this->max_size_, "size ",
	val.size(), " is not between [", this->min_size_, "; ", this->max_size_,
	"]");

	explicit InRangeImpl(T min, T max) : min_(min), max_(max) {
	FUZZTEST_INTERNAL_CHECK_PRECONDITION(min < max,
	"min must be smaller than max!");

google / fuzztest Goto Github PK

fuzztest's Introduction

FuzzTest

What is FuzzTest?

Who is it for?

Who uses it?

How do I use it?

I need help!

fuzztest's People

Contributors

Stargazers

Watchers

Forkers

fuzztest's Issues

Recommend Projects

Recommend Topics

Recommend Org