Git Product home page Git Product logo

google / clusterfuzz Goto Github PK

View Code? Open in Web Editor NEW
5.2K 122.0 535.0 98.66 MB

Scalable fuzzing infrastructure.

Home Page: https://google.github.io/clusterfuzz

License: Apache License 2.0

Python 88.06% HTML 8.26% Shell 1.50% PowerShell 0.28% Dockerfile 0.33% JavaScript 0.15% CSS 0.27% Go 0.26% C++ 0.32% Ruby 0.01% C 0.06% ANTLR 0.35% Starlark 0.03% Jinja 0.05% HCL 0.07%
fuzzing security stability vulnerabilities

clusterfuzz's Introduction

ClusterFuzz

OpenSSF Scorecard

ClusterFuzz is a scalable fuzzing infrastructure that finds security and stability issues in software.

Google uses ClusterFuzz to fuzz all Google products and as the fuzzing backend for OSS-Fuzz.

ClusterFuzz provides many features which help seamlessly integrate fuzzing into a software project's development process:

Overview

Documentation

You can find detailed documentation here.

Trophies

As of February 2023, ClusterFuzz has found ~27,000 bugs in Google (e.g. Chrome). Additionally, ClusterFuzz has helped identify and fix over 8,900 vulnerabilities and 28,000 bugs across 850 projects integrated with OSS-Fuzz.

Getting Help

You can file an issue to ask questions, request features, or ask for help.

Staying Up to Date

We will use clusterfuzz-announce(#)googlegroups.com to make announcements about ClusterFuzz.

ClusterFuzzLite

For a more lightweight version of ClusterFuzz that runs on CI/CD systems, check out ClusterFuzzLite.

clusterfuzz's People

Contributors

adetaylor avatar alhijazi avatar backes avatar davfu1 avatar dependabot[bot] avatar donggeliu avatar dor1s avatar eepeep avatar flowerhack avatar hogo6002 avatar huiziyang avatar hzawawy avatar iamarshsingh avatar inferno-chromium avatar jonathanmetzman avatar kalder avatar kennethkang avatar letitz avatar marktefftech avatar mbarbella-chromium avatar micahbales avatar mpherman2 avatar mspectorgoogle avatar mukundv-chrome avatar navidem avatar oliverchang avatar paulsemel avatar rmistry avatar vanhauser-thc avatar yuanjunh-git avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

clusterfuzz's Issues

gsutil error (missing google-compute-engine)

Running python butler.py create_config --oauth-client-secrets-path=$CLIENT_SECRETS_PATH --project-id=$CLOUD_PROJECT_ID $CONFIG_DIR from the production setup directions for Clusterfuzz on a recently provisoned GCE n1-standard-1 node caused a Python error in the gsutil defstorageclass step: ImportError: No module named google_compute_engine. I worked around it by running pip install google-compute-engine from within the virtualenv environment and re-running the command.

Support development on macOS

We should support macOS development in addition to Linux.

Current tested state:

  • Basic unit tests
  • Local server/bot instance
  • Deployment
  • All tasks

ValueError: virtualenv: cannot access third_party

I setup a local instance of clusterfuzz with git HEAD on linux. This is new setup, so no build/fuzzer/any config changes yet.

There is one bot running. Don't know it's relevant or not.

Sometimes I saw the console log of server has error messages like this:

| ERROR    2019-02-12 01:31:45,273 wsgi.py:263]
| Traceback (most recent call last):
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
|     handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 358, in __getattr__
|     self._update_configs()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 294, in _update_configs
|     self._registry.initialize()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 165, in initialize
|     import_func(self._modname)
|   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
|     __import__(name)
|   File "/home/kcwu/src/clusterfuzz/src/appengine/appengine_config.py", line 87, in <module>
|     vendor.add('third_party')
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/vendor/__init__.py", line 44, in add
|     'No such virtualenv or site directory' % path)
| ValueError: virtualenv: cannot access third_party: No such virtualenv or site directory
| INFO     2019-02-12 01:31:45,278 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 500 -

I saw this message more than once, but I don't know how to reproduce it.
BTW, usually "/_ah/start" is 404.

| INFO     2019-02-12 01:36:47,518 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 404 52

It's not only failure of "/_ah/start". I have ever seen the error with "/cleanup" and "/triage" as well. Like

| ERROR    2019-02-12 01:19:09,738 wsgi.py:263]
| Traceback (most recent call last):
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
|     handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 358, in __getattr__
|     self._update_configs()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 294, in _update_configs
|     self._registry.initialize()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/lib_config.py", line 165, in initialize
|     import_func(self._modname)
|   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
|     __import__(name)
|   File "/home/kcwu/src/clusterfuzz/src/appengine/appengine_config.py", line 87, in <module>
|     vendor.add('third_party')
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/vendor/__init__.py", line 44, in add
|     'No such virtualenv or site directory' % path)
| ValueError: virtualenv: cannot access third_party: No such virtualenv or site directory
| INFO     2019-02-12 01:19:09,741 module.py:861] cron-service: "GET /triage HTTP/1.1" 500 -

Output of gcloud version

Google Cloud SDK 233.0.0
alpha 2019.02.04
app-engine-go
app-engine-python 1.9.82
app-engine-python-extras 1.9.82
beta 2019.02.04
bq 2.0.40
cloud-datastore-emulator 2.1.0
core 2019.02.04
gsutil 4.36
kubectl 2019.02.04
pubsub-emulator 2019.02.04

BTW, I have ever "gcloud auth"ed for my other project. Don't know it's relevant or not since I run clusterfuzz locally.

No fuzzer binaries found

I created a job and uploaded my zip, however it's complaining with:

2019-02-08 13:51:08,359 - run_bot - INFO - Checking for bad build.
2019-02-08 13:51:08,654 - run_bot - ERROR - Error occurred while working on task.
Traceback (most recent call last):
  File "src/python/bot/startup/run_bot.py", line 98, in task_loop
    commands.process_command(task)
  File "/home/vadi/Programs/Mudlet/mudlet1/clusterfuzz/src/python/bot/tasks/commands.py", line 147, in wrapper
    return func(task)
  File "/home/vadi/Programs/Mudlet/mudlet1/clusterfuzz/src/python/bot/tasks/commands.py", line 360, in process_command
    run_command(task_name, task_argument, job_name)
  File "/home/vadi/Programs/Mudlet/mudlet1/clusterfuzz/src/python/bot/tasks/commands.py", line 190, in run_command
    task_module.execute_task(task_argument, job_name)
  File "/home/vadi/Programs/Mudlet/mudlet1/clusterfuzz/src/python/bot/tasks/fuzz_task.py", line 1342, in execute_task
    data_directory, testcase_count)
  File "/home/vadi/Programs/Mudlet/mudlet1/clusterfuzz/src/python/bot/tasks/fuzz_task.py", line 742, in run_fuzzer
    testcase_count)
  File "/home/vadi/Programs/Mudlet/mudlet1/clusterfuzz/src/python/bot/fuzzers/builtin.py", line 72, in run
    'No fuzzer binaries found in |BUILD_DIR| directory.')
BuiltinFuzzerException: No fuzzer binaries found in |BUILD_DIR| directory.

What exactly is it talking about?

Local development: make triage and cleanup cron more reliable (not in production)

  1. I am running local server and local bot.

  2. I add a fuzzing job, upload a custom build. The bot runs and it generates few crashes and these cases appear in "Testcases" page. But it hadn't found certain one crashing testcase I found by other means.

  3. So I uploaded the said crashing testcase (under "Upload Testcase" page)
    It is verified by the bot that the testcase indeed crash.
    This crashing testcase didn't appear in "Testcases" page. Is this expected behavior (uploaded case won't add into testcase collections) ?
    I guess yes, so I continue next steps.

  4. I uploaded a new custom build (under "Jobs" page, change file of existing jobs, then save). Just minor change which doesn't matter.

  5. Add more bots and hope they will find the said crashing case by themselves faster.

  6. The bots picked up the new build and did find the crashing testcase (*).

  7. But I still don't see the crashing test case on "Testcases" page.

(*)
I know bots found the crash because of logs in the fuzzing log inside local/storage/local_gcs/test-fuzz-logs-bucket/objects/

The new testcase is stack-overflow, which is different "crash type" to existing groups (which are timeout, assert, integer-overflow, etc.). So it's not hidden due to grouping.

corpus timeout with use_value_profile=1

I have only one fuzz target, which is libfuzzer-asan. I observed, clusterfuzz will run various tasks, like

  • asan fuzzer with full corpus (using use_value_profile=0)
  • asan fuzzer with full corpus using use_value_profile=1
  • asan fuzzer with subset of corpus

(I only discuss fuzzing with full corpus here)

There are some slow input cases in the corpus. Now the situation is:

  • normal asan fuzzer can do fuzz
  • use_value_profile=1 fuzzer timed out during importing the seed corpus.
    It always timed out at the same case (the same filename).

In other words, the computing resource on "fuzz with full corpus using use_value_profile=1" is wasted.
They are always timeout.

I am not sure this is "works as intended" or not. I think it's better to remove timeout cases from corpus for use_value_profile=1 fuzzers.

Add .lgtm.yml

@evverx thank you for adding ClusterFuzz to LGTM.com.
I created a new issue because I don't want to clutter #126.
On further thought, it seems like adding a .lgtm.yml may be a good idea (even if it means I have to conceed static analysis can be useful :-)

I'm just curious what's the advantage of adding the file if #126 fixes most of the python3-python2 issues LGTM.com is currently running into?

Feel free to add the file in a PR if you'd like. Or I can do this since I (incorrectly) preemptively rejected the PR last time :)
CC @cclauss

Frequent port bindings errors

While running the heartbleed example this error occurs pretty frequently:

| INFO     2019-02-08 13:34:16,999 module.py:861] default: "GET /_ah/stop HTTP/1.1" 503 -
| Traceback (most recent call last):
| Traceback (most recent call last):
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 96, in <module>
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 96, in <module>
|         _run_file(__file__, globals())
| _run_file(__file__, globals())
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 90, in _run_file
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 90, in _run_file
|         execfile(_PATHS.script_file(script_name), globals_)
| execfile(_PATHS.script_file(script_name), globals_)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 199, in <module>
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 199, in <module>
|         main()
| main()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 188, in main
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 188, in main
|         server.start()
| server.start()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 406, in start
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 406, in start
|         raise BindError('Unable to bind %s:%s' % self.bind_addr)
| raise BindError('Unable to bind %s:%s' % self.bind_addr)
| google.appengine.tools.devappserver2.wsgi_servergoogle.appengine.tools.devappserver2.wsgi_server..BindErrorBindError: : Unable to bind localhost:19661Unable to bind localhost:19661
| 
| Traceback (most recent call last):
| Traceback (most recent call last):
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 96, in <module>
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 96, in <module>
|         _run_file(__file__, globals())
| _run_file(__file__, globals())
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 90, in _run_file
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/_python_runtime.py", line 90, in _run_file
|         execfile(_PATHS.script_file(script_name), globals_)
| execfile(_PATHS.script_file(script_name), globals_)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 199, in <module>
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 199, in <module>
|         main()
| main()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 188, in main
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/runtime.py", line 188, in main
|         server.start()
| server.start()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 406, in start
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 406, in start
|         raise BindError('Unable to bind %s:%s' % self.bind_addr)
| raise BindError('Unable to bind %s:%s' % self.bind_addr)
| google.appengine.tools.devappserver2.wsgi_servergoogle.appengine.tools.devappserver2.wsgi_server..BindErrorBindError: : Unable to bind localhost:20010Unable to bind localhost:20010
| 
| ERROR    2019-02-08 13:34:18,749 module.py:974] Request to '/_ah/start' failed
| INFO     2019-02-08 13:34:18,750 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 500 -
| ERROR    2019-02-08 13:34:18,750 module.py:974] Request to '/_ah/start' failed
| INFO     2019-02-08 13:34:18,750 module.py:861] default: "GET /_ah/start HTTP/1.1" 500 -
| INFO     2019-02-08 13:34:18,961 instance.py:294] Instance PID: 11506
| INFO     2019-02-08 13:34:19,036 instance.py:294] Instance PID: 11519
| INFO     2019-02-08 13:34:19,087 instance.py:294] Instance PID: 11521
| INFO     2019-02-08 13:34:19,120 instance.py:294] Instance PID: 11523
| INFO     2019-02-08 13:34:19,177 instance.py:294] Instance PID: 11524
| INFO     2019-02-08 13:34:19,993 instance.py:294] Instance PID: 11513
| INFO     2019-02-08 13:34:20,082 module.py:861] default: "GET /_ah/stop HTTP/1.1" 500 1954
| INFO     2019-02-08 13:34:20,150 module.py:861] cron-service: "GET /_ah/stop HTTP/1.1" 500 1954
| INFO     2019-02-08 13:34:20,487 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 404 52
| INFO     2019-02-08 13:34:20,697 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 404 52
| INFO     2019-02-08 13:34:20,894 module.py:861] default: "GET /_ah/start HTTP/1.1" 404 52
| INFO     2019-02-08 13:34:20,965 module.py:861] default: "GET /_ah/start HTTP/1.1" 404 52
| INFO     2019-02-08 13:34:21,080 module.py:861] default: "GET /_ah/start HTTP/1.1" 404 52
| INFO     2019-02-08 13:34:21,104 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 404 52

Document monitoring workflow

e.g. how to use Error Reporting dashboard, check logs

I've also noticed a bunch of error during production testing, those might scare the people, might be good to mention some of them or maybe even downgrade to warnings?

linting in CI works differently than locally

I'm pretty sure it is pylint 1.9.4.
See #185 for a discrepancy I noticed between running pylint locally and running it in CI.

When I upgraded my local copy of pylint to 1.9.4 I was able to discover the issue showing up in CI.

While install dependencies using `local/install_deps.bash` on macOS with go installed, it won't install all dependencies.

On my macOS, I already installed python and go using brew. When I install dependencies using local/install_deps.bash, it always stops after running brew install bazel golang node nodeenv npm pkill python@2 xz command.

Logs like below:

$ local/install_deps_macos.bash
+ which brew
+ bazel_tap=bazelbuild/tap
+ brew tap
+ grep bazelbuild/tap
Error: Another active Homebrew update process is already in progress.
Please wait for it to finish or terminate it to continue.
+ brew tap --list-pinned
+ grep bazelbuild/tap
Error: Another active Homebrew update process is already in progress.
Please wait for it to finish or terminate it to continue.
+ brew install bazel golang node nodeenv npm pkill python@2 xz
Error: Another active Homebrew update process is already in progress.
Please wait for it to finish or terminate it to continue.
Warning: bazel is provided by core, but is now shadowed by bazelbuild/tap/bazel.
To refer to the core formula, use Homebrew/core/bazel instead.
Warning: bazelbuild/tap/bazel 0.22.0 is already installed and up-to-date
To reinstall 0.22.0, run `brew reinstall bazel`
Warning: node 11.10.0 is already installed, it's just not linked
You can use `brew link node` to link this version.
Warning: nodeenv 1.3.3 is already installed and up-to-date
To reinstall 1.3.3, run `brew reinstall nodeenv`
Warning: proctools 0.4pre1 is already installed and up-to-date
To reinstall 0.4pre1, run `brew reinstall proctools`
Warning: xz 5.2.4 is already installed and up-to-date
To reinstall 5.2.4, run `brew reinstall xz`
Error: go 1.5.3 is already installed
To upgrade to 1.11.5, run `brew upgrade go`
Error: python@2 2.7.15_1 is already installed
To upgrade to 2.7.15_3, run `brew upgrade python@2`

After checking the shell script, I found it starts with #!/bin/bash -ex , which would exit the script when first error occurs. It seems brew would throws error if some component with older version is installed before.

I replace #!/bin/bash -ex with #!/bin/bash -x as workaround, but it seems not perfect.

Hoping this issue could be fix.

Copying test case metadata from page removes newlines

When you copy the metadata logs:

image

It all comes out like the following:

[2019-02-08 13:06:35 UTC] mudlet1: Fuzz task : Fuzzer libFuzzer_handshake-fuzzer generated testcase crashed in 3 seconds (r1). [2019-02-08 13:09:01 UTC] mudlet1: Minimize task started. [2019-02-08 13:24:16 UTC] mudlet1: Minimize task finished. [2019-02-09 00:19:37 UTC] mudlet1: Progression task started. [2019-02-09 00:19:39 UTC] mudlet1: Progression task finished.

python process pileup, needs reproduction steps.

Terrible issue title but we'll fix it to something better (or close it) with more investigation.

I left clusterfuzz running overnight trying to find heartbleed and the fuzzer seems to have hung according to the bot logs:

2019-02-09 03:04:35,256 - run_bot - INFO - Locating generated test cases.
2019-02-09 03:04:35,257 - run_bot - INFO - Generated 4/4 testcases.
2019-02-09 03:04:35,269 - run_bot - INFO - Uploaded file to logs bucket.
2019-02-09 03:04:35,331 - run_bot - INFO - Recorded use of fuzz target libFuzzer_handshake-fuzzer.
2019-02-09 03:04:38,720 - run_bot - INFO - 503 corpus files for target handshake-fuzzer synced to disk.
2019-02-09 03:04:38,768 - run_bot - INFO - Starting to process testcases.
2019-02-09 03:04:38,768 - run_bot - INFO - Redzone is 64 bytes.
2019-02-09 03:04:38,769 - run_bot - INFO - Timeout multiplier is 1.0.
2019-02-09 03:04:38,773 - run_bot - INFO - App launch command is python /home/vadi/Programs/Mudlet/mudlet1/clusterfuzz/src/python/bot/fuzzers/libFuzzer/launcher.py .
2019-02-09 04:25:11,600 - run_bot - WARNING - Hang detected.
None
2019-02-09 04:25:11,600 - run_bot - INFO - Upto 1
2019-02-09 05:45:44,376 - run_bot - WARNING - Hang detected.
None
2019-02-09 05:45:44,376 - run_bot - INFO - Upto 2
2019-02-09 07:06:17,193 - run_bot - WARNING - Hang detected.
None
2019-02-09 07:06:17,194 - run_bot - INFO - Upto 3

(full log)

This has resulted in quite a few Python processes from clusterfuzz maxing out the CPU from the system and the clusterfuzz server from what it looks like trying to stop somewhat frantically: https://paste.ubuntu.com/p/b8JW2sX44w/

The end result of this is that the web interface is unresponsive and all calls time out with a 503.

Local development: Hide crash statistics section in testcase page

If I go to view fuzzer statistics or crash statistics on the main page, I get a nice error message saying this isn't supported on the local development environment.

However, if I have go to the testcases page, view a test case, and click the "CLICK TO SEE THE STATS" button it will spin for a while (10-20 secs) and then eventually fail with this credential related error message (at the end).

I assume this is because the feature is not supported on the local dev environment. If so, can we show that as a more explicit error message, or perhaps not show the button at all? This would be a minor UX improvement.

| WARNING  2019-02-14 04:19:12,446 urlfetch_stub.py:575] Stripped prohibited headers from URLFetch request: ['content-length']
| INFO     2019-02-14 04:19:12,610 google_auth_httplib2.py:211] Refreshing credentials due to a 401 response. Attempt 1/2.
| ERROR    2019-02-14 04:19:12,621 logs.py:327] Retrying on google_cloud_utils.big_query.Client.raw_query failed with Anonymous credentials cannot be refreshed.. Raise.
| Traceback (most recent call last):
|   File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
|     self.__bootstrap_inner()
|   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
|     self.run()
|   File "/usr/lib/python2.7/threading.py", line 754, in run
|     self.__target(*self.__args, **self.__kwargs)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/thread_executor.py", line 41, in _worker
|     result = fn(*args, **kwargs)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 116, in _handle
|     obj.communicate()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 1302, in communicate
|     req.respond()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 831, in respond
|     self.server.gateway(self).respond()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 2115, in respond
|     response = self.req.server.wsgi_app(self.env, self.start_response)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 292, in __call__
|     return app(environ, start_response)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/request_rewriter.py", line 314, in _rewriter_middleware
|     response_body = iter(application(environ, wrapped_start_response))
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 160, in __call__
|     response = self.handle_normal_request(environ)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 195, in handle_normal_request
|     self._PYTHON_LIB_DIR)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/runtime.py", line 159, in HandleRequest
|     error)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 329, in HandleRequest
|     return WsgiRequest(environ, handler_name, url, post_data, error).Handle()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 267, in Handle
|     result = handler(dict(self._environ), self._StartResponse)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1505, in __call__
|     rv = self.router.dispatch(request, response)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
|     return route.handler_adapter(request, response)
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1077, in __call__
|     return handler.dispatch()
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
|     return method(*args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/libs/handler.py", line 362, in wrapper
|     return func(self)
|   File "/home/mhl/clusterfuzz/src/appengine/libs/handler.py", line 310, in wrapper
|     return func(self, testcase)
|   File "/home/mhl/clusterfuzz/src/appengine/handlers/testcase_detail/crash_stats.py", line 54, in post
|     self.render_json(get_result(testcase, end, block, days, group_by))
|   File "/home/mhl/clusterfuzz/src/appengine/handlers/testcase_detail/crash_stats.py", line 33, in get_result
|     _, rows = crash_stats.get(query, crash_stats.Query(), 0, 1)
|   File "/home/mhl/clusterfuzz/src/appengine/libs/crash_stats.py", line 63, in get
|     limit=limit)
|   File "/home/mhl/clusterfuzz/src/appengine/python/metrics/crash_stats.py", line 163, in get
|     result = client.query(query=sql, offset=offset, limit=limit)
|   File "/home/mhl/clusterfuzz/src/appengine/python/google_cloud_utils/big_query.py", line 241, in query
|     result = self.raw_query(query, max_results=0)
|   File "/home/mhl/clusterfuzz/src/appengine/python/base/retry.py", line 101, in _wrapper
|     if not handle_retry(num_try, exception=e):
|   File "/home/mhl/clusterfuzz/src/appengine/python/base/retry.py", line 78, in handle_retry
|     total=tries)
| LogError: Retrying on google_cloud_utils.big_query.Client.raw_query failed with Anonymous credentials cannot be refreshed.. Raise.
| Traceback (most recent call last):
|   File "/home/mhl/clusterfuzz/src/appengine/python/base/retry.py", line 88, in _wrapper
|     result = func(*args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/python/google_cloud_utils/big_query.py", line 184, in raw_query
|     projectId=self.project_id, body=body).execute()
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/googleapiclient/_helpers.py", line 130, in positional_wrapper
|     return wrapped(*args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/googleapiclient/http.py", line 837, in execute
|     method=str(self.method), body=self.body, headers=self.headers)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/googleapiclient/http.py", line 162, in _retry_request
|     resp, content = http.request(uri, method, *args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/google_auth_httplib2.py", line 213, in request
|     self.credentials.refresh(self._request)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/google/auth/credentials.py", line 146, in refresh
|     raise ValueError("Anonymous credentials cannot be refreshed.")
| ValueError: Anonymous credentials cannot be refreshed.
| ERROR    2019-02-14 04:19:12,686 base_handler.py:198] Anonymous credentials cannot be refreshed.
| Traceback (most recent call last):
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
|     return method(*args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/libs/handler.py", line 362, in wrapper
|     return func(self)
|   File "/home/mhl/clusterfuzz/src/appengine/libs/handler.py", line 310, in wrapper
|     return func(self, testcase)
|   File "/home/mhl/clusterfuzz/src/appengine/handlers/testcase_detail/crash_stats.py", line 54, in post
|     self.render_json(get_result(testcase, end, block, days, group_by))
|   File "/home/mhl/clusterfuzz/src/appengine/handlers/testcase_detail/crash_stats.py", line 33, in get_result
|     _, rows = crash_stats.get(query, crash_stats.Query(), 0, 1)
|   File "/home/mhl/clusterfuzz/src/appengine/libs/crash_stats.py", line 63, in get
|     limit=limit)
|   File "/home/mhl/clusterfuzz/src/appengine/python/metrics/crash_stats.py", line 163, in get
|     result = client.query(query=sql, offset=offset, limit=limit)
|   File "/home/mhl/clusterfuzz/src/appengine/python/google_cloud_utils/big_query.py", line 241, in query
|     result = self.raw_query(query, max_results=0)
|   File "/home/mhl/clusterfuzz/src/appengine/python/base/retry.py", line 88, in _wrapper
|     result = func(*args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/python/google_cloud_utils/big_query.py", line 184, in raw_query
|     projectId=self.project_id, body=body).execute()
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/googleapiclient/_helpers.py", line 130, in positional_wrapper
|     return wrapped(*args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/googleapiclient/http.py", line 837, in execute
|     method=str(self.method), body=self.body, headers=self.headers)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/googleapiclient/http.py", line 162, in _retry_request
|     resp, content = http.request(uri, method, *args, **kwargs)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/google_auth_httplib2.py", line 213, in request
|     self.credentials.refresh(self._request)
|   File "/home/mhl/clusterfuzz/src/appengine/third_party/google/auth/credentials.py", line 146, in refresh
|     raise ValueError("Anonymous credentials cannot be refreshed.")
| ValueError: Anonymous credentials cannot be refreshed.

Put all the links in the docs at the end of file?

Fairly minor thing, but it might be good to address it later today. Right now, most of the pages have links specified at the end. Some pages though, have links (in a form of [link]: https://url) throughout the text. I think it's more convenient to put the links at the end. If you need to linkify a common word like "this" or "page", which may have collisions, just put a link in () right after the word.

Unable to run a local server

I am using macOS Mojave 10.14.3

Whenever I execute python butler.py run_server --bootstrap, the following error is returned.

| dev_appserver.py: error: unrecognized arguments: --datastore_emulator_port=9004
| Return code is non-zero (2).
| Exit.
Bootstrapping datastore...
Running: python butler.py run setup --non-dry-run --local --config-dir=configs/test
| DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "src/local/butler/run_server.py", line 106, in trigger
    response = urllib2.urlopen(request, timeout=request_timeout)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 61] Connection refused>

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "src/local/butler/run_server.py", line 106, in trigger
    response = urllib2.urlopen(request, timeout=request_timeout)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 61] Connection refused>

README clarification around language support

I know fuzzing is for testing inputs/outputs and is essentially language-agnostic, but it'd be nice if there's some mention of that in the README, or examples of running Clusterfuzz against code that's written in all sorts of languages (Chrome and C and C++ getting name-dropped is cool, but seeing Python or Ruby explicitly mentioned is cooler ๐Ÿ˜‰ )

Fuzzing a GUI application

Couldn't find a mailing list / forum, and the readme says to ask questions here, so here's a question from a beginner -

I'm just getting into fuzzing now with the open-sourcing of clusterfuzz and looking to set it up to an open-source project I contribute to, Mudlet. It's a GUI application that connects to a telnet port, gets input from the player and sends it back and forth.

I thought I'd start with the telnet class that handles incoming data. If we can ensure that can handle any input thrown at us, then that's great. I've started out by refactoring the socket data read to be fuzz-friendly.

Next, fuzz target that spawns my ctelnet class and passes in the inputs. Problem creating ctelnet - it requires an application object to read network-specific settings and to interact with the other parts of the application as a result of the incoming text stream.

What is the best advice here to continue? Should I create the whole application (which would be a more realistic scenario)?

Typo in repo description?

Hey,
shouldn't this be "all your bugs belong to us"?
If not shouldn't it be somewhere on the readme that this is intentional :P
image

Add a question on other language support to an FAQ

We've gotten enough questions on this that it makes sense to have a ready answer on whether ClusterFuzz supports languages other than C/C++.
It should mention:

  1. How rust (and possibly other LLVM langs) works and go will probably be added.
  2. Why fuzzing other languages probably won't serve the same function as fuzzing C/C++
  3. What needs to be done to add support for another language.

local/install_deps.bash seems to fail mysteriously when there isn't enough memory

I launched a VM where install_deps.bash failed with

Running: bazel run //local:create_gopath 2>/dev/null
| Return code is non-zero (1).
| Exit.

, which wasn't very helpful. Without 2 >/dev/null I was able to figure out what happened but unfortunately additionally broke something else because apparently the output is used to set GOPATH :-) I think it would be great if the output wasn't sent to "/dev/null" here.

(I relaxed the limits a bit and got past install_deps.bash and now python butler.py run_server --bootstrap seems to be stuck trying to "build templates for App Engine". The machine is totally unresponsive so it's hard to tell what's going on. I'm going to tweak the limits until it works but it would probably be useful to mention in the documentation how a "typical" machine is supposed to look like, that is something like https://github.com/google/oss-fuzz/blob/master/docs/faq.md#what-are-the-specs-on-your-machines maybe).

Did not find com_google_cloud_go//datastore package

Hit this issue while running python butler.py bootstrap. What is the happen? THANKS
| ERROR: Analysis of target '//local:create_gopath' failed; build aborted: no such package '@com_google_cloud_go//datastore': failed to fetch com_google_cloud_go: 2019/02/15 12:35:41 unrecognized import path "cloud.google.com/go"

local/install_deps.bash support http proxy on ubuntu18.04

when run on ubuntu 18.04 with root user and http proxy,there is some error occur at the end:

Running: bazel run //local:create_gopath (cwd='src')
| INFO: Invocation ID: 637ff326-b2bb-4fb5-a4ca-270784072b85
| Loading:
| Loading: 0 packages loaded
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| ERROR: /home/clusterfuzz/src/go/server/cron/BUILD.bazel:3:1: no such package '@com_google_cloud_go//datastore': failed to fetch com_google_cloud_go: 2019/02/19 15:13:43 unrecognized import path "cloud.google.com/go"
|  and referenced by '//go/server/cron:go_default_library'
| DEBUG: Rule 'org_golang_x_sys' modified arguments {"shallow_since": "1538141909 +0000"}
| DEBUG: Rule 'org_golang_x_tools' modified arguments {"sha256": "2384fa91351a7414b643c5230422ce45f5aa2be8a82727609afd4e64e6973a30"}
| ERROR: Analysis of target '//local:create_gopath' failed; build aborted: no such package '@com_google_cloud_go//datastore': failed to fetch com_google_cloud_go: 2019/02/19 15:13:43 unrecognized import path "cloud.google.com/go"
| INFO: Elapsed time: 35.316s
| INFO: 0 processes.
| FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
| ERROR: Build failed. Not running target
| FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
| Return code is non-zero (1).
| Exit.
(ENV) root@xxxx:/home/clusterfuzz

it is because can not get go packages,but i has config http proxy,so there was some other solution?

Local development: Uploading large number of input cases times out web ui for some time

How to reproduce

  1. run a server and some bots locally

  2. add a job, activate the fuzzing job. So bots started to fuzz.

  3. Upload a zip file as input testcases. The zip file contains thousands cases.

Then one bot picked up the zip and working on those cases.
According to its bot.log, the bot happily updates testcases continuously (about 5 per second).

2019-02-12 23:57:08,090 - run_bot - INFO - Updated testcase 5059196596781056 (bug -).

So far it looks good. But something went wrong:

  1. About one minute later, this message appeared in server's console output continuously (several per second).
| INFO     2019-02-12 15:45:21,771 module.py:861] cron-service: "GET /_ah/stop HTTP/1.1" 503 -

I don't know this is harmless or not. The said bot still update testcases continously according its bot.log.

  1. The web ui is broken.
    Any access to http://localhost:9000/ shows 503 - This request has timed out.
    After I accessed the above link, the server console started to output below message continuously (several per second) --- but I only access above link once.
| INFO     2019-02-12 15:50:55,732 module.py:861] default: "GET /_ah/stop HTTP/1.1" 503 -

So, the current situation is:

  1. bot still updates test cases
  2. lots of error message in server's conole
  3. I cannot access web ui.

If I restart the server, it will keep silence for a while and then repeat above issue again (output error message....).

Unable to run a local server

problem

After install successfully, I run a local server by

python butler.py run_server
# or
python butler.py run_server --bootstrap

it raise exception(with http_proxy and https_proxy set)

Running: python polymer_bundler.py (cwd='local')
| App Engine templates are up to date.
Clearing local datastore by removing local/storage.
Created symlink: source: /home/test/clusterfuzz/local/storage/local_gcs, target /home/test/clusterfuzz/src/appengine/local_gcs.
Traceback (most recent call last):
  File "butler.py", line 282, in <module>
    main()
  File "butler.py", line 256, in main
    command.execute(args)
  File "src/local/butler/run_server.py", line 162, in execute
    test_utils.setup_pubsub(constants.TEST_APP_ID)
  File "/home/test/clusterfuzz/src/python/tests/test_libs/test_utils.py", line 308, in setup_pubsub
    _create_pubsub_topic(client, project, queue['name'])
  File "/home/test/clusterfuzz/src/python/tests/test_libs/test_utils.py", line 284, in _create_pubsub_topic
    if client.get_topic(full_name):
  File "/home/test/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 193, in get_topic
    response = self._execute_with_retry(request)
  File "/home/test/clusterfuzz/src/python/base/retry.py", line 88, in _wrapper
    result = func(*args, **kwargs)
  File "/home/test/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 108, in _execute_with_retry
    return request.execute()
  File "/home/test/clusterfuzz/src/third_party/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/test/clusterfuzz/src/third_party/googleapiclient/http.py", line 837, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/home/test/clusterfuzz/src/third_party/googleapiclient/http.py", line 162, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/home/test/clusterfuzz/src/third_party/google_auth_httplib2.py", line 198, in request
    uri, method, body=body, headers=request_headers, **kwargs)
  File "/home/test/clusterfuzz/src/third_party/httplib2/__init__.py", line 1694, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/test/clusterfuzz/src/third_party/httplib2/__init__.py", line 1434, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/test/clusterfuzz/src/third_party/httplib2/__init__.py", line 1390, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 402, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

tried solutions

I updated my goodle cloud sdk to newest, not working.
I unset all my proxy setting unset http_proxy, unset https_proxy, then rerun a local server, error message shows

Running: bower install --allow-root
| bower webcomponentsjs extra-resolution Unnecessary resolution: webcomponentsjs#^v1.0.2
| bower iron-location   extra-resolution Unnecessary resolution: iron-location#1 - 2
| bower mocha           extra-resolution Unnecessary resolution: mocha#^3.1.2
| bower iron-collapse   extra-resolution Unnecessary resolution: iron-collapse#^2.0.0
Traceback (most recent call last):
  File "butler.py", line 282, in <module>
    main()
  File "butler.py", line 256, in main
    command.execute(args)
  File "src/local/butler/run_server.py", line 132, in execute
    common.install_dependencies()
  File "src/local/butler/common.py", line 278, in install_dependencies
    _install_chromedriver()
  File "src/local/butler/common.py", line 185, in _install_chromedriver
    version_request = urllib2.urlopen(constants.CHROMEDRIVER_VERSION_URL)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 101] Network is unreachable>

update:
I tried start server without install deps

python butler.py run_server --skip-install-deps

it get stucked

Running: pkill -KILL -f "dev_appserver.py"
| Return code is non-zero (-9).
Running: pkill -KILL -f "CloudDatastore.jar"
| Return code is non-zero (-9).
Running: pkill -KILL -f "pubsub-emulator"
| Return code is non-zero (-9).
Created symlink: source: /home/test/clusterfuzz/configs/test, target /home/test/clusterfuzz/src/appengine/config.
Created symlink: source: /home/test/clusterfuzz/configs/test, target /home/test/clusterfuzz/src/go/server/config.
Created symlink: source: /home/test/clusterfuzz/src/protos, target /home/test/clusterfuzz/src/appengine/protos.
Created symlink: source: /home/test/clusterfuzz/src/python, target /home/test/clusterfuzz/src/appengine/python.
Created symlink: source: /home/test/clusterfuzz/src/third_party, target /home/test/clusterfuzz/src/appengine/third_party.
Running: bazel run //local:create_gopath (cwd='src')
| INFO: Invocation ID: ccdb6b13-75dc-4c70-aa3b-ae0fc7f6d447
| Loading:
| Loading: 0 packages loaded
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| INFO: Analysed target //local:create_gopath (0 packages loaded, 0 targets configured).
| INFO: Found 1 target...
| [0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
| Target //local:create_gopath up-to-date:
|   bazel-bin/local/create_gopath
| INFO: Elapsed time: 0.243s, Critical Path: 0.00s
| INFO: 0 processes.
| INFO: Build completed successfully, 1 total action
| INFO: Running command line: bazel-bin/local/create_gopath local/gopath
| INFO: Build completed successfully, 1 total action
| /home/test/.cache/bazel/_bazel_test/5d84fedffd4e1a64944d7ddd9fdf1c97/execroot/__main__/bazel-out/k8-fastbuild/bin/local/gopath
Running: python polymer_bundler.py (cwd='local')
| App Engine templates are up to date.
Created symlink: source: /home/test/clusterfuzz/local/storage/local_gcs, target /home/test/clusterfuzz/src/appengine/local_gcs.

Heartbleed example does not work out of the box.

I was going over the documentation on this page: https://google.github.io/clusterfuzz/setting-up-fuzzing/heartbleed-example/

To do that I first followed the pre-requisites section here: https://google.github.io/clusterfuzz/setting-up-fuzzing/libfuzzer-and-afl/#compiler

That recommends installing clang 6.0 or greater. I installed it via apt and got clang version 6.0.0-1ubuntu2 as output by clang -v

The example handshake-fuzzer.cc does not compile out of the box with that version of clang as I believe it only supports the filesystem extension as experimental feature.

When I try to run the build step from the documentation:

/usr/bin/clang++ -g handshake-fuzzer.cc -fsanitize=address,fuzzer openssl-1.0.1f/libssl.a openssl-1.0.1f/libcrypto.a -std=c++17 -Iopenssl-1.0.1f/include/ -lstdc++fs -ldl -lstdc++ -o handshake-fuzzer

I get this error:

handshake-fuzzer.cc:25:10: fatal error: 'filesystem' file not found
#include <filesystem>
         ^~~~~~~~~~~~
1 error generated.

This can be fixed by changing the include to experimental/filesystem and then changing the using statement below to also include experimental at the right spot - but I imagine that will break the build on later versions of clang (I haven't tested).

A proper fix likely involves using an API that's available in clang 6 (though it may be more verbose).

Document port forwarding for 900*, maybe other ports if server accessed from remote.

I'm running a local instance of clusterfuzz. I just run the server with default settings.

$ python butler.py run_server

Then I follows this guide https://google.github.io/clusterfuzz/setting-up-fuzzing/libfuzzer-and-afl/ ,

I can "add new job". But it failed if I upload custom fuzzer zip to the job.

| ERROR    2019-02-11 15:44:09,029 base_handler.py:198] Failed to upload.
| Traceback (most recent call last):
|   File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
|     return method(*args, **kwargs)
|   File "/home/kcwu/src/clusterfuzz/src/appengine/libs/handler.py", line 286, in wrapper
|     return func(self, *args, **kwargs)
|   File "/home/kcwu/src/clusterfuzz/src/appengine/libs/handler.py", line 413, in wrapper
|     return func(self, *args, **kwargs)
|   File "/home/kcwu/src/clusterfuzz/src/appengine/handlers/jobs.py", line 126, in post
|     blob_info = self.get_upload()
|   File "/home/kcwu/src/clusterfuzz/src/appengine/handlers/base_handler.py", line 248, in get_upload
|     raise helpers.EarlyExitException('Failed to upload.', 500)
| EarlyExitException: Failed to upload.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.