Git Product home page Git Product logo

vulncode-db's Introduction

Vulncode-DB

License

Deprecation notice

Note: This project will be discontinued after December 13th, 2021.

Overview

The vulnerable code database (Vulncode-DB) is a database for vulnerabilities and their corresponding source code if available. The database extends the NVD / CVE data sets with user-supplied information regarding patch links, vulnerable code offsets and descriptions. Particularly, the database intends to make real-world examples of vulnerable code universally accessible and useful.

The main instance is hosted on vulncode-db.com and more context is provided at vulncode-db.com/about.

Why is this project deprecated?

  • Bootstrapping problem - Vulncode-DB 's usefulness depends on having unique content. We can automatically detect some vulnerability patches via CVE/NVD metadata. We can also highlight relevant sections and annotate them in a write-up fashion. We also allow users to modify or annotate content themselves. However, this by itself is insufficient to make anyone use the platform. You need much and high-quality data first to make this useful, which a prototype like ours can't attain at this stage without extensive investment.
  • Lack of community support - While there was some positive feedback there have been only a few contributors. The platform and vision seem to be inadequate to get more practical support.
  • Insufficient resources - Developing the platform and for example a feature like a version control system for user moderated content similar to Wikipedia requires much engineering work for which we, as 20% contributors, are understaffed.
  • Added value unknown - Even if all of the above would be solved it's still unclear whether the platform would provide sufficient value for individuals to justify a dedicated project. You can go to CVE details or Google for write-ups to learn more about a vulnerability. This is an established habit, hard to break and might already be good enough for individuals to learn more.

How and when?

  • This repository will be kept alive. However, we'll discontinue the https://vulncode-db.com website and API after December 13th, 2021.

Do you have feedback/ideas for how it should be continued?

Finally, thank you to all contributors and individuals who supported the project. We are very grateful for your support, time and feedback.

Best,

Ruslan and Timo

Directory structure

├── app
│   └── [submodules with Flask routes and views]
├── cert (SSL certificates)
├── data
│   ├── forms
│   └── models (Database models)
├── docker (Docker files)
├── lib (helping libraries)
│   └── vcs_handler
├── migrations (Flask-Migrate / Alembic files)
├── static (CSS, JS and other static files)
│   ├── css
│   ├── js
│   │   └── lib
│   ├── monaco
│   │   └── themes
│   └── tutorial
├── templates (Jinja2 templates)
│   └── editor
│   └── macros
├── tests (Unit tests)
├── third_party (Third-party content)
└── vulnerable_code (Temporary directory used for caching repositories)

Setup

The setup is simplified with Docker and docker-compose in particular. Having these prerequisites installed you can setup the project using the following instructions:

# Clone the repository and its (third-party) submodules.
git clone --recursive https://github.com/google/vulncode-db.git
cd vulncode-db
# Setup configuration files, the Docker images and containers.
./setup.sh
# Initialize the application and run an empty version of it.
./docker/docker-admin.sh start

Additionally, if you intend to add some data consider running:

# Fetch and insert CWE identifiers and some recent NVD entries.
./docker/docker-admin.sh init
# Search for entries with patch links and add additional application entries for them.
./docker/docker-admin.sh crawl_patches
# Run the application.
./docker/docker-admin.sh start

The main application should then be available at http://localhost:8080.

Please also see the documentation provided in docker/README.md for more details.

Terms of use

Vulncode-DB Data

This project provides data such as vulnerability annotations and mappings from vulnerability entries to corresponding patches and code. It can be self-hosted or accessed through the main project site at https://vulncode-db.com.

For any user provided content on the project's website we refer to the terms of conditions provided within this repository. Otherwise, for the project's code itself:

Vulncode-DB hereby grants you a perpetual, worldwide, non-exclusive,
no-charge, royalty-free, irrevocable copyright license to reproduce, prepare
derivative works of, publicly display, publicly perform, sublicense, and
distribute code which exclusively provided by the Vulncode-DB project. Any copy you make for
such purposes is authorized provided that you reproduce Vulncode-DB's copyright
designation and this license in any such copy.

Third-party Data

This project builds upon data provided by the CVE and NVD data sets.

Common Vulnerabilities and Exposures (CVE®)

The CVE® is maintained by the Mitre Corporation. Please see the Mitre CVE®'s Terms of use:

CVE Usage: MITRE hereby grants you a perpetual, worldwide, non-exclusive,
no-charge, royalty-free, irrevocable copyright license to reproduce, prepare
derivative works of, publicly display, publicly perform, sublicense, and
distribute Common Vulnerabilities and Exposures (CVE®). Any copy you make for
such purposes is authorized provided that you reproduce MITRE's copyright
designation and this license in any such copy.

National Vulnerabilitiy Database (NVD)

The National Vulnerability Database is maintained by the U.S. government. Please see the NVD's FAQ:

All NVD data is freely available from our XML Data Feeds. There are no fees,
licensing restrictions, or even a requirement to register. All NIST
publications are available in the public domain according to Title 17 of the
United States Code. Acknowledgment of the NVD  when using our information is
appreciated. In addition, please email [email protected] to let us know how the
information is being used.

Disclaimer

This is not an officially supported Google product.

vulncode-db's People

Contributors

3wnbr1 avatar alanyee avatar bluec0re avatar dependabot[bot] avatar elanzini avatar evonide avatar sbs2001 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vulncode-db's Issues

Ask for login when attempting to execute privileged actions and flash potential ACL errors

Some actions like clicking "propose changes" in the vulnerability view currently show a restrictive "forbidden" when you're not logged in.

Instead, we should:
a) If the user is not logged in forward to the login page after which one gets forwarded to the original page.
b) Otherwise, check if the authN/Z conditions are met and if not redirect to the previous page and flash an error which displays that this action can't be performed due to a lack of permissions.

Otherwise, the application flow breaks unexpectedly which will likely deter people from using it.

Add missing JSTree icons

The CDN changes introduced with #31 have broken the JSTree view for files. Particularly, the folder and file icons are missing. We should fix this before releasing.

Allow users to register and login

  • Implement login mechanism
    • Google Oauth
  • Implement registration (automatic user creation upon first login)
    • Require terms&condition approval
      • add terms&condition text
  • implement profile page

Deprecation warnings in SQLalchemy

=========================================================== warnings summary ===========================================================
tests/app_tests/api/test_routes.py::test_save_editor_data[query0-data0-400-expected_response0]
  /usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:1917: SAWarning: Setting backref / back_populates on relationship RepositoryFileComments.commit to refer to viewonly relationship VulnerabilityGitCommits.comments will be deprecated in SQLAlchemy 1.4, and will be disallowed in a future release.  viewonly relationships should not be mutated (this warning may be suppressed after 10 occurrences)
    (self, other),

tests/app_tests/api/test_routes.py::test_save_editor_data[query0-data0-400-expected_response0]
  /usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:1917: SAWarning: Setting backref / back_populates on relationship VulnerabilityGitCommits.comments to refer to viewonly relationship RepositoryFileComments.commit will be deprecated in SQLAlchemy 1.4, and will be disallowed in a future release.  viewonly relationships should not be mutated (this warning may be suppressed after 10 occurrences)
    (self, other),

tests/app_tests/api/test_routes.py::test_save_editor_data[query0-data0-400-expected_response0]
  /usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:1917: SAWarning: Setting backref / back_populates on relationship RepositoryFileMarkers.commit to refer to viewonly relationship VulnerabilityGitCommits.markers will be deprecated in SQLAlchemy 1.4, and will be disallowed in a future release.  viewonly relationships should not be mutated (this warning may be suppressed after 10 occurrences)
    (self, other),

tests/app_tests/api/test_routes.py::test_save_editor_data[query0-data0-400-expected_response0]
  /usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:1917: SAWarning: Setting backref / back_populates on relationship VulnerabilityGitCommits.markers to refer to viewonly relationship RepositoryFileMarkers.commit will be deprecated in SQLAlchemy 1.4, and will be disallowed in a future release.  viewonly relationships should not be mutated (this warning may be suppressed after 10 occurrences)
    (self, other),

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Allow account deactivation/deletion

We should make sure people can delete their accounts if there have been no contributions. Otherwise, I'm not sure how to approach this best as accounts might have data dependencies. For this reason, it's not possible to delete accounts from the admin UI either if an account has an existing/approved proposal.

Adding this to this milestone for now but this can likely be moved out, too.

Regularly test the overall project build process

We should make sure we avoid regressions regarding the project setup process.
Currently, we advice to execute steps like ./setup.sh and ./docker/docker-admin.sh init to get things started but we don't regularly verify if this works correctly.

Suggestion: Add a daily job which builds the project and verifies that the base version works as intended.

Move Python packages from third_party directory to virtual environment

For historic reasons when running the application outside of Docker all Python dependencies were meant to be installed to the third_party directory as AppEngine required them to be shipped.

With the newest AppEngine version this is not required anymore. We should move any third_party dependencies to be installed inside a virtual environment instead and remove third_party directory mentions from the code.

Assume a link to be a "patch link" only if tagged as `Patch` by NVD

Currently VulnCode-DB assumes every link which passes the regex

PATCH_REGEX = r".*(github\.com|\.git|\.patch|\/hg\.|\/\+\/)"
at
def get_patches(self):

as a patch link. In most cases this is correct but there are cases when this fails to identify patch links correctly such as at

https://www.vulncode-db.com/CVE-2018-7749 the patch link is identified as ronf/asyncssh@c161e26 . But this link does not contain any example of "vulnerable code".

Such type of error(s) can be avoided if we add one more condition for a link to be considered as a patch, this condition would be to check whether NVD tags the reference/link as a Patch . For CVE-2018-7749 NVD has rightly tagged it as a Third Party Advisory NOT Patch at https://nvd.nist.gov/vuln/detail/CVE-2018-7749 .

If such errors don't occur at considerable frequency close this ticket.

Add feedback history support to the review system

The review system only keeps track of one single reviewer feedback per proposal.
This means new feedback will overwrite old feedback.

We'll likely have to introduce a new table like feedback_history to keep a full history of what feedback was given so far for a new reviewer to better understand the full context.

Fix non-deterministic unit test failure

In data/models/vulnerability.py we use joinedload which in some cases seems to lead to inconsistent unit tests SQL behavior where tests sometimes pass and sometimes fail. We need to investigate this further and address this as it's impacting all automatically run tests for push and pull requests to the repository.

Implement entry annotation, review and versioning system

Allow users to annotate vulnerabilities and to propose vulnerable code for them. This requires to implement support for:

  • Modifying and suggesting new content as a logged in user
  • Reviewing proposed content as a reviewer / moderator / admin
  • Approving content and keeping versions / an archive of older versions

ModuleNotFoundError: No module named 'marshmallow.compat'

Issue

When starting the docker containers, frontend crashes beacause of an old marshmallow-sqlalchemy version. For reference marshmallow-code/marshmallow-sqlalchemy#208

To reproduce :

  • git clone https://github.com/google/vulncode-db
  • cd vulncode-db/docker
  • sudo docker-compose up --build

Stacktrace

frontend_1   | Usage: flask db upgrade [OPTIONS] [REVISION]
frontend_1   | 
frontend_1   | Error: While importing "main", an ImportError was raised:
frontend_1   | 
frontend_1   | Traceback (most recent call last):
frontend_1   |   File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 235, in locate_app
frontend_1   |     __import__(module_name)
frontend_1   |   File "/app/main.py", line 30, in <module>
frontend_1   |     from app.vulnerability.views.vulncode_db import VulncodeDB
frontend_1   |   File "/app/app/vulnerability/views/vulncode_db.py", line 18, in <module>
frontend_1   |     from app.vulnerability.views.vulnerability import VulnerabilityView
frontend_1   |   File "/app/app/vulnerability/views/vulnerability.py", line 19, in <module>
frontend_1   |     from data.database import DEFAULT_DATABASE
frontend_1   |   File "/app/data/database.py", line 22, in <module>
frontend_1   |     from data.models import *
frontend_1   |   File "/app/data/models/__init__.py", line 15, in <module>
frontend_1   |     from data.utils import populate_models
frontend_1   |   File "/app/data/utils.py", line 18, in <module>
frontend_1   |     from data.models.base import db, ma
frontend_1   |   File "/app/data/models/base.py", line 16, in <module>
frontend_1   |     from flask_marshmallow import Marshmallow
frontend_1   |   File "/usr/local/lib/python3.7/site-packages/flask_marshmallow/__init__.py", line 19, in <module>
frontend_1   |     from . import fields
frontend_1   |   File "/usr/local/lib/python3.7/site-packages/flask_marshmallow/fields.py", line 15, in <module>
frontend_1   |     from marshmallow.compat import iteritems
frontend_1   | ModuleNotFoundError: No module named 'marshmallow.compat'
frontend_1   | 
frontend_1   | Traceback (most recent call last):
frontend_1   |   File "<string>", line 1, in <module>
frontend_1   |   File "/app/main.py", line 30, in <module>
frontend_1   |     from app.vulnerability.views.vulncode_db import VulncodeDB
frontend_1   |   File "/app/app/vulnerability/views/vulncode_db.py", line 18, in <module>
frontend_1   |     from app.vulnerability.views.vulnerability import VulnerabilityView
frontend_1   |   File "/app/app/vulnerability/views/vulnerability.py", line 19, in <module>
frontend_1   |     from data.database import DEFAULT_DATABASE
frontend_1   |   File "/app/data/database.py", line 22, in <module>
frontend_1   |     from data.models import *
frontend_1   |   File "/app/data/models/__init__.py", line 15, in <module>
frontend_1   |     from data.utils import populate_models
frontend_1   |   File "/app/data/utils.py", line 18, in <module>
frontend_1   |     from data.models.base import db, ma
frontend_1   |   File "/app/data/models/base.py", line 16, in <module>
frontend_1   |     from flask_marshmallow import Marshmallow
frontend_1   |   File "/usr/local/lib/python3.7/site-packages/flask_marshmallow/__init__.py", line 19, in <module>
frontend_1   |     from . import fields
frontend_1   |   File "/usr/local/lib/python3.7/site-packages/flask_marshmallow/fields.py", line 15, in <module>
frontend_1   |     from marshmallow.compat import iteritems
frontend_1   | ModuleNotFoundError: No module named 'marshmallow.compat'
frontend_1   | [~] Executed outside AppEngine context. Manually loading config.
docker_frontend_1 exited with code 1

UI, data cleanup and removing unusable functionality from the UI

Things like code annotations are meant to be working in the next milestone.

Until then we should simplify the UI and remove any data or functionality which is not working and might cause confusion.

  • Remove "annotate code" buttons / links / mentions.
  • Look for any other functionality or links which can be cut down/trimmed down for now.
  • Remove broken or incomplete entries on the index page.

allow to search by language and vuln type

One of the main problem of the NVD / CVE raw data is that they are not enough qualified so that SAST vendors can easily find vulnerabilities corresponding to a given language or type of vuln.
Having such information will really help and speed up the development of enhancements in SAST products. For example, I would like to get a full list of XSS vulnerabilities involving Python (*.py) files.

Remove all but the initial go-cve-dictionary related alembic migrations

The go-cve-dictionary third-party dependency maintains its own tables. If there is an update it will automatically migrate/update the table structure.

Currently, Vulncode-DB explicitly ensures over alembic migrations that the data models are in sync. However, this breaks if go-cve-dictionary is run before Vulncode-DB was upgraded to the newest version as some columns or database modifications might have already happened through go-cve-dictionary.

  1. Change all initial alembic migrations in migrations/versions/82f8f48c6ee6_initial_layout.py to be create if not exist.
  2. Remove all alembic upgrade instructions related to go-cve-dictionary from all migrations after that.

This way it is expected that go-cve-dictionary will be run at least once if there have been any updates.

  • How can automatically detect and enforce this to avoid unexpected breakages?

Provide CSS and JS content directly

We're currently serving some content like specific JS files over Cloudflare and other providers like Bootstrap.

Instead, we should directly provide these files ourselves to avoid extensions which block such providers from breaking the service for some users.

Disable entry deletion for archived versions

Currently, it's possible to delete entries that have been marked as 'archived'.

This will lead to unexpected gaps in the change history.
We should fix this if this is not WAI.

Production dependency throws on login: ModuleNotFoundError: No module named '_cffi_backend'

Describe the bug
When logging in on vulncode-db.com you get an internal error.

Details seem to be:

ModuleNotFoundError: No module named '_cffi_backend'
at <module> (third_party/cryptography/hazmat/bindings/openssl/binding.py:16)
at <module> (third_party/cryptography/hazmat/backends/openssl/backend.py:117)
at <module> (third_party/cryptography/hazmat/backends/openssl/__init__.py:7)
at default_backend (third_party/cryptography/hazmat/backends/__init__.py:15)
at loads_public_key (third_party/authlib/jose/rfc7518/_backends/_jwk_cryptography.py:91)
at loads (third_party/authlib/jose/rfc7518/_backends/_jwk_cryptography.py:97)
at _load_obj (third_party/authlib/jose/rfc7517/jwk.py:50)
at _load_jwk_set (third_party/authlib/jose/rfc7517/jwk.py:62)
at loads (third_party/authlib/jose/rfc7517/jwk.py:76)
at load_key (third_party/authlib/integrations/base_client/remote_app.py:168)
at load_key (third_party/authlib/jose/jwk.py:21)
at key_func (third_party/authlib/jose/jwk.py:27)
at prepare_algorithm_key (third_party/authlib/jose/util.py:8)
at deserialize_compact (third_party/authlib/jose/rfc7515/jws.py:115)
at decode (third_party/authlib/jose/rfc7519/jwt.py:121)
at _parse_id_token (third_party/authlib/integrations/base_client/remote_app.py:198)
at parse_id_token (third_party/authlib/integrations/flask_client/remote_app.py:81)
at authorized (vulncode-db/app/auth/routes.py:128)
at decorated_function (third_party/flask_bouncer.py:66)
at dispatch_request (third_party/flask/app.py:1936)
at full_dispatch_request (third_party/flask/app.py:1950)
at reraise (third_party/flask/_compat.py:39)
at handle_user_exception (third_party/flask/app.py:1821)
at full_dispatch_request (third_party/flask/app.py:1952)
at wsgi_app (third_party/flask/app.py:2447)

I've tried adding cffi>=1.14.2 as a dependency which didn't seem to help here.

Update go-cve-dictionary submodule

Vulncode-db uses https://github.com/kotakanbe/go-cve-dictionary which is present as a git submodule. It uses https://github.com/kotakanbe/go-cve-dictionary/tree/5fe52611f0b8dff9f95374d9cd7bdb23cc5fc67a to be more specific which is 1+ year old version.

https://github.com/kotakanbe/go-cve-dictionary/tree/5fe52611f0b8dff9f95374d9cd7bdb23cc5fc67a no longer works due to some changes. Here's the log :

Sending build context to Docker daemon  608.8kB
Step 1/16 : FROM golang:alpine as builder
 ---> 1a87ceb1ace5
Step 2/16 : RUN apk add --no-cache         git         make         gcc         musl-dev
 ---> Using cache
 ---> 90867a3f9751
Step 3/16 : ENV REPOSITORY github.com/kotakanbe/go-cve-dictionary
 ---> Using cache
 ---> 2fc568001ee8
Step 4/16 : COPY . $GOPATH/src/$REPOSITORY
 ---> 28d1be732cc6
Step 5/16 : RUN cd $GOPATH/src/$REPOSITORY && make install
 ---> Running in 24a0108db67c
go get -u github.com/golang/dep/...
dep ensure -v
# Gopkg.lock is out of sync with Gopkg.toml and project imports:
github.com/pkg/errors: imported or required, but missing from Gopkg.lock's input-imports

Due to this ./docker-admin.sh init does not work too.

./docker-admin.sh init works alright after updating the go-cve-dictionary to their current master branch.
Please update go-cve-dictionary

Fix code preview bug

Fix a redirect bug for loading file content when the entry uses a custom repository.

Add releases entry to Github

Create a first release entry for the Github repository.
We could likely use:
curl -H "Accept: application/vnd.github.v3+json" 'https://api.github.com/repos/google/vulncode-db/releases'
to then fetch and generate a .html release file with markdown support that can be included in prod.

Improve overall ID handling

Currently, the application uses different identifiers such as:

  • id
  • suggested_id
  • vuln_id
  • vcdb_id
  • cve_id

We should write a wrapper class and add type annotations to avoid any confusion with this.

Link to GitHub issues tracking

We should display an icon (at the top right?) that links to this issue tracker to allow convenient bug or feature request filing.

Update navbar for prod

The navigation bar for production is deprecated.
Replace with a version that automatically considers the newest version in the repostitory.

Adding Vulnerability ID to the HTML page title

It would be a neat quality-of-life feature to have the vuln ID appear in the title, for easier navigation between browser tabs.

A quick look at the app routes shows that the vuln_view.html could be changed to update the document title, possibly inside the <script> tag.

Maybe something along the lines of:

<script>
...
document.title = "{{ vulnerability_details.id }} - Vulncode-DB";
</script>

I have not tested this edit, but I would love to see such a feature at vulncode-db.com

Consider using project kb's data

Project KB contains manually curated commit links which fix particular CVE. It contains commits which actually rectified "vulnerable code" unlike NVD which many times contains commits which tagged release.

Vulncode-db can leverage project KB's data and provide more examples of real world vulnerable code.

Data is at :

https://github.com/SAP/project-kb/blob/master/MSR2019/dataset/vulas_db_msr2019_release.csv
https://github.com/SAP/project-kb/tree/vulnerability-data/statements

FYI project KB is used by https://github.com/eclipse/steady .

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.