Git Product home page Git Product logo

victims-web's Introduction

Victims Web Service Build Status PyPI version

The victims web application.

Report an Issue

If you find an issue with the service at http://victi.ms or the code, either

Contributing

If you have a patch or a feature that you want considered to be added to the project, feel free to send us a pull request.
Make sure you run pep8 before committing.
pep8 --repeat .

Using PyPI Package

You can install and use the server by installing the PyPI Package and use the provided entrypoint. Do ensure that the required database services are available and/or configured.

pip install --user victims-web
victims-web-server

Development

This is short guide on how to work on this code base using the provided docker-compose configuration and development Dockerfile. Note that the Dockerfile provided in the base directory is not to be used in production and is only for development use.

Docker builds

Building the image

The image can be built to provide a working environment with all dependencies installed.

docker build -t local/victims-web .

Using the docker image

The docker image built as shown above will not contain the application source code but it expects the working directory to be mounted at /opt/source.

docker run --rm -it -v `pwd`:/opt/source local/victims-web

Docker Compose

The docker-compose.yml file defines services required to run a working copy of the server on your local machine. Starting the server via docker-compose will;

  1. start a supported version of MongoDB instance
  2. seed the database with test data
  3. start the web server using python -m victims.web
  4. bind to port 5000 on your localhost

Starting a server

This will start an instance of the server as described above. Note that this is started with both DEBUG and TESTING enabled. This will also ensure that your code is auto re-loaded if changed.

docker-compose up server

Executing tests against your working copy

In order to execute tests against your working copy of code, you may use the test service as described in the docker-compose.yml file. This will start up dependant services, load seed data and then execute application tests and pep8.

docker-compose up test

Usage

Secured API Access

Submission endpoints like /service/submit/archive/java are secured by an implementation similar to what is used by AWS. The authentication token is expected in a HTTP header configured via the VICTIMS_API_HEADER configuration (default: X-Victims-Api). If this is not present or if validation/authentication fails, then it falls back to BASIC AUTH.

An example using curl is as follows:

$ curl -v -X PUT -H "X-Victims-Api: $APIKEY:$SIGNATURE" -H "Date: Thu, 22 Aug 2013 15:20:37 GMT" -F archive=@$ARCHIVE https://$VICTIMS_SERVER/service/submit/archive/java?version=VID\&groupId=GID\&artifactId=AID\&cves=CVE-2013-0000,CVE-2013-0001

This can also be done using BASIC-AUTH as follows:

curl -v -u $USERNAME:$PASSWORD -X PUT -F archive=@$ARCHIVE_FILE https://$VICTIMS_SERVER/service/submit/archive/java?version=VID\&groupId=GID\&artifactId=AID\&cves=CVE-2013-0000,CVE-2013-0001

API Key and Client Secret Key

Each account on victi.ms is allocated an API Key and Secret key by default. This can be retrieved by visiting https://victi.ms/account. These can be regenerated using the form at https://victi.ms/account_edit.

Signature

The signature is generated using HTTP Method, Path, Date and the MD5 hexdigest.

Notes:

  • The Path includes the query string parameters, e.g: /service/submit/archive/java?cves=CVE-0000-0000
  • The MD5 checksum includes the data (if available) of all files that are being submitted. The checksums are sorted in ascending order before adding to the string.
  • The date is expected to be in GMT. Eg: Thu, 22 Aug 2013 15:20:37 GMT.

The following is a reference implementation in python:

from hmac import HMAC

def generate_signature(secret, method, path, date, md5sums):
    md5sums.sort()
    ordered = [method, path, date] + md5sums
    string = ''

    for content in ordered:
        if content is None:
            raise ValueError('Required header not found')
        string += str(content)

    return HMAC(
        key=bytes(secret),
        msg=string.lower(),
        digestmod=sha512
    ).hexdigest().upper()

victims-web's People

Contributors

abn avatar ashcrow avatar jasinner avatar dfj avatar pmdematagoda avatar alectolytic avatar merwok avatar

Stargazers

Justin avatar 0ri0n ☠ avatar vinodh avatar JC GreenMind avatar Tiago Moreira Vieira avatar runelabs avatar Jürgen Hermann avatar Jasper Siepkes avatar

Watchers

 avatar  avatar James Cloos avatar  avatar Ben Tipping avatar

victims-web's Issues

Implement v2 interface

Including client interface and submission interface for vulnerable JARs with server-side hashing.

SSL handshake alert - causes problems in jdk 1.7

Exception in thread "main" javax.net.ssl.SSLProtocolException: handshake alert:  unrecognized_name
    at sun.security.ssl.ClientHandshaker.handshakeAlert(ClientHandshaker.java:1292)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1954)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:515)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1299)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at java.net.URL.openStream(URL.java:1037)
    at com.redhat.victims.VictimsService$RecordStream.<init>(VictimsService.java:91)
    at com.redhat.victims.VictimsService.fetch(VictimsService.java:73)
    at com.redhat.victims.VictimsService.updates(VictimsService.java:40)
    at com.redhat.victims.VictimsService.main(VictimsService.java:138)

Hold CVE addition data

Per the conversation in #46 hold internal CVE lists as a list of dictionaries noting the cve and the date added:

{
   'cve_id': 'date_added',
 ....
}

Though any exposure of the data should still represent it back as:

['cve_id', ...]

Move to MongoEngine

We are using MongoKit at the moment but MongoEngine seems to have more general support for the things we want to do.

Revisit underlying document structure

We might need to have another look at the internal database structure. Things like embedded documents instead of arrays etc. Combination of the shift to MongoEngine and internal structure changes should give us some perf benefits and allow us to do some crazy queries from the database rather than the application.
#25
#46 will need something to indicate when the last change was made. Or do we just move this logic to the client, making sure that this record will be overwritten?
#49
#52 Might help in perf.

We might want to do this now rather later. Hopefully before any major adaptation.

Enable Auth in Service Requests

As a prerequisite for #25 the service should allow for authenticated requests. These requests should either use an API key scheme or a user/pass basic auth (which is the easiest route being users already exist).

Version 2 of the service will not provide extra functionality for a user who is authenticating. However, version 2 may be enhanced to limit anonymous requests.

v2 Service [dot] in response

Since mongodb doesn't like . in keys we are using [dot] as a replacement. The v2 service needs to remove this and push it back to an actual dot.

Add "action" key-value to returned json results

Having the web service return JSON of the following format can be helpful. We can also remove the /removes/ endpoint. This should not cause any issues with nay implementations. The following is just a sample:

# added a new record
{ "fields" : {}, "action" : "add_record"}

# removed a record
{ "fields" : {}, "action" : "del_record"}

# added a cve to a record
{ "fields" : {"cve" : ["CVE-YYYY-XXXXX"] }, "action" : "add_value"}

# removed a value from the record
{ "fields" : {"hashes" : {"sha256":{}} } }, "action" :"del_value"}

metadata not being extracted for new submissions

I just realized the victims_hash code didn't provide a generic mechanism to extract / mine for metadata within archives added to the database. This has been corrected but the following needs to be added to the submission logic.

  • from victims_hash.metadata import extract_metadata
  • metadata = metadata(filename, io=archive.stream)
  • submission.update(metadata)

Implement request based response for update/remove services

We need a way where a client connecting via the REST-API can specify what fields it needs. This will help minimize the bandwidth.

For example the maven enforcer might only require:

{ "hash":"", "hashes" : {"sha512":{"files":{}}}}

A simple sha1 based client might only require:

{ "hashes" : {"sha1":{"combined":""}}}

Outdated information in victims-websec.rhcloud.com

victims-websec.rhcloud.com issue contains old / stale data in the database.

I've been unable to synchronize using the victims-enforcer. I think it is because the json format is slightly different than what is expected. (e.g. metadata is an object instead of array).

Can you confirm what version of the server this is?

Example of expected response:

[
{
"fields": {
"cves": [
"CVE-9999-9999"
],
"date": "2013-03-15T05:47:53.445000",
"format": "Jar",
"hashes": {
"sha1": {
"combined": "e76c3b18d9a09974b4f77e37c303726a82321f29",
"files": {
"02f72bb14475d99f5fd310a2d999ed885ef26345": "junit/swingui/AboutDialog$1.class",
"--snip--" : ""
}
},
"sha512": {
"combined": "ae7b4bb76ebc7689658b5525a0888c388ca40047161a37eb2194f64858391b19c5ceba88ebda19e611e9ed7a5d2129a8bdcc05cc9e5589fdd317c725a352ac57",
"files": {
"026d92b944844bf2120140c98967e443f946c6c3b58872da518e1e3a54ea7ea15423b3c2f62ba8ac1b00a2e6457ccb63d8a293653605bb211e78225310a1e816": "junit/awtui/TestRunner$1.class",
"--snip--" : ""
}
}
},
"meta": [
{
"filename": "META-INF/MANIFEST.MF",
"properties": {
"Created-By": " Ant 1.4.1",
"Manifest-Version": " 1.0"
}
},
{
"filename": "pom.properties",
"properties": {
"groupId" : "junit",
"artifactId" : "junit",
"version": "3.8.1"
}
}
],
"name": "junit-3.8.1.jar",
"status": "SUBMITTED",
"submittedon": "2013-03-15T05:32:40.277000",
"submitter": "gm",
"vendor": "Unknown",
"version": "1.0.0"
}
}
]

Example of actual response:

[
{
"fields": {
"_v1": {
"db_version": 2.0
},
"cves": [
"CVE-2009-1523"
],
"date": "2012-12-21T01:44:29.095000",
"db_version": 2.0,
"format": "Jar",
"hash": "0e951d0b175bcbaad9c8dfcb9e481fb1dc6ef9489be9a0a42051e9cf574ee2ad137baa9e61b4e29a4b7ec1b5d877f52ba81632f674a8ad39c96a9140ada41fc9",
"hashes": {
"sha1": {
"combined": "",
"files": {}
},
"sha512": {
"combined": "0e951d0b175bcbaad9c8dfcb9e481fb1dc6ef9489be9a0a42051e9cf574ee2ad137baa9e61b4e29a4b7ec1b5d877f52ba81632f674a8ad39c96a9140ada41fc9",
"files": {}
}
},
"meta": {},
"name": "jetty",
"status": "RELEASED",
"submittedon": "2012-12-21T01:58:40.472000",
"submitter": "ashcrow",
"vendor": "Codehaus",
"version": "6.1.1"
}
},

...
]

502 when trying to fetch updates

Failed to sync database: Server returned HTTP response code: 502 for URL: https://victims-websec.rhcloud.com//service/v2/remove/1900-01-01T00:00:00

Not sure if this OpenShift's doing or the flask web-server. If it is the latter, should we consider alternatives? eg: CherryPy? (http://flask.pocoo.org/snippets/24/)

We also need to be able chunk the json response.

Update admin interface

The admin interface should understand the new workflow and allow for review/releasing submitted package details.

Implement v2 data structure

With the ability to store multiple hashes for the contents of an exploded JAR, so we can handle the exploded JAR hashing and the superset/subset case.

Verify Latest Data Import

Data was imported/updated. Just as a checkpoint the data should be lightly verified to make sure everything was stored properly.

Incremental database update for consumers

We need a way for client tools that consume the information from the Victims DB to be able to synchronize the database changes incrementally. This has to be considered as each entry (once per class file info is added) seems to be easily ~200 KB.

If we assume that there will be no updates/deletions and only additions to the database, we can rely on the entry's time-stamp. For a proper solution, we should handle all updates.

Can we maybe rely on the MongoDB change log? So send only those records that changed since TIMESTAMP?

[dev-env-scripts] Helpful?

Do these seem like something that would help?
abn@63dccaf

# creates an env for you if it does not exists, installing everything required
source ./scripts/start-dev.env.sh
# runs tests, starts/stops mongodb
./scripts/run-tests.sh 

Invalid hash value

During dev on victims-lib, came across a bad data.

{$where:"this.hash.length > 128"}

Returned:

{
  ...
  "hash" : "bc0b8aea95b8b10cb81ebff35f72d190124a83230c88171805bd2ce4d3ff4b79bd8870adfc79a9c9331622475ec06c9d8b593de12bb1c0890a4308ff61ec6fb6 ",
  ...
}

Which was of length 129.

victi.ms redirects

Would it be possible to get requests like this to be redirected to victims-websec.rhcloud.com correctly? If it can be done, we can update the defaults in enforcer and lib to this in this week's release.

https://victi.ms/service/update/2010-01-01T01:01:01

Add default service at /service

We need a transparent service location independent of the version number pointing always to the latest version, eg: /service

I am guessing this should be a one-liner to application.py

app.register_blueprint(v1, url_prefix='/service')

I am not sure if registering /service when /service/v* is registered will be an issue.

Imported data 'submittedon' format should to ISODate not string

Related to: #32 #35

{..., "submittedon": "2013-05-10T00: 00: 00", ...}

The wrong format seems to result in the following query only returning 95 results.

{"submittedon": { $gte : ISODate("2012-12-21T01: 58: 40.472Z") }}

This will also result in the service query returning invalid results:

items = current_app.db.Hash.find(
            {'date': {'$gt': datetime.datetime.strptime(
                since, "%Y-%m-%dT%H:%M:%S")}})

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.