Git Product home page Git Product logo

binarypig's Introduction

BinaryPig

Malware Processing and Analytics over Pig, Exploration through Django, Twitter Bootstrap, and Elasticsearch

Pig Libaries and Scripts

Building BinaryPig JAR

./build.sh

Installing dependencies for binarypig modules

See installation docs in "docs" directory.

Using binarypig

See pig scripts in "examples" directory.

Webapp

Installing the binarypig webapp

cd webapp

# create a local settings file
cp project/local_settings.py.example project/local_settings.py
vi project/local_settings.py

# create your mysql DB
mysql
mysql> create database binarypig;

virtualenv env
. env/bin/activate
pip install -r requirements.txt

# initialize the database
./manage.py syncdb
./manage.py migrate

Running the binarypig webapp

./manage.py runserver 0.0.0.0:8000

Issues

Some issues we encountered when running python based binarypig pig jobs

On Centos, if you have SELinux running and you disable it, you must restart arbtd or many/all of the python processes will hang and provide no output.

Reference

Getting up and Running with Vagrant

This is a mini howto on getting Binary Pig up and running on an Ubuntu 14.04 VM running over Vagrant.

Versions

  • Ubuntu 14.04
  • Hadoop 1.2.1
  • Pig 0.12.1
  • Elasticsearch 1.2.1

Steps:

# From workstation
$ git clone <repo>
$ cd binarypig
$ vagrant up
$ vagrant ssh

# now logged into VM
$ sudo su - 
$ cd /vagrant/
$ ./install-ubuntu-vagrant.sh

# press <ENTER> when asked

# agree to Oracle Java license when asked

# type "yes" when asked if you want to ssh into localhost

# enter password for root user of mysql when asked (MySQL installation)

# enter password for root user of mysql when asked (Django app install)

# walk through the django admin user creation:
#   You just installed Django's auth system, which means you don't have any superusers defined.
#   Would you like to create one now? (yes/no): yes
#   Username (leave blank to use 'root'): 
#   E-mail address: [email protected]
#   Password: 
#   Password (again): 
#   Superuser created successfully.

Now run the examples (Note: this is still a Work in Progress). This will launch a series of pig jobs that execute various BinaryPig scripts from the examples directory.

# still from VM
$ cd /vagrant/
$ ./run_examples.sh

License

Endgame, Inc.

Licensed under the Apache 2.0 license

Copyright 2013 Endgame, Inc.

Contributors

binarypig's People

Contributors

jatrost avatar crohling avatar telvis07 avatar epishkin avatar zhanif3 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.