jamesward / koober Goto Github PK

Scala 51.38% Shell 2.25% Batchfile 2.27% HTML 24.22% JavaScript 9.00% CSS 7.46% Python 3.41%

koober's Introduction

Koober

An uber data pipeline sample app. Play Framework, Akka Streams, Kafka, Flink, Spark Streaming, and Cassandra.

Start Kafka:

./sbt kafkaServer/run

Web App:

Obtain an API key from mapbox.com
Start the Play web app: MAPBOX_ACCESS_TOKEN=YOUR-MAPBOX-API-KEY ./sbt webapp/run

Try it out:

Open the driver UI: http://localhost:9000/driver
Open the rider UI: http://localhost:9000/rider
In the Rider UI, click on the map to position the rider
In the Driver UI, click on the rider to initiate a pickup

Start Flink:

./sbt flinkClient/run
Initiate a few pickups and see the average pickup wait time change (in the stdout console for the Flink process)

Start Cassandra:

./sbt cassandraServer/run

Start the Spark Streaming process:

./sbt kafkaToCassandra/run
Watch all of the ride data be micro-batched from Kafka to Cassandra

Setup PredictionIO Pipeline:

Setup PIO

Set the PIO Access Key:

 export PIO_ACCESS_KEY=<YOUR PIO ACCESS KEY>

Start the PIO Pipeline:
```
 ./sbt pioClient/run
```

Copy demo data into Kafka or PIO:

For fake data, run:

./sbt "demoData/run <kafka|pio> fake <number of records> <number of months> <number of clusters>"

For New York data, run:

./sbt "demoData/run <kafka|pio> ny <number of months> <sample rate>"

Start the Demand Dashboard

PREDICTIONIO_URL=http://asdf.com MAPBOX_ACCESS_TOKEN=YOUR_MAPBOX_TOKEN ./sbt demandDashboard/run

koober's People

Contributors

Stargazers

Watchers

Forkers

gitter-badger bjc265 eatingw gamussa berndruecker adamwhat wang-siyuan anniexcheng kayoubi veskop86 konnectnishant sanprasirt shehangamage monika101 rrohitramsen artvovc jkurup peng-sun tosheer odrianoaliveira beastneedsmoretorque yusufoguntola tanduong lmasiero ogurdima jmhockham jmisham joshlong-attic mmchougule mdrobish sabya1981 psanyal p5f8 royalghost degree kenwilcox nachinius swainl denza sahmedgithub singram seifeddine9 mailmahee victorkostyukov aaron-h samerra dmoellenbeck mattehicks gsuaki nishatdhillon praneel-gavali sassyn roussi sycyhy mowzey123 amir-hajizadeh benmingli alternegro sabrish89 zfy1355 shwanliu iamsenorespana vivek807 nmanojkumar030 vaedama charan678 foman vietnamz mseemann dkul108 kharole glothos shahaabz haonsong nareshmiriyala yajnasteju jay-li-0202 garrychen-site d0mc3k imran273 julio varantes tat-m3dicine fatehalc cs-tsui preetnitkkr gstearmit tomasmaroscikas irischan rkalyankumar nitinbly84 nangal bipinkarms rumbletumjum jay1991 bunyod zeeshanadilbutt day0ops doctorcatcodes avandenberghe

koober's Issues

Generate Demo Data Set for Rides

A ride consists of:

A driver
A rider
Location of Driver
Location of Rider
Time to pickup
Time of event

In order to do #1 we need a demo data set we can feed into PredictionIO. The data set can't be totally random otherwise our predictions might appear random. So maybe there is public taxi data we can use.

Data Visualization

To visualize the prediction data it would be nice to have a chart that showed actual demand and predicted demand.

We can use Flot for the visualization.

But we will need a way to get the actual demand over a given period. There might be a way to do this via PIO (fetch the events and process and serve them). Otherwise we will need to come up with something else.

$ ./sbt kafkaServer/run
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[warn] The global sbt directory is now versioned and is located at /Users/apple/.sbt/0.13.
[warn]   You are seeing this warning because there is global configuration in /Users/apple/.sbt but not in /Users/apple/.sbt/0.13.
[warn]   The global sbt directory may be changed via the sbt.global.base system property.
[info] Loading project definition from /Users/apple/projects/hazelcast/poc/koober/project
[warn] The global sbt directory is now versioned and is located at /Users/apple/.sbt/0.13.
[warn]   You are seeing this warning because there is global configuration in /Users/apple/.sbt but not in /Users/apple/.sbt/0.13.
[warn]   The global sbt directory may be changed via the sbt.global.base system property.
[info] Updating {file:/Users/apple/projects/hazelcast/poc/koober/project/}koober-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: commons-io#commons-io;2.4: configuration not found in commons-io#commons-io;2.4: 'master(compile)'. Missing configuration: 'compile'. It was required from org.vafer#jdeb;1.3 compile
[warn] 	:: aopalliance#aopalliance;1.0: configuration not found in aopalliance#aopalliance;1.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.sonatype.sisu#sisu-guice;3.1.0 compile
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] 	Note: Unresolved dependencies path:
[warn] 		commons-io:commons-io:2.4
[warn] 		  +- org.vafer:jdeb:1.3
[warn] 		  +- com.typesafe.sbt:sbt-native-packager:1.0.3 (scalaVersion=2.10, sbtVersion=0.13)
[warn] 		  +- com.typesafe.play:sbt-plugin:2.5.12 (scalaVersion=2.10, sbtVersion=0.13) (/Users/apple/projects/hazelcast/poc/koober/project/plugins.sbt#L1-2)
[warn] 		  +- default:koober-build:0.1-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13)
[warn] 		aopalliance:aopalliance:1.0
[warn] 		  +- org.sonatype.sisu:sisu-guice:3.1.0
[warn] 		  +- org.eclipse.sisu:org.eclipse.sisu.plexus:0.0.0.M5
[warn] 		  +- org.apache.maven:maven-plugin-api:3.2.2
[warn] 		  +- org.apache.maven:maven-core:3.2.2
[warn] 		  +- org.vafer:jdeb:1.3
[warn] 		  +- com.typesafe.sbt:sbt-native-packager:1.0.3 (scalaVersion=2.10, sbtVersion=0.13)
[warn] 		  +- com.typesafe.play:sbt-plugin:2.5.12 (scalaVersion=2.10, sbtVersion=0.13) (/Users/apple/projects/hazelcast/poc/koober/project/plugins.sbt#L1-2)
[warn] 		  +- default:koober-build:0.1-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13)
sbt.ResolveException: unresolved dependency: commons-io#commons-io;2.4: configuration not found in commons-io#commons-io;2.4: 'master(compile)'. Missing configuration: 'compile'. It was required from org.vafer#jdeb;1.3 compile
unresolved dependency: aopalliance#aopalliance;1.0: configuration not found in aopalliance#aopalliance;1.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.sonatype.sisu#sisu-guice;3.1.0 compile
	at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
	at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
	at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
	at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
	at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
	at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
	at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
	at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
	at xsbt.boot.Using$.withResource(Using.scala:10)
	at xsbt.boot.Using$.apply(Using.scala:9)
	at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
	at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
	at xsbt.boot.Locks$.apply0(Locks.scala:31)
	at xsbt.boot.Locks$.apply(Locks.scala:28)
	at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
	at sbt.IvySbt.withIvy(Ivy.scala:128)
	at sbt.IvySbt.withIvy(Ivy.scala:125)
	at sbt.IvySbt$Module.withModule(Ivy.scala:156)
	at sbt.IvyActions$.updateEither(IvyActions.scala:168)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1481)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1477)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1512)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1510)
	at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1515)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1509)
	at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
	at sbt.Classpaths$.cachedUpdate(Defaults.scala:1532)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1459)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1411)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[error] (*:update) sbt.ResolveException: unresolved dependency: commons-io#commons-io;2.4: configuration not found in commons-io#commons-io;2.4: 'master(compile)'. Missing configuration: 'compile'. It was required from org.vafer#jdeb;1.3 compile
[error] unresolved dependency: aopalliance#aopalliance;1.0: configuration not found in aopalliance#aopalliance;1.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.sonatype.sisu#sisu-guice;3.1.0 compile
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? i
[warn] Ignoring load failure: no project loaded.
[error] Expected ID character
[error] Not a valid command: kafkaServer
[error] kafkaServer/run
[error]            ^

Data Loader Improvements

Fake Data Loader: Generates a dataset that provides us obvious predictions
Sample Data Loader: Selects a given number of past months and picks a given number of records from that dataset

Improvements to the current dataloader:

Parametrize whether to use fake or sampled data
For sampled data, parametrize past months, and number of records

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

jamesward / koober Goto Github PK

koober's Introduction

Koober

koober's People

Contributors

Stargazers

Watchers

Forkers

koober's Issues

Generate Demo Data Set for Rides

Data Visualization

Data Pipeline that Sends Ride Events to PredictionIO

PredictionIO to Predict Demand

Weather Data

can't run from master

Data Loader Improvements

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent