dongfeiwww / flowlog Goto Github PK

Important: this project is only for demo purpose

The flowlog project is using flowlog-service, kafka, spark and cassandra to aggregate the realtime log. The aggregation has one-minute delay, which means the log storage is base on minute level. However, the apis only support hourly level query, it is not a hard limitation, we can easily change it to minute level log query.

For more detail, refer the Design Document, for the load test report, refer the ab testing result

• Step 1, clone the repository and change directory to flowlog
• Step 2, run command to initialize the cassandra, kafka, spark

gradle init_servers

• Step 3, build and deploy the spark job

gradle submit_task

• Step 4, launch 2nd terminal and start the service

gradle bootRun

• Step 5, launch 3rd terminal and send sample requests

curl -kvvv -XPOST http://localhost:8080/flows -H'Content-Type: application/json'  -d '[{"hour":1,"src_app":"foo","desc_app":"bar","vpc_id":"vpc-1","bytes_tx":200,"bytes_rx":600},{"hour":1,"src_app":"foo","desc_app":"biz","vpc_id":"vpc-0","bytes_tx":1000,"bytes_rx":800}]]'
# we already prepare some sample data during init servers (refer Step 2)
curl -kvvv http://localhost:8080/flows\?hour\=1
curl -kvvv http://localhost:8080/flows\?hour\=2
# after 1 minute run
curl -kvvv http://localhost:8080/flows\?hour\=1
curl -kvvv http://localhost:8080/flows\?hour\=2

• Step 6, stop the servers

gradle stop_servers

• Step 7 (optional), restart the servers

gradle start_servers

For further reference, please consider the following sections:

• Spring Reactive Web
• Apache Spark
• Apache Kafka
• Apache Cassandra