SecTweet is a Flink job that streams Twitter posts. Relevent text containing shas, file paths, etc, are aggregated and trends are determined based on the frequency of these terms over time.

For the Flink workshop, please follow the steps here instead.

Java 8

Services: Flink, Elasticsearch Kibana (Optional)

To start all containers:
runDockerContainers.sh
To start a specific container:
docker-compose up <flink-jm | flink-tm | elasticsearch | kibana>

To build the project:
./gradlew clean build

Arguments:
Live twitter mode:
--twitter-source.consumerKey <key> --twitter-source.consumerSecret <secret> twitter-source.token <token> --twitter-source.tokenSecret <tokenSecret>

File mode:
--file-source <twitter data file>
Note: there are test files available under ./data. Simply unzip them to use in sectweet.

To run the project with embedded Flink:
./gradlew run --args='<flink args here>'

To package your job for submission to Flink, use: gradle shadowJar
Afterwards, you'll find the jar to use in the 'build/libs' folder.