My project for the bachelor's thesis is predicting the user activity on Twitter using tweets and machine learning. This repository contains the source code.
Install Python and Postgresql. Don't forget to install python-dev python3-dev libpq-dev
.
Then, install pip3
packages: pip3 install setuptools nltk numpy pandas scikit-learn psycopg2 guess_language-spirit python-sql --user
.
I'm using Jupyter for working and playing with the data so install that: pip3 install jupyter
Create a new table for the processed tweets:
CREATE TABLE "tweets" (
"id" serial,
"timestamp" timestamp NOT NULL,
"user" text NOT NULL,
"length" integer NOT NULL,
"words" integer NOT NULL,
"hashtags" text[] DEFAULT '{}',
"mentions" text[] DEFAULT '{}',
"urls" text[] DEFAULT '{}',
PRIMARY KEY ("id")
);
I'm using a transformed view of the data, create it like this:
CREATE VIEW "grouped_tweets" AS SELECT
date_trunc('week', tweets."timestamp") AS week,
tweets."user",
count(tweets."id") AS tweets,
sum(tweets."length") AS total_length,
sum(tweets."words") AS total_words,
array_cat_agg(tweets."hashtags") AS hashtags,
array_cat_agg(tweets."mentions") AS mentions,
array_cat_agg(tweets."urls") AS urls
FROM tweets
GROUP BY tweets."user", date_trunc('week', tweets."timestamp");
It's using this custom aggregate:
CREATE AGGREGATE array_cat_agg(anyarray) (
SFUNC=array_cat,
STYPE=anyarray
);
Use a .pgpass
file to store the password for the database. Don't forget to chmod 0600 .pgpass
. More information in the docs.