mohitgupta3 / sparkify Goto Github PK

Sparkify is a fictitious digital streaming service by Udacity.

Here, our job is deep mining the customers' data and implement appropriate model to predict customer churn as follow steps:

• Clean data: fill the nan values , correct the data types, drop the outliers.
• EDA: exploratory data to look features' distributions and correlation with key label (churn).
• Feature engineering: extract and found customer-features and customer-behavior-features; Implement standscaler on numerical features.
• Train and measure models: I choose logistic regression, linear svm classifier, decision tree and random forest classifier to train a baseline model and tuning a better model from best of them. It is worth mentioning that this data is unbalanced because of less churn customers, so we choose f1 score as a metrics to measure models' performance.

• Python 3.6
• PySpark ML
• Jupyter

The baseline of four machine learning methods: Logistic Regression, Linear SVC, Decision Tree Classifier and Random Forest Classifier.

Though the LinearSVC spent more training time, but it can get the highest f1 score 0.702. And the LogisticRegression has a medium training time and f1 score, maybe I can tuning it to get a higher score. So I'll choose LinearSVC and LogisticRegression to tuning, and the result is as follows:

Considering this is only a quit mini dataset and our purpose is scaling this up to the total 12G dataset, so, the logistic regression is the best model from now on in this project.

Please check my blog post to get more details, here is the link.

Dataset provided by Udacity.