ORIE 4741 Learning with Big Messy Data Final Project
Kevin Van Vorst (kpv23), Manya Walia (mtw62), Saaqeb Siddiqi (ss3759)
ORIE 4741 Final Project Repository
This data analysis project aims to predict what songs will be come popular, and the qualities of a song that make it so. It achieves this by using a large dataset collected via the Spotify API of 160K+ songs and their features (e.g. name, danceability, valence, speechiness, etc). Through this project, one can begin to determine what kinds of features make a song "popular", and thus music producers can start to tailor their music in such a way as to increase its chances of being popular. It also aims to come up with a recommendation system based off of these findings.
The points I enjoyed the most about this proposal were:
The points I am most concerned about for this proposal are:
1, The description of the data is very clear and well-structures including how they clear the data, data types of each columns. And the visualization provides readers with a straightforward insight into the data.
2, I am amazed at their high R-square and low MSE both on the training set and especially on their test set. What’s more they also implement the F-test to see whether the explanatory variables are joint significant, which shows their strictness towards this model.
3, It’s worth mentioning that this group has done very deep analysis about their model. For example, they try to investigate the effectiveness excluding the “year” variable. The techniques they take could create a greater and in-depth understanding of the dataset and opportunities to lower the errors.
1, I think it could be better if they can include some explanations on each variable at the very beginning. Maybe one sentence is enough. This can help readers more clearly capture what intuition is behind this report especially for those who don’t know music a lot.
2, Discrepancy exists in the labeling of the figure. It seems this is also mentioned in some other review suggestions.
3, Maybe this report could include more explanatory variables. Because there are so many datapoints, we don’t need to worry about the overfitting problem as we do in the homework.
Summary:
This project investigates what features are relevant to popularity of a song, using data from Spotify API.
What I love about this project:
Your dataset has a good coverage of data, and you provided good description and visualization of data and features. I also love that your usage of statistics to evaluate your model and your explanation of what models are used and why. You have a great overvation on old songs popularity are predicted more correctly. I think the report is beautifully written and I especially enjoy reading the conclusion, although you could've bring more detail to that (since there is new information).
What I think could be improved:
My research interest is actually in information diffusion, i.e., how stuff gets popular, so this project really intrigues me and I'm super happy to review it. I think it would be better if you kept the artist, because popular artists tend to produce popular songs. By removing this feature, you are actually leaving out a quite important confounding variable. Additionally, some features could be better defined - what exactly are danceability and energy and how are they measured? I understand that these are coming from the dataset, but it would've been clearer to the reader if they are defined. I don't quite understand why you decided to make it a binary classification project instead of using regression, and it is not clear why you chose to use 0.8 as the cutoff for popularity. I'd also love to see your reasoning of why older songs are easier to predict. What do popularity look like by year? Are older songs generally less popular?
Overall, good modeling and a well written report!
This project's goal is to understand what makes a particular song popular on Spotify. Their intention is to build a model that correctly classifies songs as popular and not popular. In order to do this, they gather historical data for songs from the past 100 years and study it by building several kinds of models.
Some necessary feature transformations and data cleaning is performed. After this, they fit four different kinds of models (LSR, kNN SVM, and logReg) Performance from all models is high and the final model chosen can accurately predict a songs' popularity on Spotify.
Three positive things about the project are:
Three possible areas of improvement are:
All in all, the project is interesting and well explained. However, I wonder if the approach is the most accurate one as I would guess that music tastes shift through time, and the "popularity" measure predicted is decided by Spotify which may create a feedback loop. Nevertheless, I think the work was conducted in an orderly manner and the result was very insightful.
Nico Oriol
Summary
This project is about predicting the popularity of Spotify songs based on couple of features such as year, danceability, tempo, and acousticness.
Things I like
Things can be improved
What's it about?
The report is about identifying features that make a song popular on Spotify.
What data are they using?
Spotify Web API was used. In particular, the dataset consisted of 160,000 songs from 1921 to 2020.
What's their objective?
The objective is to use the Spotify API in order to perform various supervised learning analysis on it and create models that accurately predict if the song will be popular on Spotify.
Three things I like about the Midterm Report
• I find the data processing sections to be very detailed and useful as imputation methods such as dropping unnecessary columns were completely dropped.
• The explanation of the algorithms used and the process by which it was applied allowed me to easily comprehend the overall methodologies used in the Data Modeling portion.
• The use of the R2 value was unique and helped me understand how well the various models used fit the data quantitatively.
Three areas for improvement
• The data visualization section was incredibly terse due to a lack of explanation of how the various scatterplots and histograms of features help differentiate successful and unsuccessful songs.
• The use of the confusion matrix was confusing, as it seems unhelpful when figuring out how the metrics accuracy test score was obtained.
• The report does not take into consideration the changing popularities across years and the means by which identifying successful songs can be adapted with time.
What I liked:
What could improve:
This project is about to predict a song's popularity from a number of descriptive features, such as acousticness, danceability. In the midterm report, the team presents the data-cleaning and some preliminary data exploration.
I like the six histograms at the beginning. With the label “acousticness”, ”danceability”, it is really hard to understand the meaning without looking at the histogram distributions.
I like the Preliminary Analysis part, especially how you did the preliminary regressions and decided to drop some features such as “tempo” to get a more accurate prediction
The year feature is definitely worth looking at. As you brought up, there is an overreliance on the year the song was released. My personal reasoning is that the popularity is defined at the current era, and it is possible that songs from is specific time is regarded as “popular trend” now.
This project is aiming to predict a song's popularity from a number of descriptive features.
Good job with data preparation and visualizations, Maybe add a sentence restating your objective (I had to go to your proposal to figure out your target variable at first).
One thing for your preliminary analyses I might recommend is to look at a correlation matrix to get exact values for ‘largely positively correlated’. Also you could look at the correlation coefficient between features to see which ones might be redundant.
Good job creating the baseline model and reporting some stats on it. It is helpful to refer back to these and gives a good sense of baseline performance.
In your effectiveness section, you mention figure 3 and figure 4 but I was not sure where to find them (the figures on page 3 are labeled 1 and 2).
I definitely agree in using cross validation and investigating combinations or transformations of features going forward.
The over reliance on the year might be due to the fact that Spotify grew its popularity recently, and thus was not around to capture the ‘popularity’ generated from the release of a song for the older years (kind of like how the streams of a song within its first week of release greatly determine the future success of the song - I hope I’m making sense here).
I like your idea to investigate the three separate files to see if it can improve performance. I think this is a good idea (especially by genre).
It also might be beneficial to investigate a subset of artists. I would imagine there’s probably a lot of artist who have only one or two songs. Constructing a model using data from a couple artists with a lot of songs might be interesting to look at.
Overall good job! I think it's an interesting project that can be very useful and applicable.
This project is about predicting trends in Spotify song data. Key features in the dataset include metrics on danceability, energy, etc provided by the Spotify API. For the midterm report, they presented some preliminary regression analysis that gave them insights on feature selection.
Overall good start!
Summary
The project is about the popularity of songs listed on Spotify. The dataset the team will be using is from Kaggle, which includes variables describing the audio characteristics of the song as well as metadata such as name of artist, release date, etc. The main goal of the project is to determine whether or not a song will become popular based on the previously mentioned predictors.
Things I liked
Ideas and areas for improvement
This project is about predicting trends in Spotify song data. They utilized different models to try to predict whether a given song popular or not popular based on given features from Spotify's API.
Overall, the report was really well written. The structure and organization was super clear and I was able to follow the flow of their logic and the different models that they ran. Another strength was that the explanations behind why each model was used was really clear. It was clearly demonstrated that the members of this group were knowledgeable about each model and the algorithm behind it. Finally, I thought the confusion matrix was really well done and helped us understand the results of the model.
An area of improvement is in the data visualizations. In general, the histograms were useful but I thought including the scatterplots were not as useful since there were so many data points that there's no clear trend or conclusion I can draw from it. I think there could have been other visualizations that would've been better for your objective. Another area of improvement I can think of is to consider using a different popularity threshold since there are very little songs that actually pass the threshold. This might've caused the models to appear more effective than they actually are. Finally, I would've liked to see a deeper explanations to why the models worked as well as they did rather in addition to the numerical results.
Good job overall! I really enjoyed reading it :)
This project includes determining whether a song will be popular and whether to recommend it to other users through Spotify. The team use data from Kaggle which shows different characteristics of Spotify songs, such as acoustics, danceability, energy and popularity. Since there are plenty of characteristics of each song, researchers believe that they can use different supervised model to predict the popularity of songs.
Which I like about the proposal:
The project is about predicting popularity of Spotify features. The objective is to find a model using features of Spotify music that predicts whether the song is popular or not and a recommendation system that recommends similar songs. A dataset containing 169,909 data points each with 19 features are used.
I like that the results of this project can be quite useful since most of us listens to music in our life. I also think the dataset is large enough to generate a good model. Also, the feature space is quite large. It probably sufficiently covered all features that need to be considered, thus would give a good model.
I am, however, confused how you would use the model generated to produce a recommendation system. I assume in your proposal you want to use those “unique characteristics” that makes a song popular also as factors when considering whether two songs are similar. I am thinking maybe what makes a song popular is not enough to indicate whether it is similar to another song. I think if you’d like to do that, it’s better to train another model that predicts whether two songs are similar or not. In addition, I would recommend deleting some features that may not affect popularity since too many features may overfit. There are also 33375 unique values for the feature ‘artist’ which are too many. I would recommend rating artists by how popular (maybe need to find some other information) they were when the song was released and take a weighted average among all artists of this song.
Through this project, the team is trying to use features presented on Spotify to identify which song would be popular. The dataset they are using are songs available on Spotify from 1921 to 2020 and their relating features. By doing this project, they hope to shade some idea on what makes a song popular to what extent.
What I like about this project:
Some suggestions that I think could be helpful:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.