Now that you've seen some basic linear regression models it's time to discuss further how to better tune these models. As you saw, we usually begin with an error or loss function for which we'll apply an optimization algorithm such as gradient descent. We then apply this optimization algorithm to the error function we're trying to minimize and voila, we have an optimized solution! Unfortunately, things aren't quite that simple.
Most importantly is the issue of generalization.
This is often examined by discussing underfitting and overfitting.
Recall our main goal when performing regression: we're attempting to find relationships that can generalize to new cases. Generally, the more data that we have the better off we'll be as we can observe more patterns and relationships within that data. However, some of these patterns and relationships may not generalize well to other cases.
#Import the Data here.
path = '/data' #The subdirectory where the file is stored
filename = 'movie_data_detailed.xlsx' #The filename
full_path = path + filename #Alternative shortcut
df = #Your code here
#Subset the Data into appropriate X and Y features. (X should be multiple features!)
X = #Your code here
Y = #Your code here2. For each feature in X, create several new columns that are powers of that feature. For example, you could take the $budget$ column and produce another column $budget2$, a third column $budget3$, a fourth column $budget**4$ and so on. Do this until you have more columns then rows.
#Your code here.
#Create additional features using powers until you have more columns then rows.3. Use all of your new features for X. Then train a regression model using RSS as your error function and gradient descent to tune your weights.
#Your code here4. Plot the model and the actual data on the Budget/Gross Domestic Product plane. (Remember this is just a slice of your n-dimensional space!)
#Your code here#Your response here
Note: This box (like all the questions and headers) is formatted in Markdown. See a brief cheat sheet of markdown syntax here!
Here lies the theoretical underpinnings for train test split. Essentially, we are trying to gauge the generalization error of our currently tuned model to future cases. (After all, that's the value of predictive models; to predict fturue states or occurences! By initially dividing our data into one set that we will optimize and train our models on, and a second hold out set that we later verify our models on but never tune them against, we can better judge how well our models will generalize to future cases outside of the scope of current observations.
6. Split your data (including all of those feature engineered columns) into two sets; train and test. In other words, instead of simply X and respective Y datasets, you will now have 4 subsets: X_train, y_train, X_test, and y_test.
X_train, X_test, y_train, y_test = #Your code here7. Train your model on the train set. [As before use RSS and gradient descent, but only use the training data.]
#Your code here#Your code hereIterate over training size sets from 5%-95% of the total sample size and calculate both the training error (minimized rss) and the test error (rss) for each of these splits. Plot these two curves (train error vs. training size and test error vs. training size) on a graph.
View Residuals and Train Test Split on Learn.co and start learning to code for free.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent