In the previous code along, we looked all the requirements for running an ols simple regression using statsmodels. We worked with a toy example to understand the process and all the necessary steps that must be performed. In this lab , we shall look at a slightly more complex example to study the impact of spendings in different advertising channels of total sales.

You will be able to:

• Set up an analytical question to be answered by regression analysis
• Study regression assumptions for real world datasets
• Visualize the results of regression analysis

In this lab, we will work with the "Advertising Dataset" which is a very popular dataset for studying simple regression. The dataset is available at Kaggle, but we have already downloaded for you. It is available as "Advertising.csv". We shall use this dataset to ask ourselves a simple analytical question:

Which advertising channel has a strong relationship with sales volume, and can be used to model and predict the sales.

# Load necessary libraries and import the data

# Check the columns and first few rows

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

# Get the 5-point statistics for data 

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

# Describe the contents of this dataset

# For all the variables, check if they hold normality assumption

# Record your observations on normality here 

Use scatterplots to plot each predictor against the target variable

# visualize the relationship between the preditors and the target using scatterplots

# Record yor observations on linearity here 

Based on above initial checks, we can confidently say that TV and radio appear to be good predictors for our regression analysis. Newspaper is very heavily skewed and also doesnt show any clear linear relationship with the target.

We shall move ahead with our analysis using TV and radio , and count out the newspaper due to the fact that data violates ols assumptions

Note: Kurtosis can be dealt with using techniques like log normalization to "push" the peak towards the center of distribution. We shall talk about this in the next section.

# import libraries

# build the formula 

# create a fitted model in one line

Note here that the coefficients represent associations, not causations

Hint: We can use model.predict() functions to predict the start and end point of of regression line for the minimum and maximum values in the 'TV' variable.

# create a DataFrame with the minimum and maximum values of TV

# make predictions for those x values and store them


# first, plot the observed data and the least squares line

      TV
0    0.7
1  296.4
0     7.065869
1    21.122454
dtype: float64

# Record Your observations on residuals

R-Squared: 0.33203245544529525
Intercept    9.311638
radio        0.202496
dtype: float64

model.summary()

# Record your observations here for goodnes of fit 

Based on above analysis, we can conclude that none of the two chosen predictors is ideal for modeling a relationship with the sales volumes. Newspaper clearly violated normality and linearity assumptions. TV and radio did not provide a high value for co-efficient of determination - TV performed slightly better than the radio. There is obvious heteroscdasticity in the residuals for both variables.

We can either look for further data, perform extra pre-processing or use more advanced techniques.

Remember there are lot of technqiues we can employ to FIX this data.

Whether we should call TV the "best predictor" or label all of them "equally useless", is a domain specific question and a marketing manager would have a better opinion on how to move forward with this situation.

In the following lesson, we shall look at the more details on interpreting the regression diagnostics and confidence in the model.

In this lesson, we ran a complete regression analysis with a simple dataset. We looked for the regression assumptions pre and post the analysis phase. We also created some visualizations to develop a confidence on the model and check for its goodness of fit.