In this lab, you'll learn how to use the .groupby() method in Pandas to summarize datasets.

You will be able to:

• Use groupby methods to aggregate different groups in a dataframe

Let's bring in the Titantic data set.

import pandas as pd

df = pd.read_csv('titanic.csv')
df = df.drop(columns=['Name','Ticket','Embarked', 'Cabin'])
df = df.dropna()
df.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

During the Exploratory Data Analysis phase, one of the most common tasks you'll want to do is split the dataset into subgroups and compare them to see if you can notice any trends. For instance, you may want to group the passengers together by gender or age. You can do this by using the .groupby() method built-in to pandas DataFrames.

To group passengers by gender, you would type:

df.groupby('Sex')

<pandas.core.groupby.generic.DataFrameGroupBy object at 0x7fbea9576100>

# This line of code is equivalent to the one above
df.groupby(df['Sex'])

<pandas.core.groupby.generic.DataFrameGroupBy object at 0x7fbea95525e0>

Note that this alone will not display a result -- although you have split the dataset into groups, you don't have a meaningful way to display information until you chain an Aggregation Function onto the groupby. This allows you to compute summary statistics.

You can quickly use an aggregation function by chaining the call to the end of the .groupby() method.

df.groupby('Sex').sum()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

You can use aggregation functions to quickly help us compare subsets of our data. For example, the aggregate statistics displayed above allow you to quickly notice that there were more female survivors overall than male survivors.

There are many built-in aggregate methods provided for you in the pandas package, and you can even write and apply your own. Some of the most common aggregate methods you may want to use are:

• .min(): returns the minimum value for each column by group
• .max(): returns the maximum value for each column by group
• .mean(): returns the average value for each column by group
• .median(): returns the median value for each column by group
• .count(): returns the count of each column by group

You can also see a list of all of the built-in aggregation methods by creating a grouped object and then using tab completion to inspect the available methods:

grouped_df = df.groupby('Sex')

# For the following line of code, remove the `#` and then hit the tab after the period.
#grouped_df.

This is a comprehensive list of all built-in methods available to grouped objects. Note that some are aggregation methods, while others, such as grouped.fillna(), allow us to fill missing values to individual groups independently.

The .groupby() method in pandas can also run multiple different aggregations by utilizing .agg() instead of a single aggregation. A python dictionary can be passed into .agg() where the keys are the column names you want to aggregate and the values are the string representation of the exact aggregation method you want.

df.groupby('Sex').agg({'PassengerId':'count',
                       'Survived':'sum',
                       'Age':'mean'})

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

In the cell above we returned three different aggregations on three seperate columns. We counted up the number of individuals using 'PassengerId':'count'. We looked at the number of people who survived via 'Survived':'sum' and finally we also returned the mean age via 'Age':'mean', all grouped by Sex.

You can also split data into multiple different levels of groups by passing in an array containing the name of every column you want to group by -- for instance, by every combination of both Sex and Pclass.

df.groupby(['Sex', 'Pclass']).mean()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

Since the resulting object returned is a DataFrame, you can also slice a selection of columns you're interested in from the DataFrame returned.

The example below demonstrates the syntax for returning the mean of the Survived class for every combination of Sex and Pclass:

df.groupby(['Sex', 'Pclass'])['Survived'].mean()

Sex     Pclass
female  1         0.811966
        2         0.722689
        3         0.466216
        ?         0.576271
male    1         0.415094
        2         0.258503
        3         0.151436
        ?         0.376623
Name: Survived, dtype: float64

The above example slices by column, but you can also slice by index. Take a look:

grouped = df.groupby(['Sex', 'Pclass'])['Survived'].mean()
grouped['female']

Pclass
1    0.811966
2    0.722689
3    0.466216
?    0.576271
Name: Survived, dtype: float64

# Using string index label
grouped['female']['1']

0.811965811965812

# Same result as python 0 index
grouped['female'][0]

0.811965811965812

Note that you need to provide only the value female as the index, and are returned all the groups where the passenger is female, regardless of the Pclass value. The second example shows the results for female passengers with a 1st-class ticket.

In this lab, you learned about how to split a DataFrame into subgroups using the .groupby() method. You also learned to generate aggregate views of these groups by applying built-in methods to a groupby object.