Now that you've seen a more extensive example of developing a web scraping script, it's time to further practice and formalize that knowledge by writing functions to parse specific pieces of information from the web page and then synthesizing these into a larger loop that will iterate over successive web pages in order to build a complete dataset.

You will be able to:

• Navigate HTML documents using Beautiful Soup's children and sibling relations
• Select specific elements from HTML using Beautiful Soup
• Use regular expressions to extract items with a certain pattern within Beautiful Soup
• Determine the pagination scheme of a website and scrape multiple pages

This lab will build upon the previous lesson. In the end, you'll look to write a script that will iterate over all of the pages for the demo site and extract the title, price, star rating and availability of each book listed. Building up to that, you'll formalize the concepts from the lesson by writing functions that will extract a list of each of these features for each web page. You'll then combine these functions into the full script which will look something like this:

df = pd.DataFrame()
for i in range(2,51):
    url = "http://books.toscrape.com/catalogue/page-{}.html".format(i)
    soup = BeautifulSoup(html_page.content, 'html.parser')
    new_titles = retrieve_titles(soup)
    new_star_ratings = retrieve_ratings(soup)
    new_prices = retrieve_prices(soup)
    new_avails = retrieve_avails(soup)
    ...

To start, write a function that extracts the titles of the books on a given page. The input for the function should be the soup for the HTML of the page.

def retrieve_titles(soup):
    #Your code here

Next, write a similar function to retrieve the star ratings on a given page. Again, the function should take in the soup from the given HTML page and return a list of the star ratings for the books. These star ratings should be formatted as integers.

def retrieve_ratings(soup):
    #Your code here

Now write a function to retrieve the prices on a given page. The function should take in the soup from the given page and return a list of prices formatted as floats.

def retrieve_prices(soup):
    #Your code here

Write a function to retrieve whether each book is available or not. The function should take in the soup from a given html page and return a list of the availability for each book.

def retrieve_availabilities(soup):
    #Your code here

Finally, write a script to retrieve all of the information from all 50 pages of the books.toscrape.com website.

#Your code here

If you used URL hacking to generate each successive page URL, instead write a function that retrieves the link from the "next" button at the bottom of the page. Conversely, if you already used this approach above, use URL-hacking (arguably the easier of the two methods in this case).

#Your code here

Well done! You just completed your first full web scraping project! You're ready to start harnessing the power of the web!