Web scraping with python3 requests and BeautifulSoup
pip install -r requirements.txt
requirements.txt
requests==2.19.1
beautifulsoup4==4.6.3
requests module for requesting the url and fetching response and bs4 (beautifulsoup4) for making web scraping easier.
Once you have the requirements installed you can simply import and use. For now we will be requesting the yelp service and playing around with our modules
requesting_yelp.epy
import requests
from bs4 import BeautifulSoup
You can visit yelp and search for anything in search bar, for example we searched for restaurants and it returned back Best Restaurants in San Francisco, CA
Copy the url from the browser and paste it into your file
requesting_yelp.epy
url = "https://www.yelp.com/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA&ns=1"
Now requests comes to work. Since we make an GET request with our browser for this url and
we get back all the html response in form of new webpage, we are gonna do the same with
requests.get(url) and store the result.
requesting_yelp.epy
response = requests.get(url)
Now response contains the returned result from the get request to the url. We can use methods on response like, you can actually print response,
print(response)
print(response.status_code)
run the file
cmd
.../python_scraping_web> py requesting_yelp.py
<Response [200]>
200
200. The HTTP 200 OK success status response code indicates that the request has succeeded.
To actually print the whole html that the webpage contains we can print
print(response.text)
Earlier version of python requests used to print the html from response.text in ugly
way but on printing it now we can get the prettified html or we can also use the bs4 module
For that we need to create a BeautifulSoup object by passing in the text returned from the url,
soup = BeautifulSoup(response.text)
print(soup.prettify())
<img height="1" src="https://www.facebook.com/tr?id=102029836881428&ev=PageView&noscript=1" style="display:none" width="1">
</img>
</noscript>
<script>
(function() {
var main = null;
var main=function(){var c=Math.random()+"";var b=c*10000000000000;document.write('<iframe src="https://6372968.fls.doubleclick.net/activityi;src=6372968;type=invmedia;cat=qr3hlsqk;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;ord='+b+'?" width="1" height="1" frameborder="0" style="display:none"></iframe>')};
if (main === null) {
throw 'invalid inline script, missing main declaration.';
}
main();
})();
</script>
<noscript>
<iframe frameborder="0" height="1" src="https://6372968.fls.doubleclick.net/activityi;src=6372968;type=invmedia;cat=qr3hlsqk;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;ord=1?" style="display:none" width="1">
</iframe>
</noscript>
</body>
</html>
Resultant html should be something like this in both requests and BeautifulSoup case.
But BeautifulSoup gives us more advanced methods for scraping like the find() and
findall()
requesting_yelp.py
links = soup.findAll('a')
print(links)
...
<span class="dropdown_label">
The Netherlands
</span>
</a>, <a class="dropdown_link js-dropdown-link" href="https://www.yelp.com.tr/" role="menuitem">
<span class="dropdown_label">
Turkey
</span>
</a>, <a class="dropdown_link js-dropdown-link" href="https://www.yelp.co.uk/" role="menuitem">
<span class="dropdown_label">
United Kingdom
</span>
</a>, <a class="dropdown_link js-dropdown-link" href="https://www.yelp.com/" role="menuitem">
<span class="dropdown_label">
United States
</span>
...
A lot of links exists so you terminal should be full of links and html tags
We can loop over the links variable and print the individual link
for link in links:
print(link)
On running
...
<a href="/atlanta">Atlanta</a>
<a href="/austin">Austin</a>
<a href="/boston">Boston</a>
<a href="/chicago">Chicago</a>
<a href="/dallas">Dallas</a>
<a href="/denver">Denver</a>
<a href="/detroit">Detroit</a>
<a href="/honolulu">Honolulu</a>
<a href="/houston">Houston</a>
<a href="/la">Los Angeles</a>
<a href="/miami">Miami</a>
<a href="/minneapolis">Minneapolis</a>
<a href="/nyc">New York</a>
<a href="/philadelphia">Philadelphia</a>
<a href="/portland">Portland</a>
<a href="/sacramento">Sacramento</a>
<a href="/san-diego">San Diego</a>
<a href="/sf">San Francisco</a>
<a href="/san-jose">San Jose</a>
<a href="/seattle">Seattle</a>
<a href="/dc">Washington, DC</a>
<a href="/locations">More Cities</a>
<a href="https://yelp.com/about">About</a>
<a href="https://officialblog.yelp.com/">Blog</a>
<a href="https://www.yelp-support.com/?l=en_US">Support</a>
<a href="/static?p=tos">Terms</a>
<a href="http://www.databyacxiom.com" rel="nofollow" target="_blank">Some Data By Acxiom</a>
This looks a lot cleaner now.
So far we have requesting a single url. In this section we will be formatting url to request a different url.
If you look at the yelp url we used before, you might find at the very bottom that there is pagination being used.
So what we can do is visit another search page say 2 page and we find that url
changed a bit.
Specifically the url had some new value at last for the 2 page which is
https://www.yelp.com/search?find_desc=Restaurants&find_loc=los+angeles&start=30
You guessed it right
&start=30
is what is new in the url, if you have worked with django then you might have used pagination somewhere in your templates.
So that means we can actually add this value at the end of the existing url and locate to another search result page
Have a look at formatting_url.py
import requests
from bs4 import BeautifulSoup
base_url = "https://www.yelp.com/search?find_desc=Restaurants&find_loc={}"
city = "los angeles"
start = 30
url = base_url.format(city)
second_page = url + '&start=' + str(start)
response = requests.get(third_page)
print(f"STATUS CODE: {response.status_code} FOR {response.url}")
soup = BeautifulSoup(response.text, 'html.parser')
links = soup.findAll('a')
We assign 30 value to the start and add it as str(start) at the end of the url
and name it second_page and then request that page. We get 200 status code
This means that by finding the patterns in url we can request more url.
So what more could be done. We can start a loop that would request the urls and
each time increment the start value by 30
start = 0
for i in range(40):
url = base_url.format(city)
url += '&start=' + str(start)
start += 30
if start == 270:
break
...
Now how do I know that we have to increment by 30 well I checked the pattern of
urls by visiting the pages and stop at 270 so that we only request 10 pages.
You can use whatever value you want but it should be multiple of 30
Now we will be using the previous code that we wrote in formatting_url.py and
extract the particular piece of text from the html tags that we need which is the title
of the restaurant from each search page.
Visit the url and open developers tools
and point at the block of restaurant with title, rating, review etc. and find the
li tag with class regular-search-result
We will be using this class for searching the particular li tag from the response
using BeautifulSoup
reading_name.py
import requests
...
info_block = soup.findAll('li', {'class': 'regular-search-result'})
print(info_block)
Run the file and you should the whole li tag and its inner tags printed. But we want to extract the title of the restaurant from each li tag, for that we have to find the class used in the title of restaurant
The title is wrapped inside a anchor tag with class biz-name
info_block = soup.findAll('a', {'class': 'biz-name'})
print(info_block)
count = 0
for info in info_block:
print(info.text)
count += 1
print(count)
On printing the text of the html tag we get the title of the restaurant, these are
not all the title cause some block don't have biz-name class but we have what we
need.
We can also write the names of restaurants onto a file but we have to use a try except block while performing the file writing operation since the title names of restaurants contains some non str characters which could cause error.
with open('los_angeles_restaurants.txt', 'a') as file:
start = 0
for i in range(100):
url = base_url.format(city, start)
response = requests.get(url)
start += 30
print(f"STATUS CODE: {response.status_code} FOR {response.url}")
soup = BeautifulSoup(response.text, 'html.parser')
names = soup.findAll('a', {'class': 'biz-name'})
count = 0
for info in names:
try:
title = info.text
print(title)
file.write(title + '\n')
count += 1
except Exception as e:
print(e)
print(f"{count} RESTAURANTS EXTRACTED...")
print(start)
if start == 990:
break
For any questions regarding what we have done so far contact me at CodeMentor
In this section we will be go a little more further and extract the name, address, phone-number of the restaurant.
This time we will be looking for the div tag that has class biz-listing-large
that contains the restaurant details.
In writing_details.py We have reused a lot of code from other files the only
difference is that we open a new file, fetch the title, address, and phone number
from respective classes and write it into the file.
...
city = "los+angeles"
...
file_path = f'yelp-{city}.txt'
with open(file_path, 'w') as textFile:
soup = BeautifulSoup(response.text, 'html.parser')
businesses = soup.findAll('div', {'class': 'biz-listing-large'})
count = 0
for biz in businesses:
title = biz.find('a', {'class': 'biz-name'}).text
address = biz.find('address').text
phone = biz.find('span', {'class': 'biz-phone'}).text
detail = f"{title}\n{address}\n{phone}"
textFile.write(str(detail) + '\n\n')
We edit the city value so that it neither conflicts with url and our file path name.
yelp_los+angeles.txt is still doesn't has text in nice formatted way like we wanted but in the next section will be working on it.
AMF Beverly Lanes
1201 W Beverly Blvd
(323) 728-9161
Maccheroni Republic
332 S Broadway
(213) 346-9725
Home Restaurant - Los Feliz
1760 Hillhurst Ave
(323) 669-0211
...
Once we have extracted the data we want to make our data look good i.e. not without any spaces and newlines so for that we will use a simple logic
writing_clean_data.py
...
with open(file_path, 'a') as textFile:
count = 0
for biz in businesses:
try:
title = biz.find('a', {'class': 'biz-name'}).text
address = biz.find('address').contents
# print(address)
phone = biz.find('span', {'class': 'biz-phone'}).text
region = biz.find('span', {'class': 'neighborhood-str-list'}).contents
count += 1
for item in address:
if "br" in item:
print(item.getText())
else:
print('\n' + item.strip(" \n\r\t"))
for item in region:
if "br" in item:
print(item.getText())
else:
print(item.strip(" \n\t\r") + '\n')
...
We simply get the text of the item if there are any br tags else we strip the newlines, return lines, tabs and spaces / space from the text, On running the file
800 W Sunset Blvd
Echo Park
4156 Santa Monica Blvd
Silver Lake
8500 Beverly Blvd
Beverly Grove
5484 Wilshire Blvd
Mid-Wilshire
5115 Wilshire Blvd
Hancock Park
126 E 6th St
Downtown
8164 W 3rd St
Beverly Grove
7910 W 3rd St
Beverly Grove
4163 W 5th St
Koreatown
435 N Fairfax Ave
Beverly Grove
1267 W Temple St
Echo Park
429 W 8th St
Downtown
724 S Spring St
Downtown
8450 W 3rd St
Beverly Grove
2308 S Union Ave
University Park
5583 W Pico Blvd
Mid-Wilshire
'NoneType' object has no attribute 'contents'
3413 Cahuenga Blvd W
Hollywood Hills
727 N Broadway
Chinatown
6602 Melrose Ave
Hancock Park
612 E 11th St
Downtown
...
The same way we have to clean the phone number
...
for item in phone:
if "br" in item:
phone_number += item.getText() + " "
else:
phone_number += item.strip(" \n\t\r") + " "
...
except Exception as e:
print(e)
logs = open('errors.log', 'a')
logs.write(str(e) + '\n')
logs.close()
address = None
phone_number = None
region = None
Again run the file and change the start = 990 delete all content in
yelp-{city}-clean.txt run the file again. All details of the restaurant will
be written to the file
yelp-{city}-clean.txt
Tea Station Express
Bestia
2121 E 7th Pl Downtown
(213) 514-5724
République
624 S La Brea Ave Hancock Park
(310) 362-6115
The Morrison
3179 Los Feliz Blvd Atwater Village
(323) 667-1839
A Food Affair
1513 S Robertson Blvd Pico-Robertson
(310) 557-9795
Running Goose
1620 N Cahuenga Blvd Hollywood
(323) 469-1080
Howlin’ Ray’s
Perch
448 S Hill St Downtown
(213) 802-1770
Faith & Flower
705 W 9th St Downtown
(213) 239-0642
Notice that we some of the data is missing because due to some error we reduce the risk
of code crashing by setting the values to None
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent