Watch this Talk on Youtube

Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may access the World Wide Web directly using the Hypertext Transfer Protocol, or through a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying, in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis. - Wikipedia

                                            or 

A technique to extract(with the permission of authors/publishers ofcourse) information that you see on the webpages as you browse.

• To extract information
• A website may or may not have an API!
• Even if it does, you might be rate limited.

• Extract coupon information from deal of the day websites.
• Scrape reviews of your favourite films
• Capture data from Government websites and create datasets.
• Extract articles from news websites and make a news aggregator.

• Webscraping Ideas
• Webscraping Project Ideas - Medium
• 10 Mobile App Ideas that can be fueled by webscraping

• requests
• urllib

• Beautiful Soup 4
• Lxml
• re

• Scrapy

Extra credit: Selenium

• Webscraping by Wikipedia
• Khalid's Talk on Webscraping in EuroPython
• Real Python Series on Webscraping

🔔 Conference Notify

🚨 Covid 19

🚀 ISRO News