A personal scraper for developing automated testing and drawing data from sites.

Tech: Selenium
Notes:

• Selenium was developed for web testing instead of scraping, and is used for automated testing
  • carries out actions in browser environments
  • can make web requests and comes with a parser
• Advantages:
  • loads Javascript and can access data behind Javascript without necessarily going through additional requests yourself

Takeaways: Conclusion Web scraping can become handy and easy with tools such as Scrapy, BeautifulSoup, and Selenium. Each of these tools has it's own application. While Scrapy is the tool for complex projects, BeautifulSoup is used for smaller tasks. Selenium is used for Javascript featured websites – and can be used as a standalone web scraper and parser. They are all useful in their own way, and learning how to use all of them will make you a better web scraping developer.