yingdazhai / scrapy Goto Github PK

Python Scrapy is the "go-to" end-to-end framework for flexible web data colletion from data scraping and cleaning to saving and preparation, where popular utility packages such as BeautifulSoup and Selenium could be easily integrated.

Scrapy Tutorial provides a comprehensive guide on how to build scrapy web crawlers for data collection with easy-to-follow examples.

This scrapy project contains 5 example scrapy projects written in Python scrapy, demonstrating how scrapy works for web data collection, including dealing with multiple requests, scraping text data and images, and utilizing item pipelines.

1. Scraping multiple quotes: extract text data of famous quotes - quotes.
2. Scraping a bookstore extract book details and cover images from a dummy online book store - books and bookimage.
3. Scraping soccer players - extract player's information and photos registered on sofifa.com website - fifa and fifaimage.

To run these examples, Scrapy needs to be installed. Scrapy can be installed either through anaconda or pip.

$ conda install -c conda-forge scrapy

or

$ pip install Scrapy

For installing on other OS and any other installation queries, please click here.

Details on how to run individual examples are provided in dedicated README.md files inside these 5 projects.