Python web scraper based on Scrapy framework for obtaining recipe data from most popular cooking websites.
-
Create Python virtualenv
python3 -m venv recipevenv
-
Activate Python virtualenv
source recipevenv/bin/activate
-
Install Linux dependencies (for proper SQLAlchemy functioning)
sudo apt install libpq-dev libffi-dev python3-dev libxml2 libxml2-dev libxslt-dev
-
Install Python libraries
python setup.py install
-
Navigate to /cookpad directory, launch scrapy
- Save recipes to postgreSQL database :
scrapy crawl cookpadbot -a category=vegan
- Save recipes to JSON file (if you want to disable saving to postgreSQL DB, comment ITEM_PIPELINES variable in cookpad/cookpad/settings.py file):
scrapy crawl cookpadbot -o vegetarian.json -a category=vegetarian
-
To deactivate virtual environment:
deactivate
Sample recipes available in cookpad/cookpad/sample_recipes directory.