Python web scraper based on Scrapy framework for obtaining recipe data from most popular cooking websites.

1. https://cookpad.com/

1. Create Python virtualenv

   python3 -m venv recipevenv 

2. Activate Python virtualenv

   source recipevenv/bin/activate
   

3. Install Linux dependencies (for proper SQLAlchemy functioning)

   sudo apt install libpq-dev libffi-dev python3-dev libxml2 libxml2-dev libxslt-dev
   

4. Install Python libraries

   python setup.py install
   

5. Navigate to /cookpad directory, launch scrapy

   • Save recipes to postgreSQL database :

   scrapy crawl cookpadbot -a category=vegan

   • Save recipes to JSON file (if you want to disable saving to postgreSQL DB, comment ITEM_PIPELINES variable in cookpad/cookpad/settings.py file):

   scrapy crawl cookpadbot -o vegetarian.json -a category=vegetarian

6. To deactivate virtual environment:

   deactivate

Sample recipes available in cookpad/cookpad/sample_recipes directory.