washeuteessen / washeuteessen-crawler_and_parser Goto Github PK

follow each ingredient: https://www.womenshealth.de/food/gesunde-rezepte/zutatenliste

maybe page number adding by 30 is not correct

https://test.washeuteessen.de/#/search?s=bbq

e.g. https://www.essen-und-trinken.de/rezepte/61624-rzpt-bbq-sandwich is double

Text is not extracted for recipe pages, which contain step by step instructions with pictures.
E.g. https://eatsmarter.de/rezepte/zimtbluetentee, https://eatsmarter.de/rezepte/vanille-milchmix

hey! great project!

i am interested in the recipes from chefkoch and others. i tried installing the docker and run the branch develop, yet the code runs and constantly outputs:

pymongo.errors.ServerSelectionTimeoutError: mongo:27017: [Errno -2] Name or service not known
where is the data stored in the end? it did show some recipes to be crawled.

thanks for your help!

use while condition
check whether any document with parsed = False is available
if yes, parse
if no, wait for 10 min.

Check if document already in DB and dont insert again

12000 duplicates found
ToDo:

• dump db locally (oc rsync)
• delete duplicactes
• monitor number of duplicates in grafana/prometheus