Problem statement : The goal is to write a web scrapper via Node.JS and puppeteer, that will be taking the first entry of every forum post, and storing them in a MongoDB database without duplicates.
Source website: (https://forums.redflagdeals.com/hot-deals-f9/)
- install node modules - npm install
- start the script - npm start
- Data is stored in local mongoDB, database name : Koii
- Might need to do "npm start" twice as the first time it gets stuck on the first forum page and exceeds the time for buffer, should work fine after hitting npm start again.