This is a NestJS app, handling webpage crawling for requested url using Puppeteer, and managing those crawled websites using a mongoDB based REST API.
- Have Docker running on your system.
- Start up the environment with you favorite js package manager (e.g.
pnpm start
,yarn start
ornpm run start
)
- Take a website data snapshot by POSTING the api endpoint
/crawl
//snapshots
with a body like { "url": "https://google.com" }. - Fetch snapshots list (optionally with filters/sorting), show or delete a specific snapshot with the
/snapshots
REST api endpoint.
You can find API reference in the following places:
- Go to
https://david-a.github.io/web-crawler/
(Static, manually updated docs website) - Using the
/docs
endpoint when the app is running. - Import
docs/swagger.json
into your Swagger.
- .env file included for simplicity but usually it should be gitignored
- TypeORM was used for connecting and querying MongoDB. Its MongoDB driver’s a bit outdated (^3.6.0, current version is 5.0.1) Another supported option is mongoose (mongoose implementation example can be shown here - https://github.com/david-a/vending-machine).