This is a web app that can crawl a domain. It uses Java, Gradle, Spring Boot, Spring Cloud Stream, Kafka, Liquibase, Docker and JSoup.
As a pre-requisite, docker needs to be installed.
Run below command in the project's root folder to start PostgreSql, Zookeeper and Kafka in docker:
docker-compose up -d
Then run the app using below command:
./gradlew bootRun
Once the start up completed, the app will be listening at http://localhost:8080/
Import the postman collection under src/test/postman to Postman.
- Send the
Start crawling
request to start the crawler process. - Send the
Crawler Job Stats
request to check the progress of the crawling - Send the
Crawler Pages
request to get a site map of the crawled domain