This is a web app that can crawl a domain. It uses Java, Gradle, Spring Boot, Spring Cloud Stream, Kafka, Liquibase, Docker and JSoup.

As a pre-requisite, docker needs to be installed.

Run below command in the project's root folder to start PostgreSql, Zookeeper and Kafka in docker:

docker-compose up -d

Then run the app using below command:

./gradlew bootRun

Once the start up completed, the app will be listening at http://localhost:8080/

Import the postman collection under src/test/postman to Postman.

• Send the Start crawling request to start the crawler process.
• Send the Crawler Job Stats request to check the progress of the crawling
• Send the Crawler Pages request to get a site map of the crawled domain