Git Product home page Git Product logo

html-crawler's Introduction

HTML CRAWLER

This is a HTML Crawler that has been made with Java 11 and Gradle 4.10.2 The application has been developed using TDD, from the starting usecase "HtmlCrawler" with the needed entities, to the detail of the application that is used to consume the useCase..

I used a repository called DataProvider that has been developed to be a FileSystem but the app is ready to use another DataProvider with the condition of return a Document.

You can test the provided use cases using the HtmlCrawlerTest class or building and executing the app with this commands:

gradle fatJar

java -jar ./build/libs/html-crawler-all-1.0-SNAPSHOT.jar ./src/test/resources/originalSource.html ./src/test/resources/similarSource1.html make-everything-ok-button

And then you'll get something like: #root > html > body > div > div > div > div > div > div > a that it's the path to the most similar object in the similar html.

I think that this kind of answer, that it's what was required in the excercise it's not enought verbose to understand the similarity of the found element But it's not hard to extend this answer and add things like the data in the found element and the % of similarity with the original element.

The algorithm used to found similar elements was something like "if I have an element with a tag and with attributes then I want the element in the similar html that has the most similarities with those tags and attributes"

** If you want to add a similar html then you have to add it in /src/test/resources folder

html-crawler's People

Contributors

ezefrd avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.