sreedharbukya / webcrawler Goto Github PK

Web crawler

Web crawler program is plain program.

Project structure contains as follows: 

1. PROJECT LAYOUT. 

Webcrawler/

    Crawler
      |--- CrawlerProcess (Actual program)
      |--- UrlHandler (Initially started looking at URL Handler)
    
    Tests
      |--- CrawlerProcessTestCase (Test cases for Crawler Process)
      |--- UrlHandlerTestCase (Test cases for Url Handler)

|--.gitingore (git ignore files)

|-- requirements.txt (third party dependency files)


This program is developed by Sreedhar Bukya

Please feel free to write me @ [email protected]