iancostello / websitetomongo Goto Github PK

WebsiteToMongo is a configurable web crawler that downloads websites locally and then indexes the results in a mongo db dataase

##How To Configure Configuration is relatively easy and mostly requires inputting values into indexRules.settings

###Defaults #####followExternalLinks:(true/false) Whether to follow external links #####respectRobots:(true/false) Whether to respect robots.txt, but currently not implemented

###Basic Config #####homepageURL: http://www.google.com/ The URL or local file of where the webcrawler should start #####websiteURL: http://www.google.com/ URL of website (if offline this is still the actual sites url) #####workingDir: /Users/sampleUser/Documents/testWebsites/ Source Folder #####subdir: google Where to save in source folder #####saveType: (fullSite/content) fullSite downloads the entire site for offline viewing, while content just downloads certain content

###Content Config Only Required For saveType:content #####contentType:video The selector for content type to download #####contentLoc:src The tag to get the content source eg

###Database Config #####database: wikiForSchoolsTest Mongo Database Name #####collection: activities Mongo Collection Name

###Follow Rules Rules about what links to follow and files to download #####downloadFiles:jpg, png, gif, jpeg, tif, css, js File types to download #####linksToFollow:html, htm, php, asp Href links to follow #####disallow:.png.htm Example Exclude Paths #####disallow:/images/ Example Exclude Path

###Index Rules Rules about what files to Index #####includes: video Index only if has selector type eg #id .class or element eg button #####!include: .audio Don't index if has selector #####url-pattern: relevantFiles/ Index only if has url pattern #####!url-pattern uselessFiles/ Don't index if has urlPath

##To Index How Indexed Results Are loaded into the database ####### field:value ###Fields No Rules About Names ###Value ####Defaults Always Starts with $ and following default value

The filetype eg html

The size of a file in MB

The name of the file eg index.html

Full filepath from working directory google/index.html

The text of value of a link that points to the page ####Literals Denotes by text inside of ' or " it is always read as such ####Commands Grab Certain Elements inside of doc. Commands are pretty limited as of now, but you can grab any selector and any of its children. #####selector Grabs the text of a selector #####selector+child(0) Grabs the text of the first child of a selector