fairhat / kickstarter-crawler Goto Github PK

npm install kickstarter-crawler -g

npm test

Examples
API
The crawler as a service

The following example crawls a project collecting general, funding, and location
related data

'use strict';


var kickstarterCrawler = require('kickstarter-crawler');


var config, project;


// Project configurations
config = {
  url: 'https://www.kickstarter.com/projects/maxtemkin/philosophy-posters',
  fields: ['general', 'funding', 'location']
};


// Initialize the crawler
project = new kickstarterCrawler.project(config);


// Make request (crawl)
project.request(function onRequest (err, data) {

  // Something broke
  if (err) {
    console.log(err);
    return;
  }

  // Log crawled data
  console.log(data);

});

** 🍕 MORE EXAMPLES COMING 🍕 **

• {Object} project constructor
  
• Initializes the crawler and exposes its interface
  
  

• {Object} project configurations
  
• Configurations necessary to instantiate the project constructor
  
  

• {String} project profile url
  
  

• {Array} array of project data-fields, which indicate what data points will be crawled
  
• If undefined, data from all fields will be returned.
  

VALID FIELD VALUES

• general
• time
• funding
• location
• other
• media
• pledges

• Makes a HTTP request to the respective project url
  
• @param callback(err, data) {Function}
  
  

• Parses the HTML corresponding to the respective project profile page
  
• Returns a nested JSON of parsed data
• @param {String} HTML
  
• @return {Object}

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalTitle
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalCreator
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalCategory
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalSubCategory
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalProjectAvatarURL
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalProjectURL
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalCreatorURL
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} generalProjectVideoURL
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} timeNumDays
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} timeStart
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} timeEnd
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} fundingDollarsRaised
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} fundingGoal
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} fundingPercentRaised
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} fundingCurrency
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Boolean} fundingSuccess
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} fundingNumBackers
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} locationCity
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} locationState
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} locationCountry
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} otherUpdates
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} otherComments
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} otherProjectsCreated
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} otherProjectsBacked
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {String} otherWebsiteURL
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} mediaNumImages
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Array} mediaImages
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} pledgesNumPledges
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Number} pledgesNumLimitedPledges
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Array} pledgesAmounts
  
  

• @param {Function} callback(err, data)
  
• data {Object} will consist of {Object} pledgesData