internetsadboy / kickstarter-crawler Goto Github PK

Kickstarter crawler that does what you think it would

JavaScript 100.00%

kickstarter-crawler's Introduction

Getting started

const kickstarterCrawler = require('kickstarter-crawler');

// kickstarter project url
const URL = 'https://www.kickstarter.com/projects/maxtemkin/philosophy-posters/'

// returns promise
const result = kickstarterCrawler(URL);
result.then((data) => console.log(data));


// output
{
  title: 'Philosophy Posters',
  creator: 'Max Temkin',
  description: 'Ten giant philosophy posters with big ideas presented simply.',
  category: 'Graphic Design',
  location: [ 'Chicago', 'IL' ],
  duration: 30,
  startdate: 'Apr 4, 2012',
  enddate: 'May 4, 2012',
  funding: 41167,
  goal: 2000,
  backers: 1393,
  pledges: [ [ 20, 1003 ], [ 30, 359 ] ],
  thumbnail: 'https://ksr-ugc.imgix.net/assets/011/314/328/15f265a2407f6dcb311ad61bad886b01_original.jpg?ixlib=rb-2.1.0&crop=faces&w=1552&h=873&fit=crop&v=1463680857&auto=format&frame=1&q=92&s=c787b7ea241aefb1f14a5acf9b939125',
  project_url: 'https://kickstarter.com/projects/maxtemkin/philosophy-posters',
  videoDuration: '02:08'
}

Insallation

npm i kickstarter-crawler

Test

npm run test

kickstarter-crawler's People

Contributors

Stargazers

Watchers

Forkers

nivertech romansky lfagundes jivs fairhat shanxin136 ruibolun2015 rodrigobuendia rasata codymikol richtong queryee senofjohan

kickstarter-crawler's Issues

Unable to retrieve data from given Kickstarter URL

Hi @internetsadboy, Thank you for this sample module. I have recently cloned this repository and try use this as API for my project.
I am facing some issue in retrieving the data, so I tried with the URL given in the README file but still I am facing the same issue. I am sure that am doing something wrong or wrongly using this project. Could you please help me out to resolve this.

Below are the steps that I have done from my end to run this project.
In package.json file I have added the start script(start: node index.js) and then
In index.js I just copied all the stuffs from getFunds.js and ran the project using "npm start"

I am getting something like the attached screenshot

I think the values are not getting updated from the given URL from my understanding the project object is not invoked properly . Since I couldn't get any output from getters (even console.log was also not working inside the getters). eg. I tried using getBackers() and getDoolarsRaised().

Please provide some suggestions on this. Thanks in advance.

Availiable dataset

Please is it possible to make a subset of your dataset avaliable with around 3000 projects in csv or json format with all variables?

It will be very very helpful for me if you could make it because I can't find anything for my thesis. Please please please.....

some fields aren't fetched

Hi, I was making a scraper by myself while I found your git, ur lib is super nice!
as I tried to run a simple code to see what data could be fetched I realized that basic data, such as Creator and Campaign Name, aren't scraped correctly.

This is the code I used

app.get('/kick_scrape', function(req, res){

    options = {
        url: 'https://www.kickstarter.com/projects/jakehimself/the-bluelace-project-a-revolution-built-one-foot-a'
    }

    request(options, function(error, response, html){
        if(!error){
            var project = new ks.project(options)
            var title, release, rating;
            var json = { 
                title : "", 
                creator : {
                    name: "",
                    image: "",
                    fb: ""
                },
                goal: ""
            };

            project.request(function(err, data) {
              if(err) throw err
              res.send(data);
            })
        }
    })
})

and that's the result I get

{
    "general_title": "",
    "general_creator": "",
    "general_subCategory": "",
    "general_avatar": "https://s3.amazonaws.com/ksr/avatars/1937456/HEADSHOT.small.jpg?1393464758",
    "general_projectUrl": "http://www.kickstarter.comundefined",
    "general_creatorUrl": "http://www.kickstarter.comundefined",
    "location_city": "",
    "funding_dollarsRaised": 150795.34,
    "funding_fundingGoal": 25000,
    "funding_percentRaised": 603.18,
    "funding_currency": "USD",
    "funding_successful": true,
    "funding_backers": 11749,
    "time_days": 30,
    "time_start": 1384873738000,
    "time_end": 1387465738000,
    "media_numPictures": 17,
    "media_pictures": [
    "https://s3.amazonaws.com/ksr/projects/741464/photo-1024x768.jpg?1397854212",
    "https://s3.amazonaws.com/ksr/avatars/1937456/HEADSHOT.small.jpg?1393464758",
    "https://s3.amazonaws.com/ksr/avatars/1937456/HEADSHOT.small.jpg?1393464758",
    "https://s3.amazonaws.com/ksr/assets/001/351/633/d8a84934f76071e111a31d1e605c08af_large.jpg?1384900383",
    "https://s3.amazonaws.com/ksr/assets/001/351/597/30eb46bad068c4c83655c4e1abcd2b2c_large.jpg?1384900059",
    "https://s3.amazonaws.com/ksr/assets/001/458/109/f83b4cd7ed8643eb24632ed1bff50fb4_large.jpg?1387464863",
    "https://s3.amazonaws.com/ksr/assets/001/296/066/232a0a2f291f49c74888b354986423d2_large.jpg?1383863301",
    "https://s3.amazonaws.com/ksr/assets/001/350/399/6669fab85d541438fba8286b6dd4beb3_large.jpg?1384887898",
    "https://s3.amazonaws.com/ksr/assets/001/299/182/89cb4b4f42d2c4ef511008e22f794716_large.jpg?1383921864",
    "https://s3.amazonaws.com/ksr/assets/001/299/163/932e37729a5a1faecbc1d0081bed691c_large.jpg?1383921679",
    "https://s3.amazonaws.com/ksr/assets/001/299/254/596d8a5f936ac1d0bc2bd7d18fa1a7a0_large.jpg?1383922840",
    "https://s3.amazonaws.com/ksr/assets/001/299/255/4fb9e0b870745fc8355db39fa4586c8f_large.jpg?1383922859",
    "https://s3.amazonaws.com/ksr/assets/001/299/259/64fedeedf06fce670a02709209d5d5f6_large.jpg?1383922889",
    "https://s3.amazonaws.com/ksr/assets/001/299/917/0f6a4033439a6e4f9ccf09f68487d094_large.jpg?1383933076",
    "https://s3.amazonaws.com/ksr/assets/001/417/907/5ba1c4104808f112ba1937fbbc8082ee_large.jpg?1386366135",
    "https://s3.amazonaws.com/ksr/assets/001/405/895/e3b758c3c421246ebdc7a6e7597ad5de_large.jpg?1386108447",
    "https://s3.amazonaws.com/ksr/assets/001/456/799/2ea982eabaf3b7b4749c3c89264310ce_large.jpg?1387419892"
    ],
    "facebook_connected": false,
    "facebook_name": "",
    "facebook_numFriends": 0,
    "other_updates": null,
    "other_comments": 379,
    "other_projectsCreated": 0,
    "other_websiteName": "",
    "pledges_number": null,
    "pledges_limited": false,
    "pledges_percentLimited": null,
    "pledges_amounts": [],
    "pledges_data": {}
}

I'll take a look to the code and see if I can fix it. if I succeed, I'll send u a PR.

cheers!
Aaron

comments

Hi,

Good project and Thank You.

It's possible to crawler all the comments and not only the sum of comments ?

Thanks

Get project description

Hey,

I found your kickstarter-crawler while searching for an official kickstarter api (damn you, kickstarter).
It's currently possible to get some basic information about the kickstarter project, however, are there any plans on supporting the project description? I'd like to get the project description without any images, just plain text and it seems like no package supports that (yet).

Docs++

How can the documentation be improved?

Missing main project image?

Hi @GhostsNstuff

I'm trying this out via kcaas.io.

It seems to me as if the main project image is not being returned.

I'm seeing generalAvatarURL and the array mediaImages.

If I visit a kickstarter project and inspect the source, the main project image src is neither the generalAvatarURL nor any of the images in mediaImages.

For example, if you visit https://www.kickstarter.com/projects/597507018/pebble-time-awesome-smartwatch-no-compromises you will see that the src for the main image is: https://ksr-ugc.imgix.net/projects/1657474/photo-original.png?v=1425387906&w=1024&h=768&fit=crop&auto=format&lossless=true&s=9b9281dbffa42cb8cd882dd22605770e

That's not returned using kickstarter-crawler (via kcaas.io).

If I use the project url in embedly.com then it does retrieve that image URL when it scrapes kickstarter.

If you visit https://www.kickstarter.com/projects/597507018/pebble-time-awesome-smartwatch-no-compromises.json it returns an object with the html (is that what you use?) where the project image is listed (has class="project-thumbnail-img").

Collect all data

I have installed the crawler through this npm install kickstarter-crawler -g

How can I download all 61,356 kickstarter projects ?

Urls lists

This is a very nice approach and congratulations.

To be honest because I am not familiar with technologies how can I get the whole list of 61,356 kickstarter projects urls in order to use kickstarter-crawler?

As I can I understand I have to run 61,356 times in order to take the full data?

otherUpdates returning null

This also causes tests to fail