Git Product home page Git Product logo

hackathon_scraper's Introduction

Hackathon Scraper

Hi all, this repo is to help out getting OKRs for DevRel folks!

IMPORTANT NOTE

For this, you need to install selenium with firefox to run the gitcoin portion. You can find out how to do that here

Otherwise, just add the flag --no-gitcoin

Here is how it works:

  1. Every week run python scrape.py --hackathon-keyword <KEYWORD_HERE> With `<KEYWORD_HERE> being the type of hackathon you're looking for

This will save a file to next_week.json which will have a list of hackathons that you'll look to check out once submissions are availible.

  1. After each week, run python github_follow_up.py --github-keyword <GITHUB_KEYWORD>

This will pull from next_week.json and give you the github projects from submissions that have the project you're looking for!

Example: At the start of the week I run: python scrape.py --hackathon-keyword blockchain This will show me all the blockchain hackathons that are ending in 7 days or less.

Then I run: python github_follow_up.py --github-keyword chainlink This will show me all the hackathon submissions that recently ended that have their github repo have that keyword in it.

Sample output:

$ python github_follow_up.py --github-keyword chainlink
_________________________________
Total submissions: 53 + 45 = 98
2020-06-16 23:31:14.815999
 No projects had that keyword :(
$ python github_follow_up.py --github-keyword const --input-file tests/test_data.json 
_________________________________
Total submissions: 5 + 45 = 50
2020-06-16 23:40:15.635757
 Number of devpost hackathon projects 2
INFO:root:[{'url': 'https://alphavhack.devpost.com/', 'prizes': 165375, 'platform': 'devpost'}, {'url': 'https://alphavhack.devpost.com/', 'prizes': 165375, 'platform': 'devpost'}]
INFO:root: Average prize pool of submissions: 165375.0
 Number of gitcoin hackathon projects 3
INFO:root:['/hackathon/GitxChange/projects/', '/hackathon/GitxChange/projects/', '/hackathon/GitxChange/projects/']
INFO:root: Average prize pool of submissions: 110250.0

Have fun!

Testing

Note: Due to github rate limiting you have to test each one at a time

Run run cd tests; pytest -v or to run a single test cd tests; pytest -v -k test_repo_has_keyword

TODO

  • Github organizations accounted for
  • If a hackathon has multiple pages of submissions it only gets the first page
  • Gitlab accounted for
  • Devfolio Integration
  • Gitcoin integration
  • More analytics metrics
  • Github OAuth key added so we don't get rate limited and have to run induvidual tests

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.