Git Product home page Git Product logo

climate-data-aggregator's Introduction

We're working on the rhok.org project for the worldbank
Problem definition: http://www.rhok.org/problems/web-based-interface-tabulating-daily-global-climate-dataset-0

Project manager:
Hrishi Patel
Github handle: hppatel
hrishi (dot) p (dot) patel (at) gmail (dot) com

Status as of 2:00 AM 12/5/2011

Some useful knowledge we've gained:
 - Can import data into sqlite db, but only one file at a time and it needs to be unzipped first
 - SQLite DB grows at a rate of approx. 10 MB per day of data imported without indices
 - Have prototype web interface, but it's extremely basic and will probably not work in practice (can't use it to get lat-long easily for instance)
 - Tried using PostGreSQL instead of sqlite, works but appears to be much slower, at least on Amir's machine with default settings

Next steps:
 - Get web version up and running.  For now, all we need is a google map we can click on that will pass lat-long back to the server
 - Get a REST endpoint up and running for the server that accepts above lat-long requests and returns all the data available for that point
 - Import 6 years of data into the DB

Data:
 - Unfortunately, the data is 13 GB of zipped ASCII files, so it's not easy to distribute.  
 	Several of the committers for this project have copies though.  If you would like to get 
	a copy, please contact one of us.  Samples of the unzipped data files can be found in
	/data/test/ .

climate-data-aggregator's People

Contributors

skykooler avatar amirdt22 avatar zanchin avatar davewalk avatar jkrrv avatar

Stargazers

Bennet Huber avatar

Watchers

 avatar James Cloos avatar Bennet Huber avatar  avatar  avatar  avatar

climate-data-aggregator's Issues

Make spaces vs. tabs consistent in python source files

Right now, we have a mixture of .py files using either spaces or tabs depending on who originally created them. While python doesn't mind this, so long as you're consistent within the same file, it would be nice to decide on a standard so we don't have to keep switching editors back and forth.

My vote is for spaces. I find them easier to work with, and according to the python style guide http://www.python.org/dev/peps/pep-0008/ they are much preferred over tabs. I would, however, like to get other people's opinions on this, particularly skykooler's as he's the other main python dev, before I go around changing everything.

Let me know what you guys think in the comments.

Port shell scripts to python

It would be nice if we could import the various shell scripts we have to python. One of the eventual goals of this project is porting it to a Windows/ArcGIS environment, which means we'll probably have to do this eventually anyway. In the meantime, it will be much easier to use if everything is in the same language.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.