weecology / retriever-recipes Goto Github PK

Dockerfile 0.51% Python 99.49%

data-retrieval hacktobefest data-science datasets data dataset

retriever-recipes's Introduction

Retriever-Recipes

The Data Retriever earlier used a simple CLI for developing new dataset scripts. This allowed users with no programming experience to quickly add most standard datasets to the Retriever by specifying the names and locations of the tables along with additional information about the configuration of the data. The script is saved as a JSON file, that follows the DataPackage standards.

This functionality has been moved to this repository to separate the scripts from the core retriever functionalities to help with organization, maintenance, and testing. The retriever-recipes repository thus holds all the scripts which were earlier shipped with retriever and also all the script adding/editing functionalities.

To facilitate the use of recipes as a command-line utility, the user can simply clone the repository and install it. The installation steps are mentioned in the next section.

Installation

    git clone https://www.github.com/weecology/retriever-recipes.git
    cd retriever-recipes
    python setup.py install

Using the Command Line

To see the full list of command line options and datasets run retriever-recipes --help. The output will look like this:

    usage: retriever-recipes [-h] {new_json,edit_json,delete_json,help} ...

    positional arguments:
      {new_json,edit_json,delete_json,help}
                            sub-command help
        new_json            CLI to create retriever json script
        edit_json           CLI to edit retriever json script
        delete_json         CLI to remove retriever json script
        help

    optional arguments:
      -h, --help            show this help message and exit

Examples

Some example usages of the CLI interface are:

Add a new JSON script:

    retriever-recipes new_json

Delete an existing JSON script:

    retriever-recipes delete_json dataset

Edit an existing JSON script:

    retriever-recipes edit_json dataset`

Website

For more information, see the Data Retriever website.

retriever-recipes's People

Contributors

Stargazers

Watchers

Forkers

harshitbansal05 henrykironde apoorvaeternity ethanwhite shivam7569 dumbmachine ashishpriyadarshicic shubhank-saxena kadam-tushar mohitkumar6122 sohan022 aakash3101 mohamedsto7y kkothari2001 dassaniansh nageshbansal pri1311 om-doiphode

retriever-recipes's Issues

ADD COVID19-Andalucia data

https://github.com/Pakillo/COVID19-Andalucia

Add dataset with python script

I encountered ModuleNotFoundError: No module named 'retriever' when I added a script named 'west_nile.py' to scripts folder to download a dataset although it is normally imported via python shell.

Notes on failing data packages https://dashboard.data-retriever.org

baltimore-restaurants

covid-case-surveillance

https://dev.socrata.com/foundry/data.cdc.gov/n8mc-b4w4#:~:text=Export%20dataset%20as%20CSV%20%C2%BB
use json api https://data.cdc.gov/resource/n8mc-b4w4.json
or
*** we have socrata api and data id is n8mc-b4w4

fao-global-capture-product

update the excel reader and we could add all the https://www.fao.org/fishery/static/Data/

Write out retriever-recipes location on setup

We should try to write out the retriever-recipes location on setup so that retriever can use it. This is assuming most people won't move their retriever-recipes folder. In the README we could probably note that the folder should not be moved if possible, or if moved to re-run setup.py.

Version Number for version.txt

Let's add a version number for version.txt! This should be separate (i.e. starting from 1.0.0 or 0.0.1 or beta or however you want to do it) from the one in retriever, since now the two version.txts will be referencing different sets of files.

Worldclim data needs to be updated

Data has been updated to v2.1, from v2.0

URL for data:
https://worldclim.org/data/worldclim21.html

Guide to adding and testing spatial data packages

Creating a script

Use the retriever autocreate tool to read the data and create a starting script. Update the script as described in the docs

Testing a script
Make sure you have Postgres server up and running
Install Pgadmin
install QGiS

Install the new data into Postgres under the default database postgres. retriever install postgres new-data-pakage
Open QGIS and connect to the Postgres database (make sure you only select one layer at a time not to run out of memory)
Open the raw data folder ~/.retriever/raw_data/ and open the raw image with QGIS to confirm that the raw image matches the image

Example:
Output from QGIS after retriever installation.

No working - troubleshooting

Incase no data is being rendered, close QGIS and open pgadmin to check if the tables were created and they are not empty.
Open the raw data folder ~/.retriever/raw_data/ and open the raw image with QGIS to confirm that the raw data can be rendered

retriever autocreate need some fixes

The auto create misses the path in resources
We need to change the types instead of decimal to double.
In the case of char types we need to add size in quotes. I have a work-in progress branch and I guess it is a good place to start. Anyone is free to jump in.
https://github.com/henrykironde/retriever/pull/127/files

Add Economics data

IMF: several data
https://www.imf.org/en/Data

International Monetary Fund: World Economic Outlook Database from 2000-current
https://www.imf.org/en/Publications/SPROLLS/world-economic-outlook-databases#sort=%40imfdate%20descending

United Nations: National Accounts Main Aggregates Database
https://unstats.un.org/UNSD/snaama/Downloads

Link broken of Data Retriever website in Readme .

Currently the link of the Data Retriever website in the readme page of retriever-recipes.

To fix this issue we need to change the link of retriever org to its official docs as stated in Retriever readme.

Old script remains when the name of a script is changed

With edit_json, where if we edit a script and change its name, it outputs a new script with the new name, but keeps the old script with the old name. The correct functionality should be to replace the old script with the new name script, leaving no trace of the old script.

coronavirus-belgium: Error while installing

INSERT INTO coronavirus_belgium.cases_muni ( record_date, nis5, tx_descr_nl, tx_descr_fr, tx_adm_dstr_descr_nl, tx_adm_dstr_descr_fr, tx_prov_descr_nl, tx_prov_descr_fr, tx_rgn_descr_nl, tx_rgn_descr_fr, cases ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)

value too long for type character varying(5)