Git Product home page Git Product logo

ahurka / woudc-data-registry Goto Github PK

View Code? Open in Web Editor NEW

This project forked from woudc/woudc-data-registry

0.0 1.0 0.0 692 KB

WOUDC Data Registry is a platform that manages Ozone and Ultraviolet Radiation data in support of the World Ozone and Ultraviolet Radiation Data Centre (WOUDC), one of six World Data Centres as part of the Global Atmosphere Watch programme of the WMO.

Home Page: https://woudc.org

License: Other

Makefile 2.78% Python 97.22%

woudc-data-registry's Introduction

WOUDC Data Registry

Build Status Coverage Status

Overview

WOUDC Data Registry is a platform that manages ozone and ultraviolet radiation data in support of the World Ozone and Ultraviolet Radiation Data Centre (WOUDC), one of six World Data Centres as part of the Global Atmosphere Watch programme of the WMO.

Installation

Requirements

Dependencies

Dependencies are listed in requirements.txt. Dependencies are automatically installed during installation.

Installing woudc-data-registry

# setup virtualenv
python3 -m venv --system-site-packages woudc-data-registry
cd woudc-data-registry
source bin/activate

# clone codebase and install
git clone https://github.com/woudc/woudc-data-registry.git
cd woudc-data-registry
python setup.py build
python setup.py install
# for PostgreSQL backends
pip install -r requirements-pg.txt


# set system environment variables
cp default.env foo.env
vi foo.env  # edit database connection parameters, etc.
. foo.env

# create database
make ENV=foo.env createdb

# drop database
make ENV=foo.env dropdb

# initialize model (database tables)
woudc-data-registry admin setup

# initialize search engine
woudc-data-registry admin search create-index

# load core metadata

# fetch WMO country list
mkdir data
curl -o data/wmo-countries.json https://www.wmo.int/cpdb/data/membersandterritories.json
woudc-data-registry admin init -d data/

# cleanups

# re-initialize model (database tables)
woudc-data-registry admin teardown
woudc-data-registry admin setup

# re-initialize search engine
woudc-data-registry admin search delete-index
woudc-data-registry admin search create-index

# drop database
make ENV=foo.env dropdb

Running woudc-data-registry

TIP: autocompletion can be made available in some shells via:

eval "$(_WOUDC_DATA_REGISTRY_COMPLETE=source woudc-data-registry)"

Core Metadata Management

# list all projects
woudc-data-registry project list

# list all datasets
woudc-data-registry dataset list

# list all contributors
woudc-data-registry contributor list

# show a single contributor details
woudc-data-registry contributor show MSC

# add a contributor
woudc-data-registry contributor add -id foo -n "Contributor name" -c Canada -w IV -u https://example.org -e [email protected] -f foouser -g -75,45

# update a contributor
woudc-data-registry contributor update -id foo -n "New Contributor name"

# delete a contributor
woudc-data-registry contributor delete foo

Data Processing

# ingest directory of files (walks directory recursively)
woudc-data-registry data ingest -d /path/to/dir

# ingest single file
woudc-data-registry data ingest -f foo.dat -b

# ingest without asking for permissions
woudc-data-registry data ingest -d /path/to/dir -b

# verify directory of files (walks directory recursively)
woudc-data-registry data verify -d /path/to/dir

# verify single file
woudc-data-registry data verify -f foo.dat

Running Tests

# install dev requirements
pip install -r requirements-dev.txt

# run tests like this:
cd woudc_data_registry/tests
python test_data_registry.py

# or this:
python setup.py test

# measure code coverage
coverage run --source=woudc_data_registry -m unittest woudc_data_registry.tests.test_data_registry
coverage report -m

Code Conventions

Bugs and Issues

All bugs, enhancements and issues are managed on GitHub.

Contact

woudc-data-registry's People

Contributors

tomkralidis avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.