metarelate / metocean-mapping Goto Github PK

License: GNU Lesser General Public License v3.0

Python 93.39% Graphviz (DOT) 5.81% CSS 0.78% Shell 0.02%

metocean-mapping's Introduction

This Project is now in a frozen state. It is maintained for posterity. Activity has moved to https://github.com/metarelate/metarelate https://github.com/metarelate/metOcean

metOcean-mapping

metOcean mapping is a knowledge base providing information on translating meteorological and oceanographic metadata.

This project provides software to manage the knowledge base and the knowledge base content. The knowledge is stored as RDF Turtle datasets in StaticData.

To contribute to the project, the static data should be used to populate a local triple store which the management software may access.

Dependencies

Apache Jena - http://jena.apache.org/
Fuseki - http://jena.apache.org/documentation/serving_data/
Python - http://python.org/
Django - https://www.djangoproject.com/

Installation

Apache Jena - http://jena.apache.org/
1. Download the 'apache-jena' release from http://www.apache.org/dist/jena/binaries/
2. Unpack the archive
Fuseki - http://jena.apache.org/documentation/serving_data/
1. Download the 'jena-fuseki' release from http://www.apache.org/dist/jena/binaries/
2. Unpack the archive
Python - http://python.org/
1. install Python = 2.7
Django - https://www.djangoproject.com/
1. install Django >= 1.3
Configure the metarelate metocean software
1. lib/metocean/etc/metocean.config provides paths to libraries and static data
2. see lib/metocean/etc/README.md

Configuration

Local configuration files are required to set up the metOcean environment:

./lib/metocean/etc/site.cfg
see ./lib/metocean/etc/sample.site.cfg
lib/editor/settings_local.py
see lib/editor/sample_settings_local.py

Run

Run the application:
1. ./lib/editor/run.sh

metocean-mapping's People

Contributors

Stargazers

Watchers

Forkers

esc24 marqh wizardstoy pp-mo bblay bjlittle

metocean-mapping's Issues

remove um releases

Following detailed discussion with the STASH community, a path working with STASH concepts, rather than STASH UM releases is preferred.

Implement code changes to implement this

prefixes

review all prefixes to ensure consistent usage and that the prefix definitions resolve to a target.

coherent fixing is needed for this and https://github.com/metarelate/mapping-manager/issues/20 and https://github.com/metarelate/docs/issues/4

dot image

develop the metOcean-mapping editor interface to render mapping instances as dot graphs.

Mirror Iris #835 changes

Mirror the rules changed in SciTools/iris#835

Contacts List (D)

provide a TTL file containing a list of GitHub usernames as Contacts.

Constraints

This is dependent upon GRIB data [#4].
This will require mapping-manager [metarelate/mapping-manager#7] to be updated for the new dataset.

Risks

None

Architecture

New data file generated will need to be included in the loader.

peculiar API features #2

The implementation of the server 'retrieve_mappings' call seems rather odd.
It passes info -- and itself -- to queries.valid_ordered_mappings.
Which then ...

constructs a query string
calls the the server back, via the run_query method
returns the data from that back to the caller

Many of the queries behave similarly, though some also perform extra processing of the raw query result at step (3), to make the final return value .

It feels to me like any+all server ('fuseki') methods which make calls into the queries module should not be there at all (retrieve_mappings, validate, save et al.).
In effect, these methods provide management concepts specific to our database structure and/or usage. So that should really be kept separate from the generic server functions.

So, I'd expect (+ much prefer) to see the server module called by the queries module, but never the other way around.

Meanwhile, the existing 'fuseki' enquiry API methods, that I'm objecting to here, might belong in the existing queries module, or possibly in a separate layer that encodes DB usage rather than encoding concepts (I haven't seen enough to grasp it all).

implement validation rules

provide validation rules as queries with an interface from the editor

New user documentation

Documentation to provide a clear flow for a new user to obtain and use the project:

as an information resource
to contribute to

API classes

Develop Classes for the MetOcean API and update the API to improve clarity of use.

Particularly abstract the API away from the RDF encoding

Suggest Mapping, Concept

Bulk Loader (H)

To provide a bulk-loading mechanism, from a simple template based form or spreadsheet.

Constraints

This is dependent upon [#4] and upon [metarelate/mapping-manager#6, metarelate/mapping-manager#8]

Risks

None

Architecture

The simple template proposed has been a CSV file, with columns in a specified order or, perhaps, with a header row that is parsed first to determine meaning of the columns.

There are several possibilities for data feed into the system:

have a python script that takes the CSV template and produces a TTL file to be placed into the loader directories in this project
add functionality to mapping-manager to read the CSV, parse, create and insert the new triples directly into the triple-store. This is be slower that option (1) probably.

editor on redefine mappings loses propIsComp remove link

Iris integration (O)

Ensure that Iris can integrate with the mapping-manager data, in order to keep an up-to-date um_cf_map.py module.

Constraints

This is related directly to the mapping-manager project as export will be closely tied to data architecture but may be related to this project if common routines are needed for data export.

Risks

Coordination between metOcean-mapping, mapping-manager and Iris might be complex.

Architecture

Possibly a python module in this project or mapping-manager to handle the subtleties of querying a changing dataset but the actual generation script for the um_cf_map.py file should reside within Iris and operate on an 'as-requested' basis.

peculiar API features

I'm a bit suspicious that some of the API may be poorly structured -- at least, the usage of it smells rather odd.
See https://github.com/SciTools/iris-code-generators/pull/1#issuecomment-21407406 ...

using fuseki.retrieve_mapping, we get 'mappings' back from the server that require processing through further methods of the server object. So when we create our Concepts and Mapping objects, these have to embed the server object in addition to the specific data passed back from the retrieve_mappings call.

I noticed another thing when trying to understand this, but it doesn't really relate to the iris-code-generators usage.

see separate issue : #35

Trailing slash on tdb

the config file

https://github.com/metarelate/metOcean-mapping/tree/master/lib/metocean/etc

requires trailing slashes on the path names or returns errors writing to the TDB

this should be an easy Python path fix

Export TDB Changes to TTL from Mapping-Manager (F)

Possible handle session-save TTL files from mapping-manager [metarelate/mapping-manager#10].

Constraints

This is dependent upon [#4].

Risks

None

Architecture

The key point will be to hold the start time of the session and use the date stamp upon the Linkage shards to determine what Linkage (thus what Mapping) shards have changed during the session and to prompt the user if there are any unsaved changes before the session expires or is closed.
The changed shards will be written out to a file named by the MD5 sum of its contents and can be merged/loaded upon start of the next session.

collapse linkages

The Linkage graph brings complexity in a number of places whilst the flexibility it brings is not used, indeed it is actively avoided in many cases.

it is preferable to collapse this into the mapping records

a suggested structure is

http://www.metarelate.net/metocean/mapping/ed3165a689515daae3ecea862691cbad
dc:replaces "None" ;
mr:comment "Imported from external mapping resource" ;
mr:creation "2012-12-05T17:42:59.387591"^^xsd:dateTime ;
mr:creator github:marqh ;
mr:owner "None" ;
mr:reason "new mapping" ;
mr:status "Draft" ;
mr:watcher "None" .
mr:target http://www.metarelate.net/metocean/cf/1aee8fdff68b8ae1dd6e5172252189c7 ;
mr:source moStCon:m01s05i201 ;
mr:source moStCon:m01s05i202 ;

Implement Remote End Points (J)

Identify the steps to Swap out the current usage of CF-Standard-Name, Stash, Field Code etc and replace with remotely accessed sparql-endpoints or datafiles.

Constraints

This is dependent upon [#4] and strongly connected to [metarelate/mapping-manager#6, metarelate/mapping-manager#8].

Risks

Internet access unavailable would limit the detail available.

Architecture

Removal of local files.

owner watcher

restore the owner and watcher add and removing functions to the editor interface

re-implement sparql interface

The interface to the SPARQL server is implemented via a bespoke module:

lib/fusekiQuery

Consider re-implementing this using more stable code, such as:

http://sparql-wrapper.sourceforge.net/

Versioning

code should contain a 'current version' concept, which is..

composed in a standard form
incremented in a standard way during development and release processes
matched in repository release tags
available as a module version property (string)
visible in documentation

Flatten All Graphs into Single Data Pool (B)

Examine loading all data into a single dataset rather than into separate graphs.

Constraints

The is dependent upon GRIB data [#4].
This will also require a change within mapping-manager [metarelate/mapping-manager#6] for the new loading data and scripts.

Risks

With all the triples residing within a single data pool, queries will need to be carefully targeted in order to preserve current performance.

Architecture

None

new STASH stdName translations

a collection of identified translation not yet encoded in the metOcean knowledgebase

New Format for GRIB (A)

Examine the requirements for incorporating GRIB code tables and parameters with respect to the existing data and mapping shards.

Constraints

This will impact both this project and the mapping-manager [metarelate/mapping-manager#5].

Risks

None

Architecture Mods

If the GRIB format needs extensions to the data and mapping shard structure, then the initialisation data will need to be changed.

Release Process (G)

Provide a mechanism to incorporate a release snapshot of the TDB triple-store, representing all those data items regarded as appropriate for the release.

Constraints

This is dependent upon [#4] and mapping-manager [metarelate/mapping-manager#11]

Risks

None

Architecture

Shard datatype already proposed by itmh.

metarelate / metocean-mapping Goto Github PK

metocean-mapping's Introduction

metOcean-mapping

Dependencies

Installation

Configuration

Run

metocean-mapping's People

Contributors

Stargazers

Watchers

Forkers

metocean-mapping's Issues

Recommend Projects

Recommend Topics

Recommend Org