iodepo / odis-arch Goto Github PK

The goal of this issue is to review the technology, standards, and readiness level of the Pacific data portals. The following SPREP and SPC portals have been identified as potential data sources for the OceanInfoHub in the PSIDS region:

• Pacific Environmental Portal
  • technology: Drupal, CKAN (seems to harvest from national CKAN data portals)
  • data types: datasets (including maritime boundaries), publications
  • structured data: none detected using Structured Data Testing Tool, RDF export links often do not work
• Pacific Data Hub
  • technology: Drupal, DKAN, DCAT-RDF
  • content types: datasets, publications
  • structured data: none detected using Structured Data Testing Tool
• Pacific Ocean Portal
  • technology: Python, WMS
  • data types: spatial datasets
  • structured data: none
• Pacific Catastrophe Risk Assessment and Financing Initiative (PCRAFI)
  • data types: spatial datasets, documents
  • technology: GeoNode, CSW, WMS, WFS, WMTS, OAI-PMH, OpenSearch
  • structured data: none detected using Structured Data Testing Tool

Some resources:

• internetofwater/geoconnex.us#16 (comment)
• https://github.com/ESIPFed/science-on-schema.org/
• https://www.ogc.org/blog/2215
• https://schema.org/geo
• Springer Handbook of Geographic Information, see chapter 15. Geosemantic Interoperability and the Geospatial Semantic Web

@fils in our Latin America regional meeting, we discussed the importance of systematically recording and gathering memoranda of understanding.

These will be essential in transparency, and to help users understand which data flows / sharing structures are endorsed and trusted by partners.

When the document pattern is more stable, I think this should be a special case / template of this pattern, that records the IRI/DOI to the MoU, and unique identifiers for the partners involved, as well as metadata like a lifetime for the MoU.

I have started a document to collect information on metadata standards relevant to the architecture, feel free to contribute or suggest changes to the structure.

• partners are very excited to see their data in the OIH search portal, but the final JSON-LD from their endpoint lacks the partner information, to perform that search
• there is a publisher key for schema.org
• however, this will likely have to be added through the ODIS-Arch graph/processes, for each partner's endpoint
• if 'publisher' is problematic (a partner could be cascading a layer that is served initially through another publisher/endpoint), maybe the parameter could simply be named oih-partner

The Map type is just a subclass of Documents. So anything in Documents can move across to Maps.

This is not clear in the documentation and makes the Maps docs look oddly sparse.

Connect these two better.

Issues #3 and #4 here present two different takes on an information model to describe hubs, developed in the early stage of our work. They represent a bare minimum of thinking about what a hub provides to the notional ODIS network and some dynamics of its technology. We need to further flesh this idea as we move along to eventually create a database of some kind with this information that will become an important organizing part of our eventual system. Ideally, hubs will eventually advertise themselves on the web with information that can align with this model. In the near term, we can create an abstract registry of information to help us learn from what hubs are doing and work at nudging them toward the developing architecture.

Here are some of the properties/characteristics of hubs that might be important. Please comment on this issue with adjustments to these and other important properties. Keep in mind that we have to fill all this information out somewhere in the near term and maintain it for some time period and that hubs will ultimately need to push this information out on the web in some way, so we probably don't want too complicated an information model.

• Hub type: intent is to help us understand how the network will interface with a given entity; we'd want some type of vocabulary for this; one idea includes classifying hubs as primary or secondary sources of information
• Primary topics/artifacts: these are the primary concepts/artifacts that a given hub is responsible for based on its mission; we initially started building on the types of things cataloged in the ODIS-Cat prototype; we need to consider that different hubs will classify the same logical things with different terms; ODIS will need to consider a complex mapping of topics; a characteristic of these is that they will have a unique identifier (at least within the context of the given hub)
• Secondary topics/info sources: many hubs will likely have secondary information that is not the primary purpose of the hub but that is important on the network, particularly as it may help create linkages between resource pools; these items may have a unique identifier within the context of a hub but likely could/should refer to an identifier provided by some other source that is more directly responsible
• Semantic contributions: some hubs are going to provide new explicit ways of describing their primary artifacts with defined terms/concepts that add value to the overall network
  Scope: various ways of describing the overall scope of a hub; could be spatial or temporal, topical, organizational, etc.
• Machine access: what kinds of possibilities are there for interfacing with the hub's information assets in terms of APIs, web services, structured data with landing pages, etc.
• Technology family: certain types of tech platforms offer specific types of interfaces or expected capabilities that can be used to help determine how it interfaces on the network

I have started a document to collect information on data sources relevant to the architecture, feel free to contribute or suggest changes to the structure. I suppose ODISCat is supposed to fulfill that role in the long term, but this document could focus on sources that we want to include in the initial proof-of-concept implementation.

EurOcean: Research Infrastructures Database (rid.eurocean.org) is managed by EurOcean and offers a comprehensive list of all existing facilities in Europe which are dedicated to marine sciences broad range of activities. It provides the first level of knowledge and characteristics for each facility, as well as the links and contact to access the further details provided by the operator as it is the case for the Research Vessels: a list of descriptors are here attached.

List_Table_Fields_Description_SS.xlsx

I suggest to use a Jekyll theme that's a bit more suitable for documentation, hosted on GitHub Pages or somewhere else. For example:

• https://pmarsceill.github.io/just-the-docs/
• https://dox.themejack.com/

• need to add grants amounts
• existing "FundingAgency" does exist as a schema.org pattern (https://schema.org/FundingAgency) and is available through ODIS
• must also allow to specify currency type

Leverage author https://schema.org/author on Creativework

Can be Org or Person

Reference: https://developers.google.com/search/docs/advanced/structured-data/article

If you are using a controlled list for things like keywords or as additional properties of some sort it would be good if we can get a reference to them.

This would let us ensure that the OIH pattern for these is able to address them. We can keep track of those with and without machine readable versions..

For example the ISCED at http://uis.unesco.org/en/topic/international-standard-classification-education-isced

The Marine Knowledge Gate is a resource pool that we may want to consider as a hub in addition to the Research Infrastructures Database from EurOcean. This list of characteristics builds on what Pieter started in #3.

• Hub type: metadata aggregator/search portal
• Primary topics/artifacts: projects, publications, tools/software, training activity, best practices (“guidelines/standards”, “RTD protocol/technical manual”), data
• Secondary topics/info sources: organizations/institutions, “programmes”, people, countries
• Semantic contributions: benefit sectors, end user groups
• Scope: “ EurOcean provides comprehensive databases of information on topics related to marine science and technology in Europe with priority given to three main domains: Marine Knowledge Management, Marine Research Infrastructures and Ocean Public Outreach and Awareness activities.”
• Machine access: none readily apparent

• scan who is hosting the data catalogues
• report what data types are missing
• possibly flag missing datatypes on the Dashboard, or point to a diagnostic report that is generated on-demand

The Maps documentation is overly focused on Map as "document" or digital object.

Take the Maspiwo example of the bounding box and use it as an example here and better demonstrate the use of spatial with Map.

ODIS record page should include also machine readable information. The recommended format is JSON-LD based on schema.org scheme (for datasets) and additional Google fields. This information helps finding these pages using search engines, and allows also the presentation of additional information in the search results page.

The latest builds of the book have failed, this seems to be caused by a change in the TOC format, see https://executablebooks.org/en/latest/updates/2021-06-18-update-toc.html.

Running Jupyter-Book v0.11.1
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/jupyter_book/cli/main.py", line 243, in build
    parse_toc_yaml(toc)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/sphinx_external_toc/parsing.py", line 82, in parse_toc_yaml
    return parse_toc_data(data)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/sphinx_external_toc/parsing.py", line 88, in parse_toc_data
    raise MalformedError(f"toc is not a mapping: {type(data)}")
sphinx_external_toc.parsing.MalformedError: toc is not a mapping: <class 'list'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.10/x64/bin/jupyter-book", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/jupyter_book/cli/main.py", line 246, in build
    f"The Table of Contents file is malformed: {exc}\n"
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/jupyter_book/utils.py", line 48, in _error
    raise kind(box)
RuntimeError: 
===============================================================================

The Table of Contents file is malformed: toc is not a mapping: <class 'list'>
You may need to migrate from the old format, using:

	jupyter-book toc migrate /home/runner/work/odis-arch/odis-arch/book/_toc.yml -o /home/runner/work/odis-arch/odis-arch/book/_toc.yml

===============================================================================

the expert information is lacking the notion of nationality which will be needed if regional hubs want to extract only experts in their region
I propose to add https://schema.org/nationality to https://schema.org/Person

• background: I'd like to be added into the GitHub 'team' for ODIS-arch
• I believe we should setup a "Developers" team in the GitHub settings at https://github.com/iodepo/odis-arch/settings/access
  • see "Manage Access" section there, and click "Create team"
    • likely select "Write" as the role for this team, as it allows "Can read, clone, and push to this repository. Can also manage issues and pull requests"

(the admin of iodepo/odis-arch will likely have to set this up, as I don't have permissions currently, I manually wrote the url above from a different project)

• current process to generate JSON-LD: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/maspawio-harvest.py
• assigned to: @jmckenna
• CC: @jamesmmbugua & @AbigailMwalimu

the content for https://iodepo.github.io/odis-arch/ should be filled with real text, now there is only some placeholder 'lorem ipsum' text

The list of topics for the spatial data and maps is an important list of keywords.

Each one would likely need its own sub-spec (eventually) as part of a variableMeasured or "about" field.

Some of those also connect to EOVs and some, like the tracks of shipping, can cross over to the vessels pattern.

This is related to the https://book.oceaninfohub.org/appendix/list.html

So we have leveraged shema.org/knowsAbout in the Person thematic to indicate a person knows about a Thing.

A few comments.

• The range of this is Text, URL and Thing. So Thing let us connect ... well any Thing in schema.org So we can do Course, Dataset, Project, etc. quiet well.
• The domain is just Person and Organization but not sure we would use this anywhere else.

However, lets say as a group we wanted to subset the definition of this property (not saying we do though). That definition being:

Of a [[Person]], and less typically of an [[Organization]], to indicate a topic that is known about - suggesting possible expertise but not implying it. We do not distinguish skill levels here, or relate this to educational content, events, objectives or [[JobPosting]] descriptions

For argument let's say we want to imply knowAbout does indicate they have a recognized skill based on taking a course. Since knowsAbout is a Property (ie schema.org/Property we could add a new description to it.

There are a lot of semantic issues with this of course. (well.. maybe just 1 or 2 big ones). There is though... https://schema.org/disambiguatingDescription but this isn't really what it's meant for.

The real question I guess is...

How does a group indicate their interpretation of a schema.org property that they feel is simply their interpretation of schema.org's definition. Note, this is likely the power of schema.org.. to enable broad interpretation... this is simply trying to pass along additional info. Since there is no cardinality to anything in schema.org (correct?) can we simply just add in another description along with the schema.org one?

For reference:

    {
      "@id": "schema:knowsAbout",
      "@type": "rdf:Property",
      "rdfs:comment": "Of a [[Person]], and less typically of an [[Organization]], to indicate a topic that is known about - suggesting possible expertise but not implying it. We do not distinguish skill levels here, or relate this to educational content, events, objectives or [[JobPosting]] descriptions.",
      "rdfs:label": "knowsAbout",
      "schema:domainIncludes": [
        {
          "@id": "schema:Person"
        },
        {
          "@id": "schema:Organization"
        }
      ],
      "schema:isPartOf": {
        "@id": "https://pending.schema.org"
      },
      "schema:rangeIncludes": [
        {
          "@id": "schema:Text"
        },
        {
          "@id": "schema:URL"
        },
        {
          "@id": "schema:Thing"
        }
      ],
      "schema:source": [
        {
          "@id": "https://github.com/schemaorg/schemaorg/issues/1688"
        },
        {
          "@id": "https://www.w3.org/wiki/WebSchemas/SchemaDotOrgSources#TP"
        }
      ]
    },

• must verify that all patterns/entries contain author (provider) value
• strongly recommend that ODIScat ID be used
  • very important to highlight in the ODIS-Arch book
• could also be reported on the Dashboard possibly through a query

we should add https://schema.org/member -> person to the organizations

• with upcoming (now) OGCAPI effort by the major players geospatial industry, cloud-native infrastructure and standards will be leveraged by many ODIS partners
• STAC is a critical part of the OGCAPI (for querying features etc)
• STAC catalogues (JSON, lightweight and queryable) will most likely be one of the most-used methods by partners in the upcoming future/now
• ODIS must provide a path for partners that run with STAC, such as:
  • enhance the ODIS graph to support existing STAC catalogues
  • add a section in the ODIS book for partners to follow who have STAC catalogues

Need to do some repository re-organization

• Some notebooks do NOT need to be in the book section but rather in the repo directly (with reference from the book)
• Leverage the "errata" pattern in the book to still include some exploration/testing in the book to show some of work that didn't work.
• general updates to the book to clean it up and make them a bit leaner

Review the book from the POV and partner on-boarding with Jeff.

• "platform" key is lacking in schema.org
  • oil rig is example of platform
• possibly use instrument

in OceanExpert we do not have a real provider for a training but rather organizers, how do we fit this in?

we should add https://schema.org/url to the organizationsd

I have started a document to collect information on vocabularies relevant to the architecture, feel free to contribute or suggest changes to the structure.

the members (both organization and persons) should have some unique id attached so they can be matched with other elements

I still have a few remaining issues around escaped characters for the Ocean Experts Learning resources.

There are really on 4 issues. Also, The error log at the end has a double entry for each document, so it's longer than it should be.

I think all the errors are either ' \t an escaped " " or & issues. So resolving these 4 points should remove all remaining encoding issues.

So for example the ' character is fine, but the escape of it ' is not in the resource https://www.oceanexpert.org/event/1000

The interesting one is the escapes space? Take a look at https://www.oceanexpert.org/event/2019 and run it's JSON-LD through the JSON-LD playground at https://json-ld.org/playground/

I think these are the last of them! Appreciate the help of ferreting these out!

logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1000 Action: json.Unmarshal call  Error: invalid character '\'' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1000 Action: Getting normalized sha  Error: invalid character '\'' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1054 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1054 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1082 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1082 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1293 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1293 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/583 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/583 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/584 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/584 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/2004 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/2004 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/2006 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/2006 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1083 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1083 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/768 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/768 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/2018 Action: json.Unmarshal call  Error: invalid character ' ' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/2018 Action: Getting normalized sha  Error: invalid character ' ' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1311 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1311 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1085 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1085 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1086 Action: json.Unmarshal call  Error: invalid character '\'' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1086 Action: Getting normalized sha  Error: invalid character '\'' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1379 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1379 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1389 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1389 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/429 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/429 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1090 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1090 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/2019 Action: json.Unmarshal call  Error: invalid character ' ' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/2019 Action: Getting normalized sha  Error: invalid character ' ' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/2126 Action: json.Unmarshal call  Error: invalid character '&' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/2126 Action: Getting normalized sha  Error: invalid character '&' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1708 Action: json.Unmarshal call  Error: invalid character '&' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1708 Action: Getting normalized sha  Error: invalid character '&' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1191 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1191 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/224 Action: json.Unmarshal call  Error: invalid character '&' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/224 Action: Getting normalized sha  Error: invalid character '&' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/225 Action: json.Unmarshal call  Error: invalid character '&' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/225 Action: Getting normalized sha  Error: invalid character '&' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1133 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1133 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1410 Action: json.Unmarshal call  Error: invalid character '\'' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1410 Action: Getting normalized sha  Error: invalid character '\'' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/2595 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/2595 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1129 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1129 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1625 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1625 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1648 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1648 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1741 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1741 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/2403 Action: json.Unmarshal call  Error: invalid character '&' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/2403 Action: Getting normalized sha  Error: invalid character '&' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/1944 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/1944 Action: Getting normalized sha  Error: invalid character '\t' in string literal
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/306 Action: json.Unmarshal call  Error: invalid character '&' in string escape code
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/306 Action: Getting normalized sha  Error: invalid character '&' in string escape code
logger: acquire.go:151: ERROR: URL: https://www.oceanexpert.org/event/377 Action: json.Unmarshal call  Error: invalid character '\t' in string literal
logger: acquire.go:172: ERROR: URL: https://www.oceanexpert.org/event/377 Action: Getting normalized sha  Error: invalid character '\t' in string literal

the notion of the knownLanguage (https://schema.org/knowsLanguage) should be added as this can be important for some regions

• Globefinity/Chris/@dilianadineva to develop new vessels database & search tool
• need guidance for generating required JSON-LD and sitemap
• cc @fils @pbuttigieg

As we are building the book using Actions we probably want to leave out the _build directory?

• example dashboard: https://dashboard.obofoundry.org/dashboard/index.html (ODIS partners could be the rows, columns could be the monitored items)
• goal would be to generate a simple interface to display a live view of ODIS statistics, to track the status of OIH partners
• tracked items could be:
  • number of nodes active in OIH
  • percentage of possible data patterns implemented by partners
  • types of thematic patterns available (projects, training, vessels, etc) by partner
  • number of active nodes by region
• This same dashboard could also show usage activity in terms of:
  • number of indexed records (or catalogues) by ODIS, for a partner or geographical region
  • last index time of partner's services, by ODIS
  • common search (query) terms submitted (to the ODIS graph)
• Another possibility is to use this same dashboard for:
  • monitoring of partner's endpoint services (is their catalogue service available, show a big green checkmark, if not available show a big yellow question mark, and on mouse-over show when service last available)
  • view of usual web server statistics for specific ODIS urls, such as number of visits, geographic region, etc.
• Notifications could be enabled for this dashboard
  • daily/weekly/monthly report could be auto-generated and sent to team
  • notices could be sent when services are down/not available (on the partner side and the ODIS-side, as both will have problems at some point)
• Important discussion topic could be what information is private, and what tracked info should be kept as a backend admin view of this dashboard.

(all of above are just ideas, recorded here initially as a way to throw-ideas-on-the-whiteboard, and each part is fine to remove or add) Goal here is to start the discussion/planning through this ticket.

Screen capture from example site mentioned above:

we should have https://schema.org/memberOf attached to the organizations

Multilingual support in Jupyter Book is not great, but there are approaches for it.

I am reviewing executablebooks/jupyter-book#223 to see what the best approach might be,

I'm making this issue to stake this goal.

• new high-level JSON-LD key
  • is the data cross archived (GOOS, IODE)
  • is the data private?
• possibly compare SHA hashes to compare data between different repositories

• partners could enable the Action in their own repository
• Action could:
  • verify RDF/JSON-LD
  • possibly generate (or "harvest") necessary JSON-LD & sitemap for known endpoint types (GeoNode, CKAN, GeoNetwork, etc.)
  • notify ODIS dashboard / ODIS graph of status
  • output steps to submit sitemap to Google Search Console
  • ...

(similar to BeBOP-OBON/0_protocol_collection_template#5 )

Background

• many NODC/partners have catalogue services running that do not automatically generate/publish JSON-LD for each metadata record in their catalogue
• workarounds and extensions may exist, but not for all types of catalogue software
  • some partners were also providing CSV files, made by hand, that lists some of the metadata layers for their service
• since we know the (CSW) service endpoint for all partners, we can harvest the endpoint and develop a standalone process that partners can run locally (possibly nightly) to generate the JSON-LD for each record, and an RDF file containing all records

Updates:

• standalone process exists now, that:
  • hits the catalogue endpoint live
  • parses all metadata records
  • pays particular attention to keywords (as the ISO standard defines 'theme', 'place', and 'discipline' keywords)
  • initial partners processed:
    • MASPAWIO (GeoNode)
      • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/maspawio-harvest.py
      • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-maspawio
    • AMA [old African Marine Atlas Catalogue] (GeoNetwork)
      • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/ama-harvest.py
      • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-ama
    • BCC [Benguela Current Convention] (GeoNode)
      • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/benguelacc-harvest.py
      • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-benguelacc
    • Caribbean Marine Atlas (GeoNode)
      • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/caribbeanmarineatlas-harvest.py
      • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-caribbeanmarineatlas

Statistics

MASPAWIO

Parsed 149 records
    134 exported to JSON-LD
    15 were invalid records

AMA

Parsed 596 records
    596 exported to JSON-LD
    0 were invalid records

BCC

Parsed 180 records
    180 exported to JSON-LD
    0 were invalid records

Caribbean Marine Atlas

Parsed 358 records
    358 exported to JSON-LD
    0 were invalid records

Validation

• I used the schema.org validator for the JSON-LD files

Discussion

• could be useful for some partners that cannot easily expose JSON-LD for all their layers
• data folder contains separate JSON-LD files, by the actual metadata ID (so partners can examine the related record)
• parsing the keywords need discussion (the 'place' keyword could be very important for a region search, for example)
• process leverages OWSLib (the same Python library that all of these catalogue software use deep underneath)
• I was running these in Jupyter Notebook as-is, but could generate a notebook file for each partner when needed

Lessons Learned

• DublinCore response from these catalogue endpoints failed (at the catalogue server, not local script here) for metadata records with special characters in the title/name of the record (even with names with exponents: 10²)
  • therefore these harvest scripts request the ISO metadata profile record

I would add a link to the Youtube video you made for the training, found this very well done and clear.

• Python3 script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/maspawio-harvest.py
• run locally and provide feedback to OIH team here
• assigned to: @jamesmmbugua & @AbigailMwalimu

don't we need a place for the authors/contributors in https://github.com/iodepo/odis-arch/tree/master/schema/thematics/docs

• knowing which region(s) is associated with each record, is critical for any search/querying of the ODIS graph, as well as producing any summary for each region
• a top-level field in the JSON-LD for region is therefore required
• notes / challenges / thoughts:
  • multiple regions can be associated with each entry
  • global repositories may have local regional data, yet lack any 'region' definition
  • this requirement should also be added into the documentation/book, so all partners tackle this themselves early on
    • especially as adding a region value through the ODIS-Arch graph ourselves (for existing entries) could be problematic
  • the Dashboard could give a status on whether a node is defining a "region" or not

Requested edits to current documents

Old title: Ocean Info Hub
New Title: The Ocean InfoHub Project and the development of the ODIS-architecture

Old Introduction

Organizations are increasingly exposing data and resources on the Web. A popular approach to this is using web architecture to expose structured data on the web using the schema.org vocabulary. Doing this makes resources discoverable by a range of organizations leveraging this architecture to build indexes. These include major commercial indexes, large domain focused groups and community focused services.

To add to that Introduction:

The Ocean Data and Information System (ODIS) will provide a schema.org based interoperability layer and supporting technology to allow existing and emerging ocean data and information systems, from any stakeholder, to interoperate with one another. This will enable and accelerate more effective development and dissemination of digital technology and sharing of ocean data, information, and knowledge. As such, ODIS will not be a new portal or centralised system, but will provide a collaborative solution to interlink distributed systems for common goals. Together with global project partners and partners in the three regions, a process of co-design will enable a number of global and regional nodes to test the proof of concept for the ODIS.

The ODIS-architecture development is being supported by the Ocean InfoHub Project, and it has been tested initially on IOC and partner databases. However, the system and standards are open for any institution or initiative that is interested in accessing the global data ecosystem to adopt and implement.

Old next heading: Ocean Info Hub

Replaced with heading: Guidance for the implementation of the ODIS-architecture

Text edit under that heading: just delete the first sentence: "Ocean Info Hub (OIH) is an implementation of this approach for the UN ODIS community."

Old next heading: Further details

Replaced with heading: Key links to the OIH GitHub repository

https://www.w3.org/2015/spatial/wiki/ISO_19115_-_DCAT_-_Schema.org_mapping

• should define the policy (likely in the book) of how ODIS does not limit/prevent entries or queries for non-ocean related datasets
  • the decision comes down to what the partners want to share
• thoughts:
  • should this be a new pattern keyword flag in the metadata?