commondataio / dataportals-registry Goto Github PK

There are two types of datacatalogs.

First type is a classical data catalog with topics/keywords and locations linked to each single dataset. For example it's global data catalogs like Humanitarian Data Exchange. Datasets in this catalog could be linked to different topics and different countries. There is no one single country or other location for all datasets

Second type is a data catalog that has topics or location linked and these topics and locations are attributes of each dataset too. For example most regional and local government data catalogs do not have datasets for other locations except location except region / local territory. Also there are scientific repositories dedicated only to agriculture or sea exploration and we don't need to identify topic of each dataset, we could copy topics from data catalog registry entry.

This approach require manual or half-automated assignment of topics to data catalogs and adding flags that topics and location values are "transferable".

It could be following attributes:

• transfer_topics: True
• transfer_geotopics: True
• transfer_location: True

This way we could toward to replace "Country" facet search from country covered by the source, to the country of single dataset.

Current situation

Country name and country id are part of country attribute; it's mostly related to the owner of the data catalog but sometimes could be related to coverage of the data catalog. For example, https://gws-access.jasmin.ac.uk/public/odanceo/S2_L2/collection.json STAC server owner located in the UK, but it covers Ghana territory.

Another question is that some countries are big and search engine users could be interested in local datasets. So sub-regional/regional level needs to be added to the data catalog description.

Solution

1. Create owner attribute with name, link, and type sub-attributes and add location attribute
2. location attribute could include: country_id, country_name, subregion_id, subregion_name
3. Create coverage attribute as an array of the location attributes
4. Consider optional attribute address under location attribute
5. Convert automatically current countries attribute to coverage and owner attributes. If countries include more than one record, use first for owner location and all of them for coverage

Example

Result should look like

access_mode:
- public
api: true
api_status: uncertain
catalog_type: Geoportal
content_types:
- dataset
- map_layer
description: Sentinel 2 Surface Reflectance over Ghana using SIAC AC.
endpoints:
- type: stacserverapi
  url: https://gws-access.jasmin.ac.uk/public/odanceo/S2_L2/collection.json
export_standard: STAC specification
id: s2-for-ghana
langs:
- EN
link: https://gws-access.jasmin.ac.uk/public/odanceo/S2_L2/collection.json
name: S2 for Ghana
owner:
- name: "Jasmin. The UK's data analysis facility for environmental science"
- link: "https://jasmin.ac.uk"
- type: Academy
- location:
   - level: 2
   - country:
       - id: UK
         name: United Kingdom
   - subregion
       - id: GB-ENG
       - name: England
coverage:
   - location: 
      - country: 
         - id: GH
         - name: Chana
      - level: 1
software: Stac-server
status: active
tags:
- geospatial
- has_api

Map all references to EU vocabularities https://op.europa.eu/en/web/eu-vocabularies/home

It should affect:

• countries
• languages
• owner_type
• owner_link
• topics

And probably: content_type too

There is no status attribute that could help to identify if data catalog is alive. It should be similar to Fairsharing records status attributes.

Fairsharing uses following four status values:

• Ready
• In development
• Deprecated
• Uncertain

Since Fairsharing includes not only databases but also standards, policies and other content, so it could include "In development" status.

For data catalogs more likely to use only three status types: active, deprecated, uncertain.

Source https://mappingsupport.com

The dataportals.org project (Github repo) has been around for a while and has similar goals of gathering a curated list of open data portals around the world. I have maintained it for some time now, but recently I haven't been able to dedicate much time to improve it. It constantly receives requests for adding new portals. @ivbeg, if you are able and would like to, you could take on managing the project. Or, if you prefer, just specific things like reviewing submissions, improving upon the data schema, the portal frontend, etc.

If nothing else, it could be interesting to at least

• use it as a source of open data portals for this project, and / or
• write a script on the other direction to send the list of portals collected here to feed the dataportals.org database

What do you think?

Consider adding trust level score or trust level to each data catalog.

Core ideas:

• use data about trust seals from re3data
• more trust to academy and government data catalogs
• less trust to community data catalogs
• less trust to aggregators
• less trust to data catalogs without data licenses/rights

About 11 installations from SPC https://sdd.spc.int/mapping-popgis
It's a GIS - Geoportal software with API and data export

Current situation

Right now UK code is used by mistake since the valid ISO 3166 code is GB (https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2)

Solution

1. Replace by script country id for all entries with UK to GB
2. Rename entries folder UK to GB

Source https://cio.go.jp/sites/default/files/uploads/documents/opendata_lg_list.xlsx

File opendata_lg_list.xlsx

For each Regional and Local government, owner type should be set as location subregion data from ISO_3166-2 vocabulary.

It's done for Australia, Germany, Belgium, Brazil, Canada, USA, Russia, Great Britain, Argentina

The top priorities are the biggest countries:

• Indonesia
• China
• India
• France
• Italy
• Japan
• South Korea
• Kazakhstan
• Mexico
• Netherlands
• New Zealand
• Poland
• Portugal
• Sweden
• Thailand
• South Africa
• Estonia
• Chile
• Norway
• Ecuador
• Spain

But all entries should be processed.

Find and add Geonetwork servers

Add "Regional government" type of owner_type and replace all "Government" with "Regional government"

We need to identify licenses used with selected data sources. Sometimes licenses are granular at the dataset level. Sometimes there is just a one single license per data catalog.

Solution

1. Add attribute terms with sub attributes link, license_level, license_id, license_name, text.
2. license_level could be "catalog" or "dataset"
3. All fields are optional

Add ISO 3166-2 subdivisions code. Informally it's set only for US states as subfolders in entities directory. For now ISO 3166-2 code could be assigned manually as attribute with same name iso31662

There are a few hundreds of blank values owner.name and owner.link.
Owner.name displayed with the search entries results, it's important to not to have blank values.

Owner.link is not displayed by default, so it's less critical to have them without empty values, But it's still important too

Redesign website and make it more attractive, add search, add better about. Keep persistent links

Not all Dataverse instances provide open API to work with data repository and some instances require authentication and authorization to use it's API. It's not detected right now and we need to identify if Dataverse API is available and if it's not it shouldn't be registry data catalog entry. Also if authentication is required and we could detect it, probably we need to update data catalog entry schema to set authentication required flag

https://dspacecris.eurocris.org/cris/explore/dris

Add property that website is a national data catalog provded by government like data.gov, data.gov.fr, data.gov.uk and e.t.c

DKAN provides several API types including internal REST API https://demo.getdkan.org/api, CKAN compatible API /api/3/action/package_list and e.t.c. and DCAT export data.json files. We need at least one API / data export type working to harvest this data effectively. API should be detected, verified and added to registry data catalog entry.

List of Galaxy servers https://galaxyproject.org/use/
Example Galaxy server data library https://usegalaxy.eu/libraries

All these data libraries have no geographic afiiliation and theme is Bioinformatics (should be mapped to relevant topics)

Write a guide about how to find data portals installations.

Write about:

• how to find open data portals
• how to use BuiltWith
• how to find Geonetwork, GeoNode, GeoServer and ArcGIS servers
• how to identify data portal software
• how to find API

Add manually or automatically from datasets and catalogues descriptions

Add metadata registries like:

• http://metadataregistry.org/
• https://dese.aristotlecloud.io/home/
• https://www.econdata.co.za/FusionRegistry

Some are similar to data catalogs; others are more like structured databases with metadata elements.
At least some of them could be added to the registry.

Generate report on data completeness

https://v2.sherpa.ac.uk/opendoar/

Update website code, add API, and launch it at registry.commondata.io

• website code updated
• API added
• website launched

Critical features:

• full list of records
• single catalog record webpage
• persistent link to download full dataset
• API to access single datasets

Desired future features, not critical:

• country profiles
• software types profiles
• owner type profiles
• search and filtering

There are websites for all African countries at opendataforafrica.org, but there is no data from the country, only international data sources. It's used by World Bank experts, but it's not really used.

Need to decide about it

I manage a list of Brazilian local government (state and municipal) open data portals:

https://github.com/augusto-herrmann/transparencia-dados-abertos-brasil/blob/main/data/valid/brazilian-transparency-and-open-data-portals.csv

it could be useful as a data source.

Geonode instances could have multiple API endpoints like data.json export, internal REST API, CSW and other. We need to identify which API is alive and to write proper endpoints information to the registry for future use during data harvesting

More info about ARK https://wiki.lyrasis.org/display/ARKs/ARK+Identifiers+FAQ#ARKIdentifiersFAQ-WhatarethepartsofanARK?

Not all NADA instances provide REST API, but it's not verified right now. It should be detected and registry data catalogs entries updated

Include capabilities

Current software description is very short. It's described only as software name and export_formats
For future observability and crawl tasks it could be important to create detailed software profile with data added manually and automatically.

Detailed software profile creation include following tasks:

• Add data catalogs software types dictionary and/or database
• Add fields: version, website, plugins, capabilities and e.t.c.
• Automatically update registry records

Important question to answer, how to update this data regularly. Should it be part of observability tasks or registry too?

OpenAIRE identifiers of the data sources

There are many types of Geonetwork API, current API discovery project doesn't cover all API types. Need to implement pingin possible API endpoint and to identify endpoint types

https://hubzero.org/sites

1. Get list of the countries from World Bank data portal
2. Map list of the countries to the country code in catalog records geographic coverage
3. Get list of missing countries
4. Search for data catalogs: open data, geoportals, scientific repositories for each missing country
5. Update data catalogs registry

Current situation
There are a lot of metadata about data catalogs collected in Re3Data scientific data catalog.

Interesting data from re3data:

• keywords
• content type
• contact-email
• re3data identifier
• description
• persistent identifiers systems
• software
• versioning
• institutions
• repository type

This data could enrich the existing catalog and be added to the entries.

Example Re3Data entry - https://www.re3data.org/repository/r3d100010078

Possible solutions
There are a few possible strategies:

1. Extract whole re3data catalog and extend existing schema and catalog entries automatically as is under re3data attribute. That means high trust on re3data metadata quality.
2. Manually merge re3data entries with existing catalog entries and to extend existing schema without re3data attribute
3. Add automatically only re3data identifiers and consider data enrichment later using Re3data identifier-based API.

Revalidate all ArcGIS Hub endpoints. Update apidetect.py script and watch for JSON contents.

API responds with HTTP status 200 but 404 statusCode defined in JSON response body. Add additional validation code and fix existing endpoints
{"statusCode":404,"message":"Domain record(s) not found :: A domain record with hostname = 2020census-bucksgis.opendata.arcgis.com does not exist :: 404","error":"Not Found"}

Current identifier as id automatically generated from domain name. Need better identifier similar to Re3Data, ROIR or Fairsharing unique identifiers

Source https://www.base-search.net/about/en/about_sources_date.php?&search_type=Datasets