r-geoflow / geoflow Goto Github PK

Hi,

Metadata language seems to be ignored during metadata creation with geoflow.

Cheers.

Hi Emmanuel,

Some contacts (owner role for example) are ignored when metadata is published.

Thanks!

wilfried

Useful links:

• https://datacite.org/
• https://dataverse.org/
• http://guides.dataverse.org/en/latest/api/sword.html
• R package:
  • CRAN: https://cran.r-project.org/package=dataverse
  • Github: https://github.com/iqss/dataverse-client-r
  • Github: https://github.com/eblondel/atom4R

Tasks:

• add atom4R as embedded geoflow software
• add atom4R dataverse oriented action
  • manage metadata
  • support for files upload
  • support for publication

The principe is to let users configure registers. Later geoflow may embedd default common registers, eg country codelists, etc. For now no register included in geoflow, but the possibility to configure registers in the configuration with registers.

By default, a single type will be specified as generic.
For Zenodo action, several types will be managed for:

• upload: zenodoUploadType. If missing, by default, the generic type will be used
• publication: zenodoPublicationType, specified if available and if zenodo upload type is set to publication
• image: zenodoImageType, specified if available and if zenodo upload type is set to image

To manage multiple sources, the geoflow data component requires a refactoring. From the table viewpoint, the column "Data" will be refactored:

• source: now the same logic as other columns can be applied. For a given file, we will specify the syntax filename@fileurl or filename@filepath. Example: sample.pdf@http://someurl.org/sample.pdf
• sourceName: this property is removed. The source name is now the filename specified in source
• upload: this remains. Until now it was mandatory to specify it, now if not specified, upload will be set to TRUE by default. upload = TRUE means data will be uploaded to the web-tool considered. If set to false, the data files in source will not be uploaded. This is valid for Zenodo or Geoserver
• type: Given the fact it deals with upload task, it is now renamed to uploadType for more consistency. Until now it was mandatory to specify it, now if not specified, the uploadType will be set to 'other'.
• identifier: this is specific to the production of OGC web layers, this will be renamed layername

Example of a Data element for a Zenodo file upload:

source:sample1.pdf@http://www.africau.edu/images/default/sample.pdf,sample2.pdf@https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf;
uploadType:other
upload:true

Example of a Data element for a Geoserver shapefile upload:

source:shapefile1.zip@D://sandbox-geoflow/shapefile1.zip;
uploadType:shp;
upload:true;
layername:layer1

Hi Emmanuel,
Geoflow cannot publish more than one relation (http type).
Many thanks,

wilfried

is it possible to modify the line of the action in geoflow_action_zen4R_deposit_record.R where the data type is hard coded since the geoflow_entity class already assign a default value "dataset" in the line 8 ?
Something like
zenodo_metadata$setUploadType(entity$type)
This would enable to either send directly one of the values expected by Zenodo API or to let it empty and get "dataset" as a default value.

When a DOI is available as identifier of an entity, the Zen4R action should try to get existing Zenodo record based on this DOI, first with a attempt by Concept DOI (assuming the DOI is a concept DOI), next to get record by simple DOI. This will allow to get/update published records (update metadata only) including existing versions of a record that are uniquely identified with a DOI.

Hi again Emmanuel,

Spatial extent of datasets with native EPSG is not visible in geonetwork (although it's in the xml document).

Cheers,

wilfried

Cleaning should be able on a sublist of deposits, and not only all:

• by query: supported by a Zenodo API ElasticSearch query
• by doi: one or more prereserved DOIs for which we want to clean / delete the depositions
• by community: one or more communities for which we want to clean / delete the depositions

Manage upload of raster (coverage) resources including Geotiff format supported by GeoServer base distribution. other formats will be tackled in different tickets as not directly targeted by use cases now.

• ArcGrid,
• WorldImage,
• ImageMosaic.

See eblondel/geosapi#31.

Additional support required for extensions such as NetCDF.

2 options:

• update_metadata: to update zenodo record metadata properties. Default is TRUE
• update_files: to update zenodo record files. Default is TRUE

When files do not changes, either to update record metadata or to trigger the final record publication, in particular in Zenodo, we may want to skip the file download. This can be done by adding a global option skipFileDownload.

After publication, export all Zenodo supported format as metadata files for the record considered.

new data options (to use in Data entity column):

• uploadZip: TRUE/FALSE; Default is FALSE. If TRUE, data source files will be zipped, and the zip will be uploaded in Zenodo action.
• uploadZipOnly: TRUE/FALSE; Default is FALSE. Ignored if uploadZip is FALSE. If TRUE, only the zip file will be kept and upload in Zenodo action.

The action should:

• interpret entity data component to see if they are attributes and variables declared. When attributes are declared, the list of unique values will be added to the feature attribute description. If no attributes are listed, all attributes of the source features will be set by default (raw feature catalogue description).
• produce an ISO 19110 feature catalogue ISO 19139 xml object
• add FC scopes for geoflow (uploadType) and OpenFairViewer (query strategy among ogc_filters, ogc_dimensions - not yet supported in geoflow -, or ogc_viewparams) required for OFV feature catalogue handling

In addition:

• action geonapi-publish-iso-19139 should detect the produced FC and publish it
• action ows4R-publish-iso-19139 should detect the produced FC and publish it

Excluded from this task:

• handling of vocabularies/codelists to enrich feature attribute listed values with labels/definitions

When entity data features are available, a spatial representation will be added. For now bound to vector spatial representation.

For the time-being ';\n' is used, but not enough generic as ';' or '\n' are likely to be used as content.
A default line separator has to be set, and should be modifiable as option for user preferences.

I noticed that "abstract:..." values are not anymore managed properly. Sometimes the content is cut, sometimes "abstract:" prefix is not removed.
Other issue: even if II use a ";" to close the "abstract:" section, the fist line break ("\n") within the abstract section is interpreted as the end of the section.

Hi Emmanuel,

Relations (at least http links) are ignored when geosapi (WMS/WFS) is used.
Thanks in advance.

Cheers

The spatial coverage may be a simple bbox wkt but it could be also a more complex spatial coverage (eg. simplified polygons, simplified lines) less reductor than a bounding box. At now geoflow reduces the WKT into a bounding box, which results in a loss of meta information. Some logic should be added to keep such more detailed spatial coverage. In geometa action it will result in setting bounding polygons, in addition to the geographic bounding box.

The configuration of SQL layers could be done in two ways:

• as SQL file path / URL put to a datasource. In that way geoflow will try to read the SQL file to pick up the SQL query (then set using geoflow_data$setSql()), or
• directly as sql data property in the Data column

Some code will be experimented to read the SQL query, if a Db driver is set-up in the configuration. Reading the SQL query will enable dynamic metadata retrieval, as it is already done for shapefile (uploadType = 'shp')

The output CSV will contain:

• Identifier: entity identifier
• Status: simple status handling the Zenodo status draft or published
• DOI_for_allversions: generic / top level DOI (named "conceptdoi" by Zenodo).
• DOI_for_version: record version-specific DOI

Depending on readr availability in R session, gsheet::gsheet2tbl will not return the same output. Indeed as stated in the documentation: If the package readr is available, then it will be used. This can produce slightly different, but normally better, parsings.

To make sure that geoflow entity/contact loading works properly in the same way for all. readr is going to be added.

Use of metadata identifier in data identification/ citation, according to https://geo-ide.noaa.gov/wiki/index.php?title=DOI_Minting_Procedure#Third.2C_Include_the_DOI_and_citation_text_in_the_ISO_Metadata_Record

At that time it is enabled only on the geometa action. INSPIRE is tested on XML files produced locally (and eventually uploaded with actions like Zenodo), but it is not tested on the XML pushed on geonapi/ows4R. These 2 packages may need enhancements to take into account INSPIRE geometa option.

• ows4R action - eblondel/ows4R#28
• geonapi action - eblondel/geonapi#7

In some software, such as for DB connectors (#30), we need to be able to specify a handler function for a given argument.

For example, in the case of DBI, the main handler is DBI::dbConnect, but the first argument drv requires an instance of driver, that can be instantiated using DBI::dbDriver(<drivername>). The software argument definition could be extended with such handler, here in that case DBI::dbDriver.

The software function getHandlerInstance should then be able to evaluate such argument handler.

Defaut DB connectors should be made available as embedded software clients/connectors. For now this targets DBI. To start, imports for RPostgreSQL and RSQLite will be added. Others could be added if needed.