add "add Data" functionality about metasearch HOT 16 CLOSED

@tomkralidis I think the workflow you're proposing is fine.
From my point of view, it wouldn't be too difficult to identify whether data to download are in vector or raster formats. I think it is similar with what is already implemented in the plugin. If in the metadata record, there is a WFS endpoint, it is a vector. Same of raster with WCS. Then the client must handle the URL to download the data:

• endpoint: extracted from the metadata record
• bbox: extent of the Map Canvas
• request: GetFeature or GetCoverage
• version: I don't know how to extract this information, probably from the metadata record?
• typename or coverage: layer name extracted from metadata record
• srs: current projection used in the Map Canvas
  The URL should looks like: http://preview.grid.unep.ch:8080/geoserver/wfs?bbox=-84.95293 ,19.82194 ,-74.13126 ,23.19403 &styles=&request=GetFeature&version=1.0.0&typename=preview:cy_buffers&srs=EPSG:4326

Regarding where to download the data, can we imagine a temporary folder (maybe there is a default one in QGIS)?

from metasearch.

@astroimagine IF the data is served as wms/wfs/wcs service, it is already handled by QGIS/MetaSearch. A wfs indeed retrieves vectordata, and a wcs retrieves a coverage (as geotiff or whatever the server serves).

regarding a temporary folder: Qt has it's standard folders (for different platforms):
http://doc.qt.digia.com/4.6/qdesktopservices.html#StandardLocation-enum

Have a look at what your cws is serving as types of 'links' by doubleclicking on one of the results an look in the Links block.

There are a whole set of datadownload possibilities (of which we already can handle the OGS:xxx types). For most of the possibilities, see: https://github.com/geopython/pycsw/wiki/Geonode-notes#wiki-protocol-enumeration-and-description

For example these links:
WWW:DOWNLOAD-1.0-ftp--download File for download through FTP
WWW:DOWNLOAD-1.0-http--download File for download
FILE:GEO GIS file
FILE:RASTER GIS RASTER file

seem doable, but I haven't seen them yet in our national csw, where we often serve atom.xml as type:

So the crux is to transparantly and without errors handle all the different options

The link in the image above links to:
http://geodata.nationaalgeoregister.nl/brpgewaspercelen/atom/brpgewaspercelen.xml
which we could automatically handle, but we could also style it and show it in the same box (as your browser shows it). Then the user can theirself download/save it and open it in QGIS

from metasearch.

@rduivenvoorde thanks a lot for your comment.
Maybe I was not that much clear in my previous message. I was aware of the QGIS/Metasearch capabilities of adding WMS/WFS/WCS in the list of related services. But from a end-user perspective (e.g., a GIS analyst), I found a bit "frustrating" to discover a metadata record with for example a WFS (e.g, endpoint + layer name) and when I click on the "Add WFS" this adds the endpoint to the WFS service list. Then I need to browse the entire list of layers served by this WFS to try to find the layer I'm interested in.
From my point of view a metadata record describes a layer that can have a WxS endpoint with a layer name. From that it would be great to directly download the data in the Map Canvas (no need to browse the entire list of layers available in a WxS instance). Otherwise an alternative is at least to have the layer selected in the list of layer served by the instance so that we can immediately identify the layer related to the metadata record. Is it possible?

from metasearch.

a clear. Yes, I understand what you want to achieve now.

But I think we face a standardization problem here (for what I've heard, I'm not a csw standard guru): I'm told that there is no strict way defined how to define a layer name for a OWS service. So some users define it as part of the capabiliies url, or define both a capabilities and a wms/wfs url. While others (as in our national one) use the title of the service for the layername to be used.

So we could define some heuristic to determine this, but.... it would be easier if there was some common way to define this. @tomkralidis I've seen some discussion page about harmonisation between different foss csw implemenations, but was not able to find it anymore...

Besides this, it helps if the WMS administrators do not define those huge global service lists anymore. Eg geoserver has the notion of creating 'virtual OWS services': http://docs.geoserver.org/latest/en/user/services/virtual-services.html With mapserver it is even easier to achieve.
But still there is then the 'choose layer' dialog in between, but at least you do not have to scroll to the 45th layer :-)
Another thing to note: there is also not concensus what exactly is the relation between a wms layer and a csw record: some create a record for every layer, while others create a record for a group or set of layers. Though for wfs it is indeed more logical that one service corresponds to one csw record.

from metasearch.

@rduivenvoorde thanks a lot for this very interesting explanation.
You're right we are really facing standardisation problem here.
What we will probably do, in our office, is to develop some best practices documentation in order to ensure that collaborators will prepare data and metadata according some "internal" rules :-)

from metasearch.

cc @bartvde

Agreed, the problem here is the process of adding data from a service. MetaSearch (currently) supports OGC CSW CORE (Dublin Core). There's no pure way to achieve this (though GeoNetwork uses dc:URI, which is not implemented by other CSWs) using DC. Some work is going on at https://github.com/geopython/pycsw/wiki/Geonode-notes#wiki-metadata-links to help this.

The CSW ISO profile allows you to do this, but it means making two queries to the CSW. I've cc'd @bartvde who may have comments based on his extensive adventures in this area :)

from metasearch.

Right ISO AP normally requires 2 requests if I have understood things correctly (and used by INSPIRE).

Though the Dutch metadata profile has a way in ISO to embed this in the metadata record directly, but I've forgotten the details. I'm quite sure @thijsbrentjens can remind us of the details here, he is the real expert (not me ;-) ).

from metasearch.

@astroimagine do you have any metadata records in your CSW that have direct links to data? Like downloading a Shapefile, etc.?

from metasearch.

thanks @bartvde for not putting any pressure on me... I'll try to clarify what is done in the Netherlands. My knowledge of the profiles is getting a bit rusty, but indeed, in the Netherlands we do use some elements (additional to INSPIRE metadata elements) in dataset metadata to make it possible to directly access a specific layer (for WMS) or featuretype (for WFS), without an extra step. In NL we use the gmd:name element in MD_DigitalTransferOptions to refer to a WMS Layer or WFS FeatureType.

You could get these elements by one GetRecords request (assuming this is what MetaSearch uses now) and the outputschema for ISO metadata. For example with this GetRecords request:
http://nationaalgeoregister.nl/geonetwork/srv/dut/inspire?&request=GetRecords&service=CSW&version=2.0.2&resultType=results&constraintLanguage=FILTER&constraint_language_version=1.1.0&typeNames=csw%3ARecord&elementSetName=full&OutputSchema=http%3A//www.isotc211.org/2005/gmd&maxRecords=9

And look for the second record, pointing to a WFS with URL http://agentschapnl.kaartenbalie.nl/cgi-bin/mapserv?map=/srv/maps/agentschapnl/RVO_WarmteAtlas.map&version=1.0.0.

This element gmd:name for this usage is only mandatory for dataset metadata in the Netherlands though and not by definition used in service metadata. Which is a bit confusing I can imagine.

Hope this helps.

from metasearch.

@tomkralidis not yet but one of my colleagues is currently preparing metadata with link to zip files that contain shapefiles. I'll let you know as soon as it is ready.

from metasearch.

one idea that I now want to try to build:

• IF the link url is of type INSPIRE Atom (eg http://geodata.nationaalgeoregister.nl/ahn2/atom/ahn2_gefilterd.xml), we enable a 'Download data' button, AND create a memory layer in QGIS (with one feature for every polygon that is in the (geo)rss. And upon clicking on one of those features, you actually open that link in your browser (most often meaning: downloading it).

from metasearch.

@astroimagine do you have any GeoTIFF and / or Shapefile zip downloads in any metadata on CSWs we can test? I'd like to implement this for 0.4.0 release.

from metasearch.

@tomkralidis
Yes we have some resources :-)
Shapefile in a zip: http://metadata.grid.unep.ch:8080/geonetwork/srv/eng/metadata.show?id=1882&currTab=simple
GeoTiff (served via WCS): http://twapgeoportal.grid.unep.ch:8080/geonetwork/srv/eng/metadata.show?id=1112&currTab=simple

Greg

On 02 Sep 2014, at 17:45, Tom Kralidis [email protected] wrote:

@astroimagine do you have any GeoTIFF and / or Shapefile zip downloads in any metadata on CSWs we can test? I'd like to implement this for 0.4.0 release.

—
Reply to this email directly or view it on GitHub.

Dr. Gregory Giuliani
Scientific collaborator - Spatial Data Infrastructure
http://www.unige.ch/envirospace/People/giuliani.html

University of Geneva
Institute for Environmental Sciences/enviroSPACE
7 route de Drize, CH - 1227 CAROUGE
Phone: (+41 22) 379 07 09 Fax: (+41 22) 379 07 44
Web: http://www.unige.ch/envirospace
Email: [email protected]

UNEP/DEWA/GRID-Geneva
International Environment House
11, chemin des Anémones, CH - 1219 CHATELAINE
Phone: (+41 22) 917 84 17 Fax: (+41 22) 917 80 29
Web: http://www.grid.unep.ch
Email: [email protected]

from metasearch.

Hi Greg,
I am just following the discussion as I would be interested in an "add data - feature" as well. However, the links within both metadata-etities (not the links to the metadata) do not seem to work. In both cases I get "The webpage cannot be found" error when I try to access the data.
For the Geotiff it should be probably:
ftp://datastorage.grid.unep.ch/world/population_distribution_demography/bc9090_gnvgnv029_world_cities_population.zip, no idea for the shape file...
Stefan

from metasearch.

Hi Stefan,

Thanks a lot for your feedback. That’s strange because I’ve tested the ftp link and I’m able to download the zip file.
Apparently, this is not a network issue because I can successfully download the data from home.

Cheers

Greg

On 03 Sep 2014, at 11:18, ninsbl [email protected] wrote:

Hi Greg,
I am just following the discussion as I would be interested in an "add data - feature" as well. However, the links within both metadata-etities (not the links to the metadata) do not seem to work. In both cases I get "The webpage cannot be found" error when I try to access the data.
For the Geotiff it should be probably:
ftp://datastorage.grid.unep.ch/world/population_distribution_demography/bc9090_gnvgnv029_world_cities_population.zip, no idea for the shape file...
Stefan

—
Reply to this email directly or view it on GitHub.

Dr. Gregory Giuliani
Scientific collaborator - Spatial Data Infrastructure
http://www.unige.ch/envirospace/People/giuliani.html

University of Geneva
Institute for Environmental Sciences/enviroSPACE
7 route de Drize, CH - 1227 CAROUGE
Phone: (+41 22) 379 07 09 Fax: (+41 22) 379 07 44
Web: http://www.unige.ch/envirospace
Email: [email protected]

UNEP/DEWA/GRID-Geneva
International Environment House
11, chemin des Anémones, CH - 1219 CHATELAINE
Phone: (+41 22) 917 84 17 Fax: (+41 22) 917 80 29
Web: http://www.grid.unep.ch
Email: [email protected]

from metasearch.

As per #62 moved to http://hub.qgis.org/issues/11733

from metasearch.