cga-harvard / hypermap-registry Goto Github PK

@ingenieroariel any good idea about what I could test at this point?

Looks like Worldmap layers do not have the Resource.xml field populated when registered. This is reproducible when registring Worldmap to HH and doing a CSW search which will display Worldmap layers as blank Dublin Core records.

Usage stats from WorldMap layers and remote layers (ratings, number of maps a layer is in, certifications) included in results weightings in HyperSearch.

Therefore we need to index for a layer:

• rating
• number of map the layer is in
• is certified?

Those attributes must be exposed also in the WorldMap API.

In master, running a CSW HTTP POST request hangs/times out. To reproduce:

From HHypermap

python manage.py runserver 0.0.0.0:8000

From another console

import requests
print(requests.post('http://host:8000/search/csw', data='<GetCapabilities xmlns="http://www.opengis.net/cat/csw.2.0.2" service="CSW" version="2.0.2"/>').text)

• view in HHypermap: https://github.com/cga-harvard/HHypermap/blob/master/hypermap/search/views.py#L19-L58
• view in GeoNode: https://github.com/GeoNode/geonode/blob/master/geonode/catalogue/views.py#L39-L66

The timeout is rooted in reading the wsgi.input environment variable (https://github.com/geopython/pycsw/blob/master/pycsw/server.py#L229)

Observations

• sending an empty POST works fine (i.e. print(requests.post('http://host:8000/search/csw', data='').text). The problem arises when there is POST data to read
• the environment is Django's default dev WSGI setup. We are currently on Django 1.8.7
• attempting the same test in GeoNode (similar enough dev setup, endpoint is instead http://host:8000/catalogue/csw) works fine
• all other WSGI envs (Apache/mod_wsgi) work fine
• we have been advised not to rely on the django WSGI server from Django devs (http://django-irc-logs.com/2016/jul/20#2381982)
• perhaps there are slight differences in HHypermap dev setup compared to GeoNode

I realize this is the bare minimum environment and that folks will/should never deploy like this to production, however this is a quick way to stand up an instance which will also help in standing up integration tests in Travis.

This includes releasing the HTML client as a django package, adding it to Hypermap's setup.py and enabling it on the HTML version of the API.

We need to integrate MapProxy in hypermap.
MapProxy should:

1. cascade services to Web Mercator when is not available
2. cascade WMS
3. cascade Esri REST and enable as OGC services (not sure this is possible)

Date/Time mining should be extracted from metadata and stored in model.
@jmwenda already worked on similar think, so you could try to work on this at some point.

installHHM.txt
While searching for a monitoring tool to gain insight in the use of QGC web services I found both GeoHealthCheck (GHC) and HHypermap (HHM).
As i succeeded installing GHC, I keep getting errors on HHM and I wonder if someone can help me out.
Tom Kralidis replied on my mail with the request to create a ticket "for this looks like bug of some sort"

My laptop is a Ubuntu 16.04 LTS with python 2.7 (and 3.5) and following the description on the Github page I am getting to the final stage without any error.
But starting the " ./manage.py runserver " command Django crashes with an error (in the file attached)
Who can point me in the right direction to solve this? I've been googeling for the last 2 days to find a solution.

I attached the install script i made from the readme.md (which works great until the last command ;-( ) as well as the error generated by Django
ErrorHHMinstall.txt

btw: the skip_celery_task switch in the install script did not solve it.....

Thanx in advance, Cor

[2016-08-03 23:00:45,145: WARNING/Worker-2] HTTPSConnectionPool(host='ngamaps.geointapps.org', port=443): Max retries exceeded with url: /arcgis/rest/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd9122e0d50>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Input: [-60,-180 TO 60,180]

Output:
"min_x": -178.22782202180733,
"min_y": 18.910787005414388,
"max_x": -66.95000501284882,
"max_y": 71.38957425511644

should be inside bbox.

Both service and folder of services.

@bglewis : Esri REST includes much more information than WMS/WMTS... services, for example Geometry Type (Points, Lines, Polygons).
Should we track this additional information?

@jmwenda can you help with me with this?
Basically you should test what can be done with the arcrest library, and figure out a way to get a thumbnail for a layer.
After figuring it out, we can discuss how to integrate your work in code base.

Currently celery reads variables from postactivate, while uwsgi from its configuration file.

This is a common problem where some rest endpoints have projections that are not standard. Causing harvesting to stop. i.e
PROJCS["NAD_1983_StatePlane_Michigan_South_FIPS_2113_IntlFeet",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic"],PARAMETER["False_Easting",13123359.58005249],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-84.36666666666666],PARAMETER["Standard_Parallel_1",42.1],PARAMETER["Standard_Parallel_2",43.66666666666666],PARAMETER["Latitude_Of_Origin",41.5],UNIT["Foot",0.3048]]
Problem lies with the ArcRest library
http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/TaxParcel/TaxParcelQuery/MapServer

I tried to import Hypermap endpoint, and celerylog displayed me the following error:

celery_1         |   File "/usr/src/app/hypermap/aggregator/models.py", line 280, in update_layers
celery_1         |     update_layers_wm(self)
celery_1         |   File "/usr/src/app/hypermap/aggregator/models.py", line 1030, in update_layers_wm
celery_1         |     data = json.loads(response.content)
celery_1         |   File "/usr/local/lib/python2.7/json/__init__.py", line 338, in loads
celery_1         |     return _default_decoder.decode(s)
celery_1         |   File "/usr/local/lib/python2.7/json/decoder.py", line 366, in decode
celery_1         |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
celery_1         |   File "/usr/local/lib/python2.7/json/decoder.py", line 384, in raw_decode
celery_1         |     raise ValueError("No JSON object could be decoded")
celery_1         | ValueError: No JSON object could be decoded

When I did a debug on models.py I noticed trying to add hypermap as endpoint, a get resquest is made to 'http://worldmap.harvard.edu/data/search/api?start=0&limit=10', as you can see in the following code:

def update_layers_wm(service):
    """
    Update layers for an WorldMap.
    Sample endpoint: http://worldmap.harvard.edu/
    """
    response = requests.get('http://worldmap.harvard.edu/data/search/api?start=0&limit=10')
    data = json.loads(response.content)
    total = data['total']

I also did a get request to the same link and it showed me the following message:

Internal Server Error

An internal error occurred. Please try again, and if the problem continues, contact your site administrator.

So, it means there is a linking problem or it doesn't exist

Full code: https://github.com/cga-harvard/HHypermap/blob/registry/hypermap/aggregator/models.py

The logged user should be able to upload a text file containing endpoints.
The endpoints will be processed by hypermap.
If a endpoint is recognizable (as WMTS/WMS etc...) and is not already existing, then it is added as a service.
The application should keep track of uploaded endpoints with relative results (SUCCESS, ALREADY EXISTING, FAILURE)

I will check this with @ingenieroariel

• When ingesting, add an area parameter and a centroid parameter (if it does not exist)
• When searching, exclude layers that are bigger than the viewport and cover it completely (partial covering does not have to be excluded). Talk to David about the implementation. Think about the case where a layer is the same size as the bbox
• Count how many layers that cover the viewport were included in the count
• Subtract those layers from the heatmap when returning the heatmap via the API
• Add a new output parameter with the number of excluded layers.

BOP API implementation can be found at:

https://github.com/dsmiley/hhypermap-bop

The idea is to mirror the definition and implementation for time based search, spatial based search, etc. That code does not have a LICENSE file so it's better to check with David and Ben about it before starting the work.

Work can be divided in three steps:

• Implement Python functions
• Write unit tests for the Python functions
• Implement Swagger Interface

Comment by @bglewis:
The problem is with geohash the system chooses an array resolution that has a range of 1 to 32. That leads to very crude heatmaps or too accurate ones which are very slow. Quad tree provides a smaller range 1 to 4. This is what we need to implement. Talk to Smiley and perhaps Ariel about how to implement.

It would be possible to automatically classify layers from metadata parsing.
We could use a geographic names thesaurus.
We need to discuss about this before we proceed.

Add connector for harvesting for CSW for geonode (and maybe any CSW).
Maybe @tomkralidis has good ideas here.

These are WMTS layers with DPLA metadata.

In some cases no thumbnails are generated.
Most likely because of srs issues.

Currently we have the following service types enum:

SERVICE_TYPES = (
('OGC:CSW', 'Catalogue Service for the Web (CSW)'),
('OGC:WMS', 'Web Map Service (WMS)'),
('OGC:WMTS', 'Web Map Tile Service (WMTS)'),
('OSGeo:TMS', 'Tile Map Service (TMS)'),
('ESRI:ArcGIS:MapServer', 'ArcGIS REST MapServer'),
('ESRI:ArcGIS:ImageServer', 'ArcGIS REST ImageServer'),
('Hypermap:WorldMap', 'Harvard WorldMap'),
('Hypermap:WARPER', 'Mapwarper'),
)

In the Hypermap UI and in the Hypersearch client we would like to see the Hypermap:WARPER layers as OGC:WMS.

Therefore we could have another enum just for the UI like this:

UI_SERVICE_TYPES = (
('OGC:CSW', 'Catalogue Service for the Web (CSW)'),
('OGC:WMS', 'Web Map Service (WMS)'),
('OGC:WMTS', 'Web Map Tile Service (WMTS)'),
('OSGeo:TMS', 'Tile Map Service (TMS)'),
('ESRI:ArcGIS:MapServer', 'ArcGIS REST MapServer'),
('ESRI:ArcGIS:ImageServer', 'ArcGIS REST ImageServer'),
('Hypermap:WorldMap', 'Harvard WorldMap'),
)

Note that this requires a schema change or we need to modify the query to Solr.

Hyperma should provide a layer preview page, where the layer could be displayed on a Leaflet/OpenLayers map.

@jmwenda it maybe time to move your Dajngo command to hypermap. Let's discuss about it when you have some time?

Add connector for bringing in HGL layers. Need to discuss with D.S. on best option which may be via OGP

We need it for development.

Right now the ngamaps one is sometime unavailable, the demo.geonode.org is out of space and worldmap.harvard.edu is too big.

Hi, after successfully installing HHypermap on closure of #125 i ran into another problem.
loading my endpoint list (attached) results in a error (104, connection reset by peer) (also attached with .txt for uploading purpose, it is the page save of the website response)
HHM_endpoint_list.txt
error at admin_aggregator_endpointlist_add.html.txt

it could be my fault because on being unknown to how to operate HHypermap?
I am running this from my work laptop (ubuntu 16.04) with internet access (port 80).
tried our url in the list with different endings: geoserver/ows , geoserver/wms, etc....
is this a bug or am i missing something?

Thanks for your effort in advance, Cor

• q_text
• q_date
• q_geo

take for example Kotlin code.

We need a pluggable architecture to retrieve custom metadata for each service

Like http://hypersearch.cga.terranodo.io/maps/new

example:
[-50000000-01-01T00:00:00.0Z TO 2001-01-01T00:00:00.0Z]

This ticket covers the OpenSearch aspect pointing to the right location for the other endpoints.

https://gist.github.com/tomkralidis/cc764728baeaaa3ddd26722f7856252b

The endpoint should be this one, according to what I see in urls.py:

http://localhost:8000/search/csw?service=CSW&version=2.0.2&request=GetCapabilities

But I am getting:

AttributeError at /search/csw
'Csw' object has no attribute 'repository'

Exception Location: /webapps/hypermap/local/lib/python2.7/site-packages/pycsw/ogc/csw/csw2.py in getcapabilities, line 312

We need to define what the best Django package is for what we need and which Django API framework it should be based on.

The requirement is that it relies as little as possible on Django models and that it is lightweight and well integrated with Swagger.

This ticket can be closed once the right package has been added to setup.py as a PR.

Let's post the options as comments here and decide together.

GeoHealtCheck uses raphael.js
@jmwenda this should be an easy task, would you help me? Let's talk about it.

When using the hypersearch client, WMS and ESRI layers are not rendered, both in preview and in map.
We have the following error coming from MapProxy:

In home page we should be able to filter services by keyword (on title and abstract) and type (WMS/WMTS/ESRI...).
In service page we should be able to filter layers by keyword (on title, name, abstract), srs, geometry type (for ESRI)

@ingenieroariel can we add hypermap to wm-django?