cga-harvard / hypermap-registry Goto Github PK
View Code? Open in Web Editor NEWHypermap Registry, remote map services made easy for your SDI
Home Page: https://cga-harvard.github.io/Hypermap-Registry/
License: MIT License
Hypermap Registry, remote map services made easy for your SDI
Home Page: https://cga-harvard.github.io/Hypermap-Registry/
License: MIT License
@ingenieroariel any good idea about what I could test at this point?
Looks like Worldmap layers do not have the Resource.xml
field populated when registered. This is reproducible when registring Worldmap to HH and doing a CSW search which will display Worldmap layers as blank Dublin Core records.
Usage stats from WorldMap layers and remote layers (ratings, number of maps a layer is in, certifications) included in results weightings in HyperSearch.
Therefore we need to index for a layer:
Those attributes must be exposed also in the WorldMap API.
In master, running a CSW HTTP POST request hangs/times out. To reproduce:
python manage.py runserver 0.0.0.0:8000
import requests
print(requests.post('http://host:8000/search/csw', data='<GetCapabilities xmlns="http://www.opengis.net/cat/csw.2.0.2" service="CSW" version="2.0.2"/>').text)
The timeout is rooted in reading the wsgi.input
environment variable (https://github.com/geopython/pycsw/blob/master/pycsw/server.py#L229)
print(requests.post('http://host:8000/search/csw', data='').text)
. The problem arises when there is POST data to readI realize this is the bare minimum environment and that folks will/should never deploy like this to production, however this is a quick way to stand up an instance which will also help in standing up integration tests in Travis.
This includes releasing the HTML client as a django package, adding it to Hypermap's setup.py and enabling it on the HTML version of the API.
We need to integrate MapProxy in hypermap.
MapProxy should:
Date/Time mining should be extracted from metadata and stored in model.
@jmwenda already worked on similar think, so you could try to work on this at some point.
installHHM.txt
While searching for a monitoring tool to gain insight in the use of QGC web services I found both GeoHealthCheck (GHC) and HHypermap (HHM).
As i succeeded installing GHC, I keep getting errors on HHM and I wonder if someone can help me out.
Tom Kralidis replied on my mail with the request to create a ticket "for this looks like bug of some sort"
My laptop is a Ubuntu 16.04 LTS with python 2.7 (and 3.5) and following the description on the Github page I am getting to the final stage without any error.
But starting the " ./manage.py runserver " command Django crashes with an error (in the file attached)
Who can point me in the right direction to solve this? I've been googeling for the last 2 days to find a solution.
I attached the install script i made from the readme.md (which works great until the last command ;-( ) as well as the error generated by Django
ErrorHHMinstall.txt
btw: the skip_celery_task switch in the install script did not solve it.....
Thanx in advance, Cor
[2016-08-03 23:00:45,145: WARNING/Worker-2] HTTPSConnectionPool(host='ngamaps.geointapps.org', port=443): Max retries exceeded with url: /arcgis/rest/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd9122e0d50>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Input: [-60,-180 TO 60,180]
Output:
"min_x": -178.22782202180733,
"min_y": 18.910787005414388,
"max_x": -66.95000501284882,
"max_y": 71.38957425511644
should be inside bbox.
Both service and folder of services.
@bglewis : Esri REST includes much more information than WMS/WMTS... services, for example Geometry Type (Points, Lines, Polygons).
Should we track this additional information?
@jmwenda can you help with me with this?
Basically you should test what can be done with the arcrest library, and figure out a way to get a thumbnail for a layer.
After figuring it out, we can discuss how to integrate your work in code base.
Currently celery reads variables from postactivate, while uwsgi from its configuration file.
This is a common problem where some rest endpoints have projections that are not standard. Causing harvesting to stop. i.e
PROJCS["NAD_1983_StatePlane_Michigan_South_FIPS_2113_IntlFeet",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic"],PARAMETER["False_Easting",13123359.58005249],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-84.36666666666666],PARAMETER["Standard_Parallel_1",42.1],PARAMETER["Standard_Parallel_2",43.66666666666666],PARAMETER["Latitude_Of_Origin",41.5],UNIT["Foot",0.3048]]
Problem lies with the ArcRest library
http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/TaxParcel/TaxParcelQuery/MapServer
I tried to import Hypermap endpoint, and celerylog displayed me the following error:
celery_1 | File "/usr/src/app/hypermap/aggregator/models.py", line 280, in update_layers
celery_1 | update_layers_wm(self)
celery_1 | File "/usr/src/app/hypermap/aggregator/models.py", line 1030, in update_layers_wm
celery_1 | data = json.loads(response.content)
celery_1 | File "/usr/local/lib/python2.7/json/__init__.py", line 338, in loads
celery_1 | return _default_decoder.decode(s)
celery_1 | File "/usr/local/lib/python2.7/json/decoder.py", line 366, in decode
celery_1 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
celery_1 | File "/usr/local/lib/python2.7/json/decoder.py", line 384, in raw_decode
celery_1 | raise ValueError("No JSON object could be decoded")
celery_1 | ValueError: No JSON object could be decoded
When I did a debug on models.py I noticed trying to add hypermap as endpoint, a get resquest is made to 'http://worldmap.harvard.edu/data/search/api?start=0&limit=10', as you can see in the following code:
def update_layers_wm(service):
"""
Update layers for an WorldMap.
Sample endpoint: http://worldmap.harvard.edu/
"""
response = requests.get('http://worldmap.harvard.edu/data/search/api?start=0&limit=10')
data = json.loads(response.content)
total = data['total']
I also did a get request to the same link and it showed me the following message:
Internal Server Error
An internal error occurred. Please try again, and if the problem continues, contact your site administrator.
So, it means there is a linking problem or it doesn't exist
Full code: https://github.com/cga-harvard/HHypermap/blob/registry/hypermap/aggregator/models.py
The logged user should be able to upload a text file containing endpoints.
The endpoints will be processed by hypermap.
If a endpoint is recognizable (as WMTS/WMS etc...) and is not already existing, then it is added as a service.
The application should keep track of uploaded endpoints with relative results (SUCCESS, ALREADY EXISTING, FAILURE)
I will check this with @ingenieroariel
BOP API implementation can be found at:
https://github.com/dsmiley/hhypermap-bop
The idea is to mirror the definition and implementation for time based search, spatial based search, etc. That code does not have a LICENSE file so it's better to check with David and Ben about it before starting the work.
Work can be divided in three steps:
Comment by @bglewis:
The problem is with geohash the system chooses an array resolution that has a range of 1 to 32. That leads to very crude heatmaps or too accurate ones which are very slow. Quad tree provides a smaller range 1 to 4. This is what we need to implement. Talk to Smiley and perhaps Ariel about how to implement.
It would be possible to automatically classify layers from metadata parsing.
We could use a geographic names thesaurus.
We need to discuss about this before we proceed.
Add connector for harvesting for CSW for geonode (and maybe any CSW).
Maybe @tomkralidis has good ideas here.
These are WMTS layers with DPLA metadata.
In some cases no thumbnails are generated.
Most likely because of srs issues.
Currently we have the following service types enum:
SERVICE_TYPES = (
('OGC:CSW', 'Catalogue Service for the Web (CSW)'),
('OGC:WMS', 'Web Map Service (WMS)'),
('OGC:WMTS', 'Web Map Tile Service (WMTS)'),
('OSGeo:TMS', 'Tile Map Service (TMS)'),
('ESRI:ArcGIS:MapServer', 'ArcGIS REST MapServer'),
('ESRI:ArcGIS:ImageServer', 'ArcGIS REST ImageServer'),
('Hypermap:WorldMap', 'Harvard WorldMap'),
('Hypermap:WARPER', 'Mapwarper'),
)
In the Hypermap UI and in the Hypersearch client we would like to see the Hypermap:WARPER layers as OGC:WMS.
Therefore we could have another enum just for the UI like this:
UI_SERVICE_TYPES = (
('OGC:CSW', 'Catalogue Service for the Web (CSW)'),
('OGC:WMS', 'Web Map Service (WMS)'),
('OGC:WMTS', 'Web Map Tile Service (WMTS)'),
('OSGeo:TMS', 'Tile Map Service (TMS)'),
('ESRI:ArcGIS:MapServer', 'ArcGIS REST MapServer'),
('ESRI:ArcGIS:ImageServer', 'ArcGIS REST ImageServer'),
('Hypermap:WorldMap', 'Harvard WorldMap'),
)
Note that this requires a schema change or we need to modify the query to Solr.
Hyperma should provide a layer preview page, where the layer could be displayed on a Leaflet/OpenLayers map.
@jmwenda it maybe time to move your Dajngo command to hypermap. Let's discuss about it when you have some time?
Add connector for bringing in HGL layers. Need to discuss with D.S. on best option which may be via OGP
We need it for development.
Right now the ngamaps one is sometime unavailable, the demo.geonode.org is out of space and worldmap.harvard.edu is too big.
Hi, after successfully installing HHypermap on closure of #125 i ran into another problem.
loading my endpoint list (attached) results in a error (104, connection reset by peer) (also attached with .txt for uploading purpose, it is the page save of the website response)
HHM_endpoint_list.txt
error at admin_aggregator_endpointlist_add.html.txt
it could be my fault because on being unknown to how to operate HHypermap?
I am running this from my work laptop (ubuntu 16.04) with internet access (port 80).
tried our url in the list with different endings: geoserver/ows , geoserver/wms, etc....
is this a bug or am i missing something?
Thanks for your effort in advance, Cor
take for example Kotlin code.
We need a pluggable architecture to retrieve custom metadata for each service
Like http://hypersearch.cga.terranodo.io/maps/new
example:
[-50000000-01-01T00:00:00.0Z TO 2001-01-01T00:00:00.0Z]
TASK: [web | Load initial demo services] **************************************
failed: [default] => {"cmd": "python manage.py loaddata aggregator/fixtures/aggregator.json --settings=settings._vagrant", "failed": true, "path": "/webapps/hypermap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "state": "absent", "syspath": ["/home/vagrant/.ansible/tmp/ansible-tmp-1455293058.27-133680234972419", "/usr/lib/python2.7", "/usr/lib/python2.7/plat-x86_64-linux-gnu", "/usr/lib/python2.7/lib-tk", "/usr/lib/python2.7/lib-old", "/usr/lib/python2.7/lib-dynload", "/usr/local/lib/python2.7/dist-packages", "/usr/lib/python2.7/dist-packages", "/usr/lib/python2.7/dist-packages/PILcompat"]}
msg:
:stderr: Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 55, in handle
self.loaddata(fixture_labels)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 84, in loaddata
self.load_label(fixture_label)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 140, in load_label
obj.save(using=self.using)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 164, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/db/models/base.py", line 582, in save_base
update_fields=update_fields, raw=raw, using=using)
File "/webapps/hypermap/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 185, in send
response = receiver(signal=self, sender=sender, **named)
File "/webapps/hypermap/hypermap/hypermap/aggregator/models.py", line 438, in service_post_save
check_service.delay(instance)
File "/webapps/hypermap/local/lib/python2.7/site-packages/celery/app/task.py", line 453, in delay
return self.apply_async(args, kwargs)
File "/webapps/hypermap/local/lib/python2.7/site-packages/celery/app/task.py", line 560, in apply_async
**dict(self._get_exec_options(), **options)
File "/webapps/hypermap/local/lib/python2.7/site-packages/celery/app/base.py", line 354, in send_task
reply_to=reply_to or self.oid, **options
File "/webapps/hypermap/local/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task
**kwargs
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/messaging.py", line 172, in publish
routing_key, mandatory, immediate, exchange, declare)
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/connection.py", line 457, in _ensured
interval_max)
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
interval_start, interval_step, interval_max, callback)
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
return fun(*args, **kwargs)
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
return self.connection
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
self._connection = self._establish_connection()
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
conn = self.transport.establish_connection()
File "/webapps/hypermap/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
conn = self.Connection(**opts)
File "/webapps/hypermap/local/lib/python2.7/site-packages/amqp/connection.py", line 180, in __init__
(10, 30), # tune
File "/webapps/hypermap/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
self.channel_id, allowed_methods, timeout)
File "/webapps/hypermap/local/lib/python2.7/site-packages/amqp/connection.py", line 241, in _wait_method
channel, method_sig, args, content = read_timeout(timeout)
File "/webapps/hypermap/local/lib/python2.7/site-packages/amqp/connection.py", line 330, in read_timeout
return self.method_reader.read_method()
File "/webapps/hypermap/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method
raise m
socket.error: [Errno 104] Connection reset by peer
This ticket covers the OpenSearch aspect pointing to the right location for the other endpoints.
https://gist.github.com/tomkralidis/cc764728baeaaa3ddd26722f7856252b
The endpoint should be this one, according to what I see in urls.py:
http://localhost:8000/search/csw?service=CSW&version=2.0.2&request=GetCapabilities
But I am getting:
AttributeError at /search/csw
'Csw' object has no attribute 'repository'
Exception Location: /webapps/hypermap/local/lib/python2.7/site-packages/pycsw/ogc/csw/csw2.py in getcapabilities, line 312
We need to define what the best Django package is for what we need and which Django API framework it should be based on.
The requirement is that it relies as little as possible on Django models and that it is lightweight and well integrated with Swagger.
This ticket can be closed once the right package has been added to setup.py as a PR.
Let's post the options as comments here and decide together.
GeoHealtCheck uses raphael.js
@jmwenda this should be an easy task, would you help me? Let's talk about it.
In home page we should be able to filter services by keyword (on title and abstract) and type (WMS/WMTS/ESRI...).
In service page we should be able to filter layers by keyword (on title, name, abstract), srs, geometry type (for ESRI)
@ingenieroariel can we add hypermap to wm-django?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.