open-eo / openeo-grassgis-driver Goto Github PK
View Code? Open in Web Editor NEWOpenEO driver for GRASS GIS/actinia. Backend description at https://openeo.mundialis.de/.well-known/openeo
License: Apache License 2.0
OpenEO driver for GRASS GIS/actinia. Backend description at https://openeo.mundialis.de/.well-known/openeo
License: Apache License 2.0
Is there a set of example process graphs for demonstration purposes?
Currently, a Sentinel-2 collection is expected to have exactly 13 bands: https://github.com/Open-EO/openeo-grassgis-driver/blob/master/src/openeo_grass_gis_driver/collection_information.py#L209
This however only applies to Level-1C data, as Level-2A does not have a B10 anymore.
Instead, Level-2A data contains additional auxiliary bands (AOT, SCL, TCI, WVP). It would be required to
At https://github.com/Open-EO/openeo-grassgis-driver/blob/master/src/openeo_grass_gis_driver/utils/logging.py#L18 the default path for the log file is "/openeo-grassgis-driver.log"
which should be invalid on most systems: never ever create a file directly at /
.
I have used the Docker deployment approach to setup the service (also created a docker-compose file for easier deployment).
Most of the exploration endpoints are working, but the /data
endpoint returns an error. I assume that I haven't downloaded the data within the Dockerfile. Can you tell me where to store the data or how to configure the service?
I just tried latlong_wgs84.PERMANENT.strds.ndvi_16_5600m
once more (see last paragraph in #49 (comment)), this time receiving a response after 1856ms, which was the 500 error (live example).
The error page read:
ValueError: could not convert string to float: 'None'
And in the (very long) traceback the last entry was:
File "/usr/local/lib/python3.6/dist-packages/openeo_grass_gis_driver-0.0.1-py3.6.egg/openeo_grass_gis_driver/collection_information.py", line 132, in
get
extent = CollectionExtent(spatial=(float(layer_data["west"]), float(layer_data["south"]),
So I guess it's simply a missing coordinate in the bbox?
I was following the installation instructions here to deploy the GRASS GIS driver locally. Everything went fine without issues, but when running the openEO GRASS GIS driver test suite using
python3.5 setup.py test > console_out.txt
I get a number of failed tests as documented in the attached txt file. I noted that only the tests associated with "tests/test_jobs.py"
and "tests/test_processes.py"
failed - the others passed. Could someone please point out if this is expected or is there something I am doing wrong?
Additionally, I also get the following warning while running the line above:
/mnt/h/edu/semester_4/grass/openEO/venv/lib/python3.5/site-packages/setuptools/dist.py:407: UserWarning: The version specified ('0.1-poc.post0.dev15+g0cddcc2.dirty') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
"details." % self.metadata.version
Thanks!
console_out.txt
I'm failing several of the tests on a Windows install, because Windows uses "USERPROFILE" instead of "HOME".
I'm not sure if this is designed to run with Windows or not, for now I am going to replace the code on my end with this:
import os.path
home_folder = os.path.expanduser('~')
Currently, when I open https://openeo.mundialis.de/api/v0.3/ I am prompted:
One should be able to discover a backend without the need to sign up. I.e. the endpoints
should be reachable without authentication.
For example, this is essential for the Hub -- password-protected backends can't be crawled.
Using the array_element function in a Graph fails in the backend in t.rast.bandcalc with expression=(1.2 * data[0])
Running the following code from the python client (0.9)
def my_bandmath(data):
band1 = array_element(data, index=0)
return 1.2 * band1
datacube.reduce_dimension(dimension="bands",reducer=my_bandmath)
resulting in the Graph
{
"process_graph": {
"loadcollection1": {
"process_id": "load_collection",
"arguments": {
"id": "nc_spm_08.modis_lst.strds.LST_Day_monthly",
"spatial_extent": null,
"temporal_extent": [
"2015-01-01",
"2015-03-01"
]
}
},
"reducedimension1": {
"process_id": "reduce_dimension",
"arguments": {
"data": {
"from_node": "loadcollection1"
},
"dimension": "bands",
"reducer": {
"process_graph": {
"arrayelement1": {
"process_id": "array_element",
"arguments": {
"data": {
"from_parameter": "data"
},
"index": 0
}
},
"multiply1": {
"process_id": "multiply",
"arguments": {
"x": 1.2,
"y": {
"from_node": "arrayelement1"
}
},
"result": true
}
}
}
}
},
"saveresult1": {
"process_id": "save_result",
"arguments": {
"data": {
"from_node": "reducedimension1"
},
"format": "GTIFF",
"options": {}
},
"result": true
}
}
}
From https://openeo.mundialis.de/api/v0.3/:
{
"endpoints": [
{
"methods": [
"GET"
],
"path": "/processes/<string:process_id>"
},
...
From the API spec:
Variables in the paths must be placed in curly braces, e.g.
{process_id}
While it doesn't really matter for a human, this is important for the openEO Hub: It can only detect functionalities as being supported when the paths are correctly formatted. The Hub tolerates different parameter names, e.g. your prefix with string:
is okay. But the enclosing must follow the spec. Otherwise the functionality will be incorrectly reported as non-supported.
BTW your GET //collections/<string:name>
endpoint has two leading slashes, probably a typo?
I have tested the logs functionatliy within the openeo-python-client and the openeo-webeditor. But the logs
request of a job fails with an internal server error: TypeError: Object of type EoLink is not JSON serializable.
[2021-10-02 18:55:17,249] ERROR in app: Exception on /api/v1.0/jobs/user-job-6f60f678-1dee-42af-864e-ebbfa1f3364b/logs [GET]
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/lib/python3.8/site-packages/flask_restful/__init__.py", line 468, in wrapper
resp = resource(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/openeo_grass_gis_driver/authentication.py", line 76, in decorated
return f(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/flask/views.py", line 89, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/flask_restful/__init__.py", line 583, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/openeo_grass_gis_driver/jobs_job_id_logs.py", line 87, in get
return make_response(jsonify(job_logs), 200)
File "/usr/lib/python3.8/site-packages/flask/json/__init__.py", line 370, in jsonify
dumps(data, indent=indent, separators=separators) + "\n",
File "/usr/lib/python3.8/site-packages/flask/json/__init__.py", line 211, in dumps
rv = _json.dumps(obj, **kwargs)
File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.8/site-packages/flask/json/__init__.py", line 100, in default
return _json.JSONEncoder.default(self, o)
File "/usr/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type EoLink is not JSON serializable
See Open-EO/openeo-processes#331 (comment)
Check if handling of temporal intervals is correctly implemented in the processes load_collection
and filter_temporal
I tried to access the back-end at http://openeo.mundialis.de with the browser-based JavaScript client.
Unfortunately, I can't access it. It seems CORS is not implemented as I get:
Access to XMLHttpRequest at 'http://openeo.mundialis.de/api/v0.4/' from origin 'http://editor.openeo.org' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
It would be great to get CORS support as specified at https://open-eo.github.io/openeo-api/v/0.4.1/cors/
You can easily check it yourself with the Web Editor at http://editor.openeo.org .
The deployed grassgis-backend returns
{
"api_version": "1.0.1",
"backend_version": "0.2.0",
"description": "GRASS GIS Driver",
"endpoints": [
...
while the current github release is 1.0.2. This should be in sync.
The following OPENEO (1.0) core functions are not available in openeo-grassgis-driver:
Running the docker build following the documentation fails with the following errors during the testing phase:
git clone https://github.com/Open-EO/openeo-grassgis-driver.git
cd openeo-grassgis-driver
docker build -f docker/Dockerfile -t openeo_grass_gis_driver .
...
=================================== FAILURES ===================================
___________________ ActiniaInterfaceTestCase.test_list_strds ___________________
self = <test_actinia_interface.ActiniaInterfaceTestCase testMethod=test_list_strds>
def test_list_strds(self):
iface = ActiniaInterface(self.gconf)
status, layers = iface.list_strds(
location="nc_spm_08", mapset="landsat")
pprint(layers)
self.assertEqual(status, 200)
> self.assertEqual(len(layers), 1)
E AssertionError: 4 != 1
tests/test_actinia_interface.py:47: AssertionError
----------------------------- Captured stdout call -----------------------------
['lsat5_1987', 'lsat5_1987_10', 'lsat5_1987_30', 'lsat5_1987_40']
__________________ GraphValidationTestCase.test_6_graph_apply __________________
self = <test_process_graph_validation.GraphValidationTestCase testMethod=test_6_graph_apply>
def test_6_graph_apply(self):
"""Run the validation test
"""
response = self.app.post(
self.prefix +
'/validation',
data=json.dumps(APPLY_1),
content_type="application/json",
headers=self.auth)
> self.assertEqual(response.status_code, 200)
E AssertionError: 400 != 200
tests/test_process_graph_validation.py:83: AssertionError
I have also tested 2.5.3 release, it also has the same issue.
At time the image collection endpoint (https://openeo.mundialis.de/collections) gives the following error:
' "message": "The model \\"STRDSInfoModel\\" does not have an attribute \\"number_of_bands\\"", \n'
openeo-grassgis-driver test log captured:
----------------------------- Captured stdout call -----------------------------
('{\n'
' "accept_datetime": "2019-11-28 13:05:36.207693", \n'
' "accept_timestamp": 1574946336.2076917, \n'
' "api_info": {\n'
' "endpoint": "strdsmanagementresource", \n'
' "method": "GET", \n'
' "path": '
'"/api/v1/locations/latlong_wgs84/mapsets/modis_ndvi_global/strds/ndvi_16_5600m", \n'
' "request_url": '
'"http://actinia.mundialis.de/api/v1/locations/latlong_wgs84/mapsets/modis_ndvi_global/strds/ndvi_16_5600m"\n'
' }, \n'
' "datetime": "2019-11-28 13:05:36.906189", \n'
' "exception": {\n'
' "message": "The model \\"STRDSInfoModel\\" does not have an attribute '
'\\"number_of_bands\\"", \n'
' "traceback": [\n'
' " File '
'\\"/usr/local/lib/python3.6/dist-packages/actinia_core-0.99.1.post0.dev22+geb8bca3-py3.6.egg/actinia_core/resources/ephemeral_processing.py\\", '
'line 1350, in run\\n self._execute()\\n", \n'
' " File '
'\\"/usr/local/lib/python3.6/dist-packages/actinia_core-0.99.1.post0.dev22+geb8bca3-py3.6.egg/actinia_core/resources/strds_management.py\\", '
'line 594, in _execute\\n self.module_results = '
'STRDSInfoModel(**strds)\\n", \n'
' " File '
'\\"/usr/local/lib/python3.6/dist-packages/flask_restful_swagger_2/__init__.py\\", '
'line 337, in __init__\\n \'The model \\"{0}\\" does not have an attribute '
'\\"{1}\\"\'.format(self.__class__.__name__, k))\\n"\n'
' ], \n'
' "type": "<class \'ValueError\'>"\n'
' }, \n'
' "http_code": 400, \n'
' "message": "The model \\"STRDSInfoModel\\" does not have an attribute '
'\\"number_of_bands\\"", \n'
' "process_chain_list": [\n'
' {\n'
' "1": {\n'
' "flags": "g", \n'
' "inputs": {\n'
' "input": "ndvi_16_5600m", \n'
' "type": "strds"\n'
' }, \n'
' "module": "t.info"\n'
' }\n'
' }\n'
' ], \n'
' "process_log": [\n'
' {\n'
' "executable": "t.info", \n'
' "parameter": [\n'
' "type=strds", \n'
' "input=ndvi_16_5600m", \n'
' "-g"\n'
' ], \n'
' "return_code": 0, \n'
' "run_time": 0.6452951431274414, \n'
' "stderr": [\n'
' ""\n'
' ], \n'
' "stdout": '
'"id=ndvi_16_5600m@modis_ndvi_global\\nname=ndvi_16_5600m\\nmapset=modis_ndvi_global\\ncreator=mundialis\\ntemporal_type=absolute\\ncreation_time=\'2017-02-16 '
"11:31:50.189306'\\nmodification_time='2017-05-29 "
"21:04:13.222786'\\nsemantic_type=mean\\nstart_time='2013-01-01 "
"00:00:00'\\nend_time='2017-05-25 00:00:00'\\ngranularity='16 "
'days\'\\nmap_time=interval\\nnorth=90.0\\nsouth=-90.0\\neast=180.0\\nwest=-180.0\\ntop=0.0\\nbottom=0.0\\naggregation_type=None\\nnumber_of_bands=None\\nnumber_of_maps=101\\nnsres_min=0.05\\nnsres_max=0.05\\newres_min=0.05\\newres_max=0.05\\nmin_min=-2000.0\\nmin_max=-2000.0\\nmax_min=9947.0\\nmax_max=9994.0\\nraster_register=raster_map_register_73eb5d056c51424288153b87a714630a\\n"\n'
' }\n'
' ], \n'
' "progress": {\n'
' "num_of_steps": 1, \n'
' "step": 1\n'
' }, \n'
' "resource_id": "resource_id-21b3cb05-ef05-4780-87ea-cc62ed86fb5a", \n'
' "status": "error", \n'
' "time_delta": 0.6985125541687012, \n'
' "timestamp": 1574946336.9061809, \n'
' "urls": {\n'
' "resources": [], \n'
' "status": '
'"http://actinia.mundialis.de/api/v1/resources/openeo/resource_id-21b3cb05-ef05-4780-87ea-cc62ed86fb5a"\n'
' }, \n'
' "user_id": "openeo"\n'
'}\n')
________________________ DataTestCase.test_collections _________________________
self = <test_collections.DataTestCase testMethod=test_collections>
def test_collections(self):
"""Test the return of raster and vector maps in the north carolina test dataset
:return:
"""
response = self.app.get('/collections', headers=self.auth)
data = json.loads(response.data.decode())
pprint(data)
dsets = ["nc_spm_08.landsat.raster.lsat5_1987_10",
"nc_spm_08.PERMANENT.vector.lakes",
"nc_spm_08.PERMANENT.raster.elevation"]
data_id_list = []
> for entry in data["collections"]:
E TypeError: list indices must be integers or slices, not str
tests/test_collections.py:32: TypeError
----------------------------- Captured stdout call -----------------------------
[{'description': 'An internal error occurred while catching strds layers!'},
400]
Hi,
when accessing https://openeo.mundialis.de/api/v1.0/collections the current server response is:
[
{
"description": "An internal error occurred while catching strds layers!"
},
400
]
It would be nice if:
200 OK
but some 500
, so that accessing clients treat it as "gone wrong" straight away,Trying to access https://openeo.mundialis.de/api/v1.0/collections leaves one waiting and waiting, until after a minute there is a 504: Gateway time-out error. Can this be sped up? Because at the moment, the user cannot know which collections are available...
the official OPEN-EO math functions "add,substract,divide,multiply" deliver wrong results.
Calling the functions "add,substract,divide,multiply" via apply for single Band Collection and with 2 as second parameter y
results in identical internal grass call to t.rast.mapcalc with expression=(2) and resulting outputs completely filled with value 2
SubGraph: for apply :
"apply1": {
"process_id": "apply",
"arguments": {
"data": {
"from_node": "loadcollection1"
},
"process": {
"process_graph": {
"add1": {
"process_id": "add",
"arguments": {
"x": {
"from_parameter": "x"
},
"y": 2
},
"result": true
}
}
}
}
}
Currently, https://openeo.mundialis.de/api/v1.0/collections replies with:
[
{
"description": "An internal error occurred while catching mapset from location nc_spm_08!"
},
400
]
Aside from that it would be nice if this error wouldn't be there at all (:wink:), there are a couple of issues with that error response:
400
) is meant to be a HTTP status code, it is a wrong one because "internal error" should result in something in the 5xx range.200 OK
.Currently all configuration is done in config.py. When changing a value, an already built python wheel needs to be rebuild. It would be better to outsource the values to a real config file which is read during startup/runtime.
Currently, process definitions have been added manually to the openeo-grass driver. As in the example openeo python driver, process definitions can be directly taken from the official repo https://github.com/Open-EO/openeo-processes. This would make maintenance much easier.
@bgoesswe deployed a validation tool for back-ends. The results for the GRASS GIS back-end can be found here:
https://www.geo.tuwien.ac.at/openeoct/backend/validate/4
There are several issues reported:
I'm starting out on a completely fresh conda environment, following the Installation directions in https://open-eo.github.io/openeo-grassgis-driver/#installation
I am on Windows 10, using conda environments instead of virtualenv, just because I am familiar with it and by default virtualenv did not work for me.
I am failing tests because gdal is not included under https://github.com/Open-EO/openeo-grassgis-driver/blob/master/requirements.txt
I'm working out where in the process to install those for this to work.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.