Git Product home page Git Product logo

xyz-hub's Introduction

XYZ Hub

License Run XYZ Hub tests

XYZ Hub is a RESTful web service for the access and management of geospatial data.

Overview

Some of the features of XYZ Hub are:

  • Organize geo datasets in spaces
  • Store and manipulate individual geo features (points, linestrings, polygons)
  • Retrieve geo features as vector tiles, with or without clipped geometries
  • Search for geo features spatially using a bounding box, radius, or any custom geometry
  • Explore geo features by filtering property values
  • Retrieve statistics for your spaces
  • Analytical representation of geo data as hexbins with statistical information
  • Connect with different data sources
  • Build a real-time geodata pipeline with processors
  • Attach listeners to react on events

You can find more information in the XYZ Documentation and in the OpenAPI specification.

XYZ Hub uses GeoJSON as the main geospatial data exchange format. Tiled data can also be provided as MVT.

Prerequisites

  • Java 17
  • Maven 3.9+
  • Postgres 15+ with PostGIS 3+
  • Redis 7+ (optional)
  • Docker 18+ (optional)
  • Docker Compose 1.24+ (optional)

Getting started

Clone and install the project using:

git clone https://github.com/heremaps/xyz-hub.git
cd xyz-hub
mvn clean install

With docker

The service and all dependencies could be started locally using Docker compose.

docker-compose up -d

Alternatively, you can start freshly from the sources by using this command after cloning the project:

mvn clean install -Pdocker

Hint: Postgres with PostGIS will be automatically started if you use 'docker-compose up -d' to start the service.

Without docker

The service could also be started directly as a fat jar. In this case Postgres and the other optional dependencies need to be started separately.

java [OPTIONS] -jar xyz-hub-service/target/xyz-hub-service.jar

Example:

java -DHTTP_PORT=8080 -jar xyz-hub-service/target/xyz-hub-service.jar

Configuration options

The service start parameters could be specified by editing the default config file, using environment variables or system properties. See the default list of configuration parameters and their default values.

Usage

Start using the service by creating a space:

curl -H "content-type:application/json" -d '{"title": "my first space", "description": "my first geodata repo"}' http://localhost:8080/hub/spaces

The service will respond with the space definition including the space ID:

{
    "id": "pvhQepar",
    "title": "my first space",
    "description": "my first geodata repo",
    "storage": {
        "id": "psql",
        "params": null
    },
    "owner": "ANONYMOUS",
    "createdAt": 1576601166681,
    "updatedAt": 1576601166681,
    "contentUpdatedAt": 1576601166681,
    "autoCacheProfile": {
        "browserTTL": 0,
        "cdnTTL": 0,
        "serviceTTL": 0
    }
}

You can now add features to your brand new space:

curl -H "content-type:application/geo+json" -d '{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[-2.960847,53.430828]},"properties":{"name":"Anfield","@ns:com:here:xyz":{"tags":["football","stadium"]},"amenity":"Football Stadium","capacity":54074,"description":"Home of Liverpool Football Club"}}]}' http://localhost:8080/hub/spaces/pvhQepar/features

The service will respond with the inserted geo features:

{
    "type": "FeatureCollection",
    "etag": "b67016e5dcabbd5f76b0719d75c84424",
    "features": [
        {
            "type": "Feature",
            "id": "nf36KMsQAUYoM5kM",
            "geometry": {
                "type": "Point",
                "coordinates": [ -2.960847, 53.430828 ]
            },
            "properties": {
                "@ns:com:here:xyz": {
                    "space": "pvhQepar",
                    "createdAt": 1576602412218,
                    "updatedAt": 1576602412218,
                    "tags": [ "football", "stadium" ]
                },
                "amenity": "Football Stadium",
                "name": "Anfield",
                "description": "Home of Liverpool Football Club",
                "capacity": 54074
            }
        }
    ],
    "inserted": [
        "nf36KMsQAUYoM5kM"
    ]
}

OpenAPI specification

The OpenAPI specification files are accessible under the following URIs:

  • Stable: http://<host>:<port>/hub/static/openapi/stable.yaml
  • Experimental: http://<host>:<port>/hub/static/openapi/experimental.yaml

Acknowledgements

XYZ Hub uses:

and others.

Contributing

Your contributions are always welcome! Please have a look at the contribution guidelines first.

License

Copyright (C) 2017-2024 HERE Europe B.V.

This project is licensed under the Apache License, Version 2.0 - see the LICENSE file for details.

xyz-hub's People

Contributors

anonymousaccount4se avatar cfjedimaster avatar ctonhaeuser avatar dependabot[bot] avatar goshev avatar haifeng2013 avatar jwegne avatar lceni avatar marvgilb avatar mchrza avatar mujammil10 avatar qgydxby2 avatar roegi avatar rysiek2000 avatar terminaltim avatar tsteenbe avatar vladyslavhnes avatar xeus2001 avatar y-korneev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xyz-hub's Issues

Providing binary releases?

Are there any binary releases available? This would reduce the compile time quite a bit, which is nice for building hosted online environments more quickly.

Add support for de- and encryption of secret in xyz-connector libs

There is a need to store encrypted secrets in either space definitions or token metadata:

There already are several connectors that need credentials to access external resources. These credentials should not be stored as clear text with the connector's configuration within a space in order to prevent unauthorized access.

Since this functionality is already needed in several connectors today, we think it should be provided centrally as part of the XYZ infrastructure, especially since the number of affected connectors will likely rise in the future.

One possible way would be to implement this functionality inside XYZ hub so the credentials are decrypted when calling connectors that need them. The need for encryption and decryption might be signified by e.g. a prefix to field names within the connector configuration. This would be the most convenient way for connector developers. However, it has the potential to introduce significant overhead within XYZ hub which is not desirable. Additionally, great care would need to be taken in order to prevent accidental leaking of decrypted secrets during the (external) call to the connectors, e.g. by logging the request to the connector prior to sending it.

To us it is clear that the decryption process should take place inside the individual connectors. However, we think this should happen in a uniform way and would like to spare connector developers from having to implement decryption routines from scratch for every new connector.
First, getting it done right will prevent errors in the en- and decryption routines to leak credentials to the outside and second it will prevent developers writing the same boiler plate code over and over.

Therefore, we suggest to expand the xyz-connector libraries in order to add support for de- and encrypting secrets using asymmetric (public/private key) encryption. By utilizing a service like Amazon's KMS we can ensure that connector code does not get direct knowledge of the private key, thus enabling us to use a single, strong public/private key pair per environment for all connectors deployed there in order to keep things simple. Despite that, it would still be possible to use individual public/private key pairs for certain connectors if that requirement should arise.

Preliminary technical proposal for further discussion:

  • We need to configure a private/public key pair. The public key is used for encrypting the secrets either by the users or the connector. The private key should only be accessible by the connector to decrypt the secrets.
  • The public key needs to be accessible - either in the documentation of XYZ Hub or there is an REST/HTTP call to get it.
  • Add a new helper class to XYZ connectors lib that needs to know where to find the public/private key pair. We would suggest to use AWS KMS to store them. The helper class will have methods for en- and decryption the secrets. If no public/private key pair is configured, the values will be returned as is.
  • In ListenerConnector.java and ProcessorConnector.java we will check if the connector params, params or token metadata contain values that are enclosed in a certain pre- and postfix. We would suggest using "<<" and ">>".
  • If such values are found, the configured private key is used to decrypt the values and forward it to the custom connector implementation.
  • Automatic encryption using format ">>PLAINTEXT<<"
  • Automatic decryption using format "<<CYPHERTEXT>>"

error 429: Too Many Requests ,Remote function is busy or cannot be invoked

Hello, I encountered some problems when using this program. I want to ask:
When there are many concurrent requests, an error of 429 will be reported:
tile-layer.js:37

   Error: es/world/tile/web/12_3291_1673.mvt (429): Too Many Requests {"type":"ErrorResponse","error":"Exception","errorMessage":"Remote function is busy or cannot be invoked.","streamId":"z2XCiecK3A"}...
at checkResponse (response-utils.js:45:1)
at async getArrayBufferOrStringFromData (get-data.js:57:1)
at async parseWithLoader (parse.js:42:1)
at async parse (parse.js:37:1)
at async load (load.js:23:1)
at async Tile2DHeader._loadData (tile-2d-header.js:83:1). 

what should I do?

adding swagger documentation

i was expecting to see the local installation of xyz-hub to have the swagger documentation but not sure if it's available and i can't find its end point or still to be upcoming

Consider using standard http error code instead of 513

hey all!

maybe a long standing typo, but using custom http errors makes creating code stubs from swagger almost impossible. you should change your custom 513/payload too large to standard http 413 Payload Too Large

public static final HttpResponseStatus RESPONSE_PAYLOAD_TOO_LARGE = new HttpResponseStatus(513, "Response payload too large");

would also make more sense to have it as as 4xx client and not a 5xx server error

O

Testing Spatial search can lead to incorrect results

I was using the spatial search function here (https://xyz.api.here.com/hub/static/swagger/#/Read%20Features/getFeaturesBySpatial) and noticed I wasn't getting results when I should have. The reason I didn't is because the 'default' text for params included this code:

OrderedMap { "p.myProperty": "someValue", "p.otherProperty": 5 }

Other fields on the form have descriptive text for the field but they do NOT actually get sent to the API. So for example:

image

Even though you see text in limit, it's not actually passed to the API.

To fix this, we need to remove the text from the textarea so it doesn't get passed when a user doesn't notice that it is "real" text.

Please add support for `GeometryCollection`s to the HERE XYZ Hub

Currently, loading GeoJSON content with (standard) GeometryCollections leads https://geojson.tools to report errors s.a. 1 feature(s) added successfully and 1 feature(s) failed to load, without rendering said GeometryCollections.

Although the limitation is documented (GeometryCollection is not supported by HERE XYZ Hub. It is just included here for completeness.), it forces us to resort to other web applications (s.a. https://geojson.io) to get rendering - thus without HERE maps.

Could you therefore please consider supporting GeometryCollections?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.