Git Product home page Git Product logo

hop-gis-plugins's Introduction

Apache Hop GIS Plugins

This project allows you to manage GIS data in Apache Hop, Hop Orchestration Platform. It is a counterpart of Pentaho Data Integrator GIS Plugins.

Tested with Apache Hop 2.7.0-SNAPSHOT.

Building the plugins

Check out the project if you have not already done so (with git installed):

git clone git://github.com/atolcd/hop-gis-plugins.git
cd hop-gis-plugins

Run the following commands from the base project directory (with java 11 and maven installed):

mvn clean package

Or with Docker (you need docker) :

docker run --network="host" --rm \
  --name hop-gis-plugins-builder \
  -u $(id -u):$(id -g) \
  -v "$(pwd)":/app -w /app \
  -v ~/.m2:/var/maven/.m2 -v "${HOME}":/var/maven \
  -e HOME=/var/maven -e MAVEN_CONFIG=/var/maven/.m2 -e MAVEN_OPTS="-Duser.home=/var/maven" \
  \
  maven:3-jdk-11 \
  \
  mvn clean package

The built package is assemblies/target/gis-plugin-assemblies-X.X.X.zip (version can differ).

Installing/upgrading the module

Method 1 : Manual installation

Extract the content of gis-plugin-assemblies-X.X.X.zip in ${HOP_HOME}/plugins. Examples of extraction from the root directory of the project :

GIS_PLUGINS_VERSION="1.2.0"

# Use compiled version...
GIS_PLUGINS_ASSEMBLY="assemblies/target/gis-plugin-assemblies-${GIS_PLUGINS_VERSION}.zip"

# ... Or download a prepared one (must exist :)
wget https://github.com/atolcd/hop-gis-plugins/releases/download/v${GIS_PLUGINS_VERSION}/gis-plugin-assemblies-${GIS_PLUGINS_VERSION}.zip
GIS_PLUGINS_ASSEMBLY="gis-plugin-assemblies-${GIS_PLUGINS_VERSION}.zip"

# Unzip it ate the right place !
unzip ${GIS_PLUGINS_ASSEMBLY} -d ${HOP_HOME}/plugins/

To upgrade the plugins, delete files you added before and start a fresh installation.

Oracle JDBC usage

If you plan to connect to an Oracle database, add needed jars in lib folder of PDI :

  • ${HOP_HOME}/lib/ojdbc11.jar
  • ${HOP_HOME}/lib/orai18n.jar

You can get them here

Building a docker image

See dedicated page

Testing the plugins

See dedicated page

Using the plugins

See dedicated page

You will find new elements in Geospatial's directory :

  • Geospatial Group by
  • GIS File output
  • GIS File input
  • Geoprocessing
  • Geometry information
  • Coordinate system operation
  • Spatial relationship and proximity

With a french locale :

Code formatting

The java code for this project conforms to Google's code styleguide. The spotless maven plugin deals with this aspect:

# Formatting check
mvn spotless:check

# Formatting (to be done before any commit)
mvn spotless:apply

Debugging

You can debug plugins remotely in Hop GUI:

# Set debugging options (or uncomment appropriate line in hop-gui.sh to keep it active)
export HOP_OPTIONS="-Xmx2048m -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"

# Launch Hop Gui as usual
./hop-gui.sh

You will see a message like Listening for transport dt_socket at address: 5005 in your terminal.

Next, attach the debugger in your favorite IDE. Example launch.json in Visual Studio Code with Debugger for Java extension:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "java",
            "name": "Attach java debugger",
            "projectName": "hop-gis-plugins",
            "request": "attach",
            "hostName": "localhost",
            "port": 5005
        }
    ]
}

You are now able to inspect variables values at breakpoints you have set, analyse call stack and so.

Contributing

See dedicated page

LICENSE

This extension is licensed under GNU Library or "Lesser" General Public License (LGPL).

Contributors :

Our company

Atol Conseils et Développements Follow us on twitter @atolcd

hop-gis-plugins's People

Contributors

bamaer avatar cvagner avatar jtridard avatar mattcasters avatar pydirand avatar scali avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hop-gis-plugins's Issues

[Feature] : Support for GeoParquet - Geospatial data in Apache Parquet

What would you like to be added:

Apache Hop support reading and writing in Apache Parquet files.

A nice-to-have feature would be to be able to read/write geospatial fields in GeoParquet

Alternatives:

No alternative in Apache Hop, need to write the geospatial information using GDAL binaries

Has the feature been requested before?

No

If the feature request is approved, would you be willing to submit a PR?

No, because I don't have the skills to meet this demand.

Clean up old dependencies

There are some older dependencies in the Maven pom, for example from Pentaho and spring-plugins, odjbc6 and SWT.

[Feature] Add a preview rows feature to GIS File Input

What would you like to be added:

The transform GIS File Input doesn't support the data previsualization.

The feature would be very similar of what is actually possible for transforms like CSV file input or Text file input with the Preview rows feature.

Alternatives:

It is possible to view the data by runnning the pipeline.

Has the feature been requested before?

No

If the feature request is approved, would you be willing to submit a PR?

Yes, but no time to work on it soon

[Feature] Allow Metadata injection in GIS File Input transform

What would you like to be added:

The transform GIS File Input doesn't support Metadata injection, especially Filename parameter, that could be very usefull to write reusable pipelines.

The feature would be very similar of what is actually possible for transforms like CSV file input or Text file input

Alternatives:

No alternative is possible if we want to use metadata injection

Has the feature been requested before?

No

If the feature request is approved, would you be willing to submit a PR?

Yes

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.