apache / streampipes Goto Github PK

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

Home Page: https://streampipes.apache.org

License: Apache License 2.0

Java 57.88% Shell 0.61% HTML 9.08% Dockerfile 0.17% JavaScript 0.14% TypeScript 22.26% CSS 0.01% Python 2.39% SCSS 3.16% Makefile 0.02% Jupyter Notebook 3.87% Go 0.41%

iot analytics edge stream-processing iiot self-service hacktoberfest

streampipes's Introduction

Self-Service Data Analytics for the (Industrial) IoT

StreamPipes is a self-service (Industrial) IoT toolbox to enable non-technical users to connect , analyze and explore IoT data streams.

About Apache StreamPipes
User interface
Installation
Documentation
Building StreamPipes
Pipeline Elements
Extending StreamPipes
Bugs and Feature Requests
Get help
Contribute
Feedback
License

About Apache StreamPipes

Apache StreamPipes makes industrial data analytics easy!

StreamPipes is an end-to-end toolbox for the industrial IoT. It comes with a rich graphical user interface targeted at non-technical users and provides the following features:

Quickly connect >20 industrial protocols such as OPC-UA, PLCs, MQTT, REST, Pulsar, Kafka and others.
Create data harmonization and analytics pipelines using > 100 algorithms and data sinks to forward data to third-party systems.
Use the data explorer to visually explore historical data with many widgets tailored for time-series data.
A live dashboard to display real-time data from data sources and pipelines, e.g., for shopfloor monitoring.

StreamPipes is highly extensible and includes a Java SDK to create new pipeline elements and adapters. Python support is available in an early development stage - stay tuned!
Pipeline elements are standalone microservices that can run anywhere - centrally on your server or close at the edge. You want to employ your own machine learning model on live data? Just write your own data processor and make it reusable as a pipeline element.

Besides that, StreamPipes includes features for production deployments:

Assign resources such as pipelines, data streams and dashboards to assets for better organization
Monitoring & metrics of pipelines and adapters
Built-in user and access rights management
Export and import resources

User interface

Connect data from an OPC-UA server following a three-step configuration process:

Create a pipeline to detect a continuous decrease using a trend detection data processor and a Notificationsink:

Visually analyze data using the data explorer:

Installation

The quickest way to run StreamPipes including the latest extensions (adapters, pipeline elements) is by using our Docker-based installation & operation options, namely:

StreamPipes Compose - The User's Choice
StreamPipes CLI - The Developer's Favorite
StreamPipes k8s - The Operator's Dream

Important

StreamPipes CLI & k8s are highly recommended for developers or operators. Standard users should stick to StreamPipes Compose.

Please follow the instructions provided in the corresponding README.md to get started.

For a more in-depth manual, read the installation guide.

Note

TL;DR: Download the latest release, switch to the installer/compose directory and run docker-compose up -d.

Documentation

The full documentation is available here.

Quick Links:

Building StreamPipes

To properly build the StreamPipes core, the following tools should be installed:

Prerequisites

Java 17 JDK (We officially only support Java 17, JDKs above 17 might work as well, but we don't provide any guarantee)
Maven (tested with 3.8)
NodeJS + NPM (tested with v12+/ v6+)
Docker + Docker-Compose

Building

To build the core project, do the following:

    mvn clean package

To build the ui, switch to the ui folder and perform the following steps:

    npm install
    npm run build

Starting

To start StreamPipes, run docker-compose up --build -d from the root directory.

You can also use the installer or CLI as described in the Installation section.

Pipeline Elements

StreamPipes includes a repository of extensions for adapters and pipeline elements:

Connect adapters for a variety of IoT data sources as well as
Data Processors and Data Sinks as ready-to-use pipeline elements.

The source code of all included pipeline elements and adapters can be found here.

Extending StreamPipes

You can easily add your own data streams, processors or sinks. A Java-based SDK can be used to integrate your existing processing logic into StreamPipes. Pipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.

👉 Check our developer guide.

Bugs and Feature Requests

If you've found a bug or have a feature that you'd love to see in StreamPipes, feel free to create an issue on GitHub:

👉 Bugs 👉 Feature requests

Get help

If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:

👉 Mailing Lists

Or directly subscribe to [email protected]!

👉 And don't forget to follow us on Twitter!

Contribute

We welcome all kinds of contributions to StreamPipes. If you are interested in contributing, let us know! You'll get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox.

Here are some first steps in case you want to contribute:

Subscribe to our dev mailing list [email protected]
Send an email, tell us about your interests and which parts of StreamPipes you'd like to contribute (e.g., core or UI)!
Ask for a mentor who helps you to understand the code base and guides you through the first setup steps
Find an issue on GitHub which is tagged with a good first issue label
Have a look at our developer wiki to learn more about StreamPipes development.

Have fun!

Feedback

We'd love to hear your feedback! Subscribe to [email protected]

License

Apache License 2.0

streampipes's People

Contributors

Stargazers

Watchers

Forkers

dominikriemer mmbrantner caidezhi 012huang zhangxin1988 ambition119 sureshannapureddy jlleitschuh harmanpa trucnguyenlam ebida alexkarezin biaoma-ty dongbin86 selectbook daniel-gomm el-sherif dglttr r0h17 heymarco rquintin isabella232 theshadow89 mohanvive liuliuball45 wabiwabo kmdn pcnfernando charukak coderlinj5945 rwindwh apengda vesense sandeepk-dev mfaizanse samarth08 anushkrishnav neuyilan yangdh1225 ekmixon zhangjun0x01 gaojunhello patrickraoulphilipp ziyangx alany9552 sangjiexun hrushi20 jens5588 hutcheb laopeng2021 m-martin-j sonatype-lift-apache wipatrick pjfanning tenthe jbampton helenparr shell22 devinbost igit-cn lethetann hemersonrafael krausstt rullo16 cckmit kulbachcedric robertindie dprophet simaoppcastro carrotsaltdog luoluoyuyu crazy-pea4 sveno3 bossenti openio-net mmack nyodbikash sagarbk0 ctlove0523 aamirxshaikh ndace-hash grainier-macrometa grainier jadireddi wolfboys parthsali talhahere12 steveyurongsu harry262530 shyding deexidee zhenlu-wu rohankumardubey budhirajachinmay elgueromiranda osksacho samkenxstream bruyss liuxiaocs7 kshitiz-mhto

streampipes's Issues

Refactor project namespace in incubator-streampipes-extensions

Imported from Jira STREAMPIPES-3. Original Jira may contain additional context.
Reported by: riemer.
Subtask of issue #169

User Feedback

Feedback button (top right) only opens a blank window (please see attachment)

Imported from Jira STREAMPIPES-83. Original Jira may contain additional context.
Reported by: tex.

UI is not open source

I can not find UI code source? UI is not open source?Where is UI code source?thank you

Refactor project namespace in incubator-streampipes-examples

Imported from Jira STREAMPIPES-4. Original Jira may contain additional context.
Reported by: riemer.
Subtask of issue #169

Pipeline Log Button

The Pipeline Log button has no functionality

Imported from Jira STREAMPIPES-92. Original Jira may contain additional context.
Reported by: tex.

Cannot add Pipeline to Category

Cannot add an existing pipeline to a pipeline catergory (see attachment)

Imported from Jira STREAMPIPES-90. Original Jira may contain additional context.
Reported by: tex.

Data Sink for MQTT

Add a datasink for MQTT to module streampipes-sinks-broker-jvm.

Imported from Jira STREAMPIPES-71. Original Jira may contain additional context.
Reported by: zehnder.

Pipeline cannot be saved

Reproduce:

Create Pileline
Add additional Element
Delete Element again

-> Pipeline cannot be saved

Imported from Jira STREAMPIPES-86. Original Jira may contain additional context.
Reported by: tex.

Error when installing W10

Dear team,

When I try to install ".bat" in W10 I receive the following error:

C:\Users\TA-INDUS-C41581\Desktop\streampipes-installer-master\streampipes\windows10>streampipes.bat start
& : No se puede cargar el archivo
C:\Users\TA-INDUS-C41581\Desktop\streampipes-installer-master\streampipes\windows10\streampipes.ps1. El archivo
C:\Users\TA-INDUS-C41581\Desktop\streampipes-installer-master\streampipes\windows10\streampipes.ps1 no está firmado
digitalmente. No se puede ejecutar este script en el sistema actual. Para obtener más información acerca de la
ejecución de scripts y la configuración de la directiva de ejecución, consulta about_Execution_Policies en
https:/go.microsoft.com/fwlink/?LinkID=135170.
En línea: 1 Carácter: 3

& 'C:\Users\TA-INDUS-C41581\Desktop\streampipes-installer-master\stre ...

+ CategoryInfo          : SecurityError: (:) [], PSSecurityException
+ FullyQualifiedErrorId : UnauthorizedAccess

I have tried to modify this policy but I receive this error:
Set-ExecutionPolicy : Windows PowerShell actualizó la directiva de ejecución correctamente, pero el valor queda
invalidado por una directiva definida en un ámbito más específico. Debido a esta invalidación, el shell mantendrá la
directiva de ejecución efectiva actual: RemoteSigned. Escriba "Get-ExecutionPolicy -List" para ver la configuración de
la directiva de ejecución. Para obtener más información, vea "Get-Help Set-ExecutionPolicy".
En línea: 1 Carácter: 1

Set-ExecutionPolicy "Unrestricted" -Scope CurrentUser -Confirm:$false

  + CategoryInfo          : PermissionDenied: (:) [Set-ExecutionPolicy], SecurityException
  + FullyQualifiedErrorId : ExecutionPolicyOverride,Microsoft.PowerShell.Commands.SetExecutionPolicyCommand

Could you help me?

REgards

Deploy snapshots to Docker Hub

Snapshots of the backend and pipeline element containers should be published to Docker Hub to ease development of StreamPipes

Imported from Jira STREAMPIPES-10. Original Jira may contain additional context.
Reported by: riemer.

Refactor project namespace from org.streampipes to org.apache.streampipes

Imported from Jira STREAMPIPES-1. Original Jira may contain additional context.
Reported by: riemer.

Update POM files in streampipes module

Imported from Jira STREAMPIPES-31. Original Jira may contain additional context.
Reported by: riemer.
Subtask of issue #182

Can't connect Pipeline elements

Reproduce:

Create Pileline
Leave Editor (e.g. go to Data explorer)
Go back to Editor

-> sometimes elements cannot conntected anymore (see attachment)

Imported from Jira STREAMPIPES-85. Original Jira may contain additional context.
Reported by: tex.

Update POM files to meet ASF requirements

Imported from Jira STREAMPIPES-30. Original Jira may contain additional context.
Reported by: riemer.

Dashboard widget Edit

When pressing the Edit button of a Dashboard Widget only a blank window opens

Imported from Jira STREAMPIPES-97. Original Jira may contain additional context.
Reported by: tex.

Considering integrating StreamPipes and DSS?

Apache StreamPipes is a very great project in the field of tream processing frameworks, and I think it is a good way to enhance the influence of our two projects is by integrating StreamPipes with DataSphere Studio.
DataSphere Studio is a one-stop data application development and management portal open source by WeBank, which meets the needs of the entire process of data application development from data exchange, desensitization and cleaning, analysis and mining, quality inspection, visual display, regular scheduling to data output.
Github address: https://github.com/WeBankFinTech/DataSphereStudio

Data Sink for MQTT

Add a datasink for MQTT to module streampipes-sinks-broker-jvm.

Imported from Jira STREAMPIPES-71. Original Jira may contain additional context.
Reported by: zehnder.

Use internal file service in file adapters

As described in the thread: Re: STREAMPIPES-75: Extend data lake sink to store images on the mailing list. The uploaded files of the file stream and file set adapter should not be stored in the local container, instead they should be stored in the centralized file service..

Imported from Jira STREAMPIPES-80. Original Jira may contain additional context.
Reported by: zehnder.

Remove dependency to kafka-rest

Kafka-rest is licensed under the Confluent Community license. Although not being used in the code directly, Kafka-rest ist currently required to get live data in the data stream preview. We need to find another way to show real-time previews.

Imported from Jira STREAMPIPES-9. Original Jira may contain additional context.
Reported by: riemer.

Sink: HDFS

Description

Create a sink to store events into HDFS.

Input Requirements

None

Configurations

Host / Port
Folder Route
User / Password (optional)
Parameter for Buckets

Imported from Jira STREAMPIPES-49. Original Jira may contain additional context.
Reported by: zehnder.

Update POM files in streampipes-examples module

Imported from Jira STREAMPIPES-52. Original Jira may contain additional context.
Reported by: riemer.
Subtask of issue #182

Refactor project namespace in incubator-streampipes

Imported from Jira STREAMPIPES-2. Original Jira may contain additional context.
Reported by: riemer.
Subtask of issue #169

Notification Sink Placeholder

The notification sink placeholder are not working. #propertyName# are not replaced by the value of the property.

Imported from Jira STREAMPIPES-93. Original Jira may contain additional context.
Reported by: tex.

Dashboard Widget without Parameters

It should not be possible to create a new Dashboard widget unless all required paraments/configurations are set. If a widget is created without all the required paraments, the dashboard will 'crash'.

Imported from Jira STREAMPIPES-98. Original Jira may contain additional context.
Reported by: tex.

Quick Edit Parameter Update

Element parameters are not saved when edit with quick edit, also __ with pressing the save button

Imported from Jira STREAMPIPES-91. Original Jira may contain additional context.
Reported by: tex.

Update POM files in streampipes-extensions module

Imported from Jira STREAMPIPES-32. Original Jira may contain additional context.
Reported by: riemer.
Subtask of issue #182

Test Issue

This issue serves to identify the current issue ID ;-)

Consider domain property for live preview

In the live preview after starting an adapter, images and timestamps should be displayed in a human readable way.

Imported from Jira STREAMPIPES-76. Original Jira may contain additional context.
Reported by: zehnder.

Exception in adpaters when the date format is incorect

When a user marks a property as a timestamp, there is the possibility to provide a regex to convert the timestamp represented as a String into the internal representation of a UNIX timestamp in milliseconds. As long as the date in the events is formatted according to the regex this works properly. When there is an error in the data an exception is thrown and the adapter stops producing data. In this case the adapter should ignore the event instead and continue with the next event.

Imported from Jira STREAMPIPES-57. Original Jira may contain additional context.
Reported by: zehnder.

Confluence wiki for StreamPipes

Ask infra to setup the new Confluence wiki for StreamPipes targeted at contributors.

Imported from Jira STREAMPIPES-21. Original Jira may contain additional context.
Reported by: riemer.

Dashboard without name

It should not be possible to create a dashboard without a name

Imported from Jira STREAMPIPES-94. Original Jira may contain additional context.
Reported by: tex.

Extract utils class for mqtt related configurations

We have multiple adapters that connect to a MQTT broker. Therefore, they need the same configuration parameters.

To avoid having to implement these for every adapter we need a Utils class which encapsulates the configuration.

Imported from Jira STREAMPIPES-69. Original Jira may contain additional context.
Reported by: zehnder.

Password field not full Widht

The passwordfield in Adapter configuration has not full widht (see attachment)

Imported from Jira STREAMPIPES-87. Original Jira may contain additional context.
Reported by: tex.

Rename File-Download Files

Currently there is a static file name containing a space character is created. Change this to a more meaningful name (e.g. date_indexname) with no special characters

Imported from Jira STREAMPIPES-47. Original Jira may contain additional context.
Reported by: zehnder.

Dashboard Widget without required Data

should not be possible to create a dashboard widget for a data stream that does not contain the required data.

For example, select a data stream without an image, create an image widget and save the dashboard -> the dashboard "crashes".

Imported from Jira STREAMPIPES-96. Original Jira may contain additional context.
Reported by: tex.

Dashboard open in Window

The 'Window' button of a 'My Dashboard' does not work.

Imported from Jira STREAMPIPES-95. Original Jira may contain additional context.
Reported by: tex.

Fix Docker repo URLs in installer CLI

The env file of the CLI module of the installer links to some internal repos and need to be changed to a public Docker registry.

Imported from Jira STREAMPIPES-29. Original Jira may contain additional context.
Reported by: riemer.

Links are out of date

Links at "User Preferences" -> "Info" should be updated
Link "User Preferences" -> "Documentation" is not working

Imported from Jira STREAMPIPES-84. Original Jira may contain additional context.
Reported by: tex.

Add processor to filter for numerical and text values

Currently, there are Numerical Filter and Text Filter. If a user wants to apply both, i.e. filter a field for a given string AND subsequently filter a numeric field based on a given threshold, these two PE's need to be concatenated to achieve this goal. In order to improve this common scenario there should be a new processor "Numerical Text Filter" to allow doing this in one step.

Imported from Jira STREAMPIPES-72. Original Jira may contain additional context.
Reported by: wiener.

Check licenses in UI module

Do a complete license check of all packages used in the StreamPipes UI module

Imported from Jira STREAMPIPES-5. Original Jira may contain additional context.
Reported by: riemer.

Reordering 'My pipelines' by clicking on e.g. 'Name' resets the 'Stop' button to a 'Start' button even though the pipeline keeps running

Processor: Merge two data streams by timestamp

A user can use this processor to merge two data streams by their timestamp.

The processor should be implemented in Java to also run on edge devices.

Imported from Jira STREAMPIPES-59. Original Jira may contain additional context.
Reported by: zehnder.

Frontend copy past mistake

Perhaps this copy past error can be corrected:
The first Reference Point Latitude should be Reference Point Longitude.

Move CI to Apache infrastructure

Move the currently used Gitlab CI setup to Jenkins or Github Actions to make CI work in the Apache infrastructure

Imported from Jira STREAMPIPES-18. Original Jira may contain additional context.
Reported by: riemer.

Update RDF4J and empire dependencies

Bump versions of rdf4j and empire to the latest release.

Imported from Jira STREAMPIPES-82. Original Jira may contain additional context.
Reported by: riemer.

Dashboard without name

It should not be possible to create a dashboard without a name

Imported from Jira STREAMPIPES-94. Original Jira may contain additional context.
Reported by: tex.

Could not stop all pipeline elements

Hi, I have the problem that I can't stop a pipeline. Most likely I just used the Pipeline Editor wrong.
This is my pipeline:

This is the error when I try to stop the pipeline:

Support for multiple properties

According to dev Email: "Add support for multiple properties in PLC4X S7 Adapter"

Imported from Jira STREAMPIPES-22. Original Jira may contain additional context.
Reported by: tex.

Comparison to StreamSets

Hey, so I stumbled across this project and looked interesting, so digging around I see it's just a graphical editor on top of a bunch of Java SDKs, so I am really curious what the differences might be and why someone may chose your product over another?

Pipeline Update blank info box

When you save an updated pipeline, an empty info box is displayed in the top right corner (see attachment).

Imported from Jira STREAMPIPES-89. Original Jira may contain additional context.
Reported by: tex.