pablocasares / community-stack Goto Github PK

Wizzie Community Stack (WCS) is a lightweight version of Wizzie Data Platform (WDP).

Wizzie Data Platform, or WDP, is a blazingly fast, highly scalable, end-to-end solution for real-time processing, storage and visualization of any data type and format.

WCS is composed by:

• Core : Composed by Kafka, a distributed streaming platform, and Zookeeper, a highly reliable distributed coordinator.

• Data normalization tool : Composed by Normalizer, a stream processing engine based on Kafka Streams.

• Data enrichment tool : Composed by Enricher, a stream enriching engine based on Kafka Streams

• Correlation engine : Composed by ZZ-Cep, a complex event processing engine based on Siddhi and Kafka.

• Analytic data store : Composed by Druid, a high performance analytics data store for event-driven data.

• Data visualization : Composed by Wizz-Vis, an analytics platform for time series metrics using Druid.

Next you can see the WCS architecture:

You can use our Prozzie like data source. You can find more information about Prozzie in its documentation.

We recommend a machine with at least:

• 4 CPU
• 16 GB RAM
• 30 GB disk

In order to install WCS you only need download the latest release or clone the repo and run setups/linux_setup.sh script that will guide you through the entire installation.You can also use the next command if you don't want to waste your time messing with git you can run next command as root user or sudo command:

sudo bash -c "$(curl -L --header 'Accept: application/vnd.github.v3.raw' 'https://api.github.com/repos/wizzie-io/community-stack/contents/setups/linux_setup.sh?ref=1.1.0')"

The setup script will install Docker and Docker Compose and some necessary tools like curl and net-tools.

Next you can find information about WCS's configuration, execution and kafka operations.

When you install WCS, the installer will create a .env file located in $PREFIX/etc/wcs directory. In the .env file you will find three sections with some vars in it. These sections are:

• MANDATORY: You need to configure this vars before run the application.
• OPTIONAL: You can configure this vars, if you want to change some configurations or enable some features.
• STATIC: Change this variables is dangerous, you could break some components of WCS! Change it at your own risk

Note: On some linux distribution you need to open the iptables on the machine, to send data from outside to Kafka broker (port: 9092).

You can use a small tool named wcs. It's simply a lightweight CLI to do WCS and Kafka operations.

Now, you only need to run wcs up -d to start all components of WCS.

If you can check components, using wcs compose ps

root@kubeubuntu:~/community-stack# wcs compose ps
                   Name                                 Command               State               Ports
-------------------------------------------------------------------------------------------------------------------
wizziecommunitystack_druid-broker_1          /bin/sh -c druid-start.sh        Up       0.0.0.0:8080->8080/tcp
wizziecommunitystack_druid-coordinator_1     /bin/sh -c druid-start.sh        Up
wizziecommunitystack_druid-historical_1      /bin/sh -c druid-start.sh        Up
wizziecommunitystack_druid-init_1            /bin/bash -c /bin/bash -c  ...   Exit 0
wizziecommunitystack_druid-middlemanager_1   /bin/sh -c druid-start.sh        Up
wizziecommunitystack_druid-overlord_1        /bin/sh -c druid-start.sh        Up       0.0.0.0:8084->8084/tcp
wizziecommunitystack_enricher_1              /bin/sh -c exec /bin/enric ...   Up
wizziecommunitystack_kafka_1                 start-kafka.sh                   Up       0.0.0.0:9092->9092/tcp
wizziecommunitystack_normalizer_1            /bin/sh -c exec /bin/norma ...   Up
wizziecommunitystack_postgres_1              docker-entrypoint.sh postgres    Up       5432/tcp
wizziecommunitystack_redis_1                 docker-entrypoint.sh redis ...   Up       6379/tcp
wizziecommunitystack_sidekiq_1               scripts/docker-entrypoint- ...   Up
wizziecommunitystack_wizz-vis_1              scripts/docker-entrypoint- ...   Up       0.0.0.0:3000->3000/tcp
wizziecommunitystack_zookeeper_1             /docker-entrypoint.sh zkSe ...   Up       2181/tcp, 2888/tcp, 3888/tcp
wizziecommunitystack_zz-cep_1                /bin/sh -c exec /bin/cep-s ...   Up

If you need stop and, later start, WCS you can use wcs stop and wcs start.

You can use wcs down to destroy completly WCS. This operation is dangerous, you will delete database and kafka topics.

The wcs CLI, provides kafka commands in order to produce, consume and check topics. For example:

You can check all available topics running next command:

$ wcs kafka topics --list

You can consume messages from any kafka's topic using next command:

$ wcs kafka consume <topic>

You can produce messages to any kafka's topic using next commmand:

$ wcs kafka produce <topic>

WCS doesn't provide any mechanism to uninstall wcs. If you want to uninstall wcs you must do manually.

When you install WCS first time you must specify a prefix, by default It's /usr/local. In this prefix, WCS creates following directories:

• ${PREFIX}/etc/wcs : Contains configuration files.
• ${PREFIX}/share/wcs : Contains files about WCS cli and docker compose files.
• ${PREFIX}/bin : Contains the symbolic link to wcs command.

To uninstall WCS you must follow next steps:

1. Do wcs down to stop WCS and delete all databases and kafka topics.
2. Remove ${PREFIX}/share/wcs and ${PREFIX}/etc/wcs folders.
3. Remove symbolic link named wcs in ${PREFIX}/bin directory. If bin folder It isn't necessary, then you can delete it.

You can find ports binding information in next table:

Check next links to know about awesome Wizzie components and check documentation.

• Normalizer docs
• Enricher docs
• ZZ-CEP docs
• WizzVis API docs
• Druid Kafka Indexing tasks docs
• Druid Querying docs
• Kafka Operations docs