Git Product home page Git Product logo

wazicloud's Introduction

travis

WAZIUP platform

The WAZIUP platform is an IoT Big Data platform. It allows to create IoT applications and deploy them both in the Cloud and in the local gateway.

For information and documentation: https://www.waziup.io

Install

To install the platform, first clone it:

git clone https://github.com/Waziup/WaziCloud.git
cd WaziCloud

Then pull the images and run it:

sudo chmod 777 data/* -R
export PLATFORM_VER=latest
docker-compose pull
docker-compose build dashboard
docker-compose -f docker-compose.yml -f docker-compose-first-run.yml up

This will create the default users and configuration for Waziup. Once the boot up finished, you can then access Waziup platform on http://localhost:3000

To start the platform in the future, just type:

docker-compose up

You can now access the Waziup dashboard on http://localhost Keycloak is available at http://localhost:8080. Default login/password in Keycloak is admin/admin. You need to change it!

Develop

To get the source code for each submodules, you need to clone with --recursive:

# clone the WaziCloud with submodules
git clone --recursive https://github.com/Waziup/WaziCloud.git
cd WaziCloud
git submodule update --remote --recursive
docker-compose build

Tests

You can run the test campain like this:

./run_tests.sh

Or view the API documentation:

firefox localhost:800/docs

Backup

mysqldump is used to backup user database.

mysqldump -h dev.waziup.io -P 3306 -u root -proot_password --all-databases > /var/backups/mysqlbackups/`date +"%m-%d-%y"`

Restore the data:

mysql -u root -proot_password -h 127.0.0.1 <  /var/backups/mysqlbackups/`date +"%m-%d-%y"`

mongodump is used to backup sensor data:

mongodump --host dev.waziup.io --port 27017 --out /var/backups/mongobackups/`date +"%m-%d-%y"`

This command restores it in the local instance:

mongorestore /var/backups/mongobackups/`date +"%m-%d-%y"`

Debug

To export the keycloak configuration, run:

docker-compose run --entrypoint "/opt/jboss/tools/docker-entrypoint.sh -b 0.0.0.0 -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=/opt/jboss/keycloak/standalone/data/" keycloak
#Stop keycloak when start-up is finished
cd data/keycloak
mv master-realm.json master-users-0.json waziup-realm.json waziup-users-0.json ../../keycloak/imports

Rebuild Keycloak image:

docker-compose build keycloak
docker-compose push keycloak

Copyright

Copyright Waziup consortium 2018.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

wazicloud's People

Contributors

24inch avatar arbiswas avatar bures avatar cdupont avatar diopbabacar avatar diopbabacaredu avatar espritdg avatar fizarbd avatar henokzion avatar j-forster avatar joelbudu avatar malishahi avatar ousmaneo avatar pandaconstantin avatar sabrine2020 avatar zeisberg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wazicloud's Issues

Historical data API

Currently there is no interface to retrieve historical data from Waziup.
How does this interface should look like? How to retrieve the data from MongoDB?

Error "System error: mkdir /sys/system.slice: operation not permitted" when pushing and app

Receiving the error:

$ git push deis master
Error response from daemon: Cannot start container d4462fbf73b850edc23d0d841572bb6a5f50c24079281a99a7dbae662987dafe: [8] System error: mkdir /sys/system.slice: operation not permitted
To ssh://[email protected]:2222/blonde-odometer.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://[email protected]:2222/blonde-odometer.git'

Solution seems to upgrade the version of CoreOS:
deis/deis#5067
deis/deis#5074

Security and identification

Currently there is no identification/security in the platform.
Anybody can push data or an application.
How to implement identification and security?

Gateway management

At the moment there is no gateway management.
There is a small API in the controller which is not completed.
How should the gateway manager component should work? Based on which technology?

HTTP Proxy for rewrite rules

Currently the platform is configured with the following configuration for routing requests from Internet

->Internet user reachs *.waziup.io:80
->OpenStack HA Proxy forwards *.waziup.io:80 on 192.168.10.101:30999
->Kubernetes Ingress controller running in a pod on 192.168.10.101:30999 forwards to corresponding pod
-> pod receives the request

The Ingress controller allows to expose pods on nice URL such as controller.waziup.io, using a configuration file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: controller
  namespace: waziup
spec:
  rules:
  - host: controller.waziup.io
    http:
      paths:
      - path: /
        backend:
          serviceName: controller
          servicePort: 8080

This simple language allows to expose pods. But it doesn't allow to make rewrite rules, as needed for the waziup API. #23

Incompatible External Load Balancer

The external load balancer (LBaaS) in our version of OpenStack is incompatible with current version of Kubernetes.
Kubernetes version is 1.4, requiring LBaaS v2, but our version of OpenStack is Juno 2014.2.2, coming with LBaaS v1.

see http://docs.openstack.org/developer/magnum/dev/kubernetes-load-balancer.html

It is thus not possible to use Service type = LoadBalancer in Kubernetes.
Service types = ClusterIP or Service type = NodePort with an Ingress Controller works though.
The problem is for DEIS, which comes with a router which needs to connect with the external load balancer. This router creates subdomains such as myapp.waziup.io for each application pushed.

Controller data API

Currently the controller uses the following entry points for the data API (available here):

GET /data/observations: get all observations
POST /data/observations: create a new observation
GET /data/observations/{id}: get one observation

The data format is enforced to be IoT-Lite:

  {
    "id": "TemperatureSensor1",
    "location": {
      "coordinates": [[0], [0]],
      "type": "Point"
    },
    "measure": {
      "metadata": {
        "dimension": "temperature",
        "timestamp": "2016-06-08T18:20:27.873Z",
        "unit": "Celcius"
      },
      "value": 27
    },
    "platform": "UGB Field",
    "type": "SensingDevice"
  }

This interface is quite limited:

  • the entity/sensor is not mentioned in the URL, that makes browsing sensors a bit awkward
  • the attribute/measurement is not mentioned in the URL, that makes browsing sensors attributes also awkward
  • Orion has some more features such as search strings which are not reflected as well

Data subscription

Currently the WAZIUP internal Orion component supports subscription.
However the feature should be tested and exposed on the controller.
The API is already implemented but not connected to Orion.

Spec: platform admin pages

The platform should have an admin web page where you can see and manage:

  • your user profile,
  • your sensors,
  • your gateways,
  • your apps.

Automatic restart for components

Currently if one of Waziup components crashes, it is not restarted.
Apparently Kubernetes deployment/replication controller allows to do that.

controller API gateway

The role of the controller is to expose the external API and forward the requests to internal services.
Currently the controller is implemented as a Python program doing the translation.
However it would be simpler and more modular/future proof to implement it with an HTTP proxy like nginx or Apache tomcat.

Some links on the API gateway pattern and implementation:
http://microservices.io/patterns/apigateway.html
https://www.nginx.com/blog/building-microservices-using-an-api-gateway/

Currently the controller is also enforcing the IoT-lite format on the API, before translating the data to Orion.
This function should be moved to a separate wrapper located in the "broker" pod.

Bare-metal Waziup platform

The current Waziup platform is using a complex stack: OpenStack/Kubernetes/Deis.
Can we make it simpler?

App management

Currently, the components of the platform are inserted on Kubernetes container manager.
The plan is to use DEIS for the apps management. DEIS is able to compile and deploy applications in many languages. DEIS works on top of Kubernetes.
Currently, the apps are inserted manually in kubernetes or through DEIS. This process should be integrated with Waziup.
There is a small API on the controller side.

Platform often not responding

The platform is often non responsive.
For example this query to the broker sometimes hangs:

curl http://217.77.95.64:30026/version

When is hangs all other services in Kubernetes are as well not responsive.

setrlimit failed in Ingress Controller

pod contrib-nginx-ingress keep saying this:

 2016-11-06T17:22:32.051166031Z W1106 17:22:32.050907       1 utils.go:231] system net.core.somaxconn=128. Using NGINX default (511)
2016-11-06T17:22:32.214015127Z I1106 17:22:32.213796       1 command.go:76] change in configuration detected. Reloading...
2016-11-06T17:22:32.324808484Z 2016/11/06 17:22:32 [alert] 13120#13120: setrlimit(RLIMIT_NOFILE, 131072) failed (1: Operation not permitted)
2016-11-06T17:22:32.325651783Z 2016/11/06 17:22:32 [alert] 13121#13121: setrlimit(RLIMIT_NOFILE, 131072) failed (1: Operation not permitted)
2016-11-06T17:22:32.327079687Z 2016/11/06 17:22:32 [alert] 13122#13122: setrlimit(RLIMIT_NOFILE, 131072) failed (1: Operation not permitted)
2016-11-06T17:22:32.396853487Z 2016/11/06 17:22:32 [alert] 13123#13123: setrlimit(RLIMIT_NOFILE, 131072) failed (1: Operation not permitted) 

Documentation website

Create a website for platform documentation.
It should use a static website generator such as Hugo.
All content should be stored on github. Articles and documentation should be written in markdown format.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.