Git Product home page Git Product logo

gateway's Introduction

WebThings Gateway

Build Status codecov dependencies devDependencies license

Web of Things gateway.

Installation

  • If you have a Rasberry Pi, the easiest way to use the gateway is to download and flash a pre-built software image to an SD card.
  • If you prefer to use Docker, we have a prebuilt Docker image available here, for both ARM and amd64. You can also build your own image from this repository.
  • On Fedora, Debian, Raspberry Pi OS, or Ubuntu, you can install the relevant .rpm or .deb package from the releases page.
  • On Arch Linux, you can install the webthings-gateway AUR package. The PKGBUILD for this package can also be seen here.
  • Otherwise, you can build it from source yourself (see below).

Documentation

Community

Prerequisites for Building

Install OS

(If you're just installing on your PC, you can skip this step).

If you're installing on a Raspberry Pi then you may need to set up the OS on the Raspberry Pi first. See here for instructions.

Install Dependencies

Ubuntu/Debian Linux

$ sudo apt update
$ sudo apt install \
    autoconf \
    build-essential \
    curl \
    git \
    libbluetooth-dev \
    libboost-python-dev \
    libboost-thread-dev \
    libffi-dev \
    libglib2.0-dev \
    libpng-dev \
    libudev-dev \
    libusb-1.0-0-dev \
    pkg-config \
    python-six \
    python3-pip
$ sudo -H python3 -m pip install git+https://github.com/WebThingsIO/gateway-addon-python#egg=gateway_addon

Fedora Linux

$ sudo dnf --refresh upgrade
$ sudo dnf group install "C Development Tools and Libraries"
$ sudo dnf install \
    autoconf \
    bluez-libs-devel \
    boost-devel \
    boost-python2-devel \
    boost-python3-devel \
    curl \
    git \
    glib2-devel \
    libffi-devel \
    libpng-devel \
    libudev-devel \
    libusb1-devel \
    pkgconfig \
    python2-pip \
    python3-pip
$ sudo -H python2 -m pip install six
$ sudo -H python3 -m pip install git+https://github.com/WebThingsIO/gateway-addon-python#egg=gateway_addon

macOS

$ brew update
$ brew install \
    autoconf \
    libffi \
    pkg-config
$ sudo -H python2 -m pip install six
$ sudo -H python3 -m pip install git+https://github.com/WebThingsIO/gateway-addon-python#egg=gateway_addon

Install Node.js

nvm (Recommended)

nvm allows you to easily install different versions of node. To install nvm:

$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

Reinitialize your terminal session.

$ . ~/.bashrc

Manual

(If you already installed node via nvm you can skip this step)

Follow the directions from NodeJS to install on your platform.

Set up Bluetooth permissions (Linux only)

The following is required in order to let node and python3 use the Bluetooth adapter.

$ sudo setcap cap_net_raw+eip $(eval readlink -f `which node`)
$ sudo setcap cap_net_raw+eip $(eval readlink -f `which python3`)

Download and Build Gateway

  • Clone the GitHub repository (or fork it first):

    $ git clone https://github.com/WebThingsIO/gateway.git
    
  • Change into the gateway directory:

    $ cd gateway
    
  • If you have chosen to install nvm above, install and use an LTS version of node and then set the default version. The .nvmrc file will be used by nvm to determine which version of node to install.

    $ nvm install
    $ nvm use
    $ nvm alias default $(node -v)
    
  • Verify that node and npm have been installed:

    $ node --version
    v10.23.0
    $ npm --version
    6.14.9
    

    Note: these versions might differ from the LTS version installed locally.

  • Install dependencies:

    $ npm ci
    
  • Add Firewall exceptions (Fedora Linux Only)

    $ sudo firewall-cmd --zone=public --add-port=4443/tcp --permanent
    $ sudo firewall-cmd --zone=public --add-port=8080/tcp --permanent
    $ sudo firewall-cmd --zone=public --add-port=5353/udp --permanent
    
  • Set up domain:

    • If you plan to use the provided tunneling service to set up a _.webthings.io domain:

      • Start the web server:

        $ npm start
        
      • Load http://localhost:8080 in your web browser (or use the server's IP address if loading remotely). Then follow the instructions on the web page to set up domain and register. Once this is done you can load https://localhost:4443 in your web browser (or use the server's IP address if loading remotely).

    • If you plan to use your own SSL certificate:

      • The HTTPS server looks for privatekey.pem and certificate.pem in the ssl sub-directory of the userProfile directory specified in your config. You can use a real certificate or generate a self-signed one by following the steps below.

        $ WEBTHINGS_HOME="${WEBTHINGS_HOME:=${HOME}/.webthings}"
        $ SSL_DIR="${WEBTHINGS_HOME}/ssl"
        $ [ ! -d "${SSL_DIR}" ] && mkdir -p "${SSL_DIR}"
        $ openssl genrsa -out "${SSL_DIR}/privatekey.pem" 2048
        $ openssl req -new -sha256 -key "${SSL_DIR}/privatekey.pem" -out "${SSL_DIR}/csr.pem"
        $ openssl x509 -req -in "${SSL_DIR}/csr.pem" -signkey "${SSL_DIR}/privatekey.pem" -out "${SSL_DIR}/certificate.pem"
        
      • Start the web server:

        $ npm start
        
      • Load https://localhost:4443 in your web browser (or use the server's IP address if loading remotely). Since you're using a self-signed certificate, you'll need to add a security exception in the browser.

Browser Support

The Gateway requires a browser that supports:

Currently, that translates roughly to:

  • Firefox 63+
  • Chrome 54+
  • Edge 79+
  • Safari 10.1+
  • Opera 41+

Debugging

If you are using VS Code, simply use the "launch" target. It will build the gateway in debugger mode.

If you are not using VS Code, run npm run debug and it will build the gateway and launch it with --inspect.

Testing

Install Additional Dependencies

In order to run the browser tests, you'll need to install Google Chrome and a JDK (e.g. OpenJDK).

Running Tests

To run the linter and all tests:

$ npm run test

To run a single test:

$ npm run jest build/test/{test-name}.js

Source Code Structure

  • config/ - Gateway configuration files
  • deb/ - Tools for building .deb packages
  • docker/ and Dockerfile - Tools for building Docker image
  • image/ - Tools for building the Raspberry Pi image
  • rpm/ - Tools for building .rpm packages
  • src/
    • addons-test/ - Add-ons used strictly for testing
    • controllers/ - App URL routes and their logic
    • iso-639/ - Small utility for interacting with ISO-639 data, i.e. locale names
    • models/ - Data model and business logic
    • platforms/ - Platform-specific functionality
    • plugin/ - Utility classes and methods used by add-ons
    • rules-engine/ - The rules engine
    • test/ - Integration and unit tests
    • views/ - Handlebars templates
    • addon-loader.ts - Script used for starting up Node-based add-ons
    • addon-manager.ts - Manages add-ons (e.g. Zigbee, Z-Wave)
    • addon-utils.ts - Utilities for add-ons, e.g. for reading the manifests
    • app.ts - The main back end
    • certificate-manager.ts - Certificate registration and renewal, via Let's Encrypt
    • constants.ts - System-wide constants
    • db.ts - Manages the SQLite3 database
    • deferred.ts - Wraps up a promise in a slightly more convenient manner for passing around, or saving
    • ec-crypto.ts - Elliptic curve helpers for the ES256 curve
    • errors.ts - Common error classes
    • jwt-middleware.ts - Express middleware for determining authentication status
    • log-timestamps.ts - Utilities for adding timestamps to console logging functions
    • migrate.ts - User profile migration
    • oauth-types.ts - OAuth types
    • passwords.ts - Password utilities
    • platform.ts - Platform-specific utilities
    • push-service.ts - Push notification service
    • router.ts - Routes app URLs to controllers
    • sleep.ts - Small utility to implement a promise-based sleep
    • tunnel-service.ts - Utilities to determine state of tunnel and manage the PageKite process
    • user-profile.ts - Manages persistent user data
    • utils.ts - Various utility functions
    • wifi-setup.ts - Initial Wi-Fi setup code for Raspberry Pi OS
  • static/ - Static CSS, JavaScript & image resources for web app front end
  • tools/ - Helpful utilities (not part of the build)
  • package.json - npm module manifest

gateway's People

Contributors

andrenatal avatar andy-moz avatar apvenkat avatar bellayet avatar benfrancis avatar cesperanc avatar dependabot[bot] avatar dhylands avatar fjoerfoks avatar gaby2300 avatar hobinjk avatar ihorhordiichuk avatar jimsp472000 avatar karm46 avatar koehlermichael avatar lightsofapollo avatar marceloghelman avatar markh-bz avatar marsf avatar mrstegeman avatar petercpg avatar piotrdrag avatar rprys avatar rzr avatar sogaani avatar sunekuntz avatar taqbaylitassa avatar theochevalier avatar tim-hellhake avatar unghost avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gateway's Issues

The 'things' table can't be found

When I try to add a new device, I am seeing the following error in the node app. I can see the device listed in the website, but seems it's not being stored in the database, because doesn't appear in the main page (keeps showing a 'No devices yet.' message)

ZWave: node2 valueAdded: 50:Previous Reading -> 0.025
ZWave: node2 valueAdded: 50:Interval -> 70
Adapter: FooAdapter id Foo1 pairing cancelled
AdapterManager: About to resolve deferredAdd
Deferred: Resolving deferred promise id: 1 arg: { id: 'zwave-da95d8e6-2',
 name: 'zwave-da95d8e6-2-Switch',
 type: 'onOffSwitch',
 properties: [ { name: 'on', type: 'boolean' } ] }
Error saving new thing Error: SQLITE_ERROR: no such table: things

Remove Thing Integration Test

Now we have integration tests for /things, but no integration test for the "remove thing" route. We should add a test for that.

Login

As a user I want to log into my gateway so that I can access it securely.

Subscriptions with WebSockets

As a web app developer I want to subscribe to events and property changes of Things hosted by the gateway using WebSockets so that I don't have to keep polling the server.

Logout

As a user I want to log out of my gateway so that nobody else can use it from this browsing session

Add smart plug

As a user I want to connect a device to the gateway so I can control it

Add New Device Timeout

Currently the Add New Device dialog will appear to keep scanning for new devices forever.

We should add a timeout (at the adapter layer, the UI layer, or both) and close open websockets when we're done.

Adapter API - pair()

As a MozIoT web app developer I would like to call a method to tell an adapter to pair a device with the gateway.

Suggestion:

AdapterManager.pair(id)

Returns a promise which resolves once pairing is successful, or errors with a timeout or error.

Settings - View domain

As a user I want to view the current domain name/base web address of my gateway so I know how to securely access the gateway over the Internet

Settings - Edit user

As a user I want to edit the name, email address or password of a user so that their details are up to date

get/setPropertyValue should return a promise

Currently get/setPropertyValue are of the "fire and forget" variety. They should be changed to use promises.

For the ZigBee & ZWave, the stacks cache the last value so the get will always resolve right away, but that might not be the case for all adapters.

The set should send the command and resolve when the value changed is detected (or reject on a timeout or other error).

Divergence between the sqlite binding version installed by npm and the one asked by the app

When installing the gateway on Rpi, npm installs the sqlite version 51 and the app wants to run version 46. Symlinking them seems to work, but we need to figure why the divergence occurs.


node-pre-gyp ERR! Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v3.1.8/node-v51-linux-arm.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v51 ABI) (falling back to source compile with node-gyp)

pi@vaani:~/wot/gateway $ sudo node app.js
module.js:327
    throw err;
    ^

Error: Cannot find module '/home/pi/wot/gateway/node_modules/sqlite3/lib/binding/node-v46-linux-arm/node_sqlite3.node'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/home/pi/wot/gateway/node_modules/sqlite3/lib/sqlite3.js:4:15)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
pi@vaani:~/wot/gateway $ ls
adapter.js          adapters  controllers  db.sqlite3   device.js  models        package.json  static    utils.js
adapter-manager.js  app.js    db.js        deferred.js  LICENSE    node_modules  README.md     thing.py
pi@vaani:~/wot/gateway $ ls /home/pi/wot/gateway/node_modules/sqlite3/lib/binding/node-v46-linux-arm/
ls: cannot access /home/pi/wot/gateway/node_modules/sqlite3/lib/binding/node-v46-linux-arm/: No such file or directory
pi@vaani:~/wot/gateway $ ls /home/pi/wot/gateway/node_modules/sqlite3/lib/binding/
node-v51-linux-arm

Add Thing by URL

As a user I want to add a Web Thing by its URL, to be managed by my gateway.

This is for Web Things which use the direct integration pattern so provides its own Web Thing API but which the user might want to manage together with other Things using the gateway.

List smart plugs

As a user I want to see a list of devices connected to the gateway

Settings - Custom domain

As a user I want to assign a custom domain name to my gateway instead of/as well as using the tunnelling service provided by Mozilla with a free subdomain

Remove Dummy Data

Now that we can add real adapters and Things we should remove the default dummy foo adapter and default Things.

ZWave controller should be reset at start of program?

If the app crashes while the zwave dongle is inclusion mode, and you then restart the app, the dongle seems to still be in a state where many devices will timout. I observed that my Leviton devices all got timeouts.

Unplugging and replugging the dongle seemed to resolve this.

We might need to do a controller (dongle) reset. Ideally we'd detect the state that the controller is in and get it out of that state, and only reset the controller if absolutely required.

Adapter API - discoverDevices()

As a MozIoT web app developer I would like to be able to request a list of new devices which aren't yet added to the gateway.

In order for the web app to display a list of devices available on the network which can be added to the gateway, it needs to be able to get a list of devices from the adapter manager.

My suggestion would be:

AdapterManager.discoverDevices()

This method would return a Promise which resolves with a list of devices. E.g.

[{
  id: '123e4567-e89b-12d3-a456-426655440000',
  type: 'onOffSwitch',
  name: ' Aeotec Smart Switch 6'
}]
  • id - A UUID generated by the AdapterManager which internally maps this ID onto an (e.g.) Z-Wave or ZigBee identifier.
  • type - A Web Thing type, as defined in the Web Thing API specification.
  • name - A user readable name which may give a hint to the user as to which device it is (e.g. manufacturer name, model number etc.).

My expectation is that behind the scenes this method would iterate through the list of adapters and initiate a protocol specific process for listing available devices (e.g. entering a pairing or commissioning mode). Ideally the actual pairing shouldn't take place with a further user interaction and a separate AdapterManager.pair(id) method.

Need a mechanism to detect and remove failed ZWave nodes

The ZWave controller will eventually mark a node as failed if it stops responding after a while. We need a way to tag these nodes as failed, and a way to call the Manager::RemoveFailedNode C++ API.

I think that this can be reproduced by adding a node and then factory resetting it. You can then add the node a second time, and the first nodeId will stick around and should eventually get marked as failed.

Enable/disable adapter

As a user I want to see a list of adapters active on the gateway so I can choose which ones to enable.

Adapter API - setProperty()

As a MozIoT web app developer I would like to call a method to tell an adapter to a set a property on a device.

Suggestion:

AdapterManager.setProperty(deviceId, propertyName, value)

returns a Promise which resolves once the property has been successfully set, perhaps with a representation of the new value.

My suggestion is that this can just be called on the AdapterManager directly (referencing the device ID) so that the web app doesn't necessarily need to care which adapter the Thing is connected to...

Browsing to https://demo.box.knilxof.org:4443/ doesn't work properly n Chrome Desktop

In Chrome on my linux desktop, if I browse to:
https://demo.box.knilxof.org:4443/
then I get the Blue background and "No devices yet."

If I browse to the same address using Chrome on my phone then it works.

Browsing to the same address using Firefox works fine (both desktop and phone).

Browsing to my local node server (https://192.168.0.24:4443/ - using a self-signed cert) works fine after doing the security exception (in Chrome/Firefox on desktop or phone)

Web API - Add New Thing

As a developer of the MozIoT web app front end I want to add a new thing to the gateway

We need an API call/calls which can be used to add a new thing to the gateway. This includes:

  1. Adding a device to the gateway which has not yet been paired but the user wants to pair via a gateway adapter
  2. Adding a device to the gateway database which has somehow already been paired via an adapter (e.g. ZigBee/Z-Wave controller) but not yet added to the database
  3. Eventually also adding references to Web Things which are not hosted by the gateway itself but expose their Web Thing API directly

Web Thing API - Get Thing

As a Web Thing API consumer I want to get a Thing Description for a Thing via its Thing URL.

Original report:

If you do a get on /things, each thing has an href which looks like /things/:thingId, but if you do a GET on that there is no handler installed.

Remember Me

As a user I want the option to stay logged in for a period of time after I close my browser so that I don't have to keep logging in every time I use the gateway.

New device has empty name field

STR:

  • Call AdapterManager.addNewDevice()
  • print output resolved promise

Expected:

  • name field populated

Actual:

  • name field is an empty string

What's odd is that when you subsequently call AdapterManager.getDevices() the added device does have a name set.

OpenZWave segfault when pairing

Occasionally when I try to pair a device OpenZWave segfaults and takes down the Node process with it. So far I haven't been able to come up with reliable STR, it seems intermittent.

TypeError: adapter is not a constructor

STR:

  • Run "node test-server.js"

Get error:

Listening at http://:::8088
Loading adapter ./adapters/foo.js
/home/tola/Code/moziot/gateway/adapters.js:43
                adapter_manager.adapters.push(new adapter(adapter_manager));
                                              ^

TypeError: adapter is not a constructor
    at file_list (/home/tola/Code/moziot/gateway/adapters.js:43:47)
    at FSReqWrap.oncomplete (fs.js:123:15)

@dhylands Do you not see this error? Are you running test-server.js in a different way?

Settings - Add user

As a user I want to add additional users to my gateway so that multiple people can have access.

Remove Thing

As a user I want to disconnect a Thing from the gateway when I no longer need it

Settings - Remove user

As a user I want to remove a user from the gateway so that they no longer have access

Adapter API

Eventually we'd like it to be possible to write adapters (e.g. ZigBee adapter, Z-Wave adapter) in different programming languages, have them run in their own process, and talk to them via some kind of IPC mechanism.

But for our initial NodeJS prototype we’ve discussed having some kind of uniform JavaScript interface for the web app to talk to adapters.

Based on the Web Thing REST API which will be a layer above this JavaScript API, my first thoughts on the what basic underlying features of the adapter layer might be are:

Adapter

  • discoverDevices() // List available or connected devices so we can compare with list of devices already added
  • connectDevice() // Pair with the device and/or get a connected device's details to add it to the gateway database
  • disconnectDevice() // Unpair device if such a mechanism exists for this adapter

Thing

  • getProperty() // Get a property value by its name
  • setProperty() // Set a property value by its name
  • executeAction() // What kinds of functions can we execute on ZigBee/Z-Wave devices?
  • subscribe() // What kinds of events or changes we can listen or subscribe to from ZigBee/Z-Wave devices?

Does this make sense? What have I missed?

API Authentication

As a user I would like the API of my gateway to be authenticated so that only authenticated users can consume it.

Authenticate API calls (e.g. using JSON Web Tokens).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.