vmware-archive / liota Goto Github PK

Now when vrops_gateway = self.vrops.register(gateway) happens, the gateway object passed to it is changed and contains vROps specific properties.

This behavior is harmful in the package manager context, because we may want different DCCs to use the same Gateway object/representation. Therefore, the Gateway object should contain only Gateway specific configuration, parameters, etc. and should NOT contain any DCC specific parameters.

Data members required for establishing connection in Transport layer are stored in init() method of DeviceComms and DccComms, so that self._connect() can be called. This results in duplication.

Why don't we use **kwargs in abstract _connect() method, so that it can take dynamic arguments instead ?

When connection between gateway and DCC is broken, LIOTA should try establishing connection using exponential back-off mechanism. This saves some compute at gateway and also reduces load on DCC when hundreds of gateways try to re-establish connection at the same time.

Support for Java language bindings will enable the use of popular SDKs that communicate with devices to manage & extract data from them.

On Ln 58 def put_and_notify(self, item, block=True, timeout=None), it seems that it gets the smallest item every time using heapq.nsmallest(1, self.queue)[0] (Ln 64,87). It is confusing that we are using a "PriorityQueue" (Ln 52) but still needs heapq.nsmallest to sort them.

For example, WebSocket(URL "secure") is not valid anymore, vrops.register returns a value, etc.

Hello

We started working with Liota a few weeks ago, developing my own packages and sending information to my own DCC. So far, we managed to send data through HTTP but since you added the MQTT implementation we tried to work with it. Our problem comes with the Liota installation through pip, we got some errors because our packages with MQTT implementation don't find the MQTT files from Liota (can't resolve the imports), and our package files are on the package folder inside the installed Liota, are we doing maybe something wrong?

We are used to develop using virtualenv and the idea is continue the same process with Liota unless you think it's a bad idea. We've created our own requirements.txt file to create the virtualenv based on the file included with Liota, but I decided to download Liota from Github in order to have the latest Liota, work with MQTT and resolve the import issue. But through this method, Liota doesn't install correctly, doesn't appear on the virtualenv libraries and I get "ImportError: No module named liota.core.package_manager" each time I try to run the liotad.py file. Any advice on how can install Liota on a virtualenv and use MQTT?

P.D: this is how mi requirements.txt looks like
aenum==1.4.5
appdirs==1.4.0
linux-metrics==0.1.4
packaging==16.8
paho-mqtt==1.2
Pint==0.7.2
pyparsing==2.1.10
six==1.10.0
websocket-client==0.37.0
-e git+https://github.com/vmware/liota.git@e2427c533db363aeccb5ad34c5b37e97330db06f#egg=liota

Still in designing phase.

Providing MQTT Comms support in Liota for Project ICE and other DCC`s.

Contributors:
@Venkat2811

Design Approach:
@KohliDev

When initializing MQTT DCC comms, I was trying to provide an MqttMessagingAttributes object I constructed myself, using

iotcc = IotControlCenter(MqttDccComms(
    edge_system_name=edge_system.name,
    url=config['BrokerIP'],
    port=config['BrokerPort'],
    identity=identity,
    tls_conf=tls_conf,
    enable_authentication=True,
    mqtt_msg_attr=MqttMessagingAttributes( # manually construct attribute object
        edge_system_name=edge_system.name,
        pub_qos=0
    )
))

Got this error:

Traceback (most recent call last):
  File "./mqtt_debug_plus_qos.py", line 183, in <module>
    pub_qos=0
  File "/usr/local/lib/python3.5/dist-packages/liota-0.3-py3.5.egg/liota/dcc_comms/mqtt_dcc_comms.py", line 113, in __init__
  File "/usr/local/lib/python3.5/dist-packages/liota-0.3-py3.5.egg/liota/dcc_comms/mqtt_dcc_comms.py", line 122, in _connect
  File "/usr/local/lib/python3.5/dist-packages/liota-0.3-py3.5.egg/liota/lib/transports/mqtt.py", line 177, in __init__
  File "/usr/local/lib/python3.5/dist-packages/paho/mqtt/client.py", line 491, in __init__
    raise ValueError('A client id must be provided if clean session is False.')
ValueError: A client id must be provided if clean session is False.

I'm testing on various systems, and sometimes will get errors like this when running python setup.py install:

(venv) root [ /tmp/liota ]# python  setup.py  install
Traceback (most recent call last):
  File "setup.py", line 44, in <module>
    long_description = f.read()
  File "/tmp/liota/venv/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4822: ordinal not in range(128)

Turns out the README has some smartquotes. Removing those and replacing with simple quotes fixes the problem.

Package Manager absorbs the error stack trace in most of the cases. For example, https://github.com/vmware/liota/blob/master/liota/core/package_manager.py#L705

Instead of Log.error, Log.exception should be used in such places, to print the exception along with the stacktrace.

Stop is not implemented in the startup script. Causes problems when installing via ansible and restarting service after changing config files.

Connection between the EdgeSystem and the DataCenter could be broken temporarily and during this time, Liota shall store the collected data on the local disk. Once connection is reestablished, send the data to the DataCenter.

The amount of storage used per device should be configurable and once the threshold is reached, older data is deleted, replacing with newer data.

Add IBM Bluemix as DCC in Liota

Adding AWS as DCC in liota with MQTT as dcc_comms layer.

Contributors:
@Venkat2811

Design:
@KohliDev

As per OSS team we need to change path of all the liota generated files to "/var/tmp/liota" folder. The "/var/tmp" directory is made available for programs that require temporary files or directories that are preserved between system reboots:

1. "/etc/liota/conf/devs" to "/var/tmp/liota/devs"

2. "/etc/liota/conf/entity" to "/var/tmp/liota/entity"

3. "/etc/liota/conf/iotcc.json" tp "/var/tmp/liota/iotcc.json"

I think we can remove "uuid.ini" as we are going to use "iotcc.json" going forward.

AMQP communication layer support (Transports, DCCComms and DeviceComms) in Liota with DCC based example.

Contributors:
@nkanchas
@Venkat2811

Sometimes it takes time for getting the network connection established after the restart of the gateway. On the first try itself, auto re-start Liotad fails if the network link to the gateway is not up.

De-coupling IoTCC & HelixProtocol from DCC_comms layer so that we get complete plug-play architecture for integrating with other DCC_Comms.

utility.py contains multiple utility functions. Group similar utilities in separate files,

Since userdata is changed to Queue, remove it from MqttDccComm's init() and update the same in AWSIoT examples.

This cause everything that has been published to have a local copy, which is not necessary unless in debugging.

This also results in high CPU usage in pressure tests where hundreds of messages are published every second.

Logs are not readable and /n appears everywhere in case of error traceback. This issue occurs after the PR #120 merge, needs to be resolved before Next Liota release:

2017-05-05 12:34:43 UTC 5966 ERROR [MainThread] liota.lib.transports.web_socket.connect_soc(60) - Traceback (most recent call last):\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 58, in connect_soc__\n__ self.WebSocketConnection()\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 71, in WebSocketConnection__\n__ if self.identity.root_ca_cert:\n__AttributeError: 'bool' object has no attribute 'root_ca_cert'\n__
2017-05-05 12:41:54 UTC 5994 INFO [MainThread] liota.lib.utilities.utility.setup_logging(242) - created logger with /etc/liota/conf/logging.json
2017-05-05 12:41:54 UTC 5994 INFO [MainThread] liota.lib.utilities.utility.get_disk_name(140) - Disk name is : fd0
2017-05-05 12:41:54 UTC 5994 ERROR [MainThread] liota.lib.transports.web_socket.connect_soc(60) - Traceback (most recent call last):\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 58, in connect_soc__\n__ self.WebSocketConnection()\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 71, in WebSocketConnection__\n__ if self.identity.root_ca_cert:\n__AttributeError: 'bool' object has no attribute 'root_ca_cert'\n__
2017-05-05 12:59:22 UTC 6023 INFO [MainThread] liota.lib.utilities.utility.setup_logging(242) - created logger with /etc/liota/conf/logging.json
2017-05-05 12:59:22 UTC 6023 INFO [MainThread] liota.lib.utilities.utility.get_disk_name(140) - Disk name is : fd0
2017-05-05 12:59:22 UTC 6023 ERROR [MainThread] liota.lib.transports.web_socket.connect_soc(60) - Traceback (most recent call last):\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 58, in connect_soc__\n__ self.WebSocketConnection()\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 67, in WebSocketConnection__\n__ self.ws = create_connection(self.host, enable_multithread=True,\n__AttributeError: WebSocket instance has no attribute 'host'\n__
2017-05-05 13:02:19 UTC 6052 INFO [MainThread] liota.lib.utilities.utility.setup_logging(242) - created logger with /etc/liota/conf/logging.json
2017-05-05 13:02:19 UTC 6052 INFO [MainThread] liota.lib.utilities.utility.get_disk_name(140) - Disk name is : fd0
2017-05-05 13:02:19 UTC 6052 ERROR [MainThread] liota.lib.transports.web_socket.connect_soc(60) - Traceback (most recent call last):\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 58, in connect_soc__\n__ self.WebSocketConnection()\n File "build/bdist.linux-x86_64/egg/liota/lib/transports/web_socket.py", line 68, in WebSocketConnection__\n__ sslopt={"cert_reqs": ssl.CERT_NONE})\n File "/usr/local/lib/python2.7/dist-packages/websocket_client-0.37.0-py2.7.egg/websocket/core.py", line 489, in create_connection_\n__ websock.connect(url, **options)\n File "/usr/local/lib/python2.7/dist-packages/websocket_client-0.37.0-py2.7.egg/websocket/core.py", line 219, in connect_\n__ self.handshake_response = handshake(self.sock, *addrs, **options)\n File "/usr/local/lib/python2.7/dist-packages/websocket_client-0.37.0-py2.7.egg/websocket/handshake.py", line 67, in handshake_\n__ status, resp = _get_resp_headers(sock)\n File "/usr/local/lib/python2.7/dist-packages/websocket_client-0.37.0-py2.7.egg/websocket/handshake.py", line 123, in get_resp_headers\n__ raise WebSocketBadStatusException("Handshake status %d", status)\n__WebSocketBadStatusException: Handshake status 404\n__

In liota/entities/devices/thermistor_simulated.py, the __init__ method should end with self.run(), or this simulation will not start.

According to what we are doing in the bike simulator liota/entities/devices/bike_simulated.py and other demo scripts and corresponding liota package (i.e. those scripts do not call run() after creating the simulator device object), adding run() in __init__ should be the quickest fix.

I already did this in my branch. Tested. Not submitted yet.

unit = An SI Unit (work needed here)
Could any of these Python libraries for Unit support be of help?
http://www.drdobbs.com/jvm/quantities-and-units-in-python/240161101

Hi,
I can't find any information about IoTCC.
Can anyone tell me what is IoTCC and project web site or other information.

Thanks.

Most of the files listed below should not be executable --

$ find . -executable -type f -name "*.py"
./packages/graphite.py
./packages/cal_sha1sum.py
./packages/dev_disc/liota_devsim_load.py
./packages/dev_disc/dev_disc.py
./packages/liotad.py
./liota/core/device_discovery.py
./liota/core/discovery_simulator.py
./liota/disc_listeners/named_pipe.py
./liota/disc_listeners/coap.py
./liota/disc_listeners/__init__.py
./liota/disc_listeners/socket_svr.py
./liota/disc_listeners/discovery_listener.py
./liota/disc_listeners/mqtt.py
./liota/dccs/iotcc.py
./liota/dev_sims/named_pipe.py
./liota/dev_sims/coap.py
./liota/dev_sims/__init__.py
./liota/dev_sims/device_simulator.py
./liota/dev_sims/socket_clnt.py
./liota/dev_sims/mqtt.py

Marking them executable does not affect functionality when running in the normal way, but that looks misleading. Also, some text files like requirements.txt are also marked executable, which should be fixed altogether.

Working with @ApurvaH to make CollectionThreadPool size configurable

Currently, it temporarily assumes dcc pkg will register edge_system as dcc name + "_edge_system".

Docstring styled documentation for all classes and methods. Currently, only few classes have these. This will enable us to use documentation generator like Sphinx in future.

While testing reconnect back-off mechanism we see that, metrics are collected and stored infinitely in Collector Queue as send thread will always be busy with re-try mechanism. If connection is established within minutes, we'll be fine. If it takes days, it might run out of memory.

@KohliDev

Basic Edge intelligence in Liota.

Example #49

Contributors:
@Venkat2811
@nkanchas

REST as Communication layer (Transports, DCCComms and DeviceComms) in Liota. Example will be based on ThingWorx DCC

Contributors:
@nkanchas
@ashish-vmware

This implementation inherently provides receive functionality and liota will not have to implement threads internally.
https://github.com/websocket-client/websocket-client/blob/d7ada589638d00aed6b66bdf7e695d517b1ebcf0/websocket/_app.py

While loading graphite_edge_system_stats.py and graphite_edge_system_stats_with_filter.py, it fails with an error:

2017-05-04 22:13:01 PDT 8430 ERROR [PackageThread] liota.core.package_manager._package_load(602) - Exception in initialization: 'RegisteredEntity' object has no attribute 'name'

Currently, there is no way to remove/un-register/delete edge systems and devices already registered to a dcc. This functionality needs to be added in liota.

Contributors: @ApurvaH

Installing liota (0.1) on Snappy Ubuntu and Wind River IDP we noticed that the data files (in conf and example) directories are going to the wrong place in the hierarchy. They are supposed to go in /etc/liota but end up in /usr/lib/python2.7/site-packages/etc/...

Helix Protocol version needs to be checked in IOTCC DCC, otherwise it is difficult to figure out why the connection failed.

Helix Protocol fails with an error as below, if the hello message is repeated:
HelixProtocolException: HelixProtocolError(u'Unexpected Message hello during HandshakeAwaitingState',)

Helix Protocol needs to be re-designed to accept repeat messages, which will happen due to QoS=1 in MQTT.

In all the liota examples related to health stats, pre-detect the network interfaces (eth0 or ens33) and disks(sda)

@Venkat2811

Pass credentials using Identity object in all examples, instead of passing directly from config.

https://github.com/vmware/liota/blob/master/liota/dccs/iotcc.py#L578 needs to generate transactionId based on timestamp instead of a counter which gets restarted every time liota restarts.

@Venkat2811

Even change eth0 to ens33 as the network interface.

@ApurvaH has a review out for this.

Traceback (most recent call last):
File "/usr/local/bin/pip", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/pip/init.py", line 221, in main
return command.main(cmd_args)
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 252, in main
pip_version_check(session)
File "/usr/local/lib/python2.7/dist-packages/pip/utils/outdated.py", line 102, in pip_version_check
installed_version = get_installed_version("pip")
File "/usr/local/lib/python2.7/dist-packages/pip/utils/init.py", line 848, in get_installed_version
working_set = pkg_resources.WorkingSet()
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/init.py", line 619, in init
self.add_entry(entry)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/init.py", line 675, in add_entry
for dist in find_distributions(entry, True):
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/init.py", line 1942, in find_eggs_in_zip
if metadata.has_metadata('PKG-INFO'):
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/init.py", line 1463, in has_metadata
return self.egg_info and self._has(self._fn(self.egg_info, name))
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/init.py", line 1824, in _has
return zip_path in self.zipinfo or zip_path in self._index()
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/init.py", line 1704, in zipinfo
return self._zip_manifests.load(self.loader.archive)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/init.py", line 1644, in load
mtime = os.stat(path).st_mtime
OSError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/liota-0.3.1-py2.7.egg'