softwareag / cumulocity-devicemanagement-agent Goto Github PK

Using DM agent, we can register devices, however we noticed the following error in the logs, it is spamming log files.

2022-07-08 04:44:10,477 InitializerThread-DockerSensor WARNING c8ydm.client.rest_client Managed object not updated in C8Y
2022-07-08 04:44:10,793 SensorThread-DockerSensor ERROR c8ydm.core.docker_watcher The following error occured: [Errno 2] No such file or directory: 'docker'
2022-07-08 04:44:11,877 SensorThread-DockerSensor WARNING c8ydm.client.rest_client Managed object not updated in C8Y
2022-07-08 04:44:20,814 SensorThread-DockerSensor ERROR c8ydm.core.docker_watcher The following error occured: [Errno 2] No such file or directory: 'docker'
2022-07-08 04:44:21,957 SensorThread-DockerSensor WARNING c8ydm.client.rest_client Managed object not updated in C8Y
2022-07-08 04:44:30,830 SensorThread-DockerSensor ERROR c8ydm.core.docker_watcher The following error occured: [Errno 2] No such file or directory: 'docker'

When I try to run the agent on a Windows machine, I recognized VSCODE did not use the right EOL Sequence for running the *.sh scripts. So maybe put one hint in the documentation to set "LF" or "\n" for End of Line Sequence in Windows environments. "Auto" detection did not work for me (at least not in VSCODE).

The LogHandler Module needs to be reviewed

I've tried to change the required interval via Cumulocity Text-baed Configuration, which did succeed only partly. Reproduction steps:

1. Start and register agent in Cumulocity with default configuration
2. Go to the Device in Devicemanagement, note that the required Interval in Info Tab is 10
3. Go to Configuration > Text-Based Configuration and change field agent.requiredinterval to e.g. 13. Send the configuration to the device
4. Agent picks it up, operation succeeds
5. With "getting snapshot from device" you can see the new configuration got transferred to Cumulocity
6. However: The required interval in Info Tab is still at 10 Minutes, also the ManagedObject field is at 10 Minutes => seems the update of the MO has an issue.

The logs dont tell anything specific, this is the received operation:

2022-07-21 18:50:21,977 ListenerThread-ConfigurationManager INFO c8ydm.core.configuration Configuration Operation received: ['3bc64215ffe1', '"mqtt.url=mqtt.eu-latest.cumulocity.com\nmqtt.port=8883\nmqtt.tls=true\nmqtt.cert_auth=false\nmqtt.client_cert=/root/.cumulocity/certs/chain-cert.pem\nmqtt.client_key=/root/.cumulocity/certs/device-cert-private-key.pem\nmqtt.cacert=/etc/ssl/certs/ca-certificates.crt\nmqtt.ping.interval.seconds=60\nagent.name=dm-example-device\nagent.type=c8y_dm_example_device\nagent.main.loop.interval.seconds=10\nagent.requiredinterval=13\nagent.loglevel=INFO"']

In the docs:

"The script will build a docker image and starting one instance afterwards. Per default Bootstrapping is used and no other information is necessary. In this case the docker container Id is the device Id which should be entered when registering a device in cumulocity."

But how can the device id (external_id) be configured when running outside of docker?

With Debian 11 Python 3.9.2 is the desired Python version.
There might be some issues with this version as for example platform.linux_distribution was removed and leads to errors in network module and other modules using pyspectator.

When running the dm agent outside of docker

c8ydm.start

on RPi IV using the settings:

tls = true
cert_auth = true

I get the following exception:

Starting...
2022-07-17 19:10:34,411 MainThread INFO root Serial: 3332221
2022-07-17 19:10:34,412 MainThread INFO c8ydm.client.mqtt_agent Starting agent
2022-07-17 19:10:34,706 Thread-1 INFO c8ydm.client.mqtt_agent Agent connected with result code: 0
2022-07-17 19:10:35,662 MainThread INFO c8ydm.client.mqtt_agent Starting refresh token thread 
2022-07-17 19:10:35,664 TokenThread-1 INFO c8ydm.client.mqtt_agent Refreshing Token
SenseHat not available
2022-07-17 19:10:35,991 InitializerThread-AgentInitializer INFO c8ydm.agentmodules.agent_initializer Agent Initializer called...
2022-07-17 19:10:35,994 InitializerThread-DockerSensor INFO c8ydm.agentmodules.docker_watcher Docker Initializer called...
2022-07-17 19:10:36,001 InitializerThread-EventInitializer INFO c8ydm.agentmodules.event_initializer Event Initializer called...
2022-07-17 19:10:36,008 InitializerThread-Network INFO c8ydm.agentmodules.network Network Initializer called...
2022-07-17 19:10:36,032 InitializerThread-SmartRestInitializer INFO c8ydm.agentmodules.smartRest_inizializer SmartRest Template Initializer called...
2022-07-17 19:10:36,035 InitializerThread-SmartRestInitializer ERROR c8ydm.client.rest_client The following error occured while trying to check for existing SmartRest templates: 'NoneType' object is not subscriptable
2022-07-17 19:10:36,036 InitializerThread-SmartRestInitializer INFO c8ydm.agentmodules.smartRest_inizializer SmartRest Template does not exist, creating....
2022-07-17 19:10:36,036 InitializerThread-SmartRestInitializer ERROR c8ydm.client.rest_client The following error occured while trying to create SmartRest template: 'NoneType' object is not subscriptable
2022-07-17 19:10:36,071 MainThread INFO c8ydm.client.mqtt_agent Supported operations:
2022-07-17 19:10:36,240 MainThread INFO c8ydm.client.mqtt_agent {'c8y_DeviceProfile', 'c8y_MeasurementRequestOperation', 'c8y_RemoteAccessConnect', 'c8y_DownloadConfigFile', 'c8y_SoftwareList', 'c8y_Restart', 'c8y_UploadConfigFile', 'c8y_Firmware', 'c8y_Message', 'c8y_Command', 'c8y_SoftwareUpdate', 'c8y_Configuration', 'c8y_LogfileRequest', 'c8y_Docker'}
2022-07-17 19:10:36,807 MainThread INFO c8ydm.client.mqtt_agent Required interval: 10
2022-07-17 19:10:36,813 MainThread INFO c8ydm.client.mqtt_agent Model:
2022-07-17 19:10:36,814 MainThread INFO c8ydm.client.mqtt_agent ['3332221', 'raspberry', '1.0']
2022-07-17 19:10:36,950 MainThread INFO c8ydm.client.mqtt_agent Subscribing to XID: c8y-dm-agent-v1.0
2022-07-17 19:10:37,480 InitializerThread-DockerSensor ERROR c8ydm.client.rest_client The following error occured: 'NoneType' object is not subscriptable
2022-07-17 19:10:37,481 MainThread ERROR c8ydm.client.rest_client The following error occured: 'NoneType' object is not subscriptable
2022-07-17 19:10:37,548 InitializerThread-DockerSensor ERROR c8ydm.client.rest_client The following error occured: 'NoneType' object is not subscriptable
2022-07-17 19:10:37,582 MainThread ERROR c8ydm.client.rest_client The following error occured: 'NoneType' object is not subscriptable
--- Logging error ---
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/c8ydm/client/mqtt_agent.py", line 100, in run
    self.__init_agent()
  File "/home/pi/.local/lib/python3.7/site-packages/c8ydm/client/mqtt_agent.py", line 283, in __init_agent
    self.rest_client.set_operations_to_failed(ops)
  File "/home/pi/.local/lib/python3.7/site-packages/c8ydm/client/rest_client.py", line 320, in set_operations_to_failed
    if len(operations) > 0:
TypeError: object of type 'NoneType' has no len()

The registration in the cloud works, but it seems that no measurements are send to the could tenant.
What could be the issue?

In the Dockerfile the arch is set hardcoded.

Thus the building fails on arm. arch should be handed via environment of host system.

Hi,

could you please add package apt-utils to the initially installed packages?
This is required if you want to install nano in a demo.

Regards Christof

RUN apt-get update \
    && apt-get -y full-upgrade \
    && apt-get -y install \
    python3 \
    python3-apt \
    python3-pip \
    openssh-server \
    curl \
    gpg \
    lsb-release \
    apt-utils

Tobias Sommer,
Murat Bayram
Marco Stoffel
Andina Asllani

Create a Powerpoint supporting all the Device Management Features which should be demonstrated in a customer presentation.
Depends on #13

Tracking issue for:

• https://github.com/SoftwareAG/cumulocity-devicemanagement-agent/security/code-scanning/1

Especially the ca-certs handling for TLS seems to be difficult here...

Brainstorm about the Device Management with Cumulocity Storyline can look like.
Use existing presentation to get an orientation and build a structure

The self.token is only being set during the initialization of the class.
The JWT token of the mqtt agent will be set after the RestClient has been created.

Needs to be discussed as CentOS is EOL for most versions...

Some customers want to be able to host library and deb package upgrades in Cumulocity directly, including new versions of the agent, and install from there. Currently it doesn't provide any mechanism to do so, and specifying the file or path in Cumulocity is ignored by the agent in lieu of the devices own source list. This is a critical need for at least 2 customers at the moment.

Building with ARM based Machine such as Apple´s M1 does exit with an error due to an repo error in Line 43.

Changing add-apt-repo to arch=arm64 solves the problem.

Replace rest client with https://github.com/SoftwareAG/cumulocity-python-api

We have to check if all cases could be implemented with the python-api client before replacing the existing one.

When an agent has accidentally shut down after it sent 501 (EXECUTING) and before 502 / 503 (failure / success), it creates a "dangling" operation in the Cumulocity server. After the device restart, this operation is not visible from the agent since the cloud will not resend EXECUTING operations (they only resend PENDING operations).

Besides, if you create a new operation which has the same fragment as the dangling operation, the dangling operation would be set to SUCCESS and the new one is set to EXECUTING. This is because the agent would successfully send 501 then 503 to the cloud, but 503 will only update the old operation while 501 is applied to the new one.

Should at least send on startup a geo position.
Could be improved by sending geo date periodically..

PoC Agent: https://labcase.softwareag.com/projects/schindler_iot/repository/10875/revisions/debian/entry/c8ycc/agentmodules/software_management.py

Schindler Agent:
https://labcase.softwareag.com/projects/schindler-iot/repository/device-agent/revisions/master/entry/c8yda/agentmodules/listener/software_update_listener.py

I would like to run the agent in wsl2 Ubuntu.
When I follow the description in the build step:

• Build
  pip install -r requirements.txt
  pip install .

apt install python-apt -> this cannot be run without sudo

• Run
  sudo c8ydm.start -> will not be found since the agent was installed without sudo

When I do all the steps using sudo or not and execute c8ydm.start I always get:

from c8ydm.core.apt_package_manager import AptPackageManager
File "/home/mstoffel/.local/lib/python3.8/site-packages/c8ydm/core/apt_package_manager.py", line 23, in
if 'Linux' == platform.system() and ('debian' == platform.linux_distribution()[0] or 'ubuntu' == platform.linux_distribution()[0]):
AttributeError: module 'platform' has no attribute 'linux_distribution'

https://github.com/SoftwareAG/c8yMQTT/blob/master/c8yMQTT.py

https://github.com/SoftwareAG/cumulocity-thinedge-example/tree/main/Agents

If I try to debug the agent I get the following warnings and errors:

cd /home/mstoffel/cumulocity-devicemanagement-agent ; /usr/bin/env /bin/python3 /home/mstoffel/.vscode-server/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/launcher 41065 -- -m c8ydm.main
/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'c8ydm.main' found in sys.modules after import of package 'c8ydm', but prior to execution of 'c8ydm.main'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
2021-12-15 08:09:05,785 MainThread INFO root Output of SystemUtils Serial: 91764171647
Starting...
2021-12-15 08:09:05,789 MainThread INFO root Found pid file with pid 14816
2021-12-15 08:09:05,791 MainThread INFO root Checking if pid 14816 is existing...
2021-12-15 08:09:05,793 MainThread INFO root Pid 14816 does not exist
2021-12-15 08:09:05,794 MainThread INFO root Removing PID File /home/mstoffel/.cumulocity/agent.pid
2021-12-15 08:09:05,796 MainThread INFO root Serial: 91764171647
2021-12-15 08:09:05,799 MainThread INFO root No credentials found. Starting bootstrap mode.
2021-12-15 08:09:05,800 MainThread ERROR root No bootstrap credentials found. Stopping agent.

https://github.com/SoftwareAG/cumulocity-remote-access-agent