Document the metrics produced by the container monitor so it is easier for the users (and UI's) to find the correct measurements.

Create a Cumulocity IoT plugin that can be deployed to display the additional meta information about the containers and the container groups (e.g. docker compose projects).

• Display meta information about the running containers (See #1)
• Display the runtime information (e.g. cpu, memory) of the container (these will be stored as measurements)

Example service data model

The following show an example of addition container-group information that can be added to the service.

{
  "name": "example::redis", // added via service registration in format of "{projectName}::{serviceName}"
  "containerId": "1f3f0c98877a",
  "containerName": "example_redis_1",
  "projectName": "example",
  "serviceName": "redis",
  "ports": "8080",
  "command": "\"/bin/sh -c 'echo Container started\\n trap \\\"exit 0\\\" 15\\n \\n exec \\\"$@\\\"\\n while sleep 1 & wait $!; do :; done' - /lib/systemd/systemd --system --unit=basic.target\"",
  "networks": "thin-edgeio_devcontainer_default",
  "filesystem": "2.73GB (virtual 5.66GB)",
  "image": "thin-edgeio_devcontainer-tedge",
  "runningFor": "3 weeks ago",
  "state": "running",
  "status": "Up 13 days",
}

I'd like to propose an enhancement that uses the C8y's "Device Profile" function to manage containers with a docker-compose (or similar orchestrator of multiple containers/services). In other words, could we treat the docker-compose.yml as a "configuration file", rather than a "software", and combine it with container images deployed as "software" elements.

This would overcome a limitation that exists with the current approach - that containers referenced in a docker-compose.yml can only be instantiated by retrieving them from a container registry (hub.docker.com or similar), and not from the Cumulocity software repository. It would help users who's thin-edge devices cannot access the general internet (maybe they can only access the C8y platform), and do not want to deal with the complexity of setting up and managing their own private container registry alongside C8y.

Maybe this means that we'd need a new software type "container-image" - with it's own sm-plugin, who's role was simply to install the image, but not not to run it (as the "container" plugin does today)? Containers deployed in this way would only be run by docker compose, and not as standalone services?

The following error was seen in the tedge-mapper-c8y log output:

Illegal control character 0xa

It is likely that some of the messages being sent by the tedge-container-monitor service (provided by the tedge-container-plugin) whilst collecting the runtime information about the containers on the device (most likely in the command field).

Below shows an example of the command which is most likely invoking the error:

[c8y/inventory/managedObjects/update/ci_4812911111_7_thin-edgeio_devcontainer::tedge] {"containerId":"1f3f0c98877aa89cc31c6ea218cfe92fe4fb00015d7f47ab912fce6498a1a718","containerName":"thin-edgeio_devcontainer-tedge-1","state":"running","containerStatus":"Up 2 days","createdAt":"2023-03-06 12:49:34 +0000 UTC","image":"thin-edgeio_devcontainer-tedge","ports":"","networks":"thin-edgeio_devcontainer_default","runningFor":"2 months ago","filesystem":"4.39GB (virtual 7.32GB)","command":"\"/bin/sh -c 'echo Container started
 trap \\"exit 0\\" 15
 
 exec \\"$@\\"
 while sleep 1 & wait $!; do :; done' - /lib/systemd/systemd --system --unit=basic.target\"","projectName":"thin-edgeio_devcontainer","serviceName":"tedge"}

Support container commands: start/stop/restart

• Create a custom smart rest template
• Create a handler to react to the start/stop/restart messages
• Verify the user has permissions to run docker start|stop|restart commands (if not create a new sudoer.d/ file with these permissions which is added at installation time)
• Add the custom smart rest template ids during installation time (e.g. tedge config set ...)

Newer versions of Cumulocity IoT (>=10.17) seem to have a slight schema change to the manifest file (or the build process is doing something wrong), which results in the plugin not being properly recognized/installable after being uploaded.

Workaround

1. Upload the UI zip file as an extension
2. Modify the application information using the following cli command (using https://goc8ycli.netlify.app/)

c8y applications update --id tedge-container-plugin-ui --template '
{
  "manifest": {
    "author": "Felix Schaede",
    "name": "tedge-container-plugin-ui",
    "version": "0.0.3",
    "description": "Adds tab to manage docker containers to a device",
    "license": "Apache 2.0",
    "keywords": [
      "Cumulocity",
      "Plugin",
      "Widget"
    ],
    "contextPath": "tedge-container-plugin-ui",
    "key": "tedge-container-plugin-ui-key",
    "globalTitle": "Thin-edge Docker UI",
    "tabsHorizontal": true,
    "isPackage": true,
    "noAppSwitcher": true,
    "package": "plugin",
    "requiredPlatformVersion": ">=1016.0.0",
    "exports": [
      {
        "name": "Container Info Tab",
        "module": "ContainerInfoModule",
        "path": "./src/container-info/container-info.module.ts",
        "description": "Adds a tab to a container service to display all relevant container information."
      },
      {
        "name": "Container Management Tab",
        "module": "ContainerManagementModule",
        "path": "./src/container-management/container-management.module.ts",
        "description": "Adds a tab to a device to monitor the installed containers"
      },
      {
        "name": "Container Group Management Tab",
        "module": "GroupManagementModule",
        "path": "./src/group-management/group-management.module.ts",
        "description": "Adds a tab to the device to monitor container groups (aka. docker compose)."
      }
    ],
    "remotes": {
      "container-plugin": [
        "ContainerManagementModule"
      ],
      "Group-plugin": [
        "GroupManagementModule"
      ],
      "Container-Info-plugin": [
        "ContainerInfoModule"
      ]
    }
  }
}
'

Alternatively, if you have cloned the tedge-container-plugin project already you can use the ui/package.json file directly in a jsonnet template when using go-c8y-cli.

c8y applications update \
    --id tedge-container-plugin-ui \
    --template "{manifest: (import 'ui/package.json').c8y.application}" \
    --dry

Including additional information about the container would be useful to users to better monitor and troubleshoot containers running on the device.

Example service data model

The following show an example of addition container information that can be added to the service.

{
  "containerId": "1f3f0c98877a",
  "ports": "8080",
  "command": "\"/bin/sh -c 'echo Container started\\n trap \\\"exit 0\\\" 15\\n \\n exec \\\"$@\\\"\\n while sleep 1 & wait $!; do :; done' - /lib/systemd/systemd --system --unit=basic.target\"",
  "networks": "thin-edgeio_devcontainer_default",
  "filesystem": "2.73GB (virtual 5.66GB)",
  "image": "thin-edgeio_devcontainer-tedge",
  "runningFor": "3 weeks ago",
  "state": "running",
  "status": "Up 13 days",
}

There seems to be a bug where the service status of the tedge-container-monitor sometimes shows the down status even when the process is still running (and functioning correctly).

The problem seems to occur reliably when using the tedge-rugpi-image.

I suspect it might be a problem that the up status is only sent on the initial connection, but if there is an interruption, then the LWT sends the down status, however after the reconnection occurs, the up is not sent.

The bug should be easy to reproduce using the following steps:

1. Start tedge-container-monitor service
2. Restart mosquitto
3. Check the tedge-container-monitor service

When trying to install a new image version, the install-command always tries to pull that image. In my case all images are already existing in local image registry and not meant to be pulled. Currently the install command fails as it does not find the image in docker hub.

Suggestion is to only pull images if they are not already available locally. This snippet did the trick for me:

if [ -z "$(sudo docker images -q "${MODULE_VERSION}")" ]; then
   log "Pulling image: ${MODULE_VERSION}"
   "$CONTAINER_CLI" pull "${MODULE_VERSION}"
fi