How to build a connection server for Shelly 3EM
, IBM Cloud Cloudant
, and Grafana
?
Balcony power plants
are trendy, and of course, if you have one, you want to have more detailed information about your current electricity consumption and what you have saved. An easy way is to use Shelly3EM to get this data.
The main objective of this project is to explain how to implement an example using the Shelly 3EM
, IBM Cloud Cloudant
, and Grafana dashboard to perform custom visualization and store historical data for better insights. The project provides detailed information on the architecture, motivation, components, used technologies, setup for various environments, and the step-by-step procedure to configure and run the example using Podman Compose. It also includes instructions for setting up authentication for Grafana and configuring a new dashboard for visualizing the data.
Therefore, the Shelly 3EM Cloudant Grafana connection server
implements an API using FastAPI in Python. The API provides functionality to interact with and manage data in a Cloudant database, retrieve IBM Cloud configuration information, and schedule tasks to save Shelly data.
The gif below shows an example configuration for a new custom dashboard.
The image below shows a simplified architecture.
The following picture displays an instance of a running project in the Raspberry Pi
, Shelly
, IBM Cloudant Database
, shelly-3em-cloudant-grafana -connection-server
, Grafana
configuration.
The following gif shows the running project in action.
Table of Content
- Motivation and main objective
- Levels of knowledge to run the example
- Architectural overview
- Environments
- Used technologies and APIs
- Prerequisites for each environment of the Example
- Setup of the Example in the
Podman compose
environment
The motivation for the project is learning, creating custom visualization, and saving money.
The Shelly 3EM
" is a WiFi smart 3-phase energy meter with contactor control with three independent measuring channels up to 120 A each, contactor control (or load up to 10 A)".
Shelly provides a Cloud subscription in combination with the mobile app called Shelly Smart Control
you have enormous options for real-time
monitoring and managing activations.
Therefore GitHub repository contains an example to implement a Shelly 3EM-Cloudant-Grafana-connection-s contains the implements for a Shelly 3EM
-Cloudant
-Grafana
-connection-server. The implementation doesn't have the objective of monitoring real-time
data of the Shelly 3EM
.
The main objective is to save historical data hourly in the size of 0,375 kb
in a JSON
format for one Shelly data object to reduce storage for the free tier in the lite plan
of an IBM Cloud Cloudant database
.
The visualization of historical hourly data is done by using Grafana
. To display the historical data, we can create incredible custom dashboards with the Open-Source
version of Grafana
.
These are the components used in this project:
- Grafana provides a huge capability to visualize data and is also available as Open-Source for running as a container or an application.
- The
Grafana Json Datasource
is used to customize theIBM Cloud Cloudant Database
access. This usage is to avoid the usage ofenterprise plugins
for Grafana in this project. The project does implement theOpenAPI
specification for thesimPod JSON
to realize a basicdata source connector
that can access the Cloudant database on IBM Cloud.
- The
Shelly 3EM
provides a free API to connect to your device and collect data.IBM Cloud Cloudant
"is a fully managed, distributed database optimized for heavy workloads and fast-growing web and mobile apps, IBM® Cloudant® is available as an IBM Cloud® service with a 99.99% SLA. Cloudant elastically scales throughput and storage, and its API and replication protocols are compatible with Apache CouchDB for hybrid or multi-cloud architectures." and provides a lite plan for the usage of the database for development. ( at the moment for storage1GB
)
The Shelly 3EM
-Cloudant
-Grafana
-connection-server should run in three environments:
Raspberry Pi 3
Containerized
inPodman compose
on a local machineIBM Cloud Code Engine
in combination withRaspberry Pi 3
The following levels of knowledge can be helpful when you run the example:
- Programming language Python:
beginner to intermediate
- Query definition of NoSQL databases:
beginner
- Raspberry Pi:
beginner
- Containerization:
beginner to intermediate
- Grafana:
beginner
- Network and remote access:
beginner
- IBM Cloud:
beginner
- IBM Cloudant:
beginner
- Shelly 3EM:
beginner
- Bash scripting:
beginner
- Linux OS/macOS commands:
beginner
The following diagram shows the simplified dependencies of the basic architecture of the Shelly 3EM Cloudant Grafana connection server
that implements an API using FastAPI in Python. The API provides functionality to interact with and manage data in a Cloudant database, retrieve IBM Cloud configuration information, and schedule tasks to save Shelly data. It also includes endpoints related to Grafana integration for defining metrics, returning metrics payload options, and selecting queries. The code also sets up basic authentication and custom logging configurations.
Main modules of the Shelly 3EM
-Cloudant
-Grafana
-connection-server
-
Cloudant connection module
The main objective of this module is to interact with the IBM Cloud Cloudant service using REST API calls to perform various operations such as retrieving information about service instances, databases, indexes, documents, and views and searching for data. The code also handles the authentication and token management required for accessing the Cloudant service. The functions in the module handle different API calls and error handling to ensure smooth interaction with the Cloudant service. The module uses the IBM Cloud
IAM authentication
to access the database. -
Grafana connection module
The module implements the needed endpoints for the
Grafana JSON Datasource
integration to retrieve, format, and return data from a Cloudant database for integration with Grafana. The module defines functions to convert and retrieve specific data from preset search queries and views and to handle and format empty datasets if needed. Additionally, the module provides options for configuring and selecting different data sources and metrics for use in Grafana. The module also includes logging configuration based on a specified log level. -
Shelly connection module
The module implements how to get the consumption data, which will be saved in the Cloudant database; therefore, the module retrieves power consumption data from a Shelly device using REST API calls and authentication. The module includes functions to fetch overall consumption data and data for specific phases. It handles authentication, constructs the necessary headers, and validates the response from the API. The module also sets up logging based on the provided configuration. Overall, the module focuses on interacting with the Shelly device API to retrieve power consumption data.
-
IBM Cloud connection module
The module implements how to get an IAM authentication token from IBM Cloud. The module configures logging, loads IBM cloud environment variables and API keys, and then uses that information to make a POST request to the IBM cloud identity token endpoint to retrieve an access token. The module also handles the response from the token request and returns the access token and its verification status.
-
Data format for the Shelly data
{ "_id": "03e3f5f27e8742aea0035ff4bf0dd3ee", "_rev": "1-1c8c94efab24a8da925978234e7e4ce6", "result": { "emeters": [ { "power": 178.08, "pf": 0.72, "current": 1.11, "voltage": 225.86, "is_valid": true, "total": 60608, "total_returned": 320.9 }, { "power": 4.71, "pf": 0.53, "current": 0.04, "voltage": 226.14, "is_valid": true, "total": 32062.3, "total_returned": 0 }, { "power": 195.98, "pf": 0.93, "current": 0.95, "voltage": 225.08, "is_valid": true, "total": 51733.8, "total_returned": 0 } ], "date": "2025-12-25 21:41:50", "total_power": 378.77 } }
Here is a basic flow of the Shelly 3EM
-Cloudant
-Grafana
-connection-server usage:
1. Get data from `Shelly 3EM`
2. Save data in a `Cloudant` database
3. Display data in `Grafana`
This section is a basic overview of the environments, their components, and their implementation status.
-
For all environments:
Shelly 3EM
IBM Cloud Cloudant database
- Grafana data source JSON
-
Raspberry Pi 3
status (done):-
RAM:
1G
-
Python3: Version
3.9
-
Local applications:
Shelly Cloudant Grafana
server application- Grafana server
v10.2.3
Simplified architecture overview:
Detailed information related to the setup is in the following additional readmes:
-
-
Podman compose
status (done):-
Containers:
Shelly Cloudant Grafana
server application- Grafana server
v10.2.3
Simplified architecture overview:
Note: Detailed information related to the setup is in this section _5. Setup of the Example in the
Podman compose
environment" in this document. -
-
IBM Cloud Code Engine
andRaspberry Pi 3
status (done):-
Raspberry Pi 3
-
- Container including:
Shelly Cloudant Grafana
server application- Grafana server
v10.2.3
- Container including:
-
Simplified architecture overview:
Detailed information related to the setup is in the following additional readmes:
-
These are the used technologies and APIs.
- Programming language:
Python
- Python runtime:
3.9
or higher - Server framework: FastAPI
- Container runtime:
Podman
and Podman compose - APIs:
- Shelly API
- IBM Cloud Cloudant API
- Implementation of
Open API
definition forGrafana JSON Datasource
We need to know the configuration data for Shelly 3EM
, IBM Cloud
, Cloudant
, and the WLAN router
. The image below shows the simplified dependencies for the Podman compose
environment.
- Containers running in
Podman compose
Shelly Cloudant Grafana
server application- Grafana server
6.1 Get Shelly 3EM
configuration
6.2 Setup of the IBM Cloud Cloudant
service
WLAN IP
address of theShelly 3EM
- Restrict access to your
Shelly 3EM
- Get the
Shelly 3EM ID
The following values will be later saved in an environment variables file.
export SHELLY_URL=http://192.168.133.1
export SHELLY_USER=admin
export SHELLY_PASSWORD=admin
export SHELLY_ID=XXX
-
Create an IBM Cloud Cloudant service and ensure you select
Lite
as your plan option for the example usage. -
Create new IBM Cloud Cloudant service credentials as
Manager
You can inspect the details about the authentication in the IBM Cloud documentation
The following values will be later saved in an environment variables file.
# IBM Cloudant
export CLOUDANT_URL="https://XXX.cloudantnosqldb.appdomain.cloud"
export CLOUDANT_DB_NAME=shelly-3em-data
# IBM Cloud
export IBMCLOUD_APIKEY=XXX
export IBMCLOUD_URL=https://iam.cloud.ibm.com/identity/token
- The creation of the
database
,view
, andquery index
in the IBM Cloud Cloudant service will be done in section 7.3 RunGrafana
andshelly3em-cloudant-grafana-connection-server
with "Podman compose".
Note: Details are in this project's IBM Cloud Cloudant configuration README
.
The Podman compose
environment is the best to start and verify how the integration works. You can easily run it on your computer.
Prerequisites overview
Ensure you have:
- .. Python 3.9 or higher installed on your machine.
- ... (Optional) Setup a virtual environment for Python
- ... the Shelly configuration information.
- ... an IBM Cloud account.
- ... an IBM Cloud Cloudant service instance.
Setup content overview
- Get the source code and create a virtual Python environment
(Optional)
Run the server locally first- Run
Grafana
andshelly3em-cloudant-grafana-connection-server
with "Podman compose" - Create and configure the Cloudant database
- Start the schedule to save Shelly data
- Configure Grafana and create a custom dashboard
git clone https://github.com/thomassuedbroecker/shelly-cloudant-grafana.git
cd code
cat .env-environment > .env
# Shelly 3EM
export SHELLY_URL=http://192.168.133.1
export SHELLY_USER=admin
export SHELLY_PASSWORD=admin
export SHELLY_ID=XXX
# IBM Cloudant
export CLOUDANT_URL="https://XXX.cloudantnosqldb.appdomain.cloud"
export CLOUDANT_DB_NAME=shelly-3em-data
# IBM Cloud
export IBMCLOUD_APIKEY=XXX
export IBMCLOUD_URL=https://iam.cloud.ibm.com/identity/token
# APP
export APP_USER=adm
export APP_APIKEY=admin
# Options are DEBUG, INFO, and WARNING
export APP_LOG=DEBUG
# Grafana
export GF_SECURITY_ADMIN_USER=admin
export GF_SECURITY_ADMIN_PASSWORD=admin
export GF_INSTALL_PLUGINS=simpod-json-datasource
We get the source code and verify if the application is running locally.
cd shelly-cloudant-grafana/code
python3.11 -m venv shelly-cloudant-grafana-env-3.11
source shelly-cloudant-grafana-env-3.11/bin/activate
- Option 1:
python3 -m pip install --upgrade pip
python3 -m pip install "fastapi[all]"
python3 -m pip install fastapi-utils
python3 -m pip install schedule
python3 -m pip install requests
python3 -m pip install pydantic
python3 -m pip install typing
python3 -m pip install --upgrade "ibmcloudant>=0.0.27"
python3 -m pip freeze > requirements.txt
- Option 2:
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
cd code
source source shelly-cloudant-grafana-env-3.11/bin/activate
source .env
python3 shelly-3em-cloudant-connector.py
We need to ensure that the Cloudant database
exists and that the search index and the view have been created by you so that the Grafana integration is working. The configuration of the Cloudant database can be set up by you with the shelly3em-cloudant-grafana-connection-server
running in "Podman compose".
Note: Verify all environment variables are set in the .env
file.
In Podman compose
, we run Grafana
and shelly3em-cloudant-grafana-connection-server
servers and these are the main three topics of interest in this environment.
-
The
Podman compose
environment is for simple development and buildsshelly3em-cloudant-grafana-connection-server
each time when compose is started. -
The
Grafana
server depends on a successful start of theshelly3em-cloudant-grafana-connection-server
. -
Later, we connect the
Grafana server
toshelly3em-cloudant-grafana-connection-server
inside thePodman compose
networkhttps://shelly-cloudant-grafana:8081
therefore we need to configure the Cloudant database.
Here is the content of the file to configure Podman compose
:
version: "2.15.1"
services:
shelly-cloudant-grafana:
build:
context: ${PODMAN_CONTEXT}
dockerfile: ${PODMAN_CONTEXT}/docker/Dockerfile
image: shelly-cloudant-grafana:1.0.0
container_name: shelly-cloudant-grafana
ports:
- 8081:8081
environment:
- SHELLY_URL=${SHELLY_URL}
- SHELLY_USER=${SHELLY_USER}
- SHELLY_PASSWORD=${SHELLY_PASSWORD}
- SHELLY_ID=${SHELLY_ID}
- CLOUDANT_URL=${CLOUDANT_URL}
- CLOUDANT_DB_NAME=${CLOUDANT_DB_NAME}
- IBMCLOUD_APIKEY=${IBMCLOUD_APIKEY}
- IBMCLOUD_URL=${IBMCLOUD_URL}
- APP_USER=${APP_USER}
- APP_APIKEY=${APP_APIKEY}
- APP_LOG=${APP_LOG}
grafana-server:
image: docker.io/grafana/grafana:latest
container_name: grafana
environment:
- GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER}
- GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}
- GF_INSTALL_PLUGINS==${GF_INSTALL_PLUGINS}
ports:
- 3000:3000
links:
- shelly-cloudant-grafana
depends_on:
- shelly-cloudant-grafana
The following steps will guide setting up the needed environment on a macOS
.
Install the container runtime podman-desktop.
An example installation with brew
on macOS is here.
brew install --cask podman-desktop
The image below shows the running podman-desktop
on a local machine.
Step 2: Install podman-compose
The podman-compose
installation runs with Python.
python3 -m pip install --upgrade pip
python3 -m pip install podman-compose
Verify installation:
podman-compose --help
- Start
Podman compose
with the bash scriptstart-containers.sh
The script verifies that Podman
runs and loads the needed environment variables.
cd scripts/local-container
sh start-containers.sh
The image below shows the Podman compose
running the container in the Podman Desktop
application.
The database name is defined in the environment variable export CLOUDANT_DB_NAME=shelly-3em-data
in the .env
file.
- Open the
shelly-cloudant-grafana
serveropen http://0.0.0.0:8081/docs
- This has opened a browser
Swagger UI
of theshelly-cloudant-grafana
server. In this UI select the endpoint:db_create_cloudant_database
- Press
Try it out
- Press
Execute
The gif below shows the steps.
Now, we must create a view
and query index
in the database. Therefore, the documents are in the folder code/cloudant_config
of the cloned GitHub project.
These kinds of documents are called design documents
.
-
Select in the
Swagger UI
the endpointdb_create_doc_from_file
-
Select the file with the document to upload:
upload-1-view-all-data.json
,upload-2-search_index_all_data_javascipt.json
-
Press
Execute
-
Verify if the document was created correctly. After the execution, you can see in the
Response
that the template file was used correctly to create a document in the database.
To verify the created design documents.
The schedule is hard coded
to save the Shelly data hourly.
!Don't wait for a response! See notes.
- Select in the
Swagger UI
the endpointschedule_start
- Press
Try it out
- Press
Execute
- !Don't wait for a response! See notes.
- Verify the schedule is running by selecting in the
Swagger UI
the endpointschedule_get_status
Note: Using the Overload functionality for functions in Python for the function schedule_start. There are better ways to implement a job schedule; the implementation will be changed in future versions. You can find details in the https://www.scaler.com/topics/function-overloading-in-python/ blog post.
This section depends on variables configured in the environment variables for the Grafana
container.
Note: You can find examples of the environment variables in the [Grafana.ini
] file(https://github.com/grafana/grafana/blob/main/conf/defaults.ini).
- Open
Grafana
in a browser
open http://localhost:3000
- Login
User: admin
Password: admin123
- Update password
Add the simPod JSON
as a data source
.
Note: To avoid the usage of enterprise plugins
for Grafana in this project. The project does implement the OpenAPI
specification for the [
simPod JSON](https://grafana.com/grafana/plugins/simpod-json-datasource/) to realize a primary
data source connector` that can access the Cloudant database on IBM Cloud.
GitHub project with the OpenAPI specification
Now we connect our
- URL:
http://shelly-cloudant-grafana:8081
- USER:
adm
- PASSWORD:
admin
- Select
New dashboard
- Press
add Visualization
- Select
simpod-json-datasource
- Select
Shelly 3EM Metrics
- Insert a payload configuration
Here, you can configure search_option
with search
or view
.
If you select "search" you can specify topics like date
and add a search string for the Cloudant search.
The following payload will provide all the Shelly data from the date 2024-01
.
{
"search_option":"search",
"search_topic":"date",
"search_string":"\"2024-01*\""
}
- If you see that the
time field
is missing.
- We need to map the field
time
of the return values fromtype string
to thetype time
. Therefore, we are using theTransform data
functionality of Grafana.
- In
Transform data
we selectConvert field type
.
- Now, we select the
time
field and choose time astime
type.
![]() |
![]() |
- In this step, we configure the format.
Note: If you don't see any values ensure your dashboard configuration time frame fits your return values.
Overall, it is incredible that Shelly3EM provides a documented REST API so that you can implement custom solutions.
This example solution shows many setup options to use an IBM Cloud Cloudant Database to save the historical data for your Shelly data and access your custom visualization from your local network or the internet with minimal cost.
With that example, you can keep informed about your power consumption and what you return to your power provider generated by your Balcony power plant
.
The Grafana UI usage is excellent in local computer browsers or on a mobile browser.
Feel free to implement your configuration or add additional options for other databases or runtimes.