Define your custom visualization for `Shelly 3EM` with `Grafana`

How to build a connection server for Shelly 3EM, IBM Cloud Cloudant, and Grafana?

Balcony power plants are trendy, and of course, if you have one, you want to have more detailed information about your current electricity consumption and what you have saved. An easy way is to use Shelly3EM to get this data.

The main objective of this project is to explain how to implement an example using the Shelly 3EM, IBM Cloud Cloudant, and Grafana dashboard to perform custom visualization and store historical data for better insights. The project provides detailed information on the architecture, motivation, components, used technologies, setup for various environments, and the step-by-step procedure to configure and run the example using Podman Compose. It also includes instructions for setting up authentication for Grafana and configuring a new dashboard for visualizing the data.

Therefore, the Shelly 3EM Cloudant Grafana connection server implements an API using FastAPI in Python. The API provides functionality to interact with and manage data in a Cloudant database, retrieve IBM Cloud configuration information, and schedule tasks to save Shelly data.

The gif below shows an example configuration for a new custom dashboard.

The image below shows a simplified architecture.

The following picture displays an instance of a running project in the Raspberry Pi, Shelly, IBM Cloudant Database, shelly-3em-cloudant-grafana -connection-server, Grafana configuration.

The following gif shows the running project in action.

Table of Content

Motivation and main objective
Levels of knowledge to run the example
Architectural overview
Environments
Used technologies and APIs
Prerequisites for each environment of the Example
Setup of the Example in the Podman compose environment

1. Motivation and main objective

The motivation for the project is learning, creating custom visualization, and saving money.

The Shelly 3EM " is a WiFi smart 3-phase energy meter with contactor control with three independent measuring channels up to 120 A each, contactor control (or load up to 10 A)".

Shelly provides a Cloud subscription in combination with the mobile app called Shelly Smart Control you have enormous options for real-time monitoring and managing activations.

Therefore GitHub repository contains an example to implement a Shelly 3EM-Cloudant-Grafana-connection-s contains the implements for a Shelly 3EM-Cloudant-Grafana-connection-server. The implementation doesn't have the objective of monitoring real-time data of the Shelly 3EM.

The main objective is to save historical data hourly in the size of 0,375 kb in a JSON format for one Shelly data object to reduce storage for the free tier in the lite plan of an IBM Cloud Cloudant database. The visualization of historical hourly data is done by using Grafana. To display the historical data, we can create incredible custom dashboards with the Open-Source version of Grafana.

These are the components used in this project:

Grafana provides a huge capability to visualize data and is also available as Open-Source for running as a container or an application.
- The Grafana Json Datasource is used to customize the IBM Cloud Cloudant Database access. This usage is to avoid the usage of enterprise plugins for Grafana in this project. The project does implement the OpenAPI specification for the simPod JSON to realize a basic data source connector that can access the Cloudant database on IBM Cloud.
Shelly 3EM provides a free API to connect to your device and collect data.
IBM Cloud Cloudant "is a fully managed, distributed database optimized for heavy workloads and fast-growing web and mobile apps, IBM® Cloudant® is available as an IBM Cloud® service with a 99.99% SLA. Cloudant elastically scales throughput and storage, and its API and replication protocols are compatible with Apache CouchDB for hybrid or multi-cloud architectures." and provides a lite plan for the usage of the database for development. ( at the moment for storage 1GB)

The Shelly 3EM-Cloudant-Grafana-connection-server should run in three environments:

Raspberry Pi 3
Containerized in Podman compose on a local machine
IBM Cloud Code Engine in combination with Raspberry Pi 3

2. Levels of knowledge to run the example

The following levels of knowledge can be helpful when you run the example:

Programming language Python: beginner to intermediate
Query definition of NoSQL databases: beginner
Raspberry Pi: beginner
Containerization: beginner to intermediate
Grafana: beginner
Network and remote access: beginner
IBM Cloud: beginner
IBM Cloudant: beginner
Shelly 3EM: beginner
Bash scripting: beginner
Linux OS/macOS commands: beginner

3. Architectural overview

The following diagram shows the simplified dependencies of the basic architecture of the Shelly 3EM Cloudant Grafana connection server that implements an API using FastAPI in Python. The API provides functionality to interact with and manage data in a Cloudant database, retrieve IBM Cloud configuration information, and schedule tasks to save Shelly data. It also includes endpoints related to Grafana integration for defining metrics, returning metrics payload options, and selecting queries. The code also sets up basic authentication and custom logging configurations.

Main modules of the Shelly 3EM-Cloudant-Grafana-connection-server

Cloudant connection module

The main objective of this module is to interact with the IBM Cloud Cloudant service using REST API calls to perform various operations such as retrieving information about service instances, databases, indexes, documents, and views and searching for data. The code also handles the authentication and token management required for accessing the Cloudant service. The functions in the module handle different API calls and error handling to ensure smooth interaction with the Cloudant service. The module uses the IBM Cloud IAM authentication to access the database.
Grafana connection module

The module implements the needed endpoints for the Grafana JSON Datasource integration to retrieve, format, and return data from a Cloudant database for integration with Grafana. The module defines functions to convert and retrieve specific data from preset search queries and views and to handle and format empty datasets if needed. Additionally, the module provides options for configuring and selecting different data sources and metrics for use in Grafana. The module also includes logging configuration based on a specified log level.
Shelly connection module

The module implements how to get the consumption data, which will be saved in the Cloudant database; therefore, the module retrieves power consumption data from a Shelly device using REST API calls and authentication. The module includes functions to fetch overall consumption data and data for specific phases. It handles authentication, constructs the necessary headers, and validates the response from the API. The module also sets up logging based on the provided configuration. Overall, the module focuses on interacting with the Shelly device API to retrieve power consumption data.
IBM Cloud connection module

The module implements how to get an IAM authentication token from IBM Cloud. The module configures logging, loads IBM cloud environment variables and API keys, and then uses that information to make a POST request to the IBM cloud identity token endpoint to retrieve an access token. The module also handles the response from the token request and returns the access token and its verification status.

Data format for the Shelly data

{
  "_id": "03e3f5f27e8742aea0035ff4bf0dd3ee",
  "_rev": "1-1c8c94efab24a8da925978234e7e4ce6",
  "result": {
    "emeters": [
      {
        "power": 178.08,
        "pf": 0.72,
        "current": 1.11,
        "voltage": 225.86,
        "is_valid": true,
        "total": 60608,
        "total_returned": 320.9
      },
      {
        "power": 4.71,
        "pf": 0.53,
        "current": 0.04,
        "voltage": 226.14,
        "is_valid": true,
        "total": 32062.3,
        "total_returned": 0
      },
      {
        "power": 195.98,
        "pf": 0.93,
        "current": 0.95,
        "voltage": 225.08,
        "is_valid": true,
        "total": 51733.8,
        "total_returned": 0
      }
    ],
    "date": "2025-12-25 21:41:50",
    "total_power": 378.77
  }
}

Here is a basic flow of the Shelly 3EM-Cloudant-Grafana-connection-server usage:

1. Get data from `Shelly 3EM`
2. Save data in a `Cloudant` database
3. Display data in `Grafana`

4. Environments

This section is a basic overview of the environments, their components, and their implementation status.

For all environments:
- Shelly 3EM
- IBM Cloud Cloudant database
- Grafana data source JSON
Raspberry Pi 3 status (done):
- RAM: 1G
- Python3: Version 3.9
- Local applications:
  - Shelly Cloudant Grafana server application
  - Grafana server v10.2.3
  Simplified architecture overview:
Detailed information related to the setup is in the following additional readmes:
Podman compose status (done):
- Containers:
  - Shelly Cloudant Grafana server application
  - Grafana server v10.2.3
  Simplified architecture overview:
Note: Detailed information related to the setup is in this section _5. Setup of the Example in the Podman compose environment" in this document.
IBM Cloud Code Engine and Raspberry Pi 3 status (done):
- Raspberry Pi 3
- IBM Cloud Code Engine
  - Container including:
    - Shelly Cloudant Grafana server application
    - Grafana server v10.2.3
- IBM Cloud Container Registry
  
  Simplified architecture overview:
Detailed information related to the setup is in the following additional readmes:

5. Used technologies and APIs

These are the used technologies and APIs.

Programming language: Python
Python runtime: 3.9 or higher
Server framework: FastAPI
Container runtime: Podman and Podman compose
APIs:
- Shelly API
- IBM Cloud Cloudant API
- Implementation of Open API definition for Grafana JSON Datasource

6. Prerequisites for each environment of the Example

We need to know the configuration data for Shelly 3EM, IBM Cloud, Cloudant, and the WLAN router. The image below shows the simplified dependencies for the Podman compose environment.

Containers running in Podman compose
- Shelly Cloudant Grafana server application
- Grafana server

6.1 Get Shelly 3EM configuration 6.2 Setup of the IBM Cloud Cloudant service

6.1 Get `Shelly 3EM` configuration

WLAN IP address of the Shelly 3EM

Restrict access to your Shelly 3EM

Get the Shelly 3EM ID

The following values will be later saved in an environment variables file.

export SHELLY_URL=http://192.168.133.1
export SHELLY_USER=admin
export SHELLY_PASSWORD=admin
export SHELLY_ID=XXX

6.2 Setup of the `IBM Cloud Cloudant` service

Create an IBM Cloud Cloudant service and ensure you select Lite as your plan option for the example usage.
Create new IBM Cloud Cloudant service credentials as Manager

You can inspect the details about the authentication in the IBM Cloud documentation
Create an IBM Cloud API key

The following values will be later saved in an environment variables file.

# IBM Cloudant
export CLOUDANT_URL="https://XXX.cloudantnosqldb.appdomain.cloud"
export CLOUDANT_DB_NAME=shelly-3em-data
# IBM Cloud
export IBMCLOUD_APIKEY=XXX
export IBMCLOUD_URL=https://iam.cloud.ibm.com/identity/token

The creation of the database, view, and query index in the IBM Cloud Cloudant service will be done in section 7.3 Run Grafana and shelly3em-cloudant-grafana-connection-server with "Podman compose".

Note: Details are in this project's IBM Cloud Cloudant configuration README.

7. Setup of the Example in the `Podman compose` environment

The Podman compose environment is the best to start and verify how the integration works. You can easily run it on your computer.

Prerequisites overview

Ensure you have:

.. Python 3.9 or higher installed on your machine.
... (Optional) Setup a virtual environment for Python
... the Shelly configuration information.
... an IBM Cloud account.
... an IBM Cloud Cloudant service instance.

Setup content overview

Get the source code and create a virtual Python environment
(Optional) Run the server locally first
Run Grafana and shelly3em-cloudant-grafana-connection-server with "Podman compose"
Create and configure the Cloudant database
Start the schedule to save Shelly data
Configure Grafana and create a custom dashboard

7.1. Get the source code and create a virtual Python environment

Step 1: Clone the project

git clone https://github.com/thomassuedbroecker/shelly-cloudant-grafana.git

Step 2: Create an environment file from the template

cd code
cat .env-environment > .env

Step 3: Insert the needed values

# Shelly 3EM
export SHELLY_URL=http://192.168.133.1
export SHELLY_USER=admin
export SHELLY_PASSWORD=admin
export SHELLY_ID=XXX
# IBM Cloudant
export CLOUDANT_URL="https://XXX.cloudantnosqldb.appdomain.cloud"
export CLOUDANT_DB_NAME=shelly-3em-data
# IBM Cloud
export IBMCLOUD_APIKEY=XXX
export IBMCLOUD_URL=https://iam.cloud.ibm.com/identity/token
# APP
export APP_USER=adm
export APP_APIKEY=admin
# Options are DEBUG, INFO, and WARNING
export APP_LOG=DEBUG 
# Grafana
export GF_SECURITY_ADMIN_USER=admin
export GF_SECURITY_ADMIN_PASSWORD=admin
export GF_INSTALL_PLUGINS=simpod-json-datasource

7.2 `(Optional)` Run the server locally first

We get the source code and verify if the application is running locally.

Step 1: Create a virtual Python environment

cd shelly-cloudant-grafana/code
python3.11 -m venv shelly-cloudant-grafana-env-3.11
source shelly-cloudant-grafana-env-3.11/bin/activate

Step 2: Install the needed Python libs and create a `requirements.txt` file

Option 1:

python3 -m pip install --upgrade pip
python3 -m pip install "fastapi[all]"
python3 -m pip install fastapi-utils
python3 -m pip install schedule
python3 -m pip install requests
python3 -m pip install pydantic
python3 -m pip install typing
python3 -m pip install --upgrade "ibmcloudant>=0.0.27"
python3 -m pip freeze > requirements.txt

Option 2:

python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt

Step 3: Run `shelly-3em-cloudant-connector` server

cd code
source source shelly-cloudant-grafana-env-3.11/bin/activate
source .env
python3 shelly-3em-cloudant-connector.py

7.3 Run `Grafana` and `shelly3em-cloudant-grafana-connection-server` with "Podman compose"

We need to ensure that the Cloudant database exists and that the search index and the view have been created by you so that the Grafana integration is working. The configuration of the Cloudant database can be set up by you with the shelly3em-cloudant-grafana-connection-server running in "Podman compose".

Note: Verify all environment variables are set in the .env file.

In Podman compose, we run Grafana and shelly3em-cloudant-grafana-connection-server servers and these are the main three topics of interest in this environment.

The Podman compose environment is for simple development and builds shelly3em-cloudant-grafana-connection-server each time when compose is started.
The Grafana server depends on a successful start of the shelly3em-cloudant-grafana-connection-server.
Later, we connect the Grafana server to shelly3em-cloudant-grafana-connection-server inside the Podman compose network https://shelly-cloudant-grafana:8081 therefore we need to configure the Cloudant database.

Here is the content of the file to configure Podman compose:

version: "2.15.1"
services:
  shelly-cloudant-grafana:
    build: 
      context: ${PODMAN_CONTEXT}
      dockerfile: ${PODMAN_CONTEXT}/docker/Dockerfile
    image: shelly-cloudant-grafana:1.0.0
    container_name: shelly-cloudant-grafana
    ports:
      - 8081:8081
    environment:
      - SHELLY_URL=${SHELLY_URL}
      - SHELLY_USER=${SHELLY_USER}
      - SHELLY_PASSWORD=${SHELLY_PASSWORD}
      - SHELLY_ID=${SHELLY_ID}
      - CLOUDANT_URL=${CLOUDANT_URL}
      - CLOUDANT_DB_NAME=${CLOUDANT_DB_NAME}
      - IBMCLOUD_APIKEY=${IBMCLOUD_APIKEY}
      - IBMCLOUD_URL=${IBMCLOUD_URL}
      - APP_USER=${APP_USER}
      - APP_APIKEY=${APP_APIKEY}
      - APP_LOG=${APP_LOG}
  grafana-server:
    image: docker.io/grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}
      - GF_INSTALL_PLUGINS==${GF_INSTALL_PLUGINS}
    ports:
      - 3000:3000
    links:
    - shelly-cloudant-grafana
    depends_on:
    - shelly-cloudant-grafana

The following steps will guide setting up the needed environment on a macOS.

Step 1: Install the container runtime `podman-desktop`

Install the container runtime podman-desktop. An example installation with brew on macOS is here.

brew install --cask podman-desktop

The image below shows the running podman-desktop on a local machine.

Step 2: Install `podman-compose`

The podman-compose installation runs with Python.

python3 -m pip install --upgrade pip
python3 -m pip install podman-compose

Verify installation:

podman-compose --help

Step 3: Start the containers locally using `Podman compose`

Start Podman compose with the bash script start-containers.sh

The script verifies that Podman runs and loads the needed environment variables.

cd scripts/local-container
sh start-containers.sh

The image below shows the Podman compose running the container in the Podman Desktop application.

7.4 Create and configure the Cloudant database

Step 1: Create the database in the IBM Cloud Cloudant service

The database name is defined in the environment variable export CLOUDANT_DB_NAME=shelly-3em-data in the .env file.

Open the shelly-cloudant-grafana server
```
open http://0.0.0.0:8081/docs
```
This has opened a browser Swagger UI of the shelly-cloudant-grafana server. In this UI select the endpoint: db_create_cloudant_database
Press Try it out
Press Execute

The gif below shows the steps.

Step 2: Create the needed `view` and `query index` in the database

Now, we must create a view and query index in the database. Therefore, the documents are in the folder code/cloudant_config of the cloned GitHub project. These kinds of documents are called design documents.

Select in the Swagger UI the endpoint db_create_doc_from_file
Press Try it out
Select the file with the document to upload: upload-1-view-all-data.json, upload-2-search_index_all_data_javascipt.json
Press Execute
Verify if the document was created correctly. After the execution, you can see in the Response that the template file was used correctly to create a document in the database.

Step 3: Verify the created design documents

To verify the created design documents.

Select in the Swagger UI the endpoint db_get_all_design_docs
Press Try it out
Press Execute
Verify

7.5 Start the schedule to save Shelly data

Step 1: Start the schedule to save the Shelly data frequently.

The schedule is hard coded to save the Shelly data hourly.

!Don't wait for a response! See notes.

Select in the Swagger UI the endpoint schedule_start
Press Try it out
Press Execute
!Don't wait for a response! See notes.
Verify the schedule is running by selecting in the Swagger UI the endpoint schedule_get_status

Note: Using the Overload functionality for functions in Python for the function schedule_start. There are better ways to implement a job schedule; the implementation will be changed in future versions. You can find details in the https://www.scaler.com/topics/function-overloading-in-python/ blog post.

7.6 Configure Grafana and create a dashboard

Step 1: Configure authentication for `Grafana`

This section depends on variables configured in the environment variables for the Grafana container.

Note: You can find examples of the environment variables in the [Grafana.ini] file(https://github.com/grafana/grafana/blob/main/conf/defaults.ini).

Open Grafana in a browser

open http://localhost:3000

Login

User: admin Password: admin123

Update password

Step 2: Add the `JSON` data source

Add the simPod JSON as a data source.

Note: To avoid the usage of enterprise plugins for Grafana in this project. The project does implement the OpenAPI specification for the [simPod JSON](https://grafana.com/grafana/plugins/simpod-json-datasource/) to realize a primary data source connector` that can access the Cloudant database on IBM Cloud.

GitHub project with the OpenAPI specification

Step 3: Configure the `data source`

Now we connect our

URL: http://shelly-cloudant-grafana:8081
USER: adm
PASSWORD: admin

Step 4: `Save and test` the connection

Step 5: Add a new `dashboard`

Select New dashboard

Press add Visualization

Select simpod-json-datasource

Select Shelly 3EM Metrics

Insert a payload configuration

Here, you can configure search_option with search or view. If you select "search" you can specify topics like date and add a search string for the Cloudant search.

The following payload will provide all the Shelly data from the date 2024-01.

{
  "search_option":"search",
  "search_topic":"date",
  "search_string":"\"2024-01*\""
}

If you see that the time field is missing.

We need to map the field time of the return values from type string to the type time. Therefore, we are using the Transform data functionality of Grafana.

In Transform data we select Convert field type.

Now, we select the time field and choose time as time type.

In this step, we configure the format.

Note: If you don't see any values ensure your dashboard configuration time frame fits your return values.

8. Summary

Overall, it is incredible that Shelly3EM provides a documented REST API so that you can implement custom solutions.

This example solution shows many setup options to use an IBM Cloud Cloudant Database to save the historical data for your Shelly data and access your custom visualization from your local network or the internet with minimal cost.

With that example, you can keep informed about your power consumption and what you return to your power provider generated by your Balcony power plant.

The Grafana UI usage is excellent in local computer browsers or on a mobile browser.

Feel free to implement your configuration or add additional options for other databases or runtimes.

Name		Name	Last commit message	Last commit date
Latest commit History 20 Commits
code		code
deployment-log		deployment-log
images		images
scripts		scripts
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md

License

thomassuedbroecker/shelly-cloudant-grafana

Folders and files

Latest commit

History

Repository files navigation

Define your custom visualization for Shelly 3EM with Grafana

1. Motivation and main objective

2. Levels of knowledge to run the example

3. Architectural overview

4. Environments

5. Used technologies and APIs

6. Prerequisites for each environment of the Example

6.1 Get Shelly 3EM configuration

6.2 Setup of the IBM Cloud Cloudant service

7. Setup of the Example in the Podman compose environment

7.1. Get the source code and create a virtual Python environment

Step 1: Clone the project

Step 2: Create an environment file from the template

Step 3: Insert the needed values

7.2 (Optional) Run the server locally first

Step 1: Create a virtual Python environment

Step 2: Install the needed Python libs and create a requirements.txt file

Step 3: Run shelly-3em-cloudant-connector server

7.3 Run Grafana and shelly3em-cloudant-grafana-connection-server with "Podman compose"

Step 1: Install the container runtime podman-desktop

Step 2: Install podman-compose

Step 3: Start the containers locally using Podman compose

7.4 Create and configure the Cloudant database

Step 1: Create the database in the IBM Cloud Cloudant service

Step 2: Create the needed view and query index in the database

Step 3: Verify the created design documents

7.5 Start the schedule to save Shelly data

Step 1: Start the schedule to save the Shelly data frequently.

7.6 Configure Grafana and create a dashboard

Step 1: Configure authentication for Grafana

Step 2: Add the JSON data source

Step 3: Configure the data source

Step 4: Save and test the connection

Step 5: Add a new dashboard

8. Summary

About

Topics

Resources