Announcement: OpenDTU Logger release

Last year, I installed solar panels using Hoymiles micro inverters. Typically, these rely on a proprietary cloud platform and a DTU (Data Transfer Unit) for communication. The Hoymiles S-Miles cloud platform is limited in its functionality and using it comes with obvious privacy implications. Through reverse engineering, the OpenDTU and AhoyDTU projects have managed to develop open source, locally running alternatives for this platform.

Using OpenDTU, Hoymiles inverters can be monitored and their output limited dynamically. Dynamic power limits are becoming increasingly relevant for those with dynamic hourly energy pricing, as energy prices can sometimes be negative. Unfortunately, OpenDTU does not support storing historical data. To store the inverter logs retrieved by OpenDTU into a database, I am happy to announce a small helper program, OpenDTU Logger.

An ESP32-board specifically designed for the OpenDTU is the OpenDTU Fusion, which I bought to replace the existing proprietary DTU. The OpenDTU Fusion integrates an ESP32 with radio modules required to communicate with the newer Hoymiles HMS/HMT-series and the older HM-series respectively.

As the OpenDTU software is designed to run on ESP32 boards without a large amount of storage, it does not store historical production data. Instead, users are supposed to rely on individual MQTT topics to which it publishes metrics for this. For me, this was not convenient as it implies storing each metric individually as a “timestamp, metric, value” tuple. I built a proof-of-concept using this approach, but although functional, it was inefficient in terms of storage requirements and data processing.

Instead, I’d like to make use of normalised, SQL-based tables, formatted like “timestamp, metric1, metric2, etc.”. Fortunately, the web interface of OpenDTU relies on a websocket connection, which provides an output more suitable for this, as it outputs, for each inverter, all data at once, in a JSON-based structure.

After not being able to find standalone implementations capable of logging the OpenDTU output into a database efficiently, outside of integrated solutions for platforms such as Home Assistant, I decided to write something myself. I have tested it myself for the past few months and am happy with how it performs.

Using this post, I’d like to announce that I’m making my OpenDTU Logger open source. OpenDTU Logger can be found on my personal git server and is also available as a Docker image or in binary form for use on Linux, macOS and Windows.

Getting started

To parse the OpenDTU output, I first wanted to get an understanding of it. Partially, I managed to understand the output using the web API documentation. Additionally, I installed websocat, which is a command line client for WebSockets.

Using the following command, I was able to retrieve the data sent by OpenDTU

./websocat.x86_64-unknown-linux-musl "ws://192.168.1.9/livedata"

Uncovering the data structure

To store the OpenDTU output into a database, I decided to write a program using Golang, OpenDTU Logger. OpenDTU Logger currently works as follows:

1. On startup, if they don’t exist, a series of SQL tables is created.
2. OpenDTU Logger subscribes to the OpenDTU websocket
3. As messages start to come in, the JSON objects are parsed into a series of structs.
4. OpenDTU Logger detects if the amount of events has increased. If this is the case, these are inserted into the DB.
5. OpenDTU Logger detects if the age of the data is “0”. If that is the case, new log entries are created.

Visualising the data

The repository also contains a series of Grafana dashboards to visualise the data. For more information, please refer to the README.md file.

Feedback

I welcome your feedback! Please leave a comment below or contact me via email with any comments or suggestions.