Part 1 of this article provides some background information on the immerSUN solar power diverter, how it logs data to live.myimmersun.com via the immerLINK Bridge and how that data can be retrieved using HTTP for logging, graphing etc.
I’ve always found it frustrating that data is sent to the Cloud every 5 seconds but it’s not practical to retrieve it from the Internet anything like that frequently and there is no option to capture the data locally. I have been retrieving it every minute but the solar generation in particular can change a lot in a minute. For simply logging and graphing the data that hasn’t been a major problem but now I’m considering actively controlling the electric car charging rate based on the amount of excess solar generation that’s a different matter. I’ve also found there are times when live.myimmersun.com goes offline and hence no (new) data gets collected.
As hinted at in Part 1, data is sent out to the Cloud in the form of a 56-byte UDP datagram and another 56-byte datagram is sent in return. In principle it’s possible to intercept these and extract the readings.
The basic concept is simple enough – intercept the outbound UDP packets, decode the relevant bytes and publish the data locally, avoiding the round-trip to the Cloud. However, there are a few complications:
- The immerLINK Bridge gets upset if it doesn’t receive the response datagrams – and if they don’t make it to the Cloud then the myImmerSUN App stops working too. This means they can’t just be redirected to a local destination but also need to be sent to the Cloud.
- (Actually, with a bit more work to understand the make-up of the response datagrams it should be possible to replicate those from a local server, and the App could be replaced with something looking at the local copy of the data.)
- The data format is not in anyway self-describing and there is no documentation. It’s necessary to reverse-engineer which bytes represent which readings.
There are a couple of options to address the first requirement:
- A network packet capture tool could grab packets according to suitable filter criteria and make the datagram payload available for decoding. This would not require any re-plumbing of the network routing but would mean running the packet capture on a device which is routing the relevant packets – which could be something like a Raspberry Pi acting as a router, or could be the main router / firewall.
- A UDP proxy server could be placed between the source and target and the traffic routed via that. The proxy would send the datagrams on to their original target (and also forward the responses) but would extract the readings while doing so.
On balance, the proxy server approach seems the better way to go – less resource intensive, more flexible and more amenable to adapting to replace the remote server in due course.
The data format is also not an insurmountable barrier and I confirmed that it was likely the readings could be extracted before proceeding far with this project, using a capture of the packets (tcpdump on the firewall to capture the packets then Wireshark to view the captures).
It turns out the UDP datagrams have a fairly simple format – there is no encryption and the raw bytes representing the numbers are at a fixed location in every datagram, as long as you know where to look. In outline, the procedure to find these was:
- Having a hunch that the serial number of the device would be in there somewhere so reading this from the immerSUN’s UI, confirming it was numeric (rather than alphanumeric), converting to a handful of likely byte formats and looking for these in the packet captures. It turned out the serial number is roughly in the middle of the 56-byte datagram and is represented as a 32-bit Signed Integer in Little-Endian byte order.
- Looking for other parts of the datagrams that are variable (large portions of it are the same every time) and seeing how these convert in Little-Endian byte order. The ones that flip from positive to negative are particularly useful – and the ‘generating’ reading (from the solar PV inverter’s current transformer) drops to -5 or -6 overnight. This showed up almost at the end of the datagram, as a 16-bit Signed Short. The other numeric readings (‘diverting’ – which is at the very end – and ‘exporting’ which is before the serial number) are fairly easy to find too.
For the record, the following Perl code will extract the numeric values from the outbound datagram:
# The "exporting" value is a signed Short at byte 29 & 30 my $exporting = unpack( "x28 s<", $data ); # The "generating" value is a signed Short at byte 53 & 54 my $generating = unpack( "x52 s<", $data ); # The "diverting" value is a signed Short at byte 55 & 56 my $diverting = unpack( "x54 s<", $data );
(There’s more information about Perl’s
unpack function in the Perl Tutorial on pack and unpack but basically the
x28 means “skip 28 bytes” and the
s< means “interpret the next bytes as a Short in little-endian byte order”.)
A key component of the solution is a simple UDP proxy server that:
- Listens for incoming UDP Datagrams on Port 87
- Forwards these to live.myimmersun.com
- Listens for response Datagrams from live.immersun.com
- Forwards these to the original source address
While doing this, it also:
- Optionally logs the outgoing Datagrams in Hex format to a logfile
- Extracts Short Integer values containing the measured readings from the relevant positions in the outgoing Datagrams
- Publishes the extracted values as MQTT messages to a local MQTT server
Outbound Packet Redirection
Something needs to be done to get the UDP packets to hit the proxy server instead of their original destination. I use the Shorewall wrapper for ‘iptables’ on Linux as my network firewall and it’s easy enough to configure that to send packets which match various criteria to a different destination from the one they were originally destined for, though the syntax is a bit cryptic. Shorewall uses a ‘DNAT’ Rule to accomplish this and the syntax for the entry in
# source dest proto dport sport originaldest
DNAT data:172.16.40.84 data:172.16.40.81 udp 87 - 220.127.116.11
This should be read as:
Any UDP packet destined for Port 87 at IPv4 Address 18.104.22.168 and from IPv4 address 172.16.40.84 (in the ‘data’ network zone) should instead be sent to IPv4 address 172.16.40.81 (in the ‘data’ network zone), irrespective of the source port used.
Since this constitutes ‘hairpin’ routing – where a packet is sent back out the same interface on which it arrived at the firewall – the interface specification in
/etc/shorewall/interfaces needs to include the
routeback option setting.
- There’s a great online utility for decoding hexadecimal strings into various different number representations at SCADACORE’s Online Hex Converter
- The 4eco team that developed the immerSUN system before selling it to SISEM Ltd moved on to develop the myEnergi range of products (more details in this LinkedIn article). As a result, there are some similarities between the myEnergi solutions and the immerSUN and some of the posts on the myenergi.info user forums – see e.g. this post about the UDP packet format – provide some good clues as to how things work.