After giving you an introduction to this project, showing the worker service and setting up the development environment, it is now time to show you how I get data out of the meter and make some sense of it.

Serial data, you say?

Yes, the electrcity meter sends data through the P1 port over the wire that is connected to the USB port on the Raspberry Pi (see part 1) in serial form. No surprise there. You knew the 'S' in USB stands for 'Serial', right? So for each message, the meter sends a continuous stream of bytes that you need to capture, parse, check and store in some shape and form.
Now, I'm old enough to have worked with PC's that were actually equipped with serial ports and have connected a peripheral or two like mouses and modems back then, but the handeling of the data was taken care of by the PC and the program running on it. Only thing I needed to do was tell the program which port to use. Sometimes, if you were unlucky, you also needed to supply the speed to use (as in bits per second) and the number of data, parity and stop bits. Ususally it would take some fumbling to get it setup correctly. If you like to know more about the serial port and communication, I suggest to start at this Wikipedia page. 

Use of serial communication is however still wide spread (USB is ubiquitous nowdays) so .NET comes with a lot of functionality out of the box. I can create an instance of a SerialPort (part of the System.IO.Ports namespace) in my program and just as in the old days I supply it with a port, speed (baud), parity, data bits and stop bits. And just like in the old days I had to fumble with those settings to get it working :). To make it easier to change the parameters for the port creation, I defined these in the appsettings.json file. I described this proces in detail in the 'Configuration' section in part 2, so won't repeat here. 

Using the SerialPort gives me access to the stream of bytes that flows in but I still need to take care of the data and do something with it. I already described the service that continously runs in part 2. What I didn't describe there is the actual parsing of the serial data that is called upon with just one line in the EcecuteAsync method:

_telegram = await _meterReader.ReadAsStreamAsync();

Based on the configuration this will either read the messages from a file on disk (which we will not discuss here) or read the data from the USB port on the connected Raspberry Pi in an asynchronous way and stores the result in a variable of type Telegram.

The process of reading the data is fairly simple and is comprised of the following 5 steps (see lines 21-46 in MeterReader.cs):

  1. Open the SerialPort, which was configured in the constructor of the MeterReader)
  2. Create a new StreamReader, which is initialised with the stream coming from the SerialPort)
  3. Create a new Parser 
  4. Create a new Telegram (a logical representation of the data following the DSMR specification)
  5. Parse the data from the reader into the telegram
  6. return the Telegram

Lets dive a bit deeper into the telegram and the parser 

A telegram, you say?

Yes, as this descibes the package of data/message that is sent in a very good way. In it's original use, telegrams were paid for by the number of words, so there were kept as brief as possible.That is the same as what is done with the information that is coming for the meter. It just It starts with a '/', ends with a ! (and a CRC) and has labels and values in between. No other superfluous information. As said in one of the earlier parts, the data follows a well documented standard that meticulously descibes it, so it is very suitable for putting it through an automated parsing process. A telegram that comes from the meter looks like this:

/XMX5LGBBFFB231226417

1-3:0.2.8(42)
0-0:1.0.0(200605140325S)
0-0:96.1.1(4530303035303031363935303633303135)
1-0:1.8.1(007765.830*kWh)
1-0:2.8.1(001148.284*kWh)
1-0:1.8.2(005582.654*kWh)
1-0:2.8.2(003077.671*kWh)
0-0:96.14.0(0002)
1-0:1.7.0(00.000*kW)
1-0:2.7.0(00.581*kW)
0-0:96.7.21(00014)
0-0:96.7.9(00007)
1-0:99.97.0(7)(0-0:96.7.19)(181004121300S)(0000004487*s)(180608105808S)(0000001183*s)(170127203820W)(0000003656*s)(160604003643S)(0000000656*s)(160510123123S)(0000000943*s)(151126095659W)(0000002444*s)(150211091555W)(0001380715*s)
1-0:32.32.0(00001)
1-0:52.32.0(00002)
1-0:72.32.0(00002)
1-0:32.36.0(00000)
1-0:52.36.0(00000)
1-0:72.36.0(00000)
0-0:96.13.1()
0-0:96.13.0()
1-0:31.7.0(003*A)
1-0:51.7.0(001*A)
1-0:71.7.0(000*A)
1-0:21.7.0(00.000*kW)
1-0:41.7.0(00.129*kW)
1-0:61.7.0(00.014*kW)
1-0:22.7.0(00.718*kW)
1-0:42.7.0(00.000*kW)
1-0:62.7.0(00.000*kW)
!FEDE

The 'header' (first line) starts with the opening '/' and is immediately followed by the equipment identifier (i.e. the serial number of the meter). Every line up to the closing '!' is a data object as defined in the 'NEN-EN-IEC 62056-61:2002 Electricity metering – Data exchange for meter reading, tariff and load control – Part 61: OBIS Object Identification System'. Following this spec, each line starts with an OBIS code and is followed by the actual value in brackets. So in the message above the first line has OBIS code 1-3:0.2.8 and value 42 which means the version of the information coming from the P1 output is 4.2.

To be able to work with this data, I basically needed to do two things:

  • Define a class/model that represents a telegram.
  • Define a class that parses the data that is sent from the meter to the USB port on the Pi into the class mentiond in the bullet above.

I found a cool library on GitHub by Koen Ekelschot which I re-used, adapted and enhanced to do both these things. The bulk of the work is done in the Parser.cs class where each stream of data is split into seperate lines. For each line it then determines the code and the value and sets the the correct property of the Telegram class. The parser knows wihich propty to use because each one has been tagged with an attribute which tells it what the corresponding OBIS code is and what the unit of the value is. No clunky matching of strings to property names using reflection. Super elegant! An example of one of the properties in the Telegram.cs class:

[Obis("1-0:1.8.1", ValueUnit = "kWh")]
public double PowerConsumptionTariff1 { get; set; }

So whenever a line comes in from the stream that starts with such an OBIS code, the parser will recognize this and do it's thing.
Another nice thing the parser does, is it stores every line it reads as a string in an list in the Telegram class. A .ToString() method in the class does a simple string.Join on that list and gives you an easy way to output or store the complete telegram. I use this for example for saving the messages to files on disk, if this is turned on in the configuration:

private void SaveDataToFile()
{
    if (_config.SaveDataFiles)
    {
        IWriteFile fileWriter = new FileWriter();

        fileWriter.WithPath(_config.DataFilesPath)
            .WithFilename($"{_telegram.RowKey}.dsmrdata")
            .WithContents(_telegram.ToString())
            .Write();
    }
}

The FileWriter is another gem I found in a smart meter repository on GitHub. This one actually uses .NET Core to talk with the smart meter, but only retrieves a couple of values from the meter and stores them in a MySQL database on the Pi. So, not exactly what I wanted to achieve.

Next to this parser, Telegram class and attibutes, there are even some TypeConverter implementations available in the library. These, for example, convert a simple '42' read from the stream into an Enum value ObisVersion.V42.

Data storage, you say?

Yes, I built all of this to be able to store the data myself instead of just letting the meter send everything to the utility company. To store the data, there are of course a gazillion options available. I could use a traditional relational database like Azure SQL or MySQL, but there is realy only one table and there are no relations. I could use files and just dump them in an Azure Storage account, but then I have no easy way to run queries on the data. I choose to go with an Azure Table Storage solution. Azure Table storage stores large amounts of structured data. The service includes:

  • Storing TBs of structured data capable of serving web scale applications
  • Storing datasets that don't require complex joins, foreign keys, or stored procedures and can be denormalized for fast access
  • Quickly querying data using a clustered index
  • Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries

You can use Table storage to store and query huge sets of structured, non-relational data, and your tables will scale as demand increases. I could have chosen to go really fancy and store it in an Azure Cosmos DB with the Table API, which is the more premium offering of Table Storage, but I don't need Turnkey global distribution, Dedicated throughput worldwide or Single-digit millisecond latencies at the 99th percentile. Chosing this would just lead to a higher monthly bill, even when using the free offering.
The nice thing about Azure Table Storage is that it can create the structure of the table based on the data that it is fed. Only thing that is required that the objects that you send there are derived from the TableEntity class. So that is exactly what you will find in the Telegram class. By using the standard [JsonIgnore] attribute you can prevent a property from being serialized and deserialized. I used this to prevent the storage of the Lines list discussed earlier.
One you have your classsetup this way, storing a message (in this case a telegram) is fairly easy:

TableOperation insertOrMergeOperation = TableOperation.InsertOrMerge(_telegram);
TableResult result = await _table.ExecuteAsync(insertOrMergeOperation);

if (result.HttpStatusCode == (int)HttpStatusCode.NoContent)
    Console.WriteLine($"Telegram {_telegram.RowKey} stored in table {tablename}");

There is some more code around this to make sure the table exists etc., but you can find that in the repository if interested. 

The whole process is super stable, even on something as simple as a Raspberry Pi. I have this running now for a couple of months and in total stored a little over 250k telegrams. My costs for this month have been 0.02 euro so far and is predicted to end at a whopping 0.03 cents in total!

This is it for this part in the series. Questions? Remarks? Let me know in the comments below! Till the next part.

Comments


Comments are closed