I’ve just tagged a new release of the Arduino Client for MQTT – v1.9.1. This release includes an API change that will break existing sketches, something I’m very concious of doing – particular as the last release had such changes as well. But ultimately I decided this one was needed.
Previously, an instance of the client would be created with a call such as:
PubSubClient client(server, 1883, callback);
The library would then create an instance of EthernetClient under the covers. This was fine for hardware that used that class, but there are an increasing number of shields that use their own instance of the common Client API.
So, now the constructors for the client require an instance of Client to be passed in. For the vast majority of existing sketches, this just means changing the above line to something like:
EthernetClient ethClient; PubSubClient client(server, 1883, callback, ethClient);
Amongst the other functional changes are the ability to publish a payload stored in PROGMEM, connect with a username/password and also the changes described in the previous post to not fragment MQTT packets over multiple TCP packets.
There is also now a regression test suite for the library. Built using python and Ino, the suite first verifies each of the examples compiles cleanly, and if an Arduino is connected, it will upload each sketch in turn and run unit tests against it. So far the focus has been on the scaffolding of the test suite, with only a couple actual tests in place for the example sketches. I’ve tried to write it in a way to be agnostic of the library it’s testing. There’s more work to do on that side of things, but ultimately it could well be reused by others. I imagine it’ll branch into its own repository when it gets to that point.
I’ve also given the project page a tidy and moved the API docs to their own page.
Finally, I’m considering closing the comments on the project page. with 75 comments already, it isn’t the best medium through which to provide support. I’m not sure pointing to the library’s issue tracker on github is necessarily the right thing either – it’s an issue tracker, not a support forum. Perhaps I should set up a dedicated Google group for the library – or is that overkill?
Feedback, as ever, is welcome.
Cosm has had MQTT support for some time now and I’ve had my home energy usage bridged up there from my local RSMB quite happily. One thing I hadn’t played with properly was talking to Cosm directly from an Arduino – aside from a 5 minute proof-I-could-do-it.
I’m putting together a new release of the PubSubClient, and having had the occasional report of problems talking to Cosm with it, I decided to see where things stood. Sure enough, the connection was pretty unstable; often failing to connect, hanging and certainly not reliably sending or receiving messages.
Knowing the client was producing perfectly valid MQTT packets and works with other brokers, the problems had to either exist in the network connection to Cosm or with their broker implementation. The fact my RSMB bridge to Cosm work from the same network ruled out a fundamental network issue. Equally, there are plenty of MQTT brokers on-line these days to check it wasn’t a problem with the Arduino talking to a remote broker, rather than one on the same subnet.
Having debugged this sort of thing with other implementations, I had a good idea what the issue was; splitting MQTT packets over multiple TCP packets – something I knew the PubSubClient did.
When reading an MQTT packet from the network, it’s easy to assume that a read from the network will provide the next byte of the packet. When testing on local networks, that will mostly be true. But when dealing with internet-scale lag between TCP packets, the code doing the reading must handle the case when the next byte hasn’t arrived yet. This is what I think the Cosm broker is tripping over.
A quick test compared writing the packets to the network with multiple calls versus doing it with a single call and saw all of the reliability issues disappear. So I decided to update the client library to do just that.
If all you’re after is working code, then you can grab the latest library from GitHub. Note that this code includes an API change on each of the constructors – you need to provide an instance of EthernetClient rather than leave it to the library to create one. You can see what I mean in the updated examples.
In case you’re interested, here’s a run down of the changes I had to make to get this to work.
For the purposes of this exercise, an MQTT packet consists of three components; a single-byte header, between 1 and 4 bytes encoding the remaining length and finally the remaining payload bytes. The existing library provides a function that takes the header and payload, calculates the remaining length bytes and writes each of them to the network in turn – in other words, three separate network writes.
boolean PubSubClient::write(uint8_t header, uint8_t* buf, uint16_t length) {
uint8_t lenBuf[4];
uint8_t llen = 0;
uint8_t digit;
uint8_t pos = 0;
uint8_t rc;
uint8_t len = length;
do {
digit = len % 128;
len = len / 128;
if (len > 0) {
digit |= 0x80;
}
lenBuf[pos++] = digit;
llen++;
} while(len>0);
rc = _client->write(header);
rc += _client->write(lenBuf,llen);
rc += _client->write(buf,length);
lastOutActivity = millis();
return (rc == 1+llen+length);
}
The crude way to reduce the network writes would be for the function to copy all of those bytes to a single buffer and then pass that single buffer to the network. But that would involve a lot of unnecessary copying of bytes around as well as the increased memory usage for the buffer.
// Allocate yet more memory
static uint8_t packet[MQTT_MAX_PACKET_SIZE];
pos = 0;
packet[pos++] = header;
for (int i=0;i<llen;i++) {
packet[pos++] = lenBuf[i];
}
// Copy lots of bytes around
for (int i=0;i<length;i++) {
packet[pos++] = buf[i];
}
rc += _client->write(packet,pos);
There is already a buffer used for incoming packets which can be reused for the outgoing packets. The difficulty is the remaining length bytes; until the payload is constructed the code doesn’t know how long it will be, which means it does not know how many bytes are needed to encode the length. Without knowing how many bytes are used to encode the length, the client doesn’t know the offset into the buffer to start writing the payload.
The trick is knowing that there will be at most 4 bytes of length and one of header – so the payload can be written starting at the 6th byte of the buffer. Then, once the length bytes are calculated, they can be inserted into the buffer working back from the 5th byte and the header byte can be inserted ahead of the length bytes. This gets the MQTT packet into a single contiguous buffer, with minimal copying of bytes, which can be passed to the network – albeit starting at a known offset from the start of the buffer.
// llen is the number of length bytes
buf[4-llen] = header;
for (int i=0;i<llen;i++) {
buf[5-llen+i] = lenBuf[i];
}
rc = _client->write(buf+(4-llen),length+1+llen);
Simple as that.
There is one place in the library that can’t be reduced to a single network write; the recently added publish_P function allows the message payload to be stored in PROGMEN. The benefits that brings would be entirely lost if they were then copied into regular memory before writing to the network. Not much to be done about that.
I’ve been running a site for the Chale Community Project for a while now that allows members of the community monitor their energy usage online.
tl;dr; I changed some code that was using MySQL to store meter readings to use redis instead. The rest of this post describes some of the design decisions needed to make the shift.
The energy readings are published using a version of my modified CurrentCost bridge over MQTT. As well as the whole house reading, a meter might have a number of individual appliance monitors attached. Each of these readings is published to its own topic in a very simple hierarchy:
sm00/CF35D16315BF93EC053E4EFFC614E3E944C2A626/1/2
The first two topic levels identify the meter and the second two identify the individual devices attached.
With readings arriving for each device every 6 seconds, I started with simply dumping them into MySQL – with each level of the topic as a column in the table, thrown in with a timestamp. As this granularity of data isn’t needed for the site, a job could then run every 10 minutes to average the readings into 10 minute slots and put them into another table. This was good enough at the time, but it wasn’t a long term solution.
Aside from some serious normalisation of the database (adding the 40 character meter ID in every row turned out not to be the most efficient thing to do), I wanted to move to using redis for the live data. There were a few reasons for this, but primarily it was a combination of wanting to lighten the load on the (shared) MySQL server, as well as to do something more interesting with redis.
The shift from relational table to key-value store required a change in mindset as to how to store the readings. The main requirement was to minimise the work needed to store a reading as well as the work needed to calculate the average reading for each device.
Redis provides a list data type which, when using the receiving topic as the key, is a perfect fit for storing all of the readings for a particular device. But that causes problems in generating the average as the list will contain all values for the device, not just the values for a particular 10 minute timeslot.
The solution is to simply add the timeslot to the key. The timeslot is the current time, such as 2012-08-20 15:21:34 with the last three digits set to 0 – 2012-08-20 15:20:00.
This leads to keys such as:
20120820152000:sm00:CF35D16315BF93EC053E4EFFC614E3E944C2A626:1:2 20120820153000:sm00:CF35D16315BF93EC053E4EFFC614E3E944C2A626:1:2
That gets the readings stored pretty efficiently, already split into a list-per-device-per-timeslot.
The task of averaging the lists could then just look at the current time, generate the timeslot string for the previoius timeslot, retrieve all keys that begin with that string and process all the lists. But there are two problems with this approach.
First, redis doesn’t recommend using its KEYS pattern command in production. I doubt this system would ever scale to the point where the performance hit of that command became an issue, but still, it’s worth sticking to the appropriate best practices.
The second issue is that it makes a big assumption that the only timeslot that needs to be processed is the previous one. If the task fails to run for any reason, it needs to process all of the outstanding timeslots.
This needed one of the other data types available in redis; the sorted set. This allows you to have an ordered list of values that are guaranteed to be unique. When each reading arrives, the timeslot value is put into a sorted set called, imaginatively, ‘timeslots‘. As it is a set, it doesn’t matter that the same timeslot value will be put into the set thousands of times – it will only result in a single entry. As it is sorted, the task doing the averaging can process the outstanding timeslots in order.
Finally, a normal set is used to store all of the reading keys that have been used for a particular timeslot. As this is per-timeslot, the key for the set is ‘readings:‘ appended with the timeslot.
So, with that in place, the averaging task takes the follow steps.
- Gets the list of timeslots that there are readings for from the sorted set
timeslots. - For each timeslot, that is not the current one:
- Remove the timeslot from the set
- Get the list of reading keys from list
readings:timeslot - For each reading key, retrieve the list, calculate the average, insert it into the MySQL table and delete the list
- Delete the
readings:timeslotlist
And that is what I’ve had running for a few weeks now. No grand conclusion.
[ part of the orb series ]
The base of my original Orb was cut out of a thick piece of wood using a circle cutting drill bit. A pair of slightly smaller circular bits were then used to create a recess in the top of the base and finally a hole drilled through the centre. This was all it needed to glue the BlinkM in place with the lead running out of the base to the off-board arduino.
This approach wasn’t going to work for the new Orb, which needs a hollow base to house all of the boards. It also needed to be something that others could easily replicate.
I discounted 3D printing early on, as I wanted the base to be wooden. This left some sort of laser cut thing. There are a number of on-line services for getting things laser cut, such as Ponoko. They offer a wide range of materials, including 3mm thick plywood – ideal for this.
To figure out what that base was going to look like, I started by glueing a stack of cardboard together and literally carved out the shape I wanted. This let me work out what each of the individual 3mm layers needed to look like. It also helped to identify how to make the base in such a way as to allow the PCB to be accessed; I didn’t want it to be a permanently sealed unit.
The base consists of 12 layers, with 3 in the bottom ‘half’ and 9 in the top (the diagram over on the left omits most of the layers). A pair of nuts, shown in red on the diagram, are sandwiched between layers in the top half, which allow the two sections to be held together using screws. The bottom of the base also includes holes for mounting the PCB stack to it.
With the layers drawn up in inkscape, it was time to get them made. A chance conversation with Adrian McEwen at a #iotlondon meetup enabled me to take advantage of DoES Liverpool’s laser cutter.
Adrian kindly took some photos of the LASER in action.
Proving yet again, atoms are hard, this first pass wasn’t a complete success. I had made the space in the centre of the base layers the exact same size as the PCBs – so exact that the PCBs didn’t quite fit. A quick rework of the design and another pass with the lasers and I had all the bits I needed.
Some glueing and sanding later, and the base looks quite good – although I don’t appear to have a more recent photo of it to hand. It needs some more work as the design didn’t properly take into account how the orb actually attaches to the base – I have vague thoughts about the liberal use of a hot-glue gun to achieve this currently. I’ve also had some thoughts about changing the style of power-supply connector, which will change the size of opening needed in the side.
But all in all, a significant step forward as it now looks more like something people would be prepared to have sat in their home.
[ part of the orb series ]
With the Orb needing to be wireless, one of the early decisions was what sort of wireless to use.
The first option to consider was Wi-Fi. This would allow the Orb to connect straight to the Internet via the home router – no intermediary is needed. The downside is that the Orb doesn’t have any sort of UI that would allow configuration of the Wi-Fi settings. Some Arduino WiFi shields will read the configuration from an SD card – but that is not at all a nice user experience. Nor is there a lot of space available on the bottom board to fit all of the hardware in. It is also an expensive option. Likewise Bluetooth and ZigBee weren’t really a good fit.
That left a 433Mhz, or similar, radio link. I had played around with some cheap radio transceivers before and they seemed a good way to go. But then I discovered the RFM12B module via JeeLabs. Jean-Claude has written a lot about the module on his blog and has created an Arduino library for driving it – perfect for the Orb. Another benefit is that the Nanode board comes with this module – making it nicely interoperable.
The downside of the module is that it runs at 3.3V – but the rest of the Orb runs at 5V. I considered squeezing in the appropriate voltage regulators to provide both voltages from an unregulated supply. But I decided it would be easier to offload some of that by requiring a regulated 5V power supply. That way, the bottom board only needs a regulator to provide 3.3V to the radio, as well as a set of voltage dividers on each of the signal lines between the radio and the ATmega328. It also opens the option of using a micro-USB port as has become standard with mobile phone chargers – although that’s for the future.
Although I said the Orb has no configuration interface, I decided it needed something. A common pattern with radio-linked devices, such as the CurrentCost meters, is that you press and hold a switch to put the device into a pairing-mode. This mode allows you to not only establish a link between the device and a base station, but also to link multiple devices with a single base, whilst maintaining their individual addressability. To this end, I included a surface-mount switch on the underside of this board so it can be accessed through the bottom of the Orb housing.
As ever, the schematics are on github.
And that’s the final board. All soldered, stacked and ready to do its thing. What’s next? Making a base to house these boards in. But that’s for another day.