Experimentation with ZeroMQ

At the weekend I was experimenting with ZeroMQ (0MQ for short here) for handling requests for items in an online shop. The idea behind it was simple – schedule requests for items to be added to a shopping cart, and don’t allow overselling of said stock items. The legacy code handles requests by wrapping in a transaction and checking for the amount of items currently available (sales + in existing carts), then adding/rejecting as appropriate.

If however there is significant traffic, this can result in oversells as the number of items available can be reported the same to more than 1 user. If both users request the final item, your oversell happens. I wanted to remove the likelihood of this happening, and so I turned to a queueing system to see how this could make a difference. I’d seen the video of Ian Barber’s presentation at PHP UK last year, and liked the apparent simplicity of 0MQ coupled with the lightweight in-memory approach so decided to experiment a bit.

After downloading and installing the 0MQ library, and the pecl extension for PHP, I was all set to go.

The idea seemed simple:

• Take all requests for items, and pop them into a queue
• If a response from the queue isn’t available immediately, the customer should be shown a page which polls periodically for the result.
• When the result is available, redirect the customer to their cart (success) or back to the item page (failure)

The 0MQ documentation at first seemed thorough, but reading through I realised it was a bit too verbose for my liking. I can appreciate the use of metaphors and friendly language, but it seemed to hinder understanding of what 0MQ could offer, coupled with everything being on the same page. Slightly overwhelming! In contrast, RabbitMQ’s tutorial pages seemed to be a lot clearer. For a developer new to queueing system, this was definitely a win for RabbitMQ – a case of which-image-represents-best-what-I-want *click* etc. However, I stuck with 0MQ for now for the reasons outlined above.

I settled on the Push-Pull methodology, “Divide and Conquer” in the 0MQ docs. This would satisfy the requirements above of being able to dump a request into a queue and periodically (via another mechanism) check for a result. I’ve shamelessly ripped off the accompanying image from their docs here in order to illustrate my point :-)

In the above image, the part labelled “Ventilator” can be equated to the user requesting a quantity of items to add to a shopping cart, and my queue handler is a single instance of a worker. In my implementation, I went for something like the following process:

• User requests 1 of item A via form
• Memcache entry is created with a unique key for the user’s request
• 0MQ request is created, “allocate me 1 of item A, for my session ID X and unique key ABC”
• … polling happens here …
• 0MQ worker receives message and performs relevant checks
• Memcache value for supplied unique key is updated with response
• Polling for this updated value causes a redirect for the user accordingly

Polling checked the memcache key for a successful response. I decided to use memcache for this as it was quick to get going and I only technically needed a key-value setup. I also wanted to avoid unnecessary hits on a database if possible, preferring to keep this for transactional data. Plus I also wanted to play with memcache a bit ;-)

Testing

In tests, this setup seemed to work well, and the updated memcache value ended up being a serialized array with a success/error code, and some additional information eg “Sold out”, “Item is unavailable”, “You requested 2 but only 1 is available”. This enabled better feedback for the user when their request was unsuccessful.

Running this in small-scale tests was great. It was good to be able to start and stop the queue handler, and see the user being automatically bounced to the “please wait” polling page whenever the queue handler wasn’t running or had a backlog of messages to process. Fire up the handler, checks take place and the user is then redirected accordingly.

When running this under heavier loads however, we hit instances where occasionally (due to intentionally-bad Apache settings), the server processes ran out of memory and required a restart of Apache. This was done deliberately to see how the queue could recover from different scenarios. As a result, restarting Apache would cause a flood of messages to be sent to the queue handler in one go, presumably because they’d been blocked somehow previously with the large load. This was confusing and unexpected behaviour to me, however I presumed this was because Apache (with mod_php) had been struggling to send the messages out under the load, and a restart caused a flush of some sorts. I also presume this isn’t down to any fault of 0MQ. It does however pose the question of what to do in a scenario where messages can potentially be lost, but this could be solved by a message queue that persists messages somewhere until completion.

Despite this teething issue which wasn’t really down to any fault of 0MQ, the system works great. Being new to queueing like this, I’m not entirely sure how this would scale and still keep the ability to not oversell an item – database transactions from multiple workers would need to be synchronised somehow to ensure that figures are kept up to date. But overall, I’m very happy with it as a starting point for a replacement solution.

At the weekend I was experimenting with ZeroMQ (0MQ for short here) for handling requests for items in an online shop. The idea behind it was simple - schedule requests for items to be added to a shopping cart, and don't allow overselling of said stock items. The legacy code handles requests ...

From → php