Plumbing Life's Depths

• Design Theory
• Knight Errant
• Polis
• Snaking
• The Laptop
• Tuxedo
• Vindaloo

So, having worked with mDNS/Zeroconf a bit, I thought I'd take a quick look at uPNP, particularly the IGD device type. IGD here is "Internet Gateway Device", i.e. your router. There are approximately two things you want to be able to access from generic software on a router, your external IP address (so you can share it with people, though this isn't as useful as having a server somewhere that can look at the address on an incoming connection), and a way to map ports to internal hosts (i.e. IMO what 99.9% of people who know what IGD stands for are looking to do with IGD).

So, we've played with mDNS, but for those who weren't playing along, the way it works is this:

• you join a well known "group" (ip address in the multicast space)
• you send out a multicast message to the group, the content of this message is a regular dns query for a PTR record
• anyone in the group who has a PTR record that matches the request responds (to the group) with their PTR records
• you resolve each PTR record to a SRV record and eventually an A record (the machine address)

For bonus points, the members of the group can return all of the records (PTR, SRV, A and optionally TXT) in a single response to reduce network traffic (and handle broken hardware that expects all of the records at once).

The thing that makes it all work is the PTR record, which is just a specially formatted "domain name" which describes a "class" or "type" of service, rather than a particular service or machine. These look like "aastra-config._tcp.local." where the prefixed values are service-type and service-protocol components. Each machine which provides a service would create an SRV record with "My Cool Service._aastra-config._tcp.local." as the service name, and make the value in their PTR record that service-name. Their SRV record then points to their machine name, and the A name (as is normal) points to their IP address.

Conceptually, uPNP-SD is very similar:

• you join a well known "group" (ip address in the multicast space)
• you send out a multicast message to the group, the content of this message is an "http over udp" search query
• anyone in the group who has a "device" record that matches the request responds (to you, directly) with "http over udp" responses that provide locations (urls)
• for each url so received, you download an XML device/service description and traverse the description looking for the (sub)-service you actually want
• once you find the service description, you issue SOAP calls to the url in the service description

The xml file description is pretty straight-forward to parse in Python, use an etree engine, look for the device, iterate over it's services and its sub-devices's services. SOAP is pretty easy to work with too. But wow, the documentation is a PITA to traverse. The key is that the service within IGD that you need is called WanIPConnection:1 (pdf link). If you happen to wade through that document, you'll notice that the spec lets you do everything you could imagine to an IGD (that's just one of 13 services for an IGD version 1). You'll want to search for "PortMapping" in there to see the parts that might be useful.

It was right around here that I went "meh", I have a client that can find all of the IGD-enabled routers on the networks, and can tell me where the IGD WanIPConnection:1 services are on those routers. I even have the API description telling me what to do to create the port-mapping. Thing is, my own router didn't respond, because, of course, it ships by default with uPNP-IGD disabled (because it's a huge security hole to expose such a huge, featureful API when all you really want to allow is port-mapping)... and honestly, it doesn't look like a very nice API (why are all of the parameters prefixed with "New"?)... but I'm almost there, why not finish the experiment? Well, because it wouldn't even work in my own network, and really, my itch stopped itching around then.

I have been writing some acceptance tests for a Flex application in Selenium using Nose, and VirtualBox. To do so, I had to fix quite a few bugs in the Selenium Flex API. Are all Flex devs over-engineers? The SFAPI had a scary series of interacting methods, a helper class, multiple pieces of shared state mutated by what appeared to be simple query methods, and two entirely different ways of handling two objects which have the same formal interface, all of which was just to walk a tree of nodes (at least, as far as I could see).

I spent quite a bit of time staring at and debugging this code, trying to fathom why it was so complex. When you're new to an environment you often think there must be some reason you just don't understand yet. Thing is, it didn't *work*, so I had to fix it (on a deadline, no less).

Turns out the whole edifice really is just walking a tree of nodes, and all the messy side-effect laced, implicit operations were just attempting to eliminate duplicate visits (with a broken algorithm, as it turned out). In my hastily bashed out code, the tree traversal is a 48 line function that takes a functor and applies it to each node in the tree (with some logic to allow for short-circuits). It replaces all of the complex, error-prone, side-effect-laced methods, and the pointless helper class and can be used in all locations where traversal is needed in the code-base (with a single line of code) instead of requiring that the traversal logic be duplicated for each traversing method.

Thing is, all of the documentation I read on Flex, all of the examples, they seem to follow these same overly formal and rather obtuse patterns. I want a form that submits data to a web server and updates with error messages and the like... of course I need a model, a presentation model, a view, a controller, custom error handlers, a few classes to do the http submission, observers to handle data-change events, etceteras. Argh.

Problem that looks like it will be coming up soon... you are always wanting to be able to test your code against a (huge, PostgreSQL) LIVE DB before you release/promote the code. Loading a DB dump can take hours. Your test code can't all just run in a single transaction.

First approach: run the DB server in a virtual machine, sync the LIVE DB down to the virtual machine, snapshot just before you run tests, throw away the snapshot.

Second approach: run the DB server on real hardware, sync the LIVE DB down to a template DB on the real hardware, drop the database and recreate based on the template (note: you have to stop the sync before you create the new db). In my spike test the create-from-template approach still takes a few minutes, but it's far faster than loading an SQL dump.

create database moo with template moo_template;

Both approaches, of course, require a DB sync operation... that's the part I still need to research.

ssh dbhost.example.com "pg_dumpall | gzip -c" > ~/cluster.sql.gz

ssh dbhost.example.com "gunzip -c | pg_restore" < ~/cluster.sql.gz