Talk in Helsinki this week: the Open Data Movement

UPDATE: video from the talk, expertly shot by Jyri Engestrom, is now available.

I'm heading to Helsinki in a few days for the next Thinglink workshop. My lovely hosts Ulla-Maaria and Jyri have organised a chance for me to give an Aula talk on "the Open Data Movement". I'm honoured to be part of a series that has included Ben Cerveny, Henry Jenkins, Joi Ito and Lawrence Lessig.

Here's the invitation:

OPEN DATA MOVEMENT - THE NEXT WAVE OF OPEN SOURCE

Matt Biddulph
www.hackdiary.com

Thursday August 17th at 18:00
Helsinki Institute for Information Technology (HIIT)
6th floor, Pinta-building, High Tech Center (HTC), Ruoholahti
Tammasaarenkatu 3, Helsinki

Tilaisuus on avoin ja maksuton. Luento on englanninkielinen.
Welcome - Tervetuloa!

* * *

The Wikipedia is only the tip of the iceberg of information that is becoming freely accessible on the internet. Following the success of open source, an open data movement is occurring online that seeks to gather, publish and enable the reuse of rich machine-readable datasets - like all programs ever broadcasted by the BBC.

By opening up these wellsprings of information, which were previously only accessible to large institutions, the open data movement has unleashed a new wave of creativity on the Web. Programmers, students, and companies are building mashups by overlaying photos, blog posts, and other objects to open datasets like the BBC Programme Catalogue, Wikipedia, Open Streetmap, and Thinglink.

As a case in point, Biddulph will describe how the BBC's database of programming from the 1950s to the present day was transformed from an internal green-screen application to a public Web 2.0 service using Ruby on Rails. Expect to see some playful examples of what you could do with it and other open datasets.

* * *

Matt Biddulph is a freelance software developer based in London. He previously worked at BBC Radio and Music Interactive as the leader of the software architecture team, aka Head of Plugging Things Into Other Things. He blogs on Hack Diary (www.hackdiary.com).

* * *
The event is organized by Aula (www.aula.org) in collaboration with HIIT. Aula is an open network that promotes the exchange of ideas across boundaries.

Permalink

Matt Biddulph, August 13, 2006 02:02 PM

Extended trip to San Francisco

The nice people at O'Reilly invited me to Foo Camp this year. On August 24th I'll be packing up my bags and leaving Amsterdam for sunny California.

After Foo Camp, I'm planning to stay in San Francisco until at least mid-October, and (having worked my arse off for the last couple of months) I'm going to be able to mostly treat it as a holiday.

Shockingly, I've never visited San Francisco before, so I'm looking out to my extended network for fun people to meet, exciting companies to visit, and places to stay in SF.

If anyone has a housemate who's going to be out of town, a geek apartment they want to sublet or just a comfy couch they can lend me for a few nights, please do let me know. Tips on cheap hotels also very much appreciated.

This is going to be so much fun. Hope I see you in California.

Permalink

Matt Biddulph, August 06, 2006 02:48 PM

Using Omnigraffle to visualise Rails model associations

This week I'm doing some Rails consulting work for a company that's already developed and deployed a major application. Getting to know a new codebase takes a little time and every diagram or visualisation helps. To help me understand their ActiveRecord model relationships, I knocked together a quick script to scan the associations between models and output it in the Graphviz DOT format.

A quick Omnigraffle import later, and I get useful diagrams like this fragment from the BBC Programme Catalogue codebase:

Here's the code, ready for running from the top of any Rails project (ymmv, etc):

#!/usr/bin/env ruby
require "config/environment"
Dir.glob("app/models/*rb") { |f|
    require f
}
puts "digraph x {"
Dir.glob("app/models/*rb") { |f|
    f.match(/\/([a-z_]+).rb/)
    classname = $1.camelize
    klass = Kernel.const_get classname
    if klass.superclass == ActiveRecord::Base
        puts classname
        klass.reflect_on_all_associations.each { |a|
            puts classname + " -> " + a.name.to_s.camelize.singularize + " [label="+a.macro.to_s+"]"
        }
    end
}
puts "}"

Permalink

Matt Biddulph, August 02, 2006 02:57 PM

My first week with Thinglink

The first Thinglink technical workshop took place in Amsterdam this month. Ulla-Maaria Mutanen and I spent an incredibly productive week at the Mediamatic offices thrashing out ideas that will form a Thinglink technical whitepaper over the next few months.

I'd like to highlight two things in particular that came out of this activity.

Firstly, I wrote up some ideas on the Thinglink blog entitled Design patterns for building with web APIs. This came out of a discussion on the different kinds of integration model that occur in a Web 2.0 world. Although the patterns are described using stories from a Thinglink perspective, I think they're generally applicable. I'd be interested in hearing anyone's thoughts on how they can be refined, and whether they seem to fit the kinds of systems you're building.

Secondly, go take a look at the Thingtagging site. It's an aggregator for photos on flickr that have been thingtagged. We think it's a lot of fun. We also think that you should start thinglinking and thingtagging your unique and interesting objects right now.

I'd like to thank Tom Coates not only for the gorgeous design of the site, but also for pointing out that 'mashup' is really quite a tired word these days. We think that an exquisite combination of two great services deserves better, and so we name this a Flickr/Thinglink intimacy.

Permalink

Matt Biddulph, July 23, 2006 11:32 PM

My last.fm hack discussed on BBC Radio

Thanks to Tristan for pointing out a little snippet on the Phill Jupitus 6Music Breakfast Show this week. The Internet's Dave Green talked a bit about my last.fm hack as part of his "zomg teh internets!" section.

I've grabbed a little MP3 segment, which you should be able to listen to here (duration 01:58):

I'm very proud to be known as a "rogue BBC employee."

Permalink

Matt Biddulph, July 14, 2006 05:38 PM

RailsConf 2006 is over

RailsConf wrapped up this afternoon and I'm heading back to reality. Before I fly home to Amsterdam, I'm staying one night in actual downtown Chicago. It's a great relief after so many nights in O'Hare Airportsville. Luckily the quality of the conference made up for the terrible location.

It's been a fun and worthwhile trip that's introduced me to a whole new community. Unlike an ETech or an XTech, there were only a few familiar faces from my usual conference circuit. I think I made a reasonable impression on my new friends with my talk on Saturday morning: Putting the BBC's Programme Catalogue on Rails. There was a great turnout and people hung around after the talk to ask some very interesting and intelligent questions. I spotted Martin Fowler at the back of the room, which threw me for a few seconds. His books on refactoring and software patterns are essential reading for any developer and I have enormous respect for his opinions.

Without the audio, I'm afraid the slides are just a vague visual outline. There was a dedicated team recording the talks throughout the conference, and I'm told that the audio will be available soon. Sadly you'll have to pay for access to this, although the cost may be justifiable judging from the quality of the talks that I was able to attend.

Permalink

Matt Biddulph, June 25, 2006 11:19 PM

Annotated source code for Second Life flickr screen

I've had quite a few requests for the source code of the Flickr screen for Second Life that I wrote about a few weeks ago. Here's the code, annotated with links to the Second Life coding wiki and a few notes.

I'm only publishing the LSL code, because the serverside code isn't very interesting and does pretty much exactly what it says in the previous post. If you want to run your own flickr screen, the URLs in the code below should work just as well for your objects as they do for mine. If you have problems, let me know. I reserve the right to switch off the code if the traffic gets too high, but I'll post here if I have to do that.

The code uses SL's streaming media feature to load the jpeg into a texture. This comes with a number of restrictions: 'You are allowed one movie (or "media" resource) per land parcel. The movie will be played by replacing a texture on an object with the movie. Users will only see the movie when they are standing on your land parcel. Otherwise they will see the static texture. Script functions only work for objects owned by the land owner or deeded to the group that owns the land. (Remember to set asset permissions on your script and object as well as sharing it with the group!)'. I'm hoping for much better media support than this in future Second Life versions.

// A place to remember the ID for the latest http request we made, so the callback doesn't process out-of-order responses
key http_id; 
// Test pattern - Used as default video texture when one is missing in parcel media
key VIDEO_DEFAULT = "6e0f05ad-1809-4edc-df29-fae3d2a6c9b8";  
// Set the texture to the jpeg provided by url
seturl(string url) 
{
    key video_texture = llList2Key(llParcelMediaQuery( [PARCEL_MEDIA_COMMAND_TEXTURE]), 0);
    if(video_texture == NULL_KEY)
    {
        video_texture = VIDEO_DEFAULT;
        llParcelMediaCommandList([PARCEL_MEDIA_COMMAND_TEXTURE, VIDEO_DEFAULT]);
    }
    llSetTexture(video_texture,ALL_SIDES);
    llParcelMediaCommandList([PARCEL_MEDIA_COMMAND_URL,url]);
    llParcelMediaCommandList([PARCEL_MEDIA_COMMAND_PLAY]);
    llParcelMediaCommandList([PARCEL_MEDIA_COMMAND_AUTO_ALIGN,TRUE]);
}
default
{
    state_entry()
    {
        // set a default jpeg
        seturl("www.flickr.com/images/flickr_logo_gamma.gif.v1.2");
        // start listening for nearby speech
        llListen(1,"",NULL_KEY,"");
        // start sensing nearby agents once a minute
        llSensorRepeat("",NULL_KEY,AGENT,10,10,60);
    }
    sensor(integer num_detected)
    {
        integer i;
        for (i = 0; i < num_detected; i++)
        {
             // ping the server so we know who's around
             string url = "www.hackdiary.com/secondlife/experiments/seen?name="+llDetectedName(i);
             // but don't record the request ID, because we don't need 
             // http_response to care about this request
            llHTTPRequest(url,[],"");
        }
    }
    listen(integer channel, string name, key id, string message)
    {
        llSay(0,"Setting the tag for "+name+" to "+message);
        string url = "www.hackdiary.com/secondlife/experiments/set_tag?name="+name+"&tag="+message;
        llHTTPRequest(url,[],"");
    }
    touch_start(integer total_number)
    {
        // ask the server for a jpeg appropriate to the agent who touched us
        string url = "www.hackdiary.com/secondlife/experiments/sensed?name="+llDetectedName(0);
        llSetText("Finding a picture for "+llDetectedName(0),<0,0,1>,1);
        llParcelMediaCommandList([PARCEL_MEDIA_COMMAND_STOP]);
        http_id = llHTTPRequest(url,[],"");
    }
    http_response(key request_id, integer status, list metadata, string body) {
        // make sure we're processing a response we care about
        if(request_id == http_id) {
            // only on request success
            if(status == 200) {
                // data is coming back as pipe-delimited
                list data = llParseString2List(body,["|"],[]);
                // url is the first field
                string url = llList2String(data,0);
                if(url == "UNKNOWN") {
                    llWhisper(0,"I don't know what kind of picture to show you. 
                    Type '/1 sometag' to tell me what tag to search for on flickr and I'll remember it.");
                } else {
                    // name is the second field
                    string name = llList2String(data,1);
                    llSetText("Showing a picture for "+name,<0,0,1>,0.5);     
                    seturl(url);
                }
            }
        } 
    }
}

Permalink

Matt Biddulph, June 19, 2006 06:17 PM

What I do for a living

My name is Matt Biddulph. Since October 2005, I've been an independent freelancer, open to new work. I'm currently based in Amsterdam but I'm open to travel and have worked remotely for clients in a number of countries. I specialise in the design of software systems for the internet. I have a particular interest in digital media, social software and data on the web.

I'm an experienced developer in most of the languages and systems you'd expect from more than ten years of work in the industry: Ruby (and Rails), Python, Java, Perl, Unix, databases, web servers and so forth. These days I spend as much time consulting on the design and modelling of systems as I do writing code for them. I write about my personal experiments in technology here on hackdiary, and it's always a good reflection of my interests at any given time.

Before I went freelance, I spent several years working at the Press Association and the BBC. There's a lot more about that period in my CV.

Notable freelance projects

Nature publishing group

I developed a number of patches to Movabletype and modules for Ruby on Rails to enhance Nature's ability to publish over XML/RPC and Atom Publishing Protocol in their internal editorial systems.

BBC Programme Catalogue

Using Ruby on Rails, I was the developer and information architect on the first public experimental prototype of the BBC Programme Catalogue. The site exposes the database of the BBC's enormous archive to the internet via a search engine and web APIs.

Thinglink

Consulting on the technical architecture and implications of Thinglink, the free product code for creative work.

Permalink

Matt Biddulph, June 05, 2006 06:42 PM

Alas, Second Life! Web 2.0 in a virtual world

Second Life has been my new hacking obsession ever since I bought a laptop fast enough to run it. I don't spend a lot of time socialising in the gameworld, but I am fascinated by the possibilities for makers of new user interfaces, useful virtual objects and playful toys. With every object being scriptable, aware and active, it's a proving ground for Everyware.

Version 1.10 was released last week, and hidden among the exciting new visual modeling possibilities of shiny rendering and flexible objects was Second Life's own XMLHTTPRequest: llHTTPRequest. Using asynchronous callbacks, it gives the platform an important new capability: communication with the web on demand. A lot of what we are learning about AJAX makes sense here, in this world of Asynchronous Lindenscriptinglanguage And Some-sort-of-data (ALAS!)

I've spent a few hours hacking on some toy objects with this new capability, starting with the mashup de rigeur: Flickr integration. My home in SL now sports a simple picture frame. Touch it and it looks up your avatar name to see what your favourite Flickr tag is, picks a random picture with that tag from Flickr and displays it on its surface. If it hasn't met you before, it asks you to tell it what tag to use.

Because Second Life is so wonderfully visual, here's a little demo movie that I recorded with Tom Coates:

how it works

UPDATE: source code is available

LSL is a rather limited C-like scripting language with a couple of interesting features (like its native support for state-machines to respond to events). As far as I've been able to work out, it lacks the ability to parse XML or JSON, and has no native associative array type. Objects can store some state, but it is a little fragile in the face of resets. I decided not to try accessing the Flickr API directly, or using local storage. Instead, I created a stateful web companion for my new object with a bit of serverside Rails.

Strings and lists are the most useful datatypes available, and there's a function similar to perl's 'split' for turning strings into lists. This makes pipe-delimited a reasonable format for simple data. I created the following web resources:

/seen?user=SomeAvatar: Records that the object has sensed the presence of SomeAvatar

/touched?user=SomeAvatar: In response to a 'touch' event, consults the database for the user's tag and asks the Flickr API for a random photo with that tag. Returns a string such as flickr.com/some/photo.jpg|SomeAvatar|, or UNKNOWN.

/set_tag?user=SomeAvatar&tag=sausages: Records SomeAvatar's favourite tag

The HTTP system is nicely responsive: using the web as my object's outboard brain added only a tiny bit of latency to the mix. The asynchronous model allowed other processing to continue while waiting for a response. With these URLs ready to respond, I wired up the appropriate Second Life events using llSensorRepeat and sensor for presence, llListen and listen to respond to spoken commands, and touch_start for the physical interface. The llParcelMediaCommandList features are confusing (and only work on land you own, with movie streaming enabled in the client), but I found the source code for Freeview to be a useful reference.

If you're interested in this code, or would like to see a demo, you can find me from time to time in SL as Matt Basiat.

Permalink

Matt Biddulph, May 29, 2006 11:37 PM

XTech 2006 and Barcamp Amsterdam II

XTech 2006 and Barcamp Amsterdam II just wound up. It's been a mind-expanding, productive and wonderfully fun week. It was my second time on the XTech programme committee, and my first ever Barcamp.

Looking back at the schedule and the flickr photos and taking stock, I'm even more thrilled about the lineup and the people than I was before it all started. I'm very proud that I could play a small part in putting together the XTech programme.

My talk (on the BBC Programme Catalogue work) and our tutorial on Ruby on Rails went pretty well according to the attendee evaluations. I saw a ton of really thought-provoking talks and got to hang out with some very smart and lovely people.

Even putting together a list of URLs so I could name-drop my old and new friends took about 15 minutes; here goes: Marco Wessel, Ulla-Maaria Mutanen, Andy Smith, Freeman Murray, Suw Charman, Paul Hammond, Deb Bassett, Rob Lee, Tom Coates, The Ryan King, Brian Suda, Ralph Meijer, Thomas Vander Wal, Uche Ogbuji, Edd Dumbill, Jeremy Keith, Liz Turner, Simon Willison, Nat Downe, Dave Beckett, Matt Patterson, Max Carlson, Paul Prescod, Libby Miller and Pete Ferne. There were many more, and surely more still that I didn't get a chance to meet. If you didn't make it to either of the events, keep an eye out for the next time round. You won't regret it.

Permalink

Matt Biddulph, May 21, 2006 10:54 PM

Last.fm isn't just for humans

While I'm talking BBC, here's a story from a little while ago. I'm a big fan of last.fm, and I've been using it for a few years. Because I used to play my MP3s on a headless linux box in my flat, I wrote a commandline python mp3 player that could ping last.fm. My profile is a pretty good picture of my listening habits.

At BBC Radio, the radio stations are moving steadily from traditional analogue studios to fully digital systems that play nearly all their music from hard disk. As a member of the Architecture Team there, I had access to experimental data feeds from these systems. One day at work I asked myself a question: what happens when you plug behavioural data generated by an automatic process into social software designed for humans?

Half an hour later, I'd rigged my last.fm plugin into the feed system and switched it on. Over a year later, when I left the BBC, sekrit had accumulated a record of more than 50,000 tracks played on BBC 6Music.

Bear in mind when looking at this data that only the most mainstream and automatable parts of this admirably diverse radio station are visible in the feed. Every dusty Ska 7" played by Phill Jupitus on the Breakfast Show is invisible here. Even with this proviso