What is Plotter?

Plotter is an open-source tool for designers to map their data into the real world. Plotter asks the you – the designer – to define that connection between data and visual presentation before making anything. In other words, whatever else you make with Plotter, you’ll be making a map.

The system is designed so that in theory you could map anything you wanted – buildings, animals, or conceptual frameworks.

Plotter asks that whatever other characteristics each individual component of the dataset contains, that it also include a specific timestamp and location vector. The timestamp and location can be of any scale – seconds to centuries, millimeters to light-years – but they need to be embedded within the data. Using the two characteristics of space and time, Plotter then asks the user to provide the imagery to layer data on top of. This is the context, and can take the form of any raster-based photograph, scan, sketch, rendering etc. There are plans to allow Plotter to accept vector imagery such as KML files and 3D models as well.

After appropriate data and map imagery have been assembled, the core of Plotter goes into effect – a 3D engine for rendering, displaying, and exploring data in real time.

Prototyping

Plotter – Early Prototype from Peter Esveld on Vimeo.

This project didn’t set out to be a visualization tool. Originally, Plotter was conceived as a client project. A manager at the U.N. in New York had asked for a sophisticated way to track and view U.N. flights all around the world. The goal was to be able to see individual routes and fuel consumption – and get a really detailed look at the carbon footprint generated by all this air travel on both a local and a global scale. This would have been really simple to accomplish with charts and graphs, but I wanted to build something more sophisticated. The first attempt is what you see in the video above.

So, prototyping started not by creating data visualizations in Processing, but in trying to pick a technology to parse data in Excel Documents and upload that into something usable (A CouchDB database, as it turns out). All the U.N. flight records were in relatively unstructured Excel format – lots of extra junk in there, extra cells and rows, gif images, etc. Here’s an example, freely available online. So, I needed a really flexible way to program something to pull this data out. Where I wouldn’t have to write thousands of lines of code, and could hopefully leverage existing libraries to deal with the primitive nuts and bolts. This is where Clojure fits the bill. I wasn’t really anticipating learning a Lisp language or the basics of functional programming, but that’s what happened. Clojure is built on top of the Java Virtual Machine, which means that you can call methods from any Java library – including JExcel, Java’s Excel document library. Clojure also lets you accomplish tasks with 5 lines of code where Java might require you to write 30. Here’s an example that uses JExcel to open up an Excel doc, look at each sheet, and print each row of that sheet to the console:

(defn parse-it [file]
  (let [wb (Workbook/getWorkbook (File. file))]
    (doseq [sheet (.getSheets wb)]
      (let [num-rows (range 0 (.getRows sheet))]
        (doseq [r num-rows]
          (let [row (.getRow sheet (r))]
          (prn (format-temp row))))))))

[Link to full parsing code here]

Once the Excel docs are parsed, they are pushed up into a CouchDB database. If a database doesn’t exist, one is created on the fly. This is all made possible by the excellent Clutch library for Clojure. CouchDB isn’t super well documented yet, but it’s basically a very fast database that allows versatile indexing and searching. More information can be found here. CouchDB is excellent for data visualization, because it allows for the creation of specific views, or subsets of data, to that you can use to pull in only the data you need for a given application state. For purposes of testing and design, I used a local installation of CouchDBX, which provides a bare-bones GUI so that you can interact with Couch, browse the records, and write views—all inline.

The core of Plotter is built using Processing, a Java library for visual programming. I further clouded the issue by using a more obscure fork of Processing called Ruby-Processing, developed by Jeremy Ashkenas. Ruby-Processing runs on the JVM as well, using JRuby, and is great for building cool stuff very quickly. Plotter is, at its heart, a big, tile-based, scaling interactive map. As such, it owes a heavy debt to Modest Maps, developed by the amazing guys over at Stamen, a design studio in San Francisco. Plotter’s basic functionality comes from Modest Maps, although most of it was completely re-worked to work as a concurrent, thread-safe 3D mapping engine (written in Ruby, not Java).

A typical useful Jruby paradigm might be something like creating a new Java thread pool:

# Thread Pool
@executor = java.util.concurrent.Executors::newFixedThreadPool(13)
# Array to hold queued threads.
@future = Array.new

Later, add a thread to the pool (Loader needs to implement Java’s Callable)

@future.push(@executor.submit(Loader.new(v, @list[v[2]][v])))

Here’s the Loader class

class Loader

  include Processing::Proxy
  include java.util.concurrent.Callable

  def initialize(key, path)
    @key = key
    @path = path
    #puts "loader #{@key}"
  end

  def call
    img = PImage.new
    img = load_image(@path)
    #puts "the key is: #{@key}, the path is #{@path}, the image is #{img}"
    return @key, img
  end
end

Then you could check to see if those threads have finished, and pop the return values (an image and hashtag, in this case) into another array:

@future.each_index do |i|
  if(@future[i].isDone())
    key, val = @future[i].get
    @images[key] = val
    @future.delete_at(i)
  end
end

Look & Feel

Plotter’s look and feel is largely up to the developer. In general, it attempts to keep a very neutral design and let the interaction and data take center stage. Even so, things should look extremely clean and tidy, and very modern. Plotter’s design aesthetic tends toward the work of Edward Tufte, Herbert Bayer, and contemporary Dutch graphic designers. Here are a few sample images to better explain the ideal Plotter mapping aesthetic.

Image from this amazing Flickr set.

5.14.2010 at 6:00 pm

What is Plotter?

Prototyping

Look & Feel

All original content © Friend 2010.