Geo Code

The following post was written by our newest team member @deland, and it is something between an introduction to the Hyperpublic API and a documentation of a first experience with it. The actual use case described below is just a toy application - but it provided an opportunity for some cool visualizations. We hope you enjoy it and come up with some cool uses of your own!

If you haven't already, the first thing you're going to need to do to get started with the API and follow along (or experiment on your own) is to register for a Hyperpublic API key. It's free! It's fun! There are API wrappers for many programming languages so you can choose your favorite and get started right away. I'm going to be describing my method using Python, but the syntax in all the languages is pretty straightforward.

We're going to focus on pizzerias in New York City because everyone loves pizza (and because we had to choose something). The Hyperpublic API is a window into the world indexed by (exact) location. We can query the API by sending it addresses, zip codes, phone numbers, or latitudes and longitudes as well as categorical information (more on this in a moment). The query we send the API is then resolved to a specific latitude and longitude, and nearby places satisfying the query are returned. This means if we send the API a generic location like "New York, NY", a specific latitude and longitude is set, and then the search is executed. What this really means is you may not get what you wanted if you use such a generic location! The parameters available to query the API are described here, as is the structure of the documents that will be returned to you.

The categorical information in included in the query quite easily. We ask for places with a specific category (restaurant, office, etc) or we can ask for the database to be searched for other text (via the parameter 'q', for query). All locations with a place-name or tag matching your text will be returned. So we will query the API using the location parameter and also q = 'pizza'. For reasons that will become clear soon, I took the result of the queries and stored them in a local Mongo database . To be able to get all pizzarias in new york city, I looped over all the zipcodes in new york. The API only returns 50 locations at a time, but we can ask for more. Here is the relevant code:

 

from hyperpublic import *
...
for zip in zipcodes:
   inserted = 0
   notinserted = 0
   for p in range(1,rounds + 1):
      try:
         items = hp.places.find(location=str(zip),q="pizza", page = p, page_size = 50)
      except:
         break
      for item in items:
         try:
            dbh.nypizza.insert(item, safe = True)
            inserted += 1;
         except DuplicateKeyError:
            notinserted += 1
    print "%d items inserted and %d duplicates found for zipcode %d" % (inserted, notinserted, zip)


Here, my local mongodb connection is called dbh and the collection is called nypizza. Because some zip codes don't cover that much area - many of these queries will return the same pizzerias as other queries. This may not be a problem depending on your application, but you also might want to be aware of it (I just prevented mongo from duplicating records by inserting a key on the id field). Now we have the location of all (I get 2439) pizza places in the city - how easy was that?!

Probably we want to visualize all the locations at once. There are many options for doing this - and many existing python libraries to make our lives easier. For visualization only, my favorite tool (so far) was created by Seth Golub. It creates a heatmap based on density of points, and can overlay the 'heat' on top of maps coming from OpenStreetMap . This image is available under CC-BY-SA.

Depending on your needs, you can also overlay this information onto Google Earth maps or Gmaps. The nicest tool I found for this is hosted here. The interface is incredibly easy. Here is the output (it spits out kml data which google earth can read, and can also be accessed by google maps):

 Let's visualize the network of pizzerias in the city. I inserted a geo index on the mongo database. Then for every pizza place X, I can query the database for all other pizzerias within .1 miles (roughly) of X. Every time I find a relationship like this, I create an edge in a graph using the awesome network/graph software package at NetworkX. Here is the result:

You might start wondering, "if I wanted to go on a 'pizza-slice' tour of new york city, and I wanted the longest sequence of pizzerias separated by .1 miles without visiting any place twice - what would I do?". Coincidentally, I wondered the exact same thing, and wrote a short function to search the graph for the longest such path. (For those who care, finding the longest path in a graph with cycles is known to be NP-hard. So don't try this on your big graphs.) Here is the result, you can visit 32 pizzerias (labeled, in order,it A-Z, then 0-5). Best of luck.