It seems when we launched version 2.0 of our Flickr shapes, we posted them with a flaw which made them useless to most popular geo applications.

Awwwww…

Luckily, Christopher Manning wrote a python script which makes them useful.

Yaaaayyyyy!

The least we can do is post an update which has already been christopher-manning-ified, So, we are very happy to announce version 2.0.1 of the Flickr shape files which can be downloaded here:
www.flickr.com/services/shapefiles/2.0.1/

Look, it works:

Flickr Shapes 2.0.1 in TileMill

A very hearty THANKS! from your friends at Flickr, Christopher.

The best thing about working at Flickr is that my coworkers all love the site and product ideas can come from anyone.

Recently, the Flickr staff had to work from home while our office was disassembled and relocated a few floors down. A chance to sleep in and start the weekend early? Or get together with a few ambitious coworkers and hack together a new feature?

We met at Nolan’s house, ate a farmer’s breakfast, and brainstormed.

We wanted to build something fun, which a few of us could start working on that morning and have a demo ready by the end of the day. Something suited to the talents and interests of the people in the room. Secret Faves? Risqué Explore?

Bert wanted to help people geotag more photos, but he wanted more sophisticated privacy controls first. I’d been using a simple web app that I built with the Flickr API to manage the geo privacy of my photos, and it seemed like something more people should have access to.

So we had a need. We had a proof of concept. We had enough highly caffeinated engineers to fill a small dinner table. “Let’s build geofences.”

What the beep is a geofence?

You probably know what geotagging is. It’s nerd-speak for putting your photos on a map. Flickr pioneered geotagging about five years ago, and our members have geotagged around 300 million photos and videos.

We’ve always offered the same privacy settings for location data that we offer for commenting, tagging, and who can see your photos. You have default settings for your account, which are applied to all new uploads, and which you can override on a photo-by-photo basis.

This works well for most metadata. I have a few photos that I don’t want people to comment on or add notes to, but for the most part, one setting fits all my needs.

But geo is special. I often override my default geo privacy. Every time I upload a photo taken at my house, I mark it “Contacts only”. Same for my grandma’s house. And that dark place with the goats and candles? Sorry, it’s private.

Managing geo privacy by hand is tedious and error prone. Geofences make it easier.

Geofences are special locations that deserve their own geo privacy settings. Simply draw a circle on a map, choose a geo privacy setting for that area, and you’re done. Existing photos in that location are updated with your new setting, and any time you geotag a photo in that area, it gets that setting too.

Geofences are applied at upload time, or when you geotag a photo after uploading it. It doesn’t matter how you upload or tag your photos: The Organizr, the map on your photo page, and the API all use geofences.

If you’re ready to dive in, visit your account geo privacy page and make your first geofence.

Boring details

When dealing with privacy, we need to be conservative, reliable, and have clearly defined rules. The geofences concept is simple, but the edge cases can be confusing.

No geofencesCommon use case

• What happens when you have overlapping geofences?
• What if you move a photo from outside a geofence to inside one?
• Where does your default geo privacy setting fit in?

The vast majority of Flickr members will never encounter these edge cases. But when they do, Flickr plays it safe and chooses the most private setting from the following options:

• The member’s default geo privacy
• The current geo privacy of the photo (if it was already geotagged)
• Any geofences for the new location

If your account default is more private than your geofences, the geofences won’t take effect. If you have overlapping geofences for a point, the most private one will take effect. If you move a photo whose location was private into a contacts-only geofence, it will stay private.

Overlapping geofencesMore private account default

As a reminder, here’s the ranking of our privacy settings:

Note that friends and family are at the same level in the hierarchy. Your family shouldn’t see locations marked as friends only, and vice versa.

With that in mind, what should Flickr do when someone geotags a photo where friends and family overlap? Maybe he wants both friends and family to see it, or maybe he wants neither friends nor family to see it. To really be safe, we must make that location completely private.

Friends and family overlap

Go forth and geotag

A few years ago, privacy controls like this would have been overkill. Geo data was new and underused, and the answer to privacy concerns was often, “you upload it, you deal with it.”

But today, physical places are important to how we use the web. Sometimes you want everyone to know exactly where you took a photo. And sometimes you don’t.

An embarrassingly long time ago we released the first public version of the Flickr Shapefiles. What does this have to do with Captain America and a cat? Nothing, really.

Anyway, we haven’t completely forgotten about shapefiles and have finally gotten around to generating a new batch (read about Alpha Shapes to find out how it’s done). When Aaron did the first run we had somewhere around ninety million (90M) geotagged photos. Today we have over one hundred and ninety million (190M) and that number is growing rapidly. Of course lots of those will fall within the boundaries of the existing shapes and won’t give us any new information, but some of them will improve the boundaries of old shapes, and others will help create new shapes where there weren’t any before. Version 1 of the dataset had shapes for around one hundred and eighty thousand (180K) WOE IDs, and now we have shapes for roughly two hundred and seventy thousand (270K) WOE IDs. Woo.

The dataset is available for download today, available for use under the Creative Commons Zero Waiver:

www.flickr.com/services/shapefiles/2.0/

Little Johnny JSON

Originally we provided the full dataset in our own home-grown XML format because, well, it seemed like a good idea. For version two we’re releasing the shapes in GeoJSON format. We think this is a Good Thing because unlike our  old XML format, at least one other person in the world already knows how to read and write GeoJSON. For example, Our friends over at Stamen Design and SimpleGeo have created a ridiculously easy-to-use JavaScript library called Polymaps which of course reads GeoJSON out of the box. With a few lines of JavaScript you can render the Flickr shapefiles and start using them without all that pesky XML parsing stuff:

Or if GeoJSON doesn’t suit you you can use a free tool like ogr2ogr to convert it to something that does.

Layers

(photo by doug88888)

The GeoJSON format allows grouping of features (and their related geometries) into FeatureCollection objects. A FeatureCollection seems to be roughly equivalent to a layer in a typical GIS, or a placetype in WhereOnEarth-speak. To make the dataset a little easier to manage we decided to break the shapes up into FeatureCollections based on placetype; one each for continents, countries, regions (states), counties, localities (cities), and neighbourhoods. Each of these is its own GeoJSON file in the dataset.

Here’s an example of what one of our GeoJSON objects looks like:

{
  "type": "FeatureCollection",
  "name": "Flickr Shapes Public Dataset 2.0 - Regions",
  "features": [
    {
      "type": "Feature",
      "id": 2344541,
      "properties": {
      "woe_id": 2344541,
      "place_id": "Cxf0SmObApi9R9T8",
      "place_type": "region",
      "place_type_id": 8,
      "label": "Barbuda, AG, Antigua and Barbuda",
    },
    "geometry":
    {
      "type": "MultiPolygon",
      "created": 1292444482,
      "alpha": 0.03,
      "points": 118,
      "edges": 17,
      "is_donuthole": 0,
      "link": {
        "href": "farm6.static.flickr.com/5209/shapefiles/2344541_20101215_724c4cae47.tar.gz",
      },
      "bbox": [-62.314453125,17.086019515991,-61.690979003906,17.93692779541],
      "coordinates": [
        [
          [[-61.739044,17.587740], [-61.735268,17.546171], [-61.690979,17.426649], [-61.765137,17.413546]
... etc

A file is a single FeatureCollection object which holds an array of Features, which each hold a Geometry which is a MultiPolygon, which holds an array of Polygons which in turn each consist of an array of LinearRings. Got it?

You’ve Been Superseded

We’ve also included a separate file/layer called flickr_shapes_superseded.geojson. This is a FeatureCollection that consists of all the WOE IDs that have been “superseded”. Occasionally (too often?) a WOE ID needs to be retired and replaced with a new one. We keep up to date with these and are always reverse-geocoding against the latest WOE IDs. However, there are plenty of old photos (and Flickr shapes) that have been assigned to one of these old IDs, and we have shapes for them (currently a little more than nine thousand). A simple solution might be to just re-assign these photos to the new WOE IDs (when a WOE ID is retired its replacement is specified), or even just re-run the reverse-geocoding process. This may not be what the owner of the photo wants; it might come out with a different result than they had at first (which they may have been perfectly happy with), if the size and location of the WOE rectangle changed (which it probably did). So it’s a problem without a clear solution. And since we have data for these old WOE IDs we’ve included their associated shapes in the dataset. In most cases there will be shapes corresponding to the new WOE IDs that will over time become more and more accurate as more photos get uploaded and end up being assigned to the new WOE IDs. But you can have the old ones too.

WTF

Sometimes, Clustr just gets it wrong. As mentioned in previous posts many of the Flickr shapes are just plain weird. It may be due to a lack of data (i.e. source photos), a weakness of the algorithm, an inappropriate choice of the alpha parameter or (shame!) a plain old bug. One of the things that surprised us was that Clustr was not supposed to output inner rings, or polygons with holes. It turns out that it does. In the GeoJSON output this can cause some weirdness (depending on what you use to render the shapes) since the GeoJSON is formatted with the assumption that each ring is a distinct polygon, instead of possibly one part of a single polygon with holes in it. This and other weirdness are known issues and something we shall strive to fix in the future, however we felt it best to release the existing dataset now rather than spend forever trying to get it perfect, and end up not releasing anything at all.

You may also notice that unlike version 1 of the dataset, there is only a single shape per WOE ID. All of the previous versions of the shapes are still available via the Flickr API, but in the interests of keeping the file size down we’ve limited this download to just the latest versions of each shape.

And as always, there’s lots more to do.

Recently Yahoo!! released new map tiles … ah heck, actually it was a couple of months ago, I just suck at writing blog posts … anyway, here’s their post from last December: Yahoo! Maps – New International Coverage which goes on to say …

“We’ve added detailed coverage to 45 new countries, with new data in a further 30 countries, to make it easier for you to navigate to exotic locales as you plan for your winter travel.”

I’m as keen as the next person to have 45 new countries, so I bumped our version of the maps API used over here at Flickr to make use of the new tiles.

Which gives us more to work with for many bits of Albania, Andorra, Argentina, Australia, Austria, Bahrain, Belarus, Belgium, Bosnia & Herzegovina, Botswana, Brazil, Bulgaria, Canada, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Gibraltar, Greece, Hong Kong, Hungary, Iceland, India, Indonesia, Ireland, Italy, Kuwait, Latvia, Lesotho, Liechtenstein, Lithuania, Luxembourg, Macau-China, Macedonia, Malaysia, Malta, Mexico, Moldova, Monaco, Montenegro, Namibia, Netherlands, New Zealand, Norway, Oman, Philippines, Poland, Portugal, Qatar, Romania, Russia, San Marino, Saudi Arabia, Serbia, Singapore, Slovakia, Slovenia, South Africa, South Korea, Spain, Swaziland, Sweden, Switzerland, Taiwan, Thailand, Turkey, US, United Arab Emirates, United Kingdom and [catch breath], Vietnam.

Often we’ll try to improve our coverage with the help of (the frankly awesome) OpenStreetMap (OSM) peeps, as previously mentioned in; Around the World and Back Again, Flickr [heart] Burning Man [heart] OpenStreetMap and More new map tiles.

With this roll out we’ve actually removed some OSM tiles. Mexico City, Rio de Janeiro, Sao Paulo are now covered in more detail by Yahoo! … here’s the tiles for Mexico City …

… which as you can see are pretty good.

But from OpenStreetMap we’ve added Accra, Algiers, Cairo, Harare, Kinshasa, Mogadishu and Nairobi, you can see the before and after for Nairobi below …

Before

After, with OSM Tiles

… which as you can see are also pretty good

Clearly we can’t spend forever swapping tiles in and out, well maybe we can, but at some point there’ll probably be a better way of managing this kind of thing.

In the meantime though, this being the code blog, and just for fun, here’s a little bit of what happens in the world of JavaScript.

Warning, some code ahead!

This check_map function is called each time you move, zoom or switch map views …

check_map: function(map_obj, parent_node) {
	
	//	Grab the focus, zoomlevel and maptype of the current view
	var lat_lon	=	map_obj.getCenterLatLon();
	var zl 		=	map_obj.getZoomLevel();
	var map_type 	=	map_obj.getCurrentMapType();

Then we check the location against a list of places we want to load OSM tiles for, we could do this any number of ways, but pragmatically this just works (scroll the below code all the way to the right for the interesting bits, if you’re not reading this in an RSS feed) …

	//	Now see if we match any of these following tests to work out if we need to switch to osm or not.
	//	The meta-test is if we are in MAP mode.
	var in_the_zone = false;
	if (map_type == 'YAHOO_MAP') {
		if (zl < 8 && lat_lon.Lat >= 39.8558502197 && lat_lon.Lat <= 40.0156097412 && lat_lon.Lon >= 116.2662734985 && lat_lon.Lon <= 116.4829177856) in_the_zone = true;		// Beijing
		if (zl < 7 && lat_lon.Lat >= 40.735551 && lat_lon.Lat <= 40.807533 && lat_lon.Lon >= -119.272041 && lat_lon.Lon <= -119.163379) in_the_zone = true;				// Black Rock City, 2008
		if (zl < 9 && lat_lon.Lat >= 35.46290704974905 && lat_lon.Lat <= 36.02799998329552 && lat_lon.Lon >= 139.21875 && lat_lon.Lon <= 140.27069091796875) in_the_zone = true;	// Tokyo
		if (zl < 8 && lat_lon.Lat >= -34.6944313049 && lat_lon.Lat <= -34.4146499634 && lat_lon.Lon >= -58.6389389038 && lat_lon.Lon <= -58.2992210388) in_the_zone = true; 		// Buenos Aires
	// Yahoo Better	if (zl < 8 && lat_lon.Lat >= 18.9466495514 && lat_lon.Lat <= 19.6985702515 && lat_lon.Lon >= -99.5071487427 && lat_lon.Lon <= -98.7429122925) in_the_zone = true; 	// Mexico City
	// Yahoo Better	if (zl < 8 && lat_lon.Lat >= -23.0837802887 && lat_lon.Lat <= -22.7662200928 && lat_lon.Lon >= -43.7946395874 && lat_lon.Lon <= -43.1328392029) in_the_zone = true; 	// Rio de Janeiro
	// Yahoo Better	if (zl < 8 && lat_lon.Lat >= -24.0083808899 && lat_lon.Lat <= -23.3576107025 && lat_lon.Lon >= -46.8253898621 && lat_lon.Lon <= -46.3648300171) in_the_zone = true; 	// Sao Paulo
	
		if (zl < 8 && lat_lon.Lat >= -35.2210502625 && lat_lon.Lat <= -34.6507987976 && lat_lon.Lon >= 138.4653778076 && lat_lon.Lon <= 138.7634735107) in_the_zone = true; 		// Adelaide
		if (zl < 8 && lat_lon.Lat >= -34.1896095276 && lat_lon.Lat <= -33.5781402588 && lat_lon.Lon >= 150.5171661377 && lat_lon.Lon <= 151.3425750732) in_the_zone = true; 		// Sydney
		if (zl < 8 && lat_lon.Lat >= -27.8130893707 && lat_lon.Lat <= -27.0251598358 && lat_lon.Lon >= 152.6393127441 && lat_lon.Lon <= 153.3230438232) in_the_zone = true; 		// Brisbane
		if (zl < 8 && lat_lon.Lat >= -35.4803314209 && lat_lon.Lat <= -35.1245193481 && lat_lon.Lon >= 148.9959259033 && lat_lon.Lon <= 149.2332458496) in_the_zone = true; 		// Canberra
		if (zl < 8 && lat_lon.Lat >= -38.4112510681 && lat_lon.Lat <= -37.5401115417 && lat_lon.Lon >= 144.5532073975 && lat_lon.Lon <= 145.5077362061) in_the_zone = true; 		// Melbourne
	
		if (zl < 8 && lat_lon.Lat >= 33.2156982422 && lat_lon.Lat <= 33.4300994873 && lat_lon.Lon >= 44.2592010498 && lat_lon.Lon <= 44.5364112854) in_the_zone = true; 		// baghdad
		if (zl < 8 && lat_lon.Lat >= 34.4611015320 && lat_lon.Lat <= 34.5925598145 && lat_lon.Lon >= 69.0997009277 && lat_lon.Lon <= 69.2699813843) in_the_zone = true; 		// kabul
	
		if (zl < 10 && lat_lon.Lat >= -4.3634901047 && lat_lon.Lat <= -4.3009300232 && lat_lon.Lon >= 15.2374696732 && lat_lon.Lon <= 15.3460502625) in_the_zone = true; 		// kinshasa
		if (zl < 10 && lat_lon.Lat >= 2.0093801022 && lat_lon.Lat <= 2.0614199638 && lat_lon.Lon >= 45.3139114380 && lat_lon.Lon <= 45.3669013977) in_the_zone = true; 			// mogadishu
		if (zl < 10 && lat_lon.Lat >= -17.8511505127 && lat_lon.Lat <= -17.7955493927 && lat_lon.Lon >= 31.0210304260 && lat_lon.Lon <= 31.0794296265) in_the_zone = true; 		// harare
		if (zl < 10 && lat_lon.Lat >= -1.3165600300 && lat_lon.Lat <= -1.2379800081 && lat_lon.Lon >= 36.7483406067 && lat_lon.Lon <= 36.8735618591) in_the_zone = true; 		// nairobi
		if (zl < 10 && lat_lon.Lat >= 5.5237197876 && lat_lon.Lat <= 5.5998301506 && lat_lon.Lon >= -0.2535800040 && lat_lon.Lon <= -0.1586299986) in_the_zone = true; 			// accra
		if (zl < 10 && lat_lon.Lat >= 30.0068798065 && lat_lon.Lat <= 30.1119003296 && lat_lon.Lon >= 31.2149791718 && lat_lon.Lon <= 31.3111705780) in_the_zone = true; 		// cairo
		if (zl < 10 && lat_lon.Lat >= 36.6997604370 && lat_lon.Lat <= 36.8181610107 && lat_lon.Lon >= 2.9909501076 && lat_lon.Lon <= 3.1476099491) in_the_zone = true; 			// algiers
	
	}

If we found a match up there, then "in_the_zone" would have been set, in which case we'll check to see if were already showing osm tiles, if not we'll tell the YAHOO API to use our OSM tiles rather than the YAHOO ones ...

	//	ok, if we've match one of the above tests then we are in osm land ...
	if (in_the_zone) {
	
		//	if we aren't already osming, then we need to switch the map tiles.
		if (!this.osming) {
			
			//	Say that we are now osming
			this.osming = true;
			//	Put the new tiles in
			YMapConfig.tileReg=['/our_openstreetmap_tile_broker.xxx?t=m&','/our_openstreetmap_tile_broker.xxx?t=m&'];
	
			//	Tell the maps to clear out the tile cache and load in the new tiles.
			map_obj._cleanTileCache();
			map_obj._callTiles();
		}
	} else {
 

"YMapConfig.tileReg=" tells the map API to load tiles from us rather than the default ones, hint, you should set this to your own tile server

"_cleanTileCache()" and "_callTiles()" are two undocumented functions (and therefor may break at some point) that allows you to force the map to load in new tiles. Otherwise the current view of the map will still show the Yahoo tiles and not use the new ones until you next pan the map around. Again there's probably a better way of doing this, but there's still a lot to be said for it-just-works!

The next bit runs if we're not "in the zone" to see if we need to turn the OSM tiles back off.

	//	otherwise we are not looking at an osm spot, in which case check to see if
	//	we *were* looking at one, and if so, turn it all off again
		if (this.osming == true) {
			//	say that we are no longer osming
			this.osming = false;
			//	turn the tiles back
			YMapConfig.tileReg=this.oldTileReg;
			
			//	Tell the maps to clear out the tile cache and load in the new tiles.
			map_obj._cleanTileCache();
			map_obj._callTiles();
		}
	}

Which is roughly the opposite of turning them on. The only thing to note is that is the line ...

YMapConfig.tileReg=this.oldTileReg

... when we created the map I used this ...

//	record what the old tiles were
this.oldTileReg=YMapConfig.tileReg;

... to record where YMapConfig thinks it aught to be loading tiles from when in map view (as opposed to hybrid or satellite view) so it can be reset when you need it.

And that's pretty much it. I've chopped out some distracting bits here or there to make it clearer and no-one else is likely to do it like this, but I find sharing to be cathartic (adjective not noun)

For a smidge of further reading this is a handy article from A List Apart: Take Control of Your Maps by Paul Smith.