Azavea Labs

I was recently introduced to Steve Gifford at Mousebird Consulting, a software firm based in San Francisco that builds mapping tools for the iOS platform.  Steve and his colleagues are the developers of the open source iOS mapping framework, WhirlyGlobe Maply.  The framework enables them to build both 2D and 3D mapping applications for iPhones and iPads.  It’s slick, impressive technology that is sort of a combination of the Google Earth globe and a conventional, web-based mapping application.

Mousebird Consulting joined the LocationTech working group at the Eclipse Foundation in March.  LocationTech is a young organization and while there are now several projects moving through the incubation process (GeoTrellis is one of them), there is not yet a lot of coordination or integration between projects.  So I was really excited to see Steve take the initiative to integrate one of our GeoTrellis examples, the GeoTrellis Transit API demo, into Mousebird’s WhirlyViz application. GeoTrellis Transit is an extension of the core GeoTrellis framework.

While the core GeoTrellis is primarily focused on fast, distributed raster data processing, the GT Transit project adds support for fast network routing and incorporates both GTFS and OpenStreetMap parsing, a high performance network data structure and support for routing and calculation of time-dependent “travelsheds”, the area a traveler can reach within X minutes.  By “time-dependent”, I mean that GT Transit can calculate transit access areas for a specific time of day and days of week using the schedule information encoded in a GTFS data set.  All of this is wrapped by an API.  When we launched GeoTrellis Transit, we also set up a couple of demos using data for Philadelphia – a travelshed calculator and a “scenic route” demo that shows where you can wander between a starting and ending point and still arrive on time. The WhirlyViz app has some nice design features.  It’s a native iOS app, but it uses JSON and Javascript for configuration, and Steve was able to add a new configuration without having to roll out a new application.  Steve picked up the Travelshed API and turned it into a new configuration of the WhirlyViz app.  It’s pretty cool.  In addition to showing the travelsheds, you can set the day-of-week, time-of-day and transit modes.  He wrote up some details in a blog post he published last week.  Here are a few screenshots.

GeoTrellis Transit uses OpenStreetMap and a GTFS file to enable generation of “travel-sheds”. This one shows walking distance are around downtown.

The accessible area changes a great deal when we add access to regional rail.

References

• WhirlyGlobe on GitHub: mousebird.github.io/WhirlyGlobe/
• GeoTrellis on GitHub: https://github.com/geotrellis
• GeoTrellis Transit on GitHub: https://github.com/geotrellis/geotrellis-transit
• GeoTrellis Transit API demo: transit.geotrellis.com/
• GeoTrellis Transit Travelshed demo:  transit.geotrellis.com/travelshed.html
• GeoTrellis Transit Scenic Route demo:  transit.geotrellis.com/scenic.html

The Legendary Island of Avalon (image credit)

 

The GeoTrellis team is very excited to announce the availability of GeoTrellis 0.9 (codename “Avalon”), a significant new release that is a big step forward towards our goal of a general purpose, high performance raster geoprocessing library and runtime designed to perform and scale for the web.

First of all, we’ve significantly revised the documentation site — geotrellis.io/.  Props to Rob Emanuele for the new site and the Azavea design team for revised styling.  The new site includes both case studies and some samples we’ve developed since the 0.8 release.  There is a full set of release notes available, but here are some highlights:

• API Refactor: We’re moving away from requiring users to manually create operations and pass in rasters as arguments, and instead having objects called ‘DataSources’ that represent the source of data, with the operations to transform or combine that data as methods on those objects. These methods are not stateful, they return new DataSources per method call. Similar to the ‘Future’-like model of Operations, transformations on a DataSource are not actually run until the server is told to ‘run’ the source, either explicitly or through a method call on DataSource and an implicit server parameter. Special thanks to joshmarcus for his vision of this API change and all the work he put into making it happen. This API change also means that any code that currently runs on an 0.8 release will probably be very broken. The ability to create and run Op[T]’s is still there, but some of the functionality, especially dealing with parallelizing over tiles, was stripped from them. Let us know on the mailing list or in #geotrellis on freenode IRC if you’re upgrading and we’ll lend a hand with the transition
• File I/O: Reading ARGs from disk has been sped up, and in some cases, you’ll find improvements of an order of magnitude or more.
• Spray.io:  Replaced Jetty with Spray.io, a fast HTTP server for Akka Actors
• Tile operation improvements: Running multiple operations over tiled data has been greatly improved. For example, if you were to multiply a raster by an integer, add the result to another tiled raster, and then run a zonal summary (such as taking the mean of values within a polygon) on that result, GeoTrellis 0.8 would unnecessarily reduce to a whole raster in memory between the different transformations (see issue #517). In 0.9, you’ll get the desired behavior, where the multiplication, addition, and zonal summary all all done in parallel per tile, before the final zonal summary result is created from the reduction of the tile zonal summary results.
• Clustering improvements: We took several steps to make it easier to distribute operations over a cluster using Akka clustering. There’s a .distribute call on DataSource which will distribute all of the operations of the DataSource’s elements across the cluster, based on the configuration.
• Macros: A new geotrellis-macros project was created to deal with issue #624, based on the discussion of #324. This includes macros for checking whether a value is or isn’t NoData, independent of what type that data is. And these checks are inlined through macro magic, so there’s no performance hit for the nicer syntax.
• Revised Operations: Added double support to Focal Standard Deviation, Focal Min, Focal Max, Focal Sum operations; added 8-neighbor connectivity to RegionGroup;
• New Operations: Ordinary Kriging Interpolation, Hydrology operations (Fill, Flow Accumulations, Flow Direction), IDW Interpolation

Please let the team know — via the #geotrellis channel on Freenode IRC or the geotrellis-user Google Group mailing list — if you have any comments or suggestions.

Version 0.10 Plans

We’re hard at work on a GeoTrellis 0.10.  The major plans for this release include:

• Integrate Apache Spark 
• Support for operating on data stored in the Hadoop Distributed File System (HDFS)
• Support for multi-band rasters
• Develop a scala wrapper for JTS
• Add more operations
• Formal move to LocationTech working group at the Eclipse Foundation

More Info

• Github: github.com/geotrellis/geotrellis
• Maven repository: https://oss.sonatype.org/content/repositories/releases/
• API Scaladocs: geotrellis.github.com/geotrellis/latest/api
• Issue tracker for 0.10: https://github.com/geotrellis/geotrellis/issues?milestone=6&page=1&state=open
• Mailing list: https://groups.google.com/group/geotrellis-user
• IRC: #geotrellis on freenode

GeoTrellis is released under the Apache2 license.

A few months ago we set out to create a new system to forecast spacetime events that would be scalable, adaptable, and accurate.   Such a system has many potential uses such as  modeling global conflict or forecasting crime.  To accomplish our objectives we required tools with deep capabilities.   For instance, processing geospatial data of significant scale in a timely manner is a great fit for our GeoTrellis project.   Ready access to sophisticated modeling and machine learning capabilities is a natural fit for projects such as R and Apache Mahout.  How, though, can such diverse tools function together? CSVs.

While machine learning and “big data” may be all the vogue in the analysis community, at the end of the day these systems mostly boil down to a string of components that each transform tabular data into new tabular data.  While there are more efficient data formats that can be utilized with specific tools, by focusing on CSVs when possible, we designed a system that is modular and allows individual components to be replaced at whim.  A loosely coupled system allows greater flexibility for distributed and resilient processing since each step in the workflow can be monitored for success while work can be spread across multiple servers.

Our approach to predicting spacetime events requires a few steps:

• Process outcomes (events) and covariates into tabular format
• Build a model
• Make predictions using the model

Processing Training Data

In our case, the outcomes we want to predict are events in space and time.  Our covariates might be prior knowledge of event frequencies at a particular location, the distance or density of geographic features, or temporal variables such as the day of week and weather.   We decided to use Scala and GeoTrellis to fulfill these needs.  Our inputs to this step are CSVs (events, temporal variables), Shapefiles (points, lines, polygons), and GeoTIFFs (rasters).   Our outputs are simply CSVs.

In the process of working on this project, we extended GeoTrellis to support some geographic operations over rolling temporal windows.   For instance, rapidly calculating counts and Kernel Densities for the last 28 days for every time period within a range.   We also implemented a simple raster to CSV conversion operation to support the use of diverse modeling packages alongside GeoTrellis.

Model Building

For machine learning, we are leveraging the R project and Apache Mahout.  While our developers were initially skeptical to use R in a production system, we’ve been pleasantly surprised with its scale and speed.   Many R packages contain C++ or Fortran code under-the-hood that is quite fast.  R also allowed us to easily generate graphical output to examine our results.

We also spent some time working with Apache Mahout.   While Mahout will scale much more than R, we found that R was better for rapid prototyping and handled most of the scale we needed for this project (with some tricks).   If we need greater scale in the future, we can incorporate Mahout more directly without changing our overall workflow.

Predictions & Glue

Measuring accuracy in meaningful ways is an essential part of predictive modeling.  Ideally, it should be easy to iterate over new ideas and measure how your results progress.   Since we built many individual components for the project, we needed a tool to glue everything together and to allow a user to easily accomplish particular tasks.  We decided to use Python to build a command line tool for this part of the project.

To summarize, here is our toolchain:

• Python – primary interface to the project; glue that combines different tasks into simple commands
• Scala / GeoTrellis / R – geoprocessing and data preparation
• R / Mahout – modeling

I highly recommend such a loosely coupled approach to others working on geospatial modeling.   The flexibility of this approach has worked well for us and allowed us to use best-of-breed tools across all of our requirements.

GeoTrellis 0.8.1 is out, and with it is a new way to view your Raster data in an easy an informative way. In this blog post, we’ll take a look at some Pennsylvania elevation data using the new GeoTrellis admin tool.

The data that we’ll be viewing is from the National Elevation Database (ND) developed by the U.S. Geological Survey. In a tutorial on the GeoTrellis documentation site, we saw how to get the data and convert the GeoTIFF rasters into the ARG format required by GeoTrellis. With GeoTrellis 0.8.1, when you use the geotrellis-server project to develop REST services with GeoTrellis, you have the option to also serve an administration tool on the /admin path of the Jetty server. You only have to make sure that some options in your application configuration are set correctly (or not set, as these are the defaults). The settings are geotrellis.admin.serve-site = “yes” and geotrellis.admin.serve-from-jar = “yes”. The first setting tells the GeoTrellis server to start the services required by the admin site. The second tells the GeoTrellis server to serve up the static files that make up the admin site. The javascript, css and html files that make up the site are packaged into the goetrellis-server JAR.

Starting the admin tool

If we run the server (see the tutorial or the README on the github page for the code for how to run the server), we can go to localhost:8880/admin/ and get to the admin site:

In this case, the admin site is showing data from a catalog that I have locally. The catalog’s name is “Catalog of Rob’s Data”, and it has a number of data stores (indicated by the folders on the left). The catalog that the admin site displays is the one set in the configuration as geotrellis.catalog. For more information about the GeoTrellis catalog, see the overview section on the subject in the documentation.

We can open up the ‘NED data’ data store and select Philadelphia to visualize the raster:

The raster fits over the base map perfectly. This is because both the base layer and the raster have a projection of Web Mercator (ESPG:3857); if the raster was not in that projection, it would be places in an incorrect spot on the map (although you would still be able to view it).

Changing the base layer

We can choose another base layer using the Leaflet layer chooser on the right side of the map:

Let’s change it to the Stamen Toner layer, and increase the opacity of the raster using the Opacity slider, in order to get a more clear view of the coloring:

Changing the Color Ramp

Seeing Raster Values

The Mythical City of Atlantis – just as legendary as GeoTrellis 0.8’s advanced geoprocessing power! (image credit)

The GeoTrellis team is very excited to announce the availability of GeoTrellis 0.8 (codename “Atlantis”), which is a major new release that is a huge step forward towards our goal of a general purpose, high performance geoprocessing library and runtime designed to perform and scale for the web.

First of all, you should check out our new documentation site — geotrellis.github.com/ — which features a great deal of new documentation and tutorial material. Credit is due to Rob Emanuele for the terrific new organization and layout.

Secondly, take a look at the Azavea Labs blog Adam Hinz recently published (example running above) which describes process of building a kernel density service using GeoTrellis that features some of the new 0.8 functionality. Adam’s post details step-by-step the development of a WMS service that should be helpful as you think about how you might develop your own services with GeoTrellis. It is a companion blog to the Azavea Atlas blog series being written by John Branigan that explores map algebra concepts through the process of creating a site suitability model using wildlife habitat preferences.

As you delve into GeoTrellis 0.8 in more depth, here are some new features you may want to explore:

• A new suite of raster operations in the focal category of map algebra.
  • For more information, see: geotrellis.github.com/operations/raster/focal.html
• Simplified raster rendering with a built-in suite of color ramps for data cartography.
  • For more information, see: geotrellis.github.com/overviews/rendering.html
• Zonal summary (by feature) operations, including a caching mechanism for tiled rasters for improved performance.
  • For more information, see: geotrellis.github.com/operations/raster/zonal.html
• New vector feature framework for operations on points, lines, polygons with a suite of vector operations, as well as re-engineered rasterization operations.
  • For more information, see: geotrellis.github.com/overviews/vector.html and geotrellis.github.com/operations/feature.html

Please let the team know — via the #geotrellis channel on Freenode IRC or the geotrellis-user Google Group mailing list — if you have any comments or suggestions. We will likely release a minor bugfix release (0.8.1) in the future.

Github: github.com/geotrellis/geotrellis
Maven repository: https://oss.sonatype.org/content/repositories/releases/
API Scaladocs: geotrellis.github.com/geotrellis/latest/api
Issue tracker: https://github.com/geotrellis/geotrellis/issues?milestone=3&state=open
Mailing list: https://groups.google.com/group/geotrellis-user
IRC: #geotrellis on freenode

GeoTrellis is released under the GPL V3 license.