Upgrading From REE 1.8.7 to Ruby 1.9.3

In a previous blog post, we discussed our path to upgrading to Rails 3.0 from Rails 2.3. At the time, a number of comments asked about our upgrade path from 1.8.7 to 1.9.3. We waited until the Rails 3.0 upgrade was complete and in production before beginning the Ruby upgrade. It is probably a good thing, since upgrading our Ruby version required significantly more work than we had anticipated.

We were really excited about the potential performance improvements that a number of other companies have reported after upgrading to Ruby 1.9.3. Harvest, ZenDesk, UserVoice, NewRelic, and Ngin all have released great blog posts reporting pretty significant performance gains after making the upgrade.

Getting Started

The first major milestone was getting our Rails app to start locally in Ruby 1.9.3. We had to upgrade a number of our gems (e.g. Zookeeper,libxml-ruby, hpricot) so that they would work in Ruby 1.9.3. For some gems, we only needed them in one environment. Gemfiles have a useful feature where you can specify the platform that you want a particular gem installed, like so:

group :development do
  gem 'ruby-debug', :platforms => :ruby_18
  gem 'debugger', :platforms => :ruby_19
end

Sharing Sessions

Currently, Airbnb utilizes Ruby on Rails’ Cookie Based Session Store. By default, the cookie based session store serializes data from the session using Ruby’s Marshal. While this provides you with the ability to store complex objects in the session, it limits the portability of that data. For example, a Date object serialized by Ruby 1.8.7’s Marshal will throw an exception if you try to deserialize it using Ruby 1.9.3.

To make the session cookie portable between Ruby versions, we monkey patched the code that serializes the session to use JSON instead. Interestingly, the MessageVerifier class in Ruby on Rails 3.2.3 provides support for specifying the serializer; however, ActionDispatch:: Cookies:: SignedCookieJar does not. So we pulled in the MessageVerifier from Rails 3.2.3 into our Rails 3.0 app, and monkey patched ActionDispatch:: Cookies:: SignedCookieJar to use JSON as the serializer. To minimize session resets during the transition period while we rolled this out, before loading the session from the cookie, we try to infer whether it was serialized with Marshal or JSON by reading the first couple characters of it. The code is included in this Gistfor what we call our “Ruby on Rails JSON Cookie Session Store.”

We had to clean up our codebase to make sure that we were only storing objects that could be serialized using JSON. Switching Date objects to be ISO strings was rather trivial. Somewhat surprisingly, the FlashHash, the object that Ruby uses when you call something like `flash[:notice] = “Success!”`, isn’t portable between Ruby versions. To get around this, we use a custom middleware to move the flash messages to a separate cookie, where it is serialized using JSON. This bypasses ActionDispatch::Flash, which looks in the session for a FlashHash. At Airbnb, many of our pages are cached, so we actually use JavaScript to read this separate cookie with the flash and add it to the DOM in the client.

It’s worth noting that we initially wrote this code so that we could share the session between services. Like many Ruby on Rails apps that reach some amount of scale, we’re moving towards a Service Oriented Architecture. When we launched Airbnb’s “Communities” feature, which was built as its own service on Rails 3.2.3 and Ruby 1.9.3, it shared the session with Airbnb’s main monolithic Rails application (a.k.a. monorail) which was running Rails 3.0 and Ruby 1.8.7 at the time. Using JSON to serialize the session will allow us to share the session with services written in other frameworks and languages altogether, like Node.js.

Memcached

Rather than dealing with sharing data between Ruby versions in memcached, we setup a completely separate memcached cluster for the Ruby 1.9.3 servers. In general, this worked out pretty well for us.

One rather obscure issue that created some major headaches for us involved the fact that data serialized using Ruby Marshal apparently takes up more space in Ruby 1.9.3 than in Ruby 1.8.7.  The default maximum object size in memcached is 1MB, and some data that we were serializing in memcached no longer fit when we switched to Ruby 1.9.3.  Code that once cached values suddenly failed silently when we switched to Ruby 1.9.3.

Ruby Syntax Upgrade Guide

We gradually updated our codebase so that it would work in both Ruby 1.8.7 and Ruby 1.9.3.  The following is a guide on how to write code that works in both environments:

Encoding:

As many people have pointed out, encodings will be the biggest pain point when upgrading to Ruby 1.9.3 from 1.8.7.  You’ll have to add the “magic encoding comment” on the top of every file that uses UTF-8 encoded characters.

# encoding: utf-8

Dates:

Ruby 1.8 supports the American style date format, MM/DD/YYYY, so calling Date.parse on the string “10/11/2012″ will return a Date object representing October 11th, 2012.  But in Ruby 1.9.3, American style dates are no longer supported, and Ruby 1.9 appears to parse them in the European format of DD/MM/YYYY:

require 'rubygems'
require 'date'

Date.parse("10/13/2012").to_s
# 1.8.7 > "2012-10-13"
# 1.9.3 > ArgumentError: invalid date

Date.parse("10/11/2012").to_s
# 1.8.7 > "2012-10-11"
# 1.9.3 > "2012-11-10"

We use Jeremy Evans’ American Date gem to keep this functionality consistent between Ruby 1.8.7 and Ruby 1.9.3.

Checking an object’s methods:

Calling .methods on a object in Ruby 1.8.7 returns an array of strings, while in Ruby 1.9.3, an array of symbols is returned.  Instead of doing something like this:

String.methods.include?("freeze")
# 1.8.7 > true
# 1.9.3 > false

Do this:

String.respond_to?(:freeze)
# 1.8.7 > true
# 1.9.3 > true

Regular Expressions:

There is a very subtle difference in how Ruby 1.8.7 and Ruby 1.9.3 handle regular expressions with UTF-8 encoded strings.  In the example below, we attempt to write a regular expression that can isolate the name part from the greeting of a message “Hello Chloë,”:

# encoding: UTF-8

puts "1: " + "Hello Chloë,".gsub(/[\w]*,/u, '')
puts "2: " + "Hello Chloë,".gsub(/[[:alnum:]]*,/u, '')
puts "3: " + "Hello Chloë,".gsub(/[\w[:alnum:]]*,/u, '')

# ruby 1.8.7 output:
# 1: Hello 
# 2: Hello Chloë
# 3: Hello 

# ruby 1.9.3 output:
# 1: Hello Chloë
# 2: Hello 
# 3: Hello 

As you can see, the third approach is the only version that works consistently between Ruby 1.8.7 and Ruby 1.9.3.

The meaning of the POSIX character class [:punct:] is subtly different between Ruby 1.8.7 and Ruby 1.9.3.  In the following example, we attempt to replace all of the punctuation characters with a Unicode snowman:

# encoding: UTF-8

PUNCTUATION_CHARS = '!"#$%&\'()*+,-./:;<=>?@\[\]^_\`{|}~'
puts PUNCTUATION_CHARS.gsub(/[[:punct:]]/, "☃")

# 1.8.7 > ☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃☃
# 1.9.3 > ☃☃☃$☃☃☃☃☃☃+☃☃☃☃☃☃<=>☃☃☃☃☃☃^☃☃`☃|☃~

As you can see, in Ruby 1.9.3, the characters $+<=>^|~` don’t get replaced.  To get around this, we defined a constant with all the punctuation characters and match against that.

I recommend reading the “Regular Expressions” chapter in the Pickaxe book, which is offered as a sample PDF here: pragprog.com/book/ruby3/programming-ruby-1-9

Strings:

The String class no longer supports the #each method.  In Ruby 1.8.7, this method would allow you to iterate on each line of a string.  This (odd) functionality was dropped in Ruby 1.9.3.

Hashes:

Hash#select returns an array of arrays in Ruby 1.8.7, but a proper Hash in Ruby 1.9.3.  You can write code that is compatible with both 1.8.7 and 1.9.3 by wrapping the call in Hash[] like so:

h = {1 => "a", 2 => "b", 3 => "c"}

puts h.select{|x,y| true}.inspect
puts Hash[h.select{|x,y| true}].inspect

# 1.8.7 output:
# [[1, "a"], [2, "b"], [3, "c"]]
# {1=>"a", 2=>"b", 3=>"c"}

# 1.9.3 output:
# {1=>"a", 2=>"b", 3=>"c"}
# {1=>"a", 2=>"b", 3=>"c"}

Ruby 1.8.7 had syntax for constructing a simple hash, but that has been removed in Ruby 1.9.3:

h = {"foo", "bar"}
puts h.inspect

# 1.8.7 > {"foo"=>"bar"}
# 1.9.3 > syntax error, unexpected ',', expecting tASSOC

Case Statements:

Colons are no longer valid after “when” in a case statement.  We prefer to use “then” or a newline instead.

class Cocktail
  attr_accessor :name
  def initialize(name)
    @name = name
  end

  def liquor
    case self.name
    when "journalist": "gin"
    when "sazerac": "whiskey"
    when "mojito": "rum"
    end
  end
end

cocktail = Cocktail.new("sazerac")
puts cocktail.liquor

# 1.8.7 > whiskey
# 1.9.3 > syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'

Load Paths:

In Ruby 1.9.3, LOAD_PATH no longer includes . because it was deemed a security risk.  You can explicitly add it when requiring files, use absolute paths, or use require_relative.

Member:

In Ruby 1.9.3, Range#member? and Range#include? behave differently for ranges that are defined by begin and end strings. In Ruby 1.9.3, those methods only return true if an exact match is in the range, not just a prefix of the string.

puts ('a'..'z').member?("airbnb")
# 1.8.7 > true
# 1.9.3 > false

CSV:

In order to have consistent behavior between Ruby 1.8.7 and Ruby 1.9.3, we created a class called CSVBridge, and use that instead of CSV or FasterCSV:

require 'csv'
require 'fastercsv'

if CSV.const_defined?(:Reader)
  class CSVBridge < FasterCSV
  end
else
  class CSVBridge < CSV
  end
end

Miscellaneous other changes:

• ‘retry’ is no longer supported in iterators and loops
• Symbol#to_i is no longer supported (pragdave.blogs.pragprog.com/pragdave/2008/05/ruby-symbols-in.html)
• In Ruby 1.8.7, you could find out the current Ruby version with VERSION and RUBY_VERSION.  In Ruby 1.9.3, VERSION is now gone, but RUBY_VERSION is still supported.
• Object#type has been removed, so instead, use Object.class.name

RubyBridge:

We created a nothington class called RubyBridge to encapsulate a bunch of helper methods that we found ourselves using repeatedly to make our code compatible:

# This file is for methods meant to bridge Ruby 1.8 and Ruby 1.9 code

class RubyBridge

  # This is an attempt to fix issues with strings that are SafeBuffers breaking URI.escape and RightAws::AwsUtils.URLencode
  def self.regular_string(s)
    if RUBY_VERSION >= '1.9'
      (s.nil? || s.class.to_s == 'String') ? s : s.to_s.to_str
      # do not check is_a?(String) here since ActiveSupport::SafeBuffer and ActiveSupport::OutputBuffer return true
    else
      s.to_s
    end
  end

  # for reference, see www.zendesk.com/blog/upgrade-the-road-to-1-9
  def self.force_utf8_encoding(str)
    if str.is_a?(String) && str.respond_to?(:force_encoding)
      str = str.dup if str.frozen?

      str.force_encoding(Encoding::UTF_8)

      if !str.valid_encoding?
        #logger.warn("encoding: forcing invalid UTF-8 string; text is #{str}")
        str.encode!(Encoding::UTF_8, Encoding::ISO_8859_1)
      end
    end

    str
  end

  # for reference, see www.zendesk.com/blog/upgrade-the-road-to-1-9
  def self.force_binary_encoding(str)
    if str.is_a?(String) && str.respond_to?(:force_encoding)
      str = str.dup if str.frozen?

      str.force_encoding(Encoding::BINARY)
    end

    str
  end

  # Encodes a string from encoding "from" to encoding "to" in
  # a way that works for both ruby 1.8 and 1.9
  def self.convert_string_encoding(to, from, str)
    if "1.9".respond_to?(:force_encoding)
      str = str.dup if str.frozen?
      str.encode(to, from, :undef => :replace)
    else
      require 'iconv'
      Iconv.conv(to, from, str)
    end
  end

end

Force UTF-8 Params:

The following is a method that we added to application_controller as a before_filter for all actions to ensure that params were encoded with UTF-8:

# See stackoverflow.com/questions/8268778/rails-2-3-9-encoding-of-query-parameters
# See https://rails.lighthouseapp.com/projects/8994/tickets/4807
# See jasoncodes.com/posts/ruby19-rails2-encodings (thanks for the following code, Jason!)
def force_utf8_params
  traverse = lambda do |object, block|
    if object.kind_of?(Hash)
      object.each_value { |o| traverse.call(o, block) }
    elsif object.kind_of?(Array)
      object.each { |o| traverse.call(o, block) }
    else
      block.call(object)
    end
    object
  end
  force_encoding = lambda do |o|
    RubyBridge.force_utf8_encoding(o)
  end
  traverse.call(params, force_encoding)
end

More Monkey Patches:

Some additional monkey patches related to handling data serialization in ActiveRecord and Thrift are included in this gist.

Deployment

In line with experience of others, the bulk of the problems that we encountered with upgrading to Ruby 1.9.3 involved encodings.  Once we got all of our specs passing, we needed to test the app with production traffic to uncover the more insidious encoding problems.  We configured our build server so that we could maintain builds for both Ruby 1.8.7 and Ruby 1.9.3 at the same time.  Rather than making the switch all at once, we deployed the Ruby 1.9.3 build to a handful of instances in our cluster so that they could get production traffic, added them into the load balance and then watched for exceptions.  We’d take the instances out of the load balancer, fix the errors and repeat.

Conclusion

With over 100,000 lines of code in our main Rails app and support for 21 different end-user languages, upgrading Airbnb to Ruby 1.9.3 was a significant undertaking.

Pros

• our test suite runs 2-3 times faster
• we can use the latest gems
• general performance improvements, we now need fewer servers

Cons

• roughly 6 months of development time on and off
• lots of very subtle bugs and syntax changes

Was it worth it?  We were hoping to see the type of performance gains that Zendesk and Harvest reported after they upgraded.  While Zendesk reported 2-3x improvement in response time, we saw only a 20% improvement.

However, in the past couple months, we have been able to tune our application in numerous ways (which we hope to document in a future blog post).  As a result, our performance has improved by a margin more in line with what we had hoped for:

• 
• Published in Code
• December 11, 2012 by Jeff Yip
• 8 Comments

8 Comments

1. Jeff,

   Thanks for sharing your experience. We’re about to make the same transition from REE 1.8.7 to MRI 1.9.3 and posts like yours and other’s you reference are very valuable.

   I just noticed the code snippets you’re pulling from GitHub are not displaying. When I move on your GitHub account, I can see them but they are not successfully embedded in this post. At least in Firefox.

   Best.

   Reply

   1. Ooops… Now it works. Maybe just a glitch with GitHub.

      Reply

2. [...] You might decide to switch from Ruby 1.8 to Ruby 1.9, depending on how quickly you jump up the Rails version ladder.  That will make it a very, very painful transition for you, because the standard library API has breaking changes and the language syntax itself has breaking changes.  There are also non-breaking changes where the behavior of code will just change overnight — e.g. Date.parse(“5/6/2013″) can be May 6th or June 5th depending on what version of Ruby you are running.  And the encoding issues.  Oh God. [...]

   Reply

3. [...] Upgrading From REE 1.8.7 to Ruby 1.9.3 from Airbnb Nerds blog [...]

   Reply

4. I had the same session issue (was trying to make my sessions safer and shared between a Rails app and a Go app). Here is a Rails 4 compatible patch https://gist.github.com/mattetti/7624413
   the gist contains a link to a Rails issue discussing the patch. Check there for updates.

   Reply

5. […] nerds.airbnb.com/upgrading-from-ree-187-to-ruby-193/ […]

   Reply

6. Wow – that is some serious response times improvement. Impressive.

   Cheers,
   Thomas
   TSR

   Reply

7. Awesome write up on your experience upgrading. Thanks for taking the time to be so detailed

   Mark

   Reply