Mixpanel + Ruby on Rails = Wonderful

My name is Patrick McKenzie and I am an analytics junkie. Hiya, Patrick! I have an almost unhealthy interest in how my business' web site (you're on it) is performing on a day to day basis. I say "almost" unhealthy because good analytics leads to good decisionmaking and good decisionmaking makes me money, and money is generally healthy, or at least helps me buy more tofu.

Recently I've added a new analytics service (Mixpanel) to the suite I already use (Google Analytics, CrazyEgg, and Clicky) and I have been so happy with it I had to share. This article explains:

• a bit of background about analytics
• how I integrated my Rails site with Mixpanel (steal my code)
• how I used funnel data to improve the experience of my customers
• (and why that makes me money)

Show/hide Table of Contents

Using Mixpanel with Ruby on Rails

Analytics: The Rationale

• Why Analytics?
• What To Track?
• How To Track?

Mixpanel Implementation

• How Mixpanel is different
• The API
• OSS Code

Changes to Site

• Rationale
• My Funnel
• Initial Funnel Results
• Data Driven Improvements
• Results of Improvements

Why You Should Use Analytics

The fundamental reason you want to use analytics is because it lets you know how your customers interact with your site and that this drives decisionmaking about your site design, future business directions, and the like.

You might say "Wait, why don't I just talk to customers?" You should talk to your customers, but customers:

• have more important things to do than talk to you
• are not experts about your site
• misremember things
• resent being tested on like rats in a twisted science experiment ("Would you hit this bar for cheese more if it was green, or blue?")

• is always there, waiting for your analysis
• is totally expert on use of your site
• is as accurate as your collection method
• has no issue with answering arbitrary, inane questions

Back to the Table of Contents

What To Track

The mad scientist in me wants to say Track it all! while cackling madly, but over the years I have found that having too much data (cough Google Analytics cough) is almost as bad as having none, because it numbs me to the import of the data. If it isn't driving good decisions, it is worthless. If I have to swim through a million and one screens to drill down to the information of interest to me, I don't do it very often, and data which is not read might as well not be collected.

So collect manageable amounts of data and prioritize your collection around key business goals. For example, if you sell stuff, then selling stuff is a core business goal. (If you don't sell stuff, get your Web 2.0 whippersnapper feet off my lawn and don't come back until you sell stuff.) The usability of the core interaction between your users and your web application is a core business goal. Focus your attention on places where change moves the needle, not on irrelevancies like "Does my blog archive page for August 2006 get a higher click through rate to the category page when the text is fuscia or purple?"

Back to the Table of Contents

How To Track

To the maximum extent possible, use pre-existing solutions rather than writing your own analytics code. I've done both, and believe me, the engineering effort to collect, record, and display custom statistics is immense.

For example, I have published some statistics about my business for the last few years, and that is all custom code. That custom code is roughly 10% of my code base. While I don't regret having written it, that 10% could have been spent on new features to impress customers. Only use custom analytics code when you have compelling business requirements for it.

By comparison, integrating pre-existing web analytics packages is easy. Google Analytics and CrazyEgg require copy/pasting perhaps 5 lines into your most general template. Mixpanel took a bit more work, but much less than custom code, and since I already did it for you you don't really have to worry about it.

Use the package optimized for the report data you want to see. For example:

• Google Analytics tracks comprehensive web stats across time and visitor source very well.
• CrazyEgg produces very visual, very actionable reports about interactions with particular pages' UI.
• Mixpanel tracks arbitrary events and funnels very, very well.

Back to the Table of Contents

How Mixpanel is Different

Unlike "copy/paste and you're done" analytics solutions, Mixpanel lets you programmatically define what interests you via their API. This increases setup time versus copy/pasting a line of Javascript, but it gives you very fine control over integration with your site and software, and moves the data filtering from analysis time to design time. Since you'll be looking at reports a heck of a lot more than you'll be identifying new things to look at, that will save you time overall.

Mixpanel tracks events, not page views. (If you're coming from a traditional analytics package like Google Analytics, you're probably used to thinking exclusively in terms of page views. In GA, to track something like a click on a button, you create a virtual page view on a URL like /clicks/my-button, and then track hits on that virtual URL. No, really.) An event can be anything you want: a click on a button, a page view, a customer fulfilling some sort of condition ("This marked his 200th login"), etc.

Events can have properties. Everyone here a programmer? Good: properties are Map<String, String>. That is all you need to know about them. (This introduces a subtlety: you can stuff a number into the properties, but since it is operated on like it is a string, you can't currently e.g. calculate averages with it in their default reports.

[Editor's note: Some years after original publication, Mixpanel got a feature which changes this. See here.]

The fundamental form of analysis Mixpanel supports is, for a certain date range and event:

• how many events occurred
• what properties and what counts of those properties did the events have

• users are only counted at each stage once
• you can segment a funnel based on properties

Back to the Table of Contents

API Access Considerations

To track with Mixpanel, you make HTTP POSTs to particular URLs with parameters set a certain way. If you're interested in the technical details of what variable goes where, go read them. After you understand them, you can understand the following design considerations:

Access the API ascychronously: If you make an HTTP POST while rendering a page for a live user, and wait on the result, you're causing them to wait for something that does not add value to them. Don't do that. In particular, if the Mixpanel API should hiccup, you could potentially be waiting several seconds for it to timeout. Instead, fire and forget API requests.

My implementation uses DelayedJob for this: rather than having a Mongrel process (busy serving users) do the API request, I schedule it as a DelayedJob. A DelayedJob worker will, several seconds after scheduling, see the API request in the queue, fire it off, and if either my code or Mixpanel bugs out he won't care, he'll just retry again later. (An interesting side effect of this: if I throw an exception because my tracking code is bad, that doesn't cause the web page to show the user an error. I wouldn't suggest relying on this, but it is a nice safety net to have if you also push changes to production at 2 AM in the morning.)

Assigning unique IDs: The naive approach to assigning unique IDs to users is to use their IP address or associate a unique ID with their account. However, users frequently access the same site from different IPs (school and home, for example). One IP address may represent hundreds of users -- there is an AOL gateway in Kansas which accesses my site several dozen times a week, and I was honestly very concerned that I had an addicted schoolmarm before figuring that out. Assigning IDs to accounts is a good solution, and trivial in Rails (you get @user.id for free, after all), but if you want to track people across a signup you have to be tricky.

Assigning unique IDs is important in Mixpanel for funnel tracking. If you report two different stages of a funnel with two different unique IDs, Mixpanel will assume they are unrelated and junk the second data point.

Track durable data yourself: Mixpanel's Javascript implementation has a concept called Super Properties, which is an abstraction for a per-user cookie. They're automatically added as properties to all events and funnels for that user. To do this with their API, you have to track the data you want to persist yourself. In Rails, you can stuff it in the session, the cookie, Memcached, the database, wherever. For my particular needs I put most of it in the session, which since I use the CookieStore is really just a convenient way to set a cookie anyhow. However, you have a lot of flexibility to grab different bits from different places.

Back to the Table of Contents

OSS Mixpanel Rails Code

I split the Rails code for Mixpanel into two places. The first, which you can just copy/paste into your own project (remember to replace your API key), encapsulates the Mixpanel API in a model mixpanel.rb. This encapsulation keeps the details of the API away from your code in case they change later, and also means that you can conveniently serialize Mixpanel API requests as DelayedJobs (which is automatic if you have it installed).

This code is available under the MIT license, just like Rails. Go nuts. If you'd rather read it in your IDE, click the little icon in the top-right corner to copy/paste.

Show/hide code for mixpanel.rb

#Encapsulates the Mixpanel API as a Ruby (Rails) model.
#See www.bingocardcreator.com/articles/tracking-with-mixpanel.htm for usage notes.
#This code is copyright Patrick McKenzie, 2009.
#It is released under the MIT license (same as Rails).
#Short version: do whatever you want, just don't sue me.
#Long version: www.opensource.org/licenses/mit-license.php

class Mixpanel
  require 'net/http'

  TOKEN = "Replace me!"  #See your Mixpanel dashboard
  
  attr_accessor :options
  attr_accessor :event

  #This is the URL for the mixpanel API call.  It is frozen when track! is called on this object.
  #The reasons for this are subtle, and mostly related to delayed_job serialization.
  attr_accessor :saved_url

  def self.logger
    RAILS_DEFAULT_LOGGER
  end

  
  def self.log_event(event, user_id, ip, opts = {})
    opts.merge!({:event => event, :id => user_id, :ip => ip})
    logged_event = Mixpanel.new(opts)
    self.logger.info "Mixpanel: #{logged_event.inspect}, #{logged_event.url}"
    logged_event.track!
  end

  def self.log_funnel(funnel_name, step_number, step_name, user_id, ip, opts = {})
    funnel_opts = opts.merge({:funnel => funnel_name, :step => step_number, :goal => step_name})
    self.log_event("mp_funnel", user_id, ip, funnel_opts)
  end

  def initialize(opts = {})
    @options = {}
    @options['ip'] = opts[:ip] unless opts[:ip].nil?
    @options['time'] = Time.now.to_i
    @options['token'] = TOKEN
    @options['distinct_id'] = opts[:id] unless opts[:id].blank?
    @event = opts[:event] unless opts[:event].nil?
    opts.each do |key, value|
      unless [:ip, :id, :event, :step].include? key
        @options[key.to_s] = value.to_s
      end
      if [:step].include? key
        @options[key.to_s] = value.to_i
      end
    end
  end

  def raw_data
    hash = {}
    hash['event'] = @event
    hash['properties'] = @options
    ActiveSupport::JSON.encode(hash)
  end
  
  def serialize_data
    Base64.encode64s(raw_data)
  end

  def url
    ret = "api.mixpanel.com/track/?data=#{serialize_data}"
    if (@options['ip'].nil?)
      ret = "#{ret}&ip=1"
    else
      ret
    end
  end

  def track!()
    @saved_url = "data=#{serialize_data}"
    if (@options['ip'].nil?)
      @saved_url = "#{@saved_url}&ip=1"
    end

    #If you have DelayedJob installed, this will use it, otherwise it fires the request *immediately*.
    #This is a very bad idea for most uses because it will result in *your* site blocking while waiting
    #for the Mixpanel API to return.  Be smart: install DelayedJob.
    if (respond_to? :send_later)
      dj = send_later :access_api
      dj.priority = 10 #This puts the Mixpanel DJs on the lowest priority so that
      dj.save          #they don't block print jobs on BCC if Mixpanel times out.
    else
      access_api
    end
  end

  def access_api
    if (RAILS_ENV == "production")
      res = Net::HTTP.start("api.mixpanel.com", 80) {|http|
        http.read_timeout = 3
        http.request_post("/track/", @saved_url)
      }
    end
  end
end

    

Then, I added a bit of code to my ApplicationController (application.rb) to give me a quick and easy wrapper around the API anywhere. You'll want to write your own code here specific to your own needs for associating IDs with users and automatically initializing properties for them. I'll talk you through it.

Show/hide code in application.rb

  #Fetches the mixpanel ID for this user.
  #@user is set by a before_filter that works for all logged in users.
  #Non-cookied users get a random ID set in their session cookie.
  #Returning users get reassigned the ID we associated their account with.
  #We could put that in the database, but putting it in memcached lets me blow
  #this away with new code releases so that Mixpanel doesn't have to track inconsistent state.
  #
  #Cache evictions won't happen until I have approximately 2 million users.
  #If that happens, I'll pay someone to fix this while I sip iced tea on the beach.
  def fetch_mixpanel_id
    srand  #Explicitly call srand in case you set it to a known value elsewhere (which would be catastrophic).
    user_id = @user ? @user.id : session[:user_id]

    @mixpanel_id = session[:mixpanel_id]
    if (user_id)
      @mixpanel_id = Rails.cache.read("mixpanel_id_for_#{user_id}") || @mixpanel_id
      @mixpanel_id ||= user_id
      Rails.cache.write("mixpanel_id_for_#{user_id}", @mixpanel_id)
    else
      @mixpanel_id ||= rand(10 ** 10).to_i
    end
    session[:mixpanel_id] = @mixpanel_id
    #Only useful if you use A/Bingo Rails A/B testing, see www.bingocardcreator.com/abingo
    if (request.user_agent =~ /\b(HttpClient|Baidu|Gigabot|Googlebot|libwww-perl|lwp-trivial|msnbot|SiteUptime|Slurp|WordPress|ZIBB|ZyBorg)\b/i)
      #This prevents robots from occupying more than 1 participant slot in A/B tests.
      Abingo.identity = "robot"
    else
      Abingo.identity = @mixpanel_id
    end
  end

  #Log a new Mixpanel event
  #event: name of event
  #opts: Properties defined for this event only.
  #perma-opts: Properties to associate for this and all later events/funnels for this user.
  def log_event(event, opts = {}, perma_opts = {})

    if session[:mixpanel_opts]
      perma_opts.merge!(session[:mixpanel_opts])
    end

    #Permanent options are stored in the session, and persist until the user
    #clears their cookie.
    session[:mixpanel_opts] = perma_opts

    if (@user && (@user.is_admin? || @user.is_editor?))
      return #don't pollute data with my actions
    end

    #Tracks trial users vs. guests vs. registered users
    if (@user)
      opts.merge!({:role => @user.role})
      opts.merge!({:source => @user.source})
    end

    participating_ab_tests = Abingo.participating_tests rescue {}

    opts.merge!(participating_ab_tests)
    
    #Fold in permanent options
    opts.merge!(perma_opts)
    unique_id = fetch_mixpanel_id
    Mixpanel.log_event(event, unique_id, request.remote_ip, opts)

    #the following code is for Kissmetrics.
    KM.init  #Normally requires API key, but hacked to avoid it.
    KM.identify(unique_id)
    KM.record(event, opts)

  end

  #Log a new Mixpanel funnel step
  #funnel_name: what it says on the tin
  #step_number: 1, 2... etc.  Must be followed in order.  Mixpanel calls this "step".
  #step_name: String describing what user did to advance to this step.
  #           Mixpanel calls this "goal", for some reason.  Name must always be the
  #           same for a particular funnel and step number.
  #opts:      Properties defined for this event only.
  #perma-opts: Properties to associate for this and all later events/funnels for this user.
  #            Analagous to Mixpanel "Super Properties".
  def log_funnel(funnel_name, step_number, step_name, opts = {}, perma_opts = {})

    if session[:mixpanel_opts]
      perma_opts.merge!(session[:mixpanel_opts])
    end
    #Permanent options are stored in the session, and persist until the user
    #clears their cookie.
    session[:mixpanel_opts] = perma_opts

    if (@user && (@user.is_admin? || @user.is_editor?))
      return #don't pollute data with my actions
    end

    #Tracks trial users vs. guests vs. registered users
    if (@user)
      opts.merge!({:role => @user.role})
      opts.merge!({:source => @user.source})
    end

    #fold in permanent options
    opts.merge!(perma_opts)
    unique_id = fetch_mixpanel_id
    Mixpanel.log_funnel(funnel_name, step_number, step_name, unique_id, request.remote_ip, opts)

    #For Kissmetrics
    #the following code is for Kissmetrics.
    KM.init  #Normally requires API key, but hacked to avoid it.
    KM.identify(unique_id)
    KM.record(step_name, opts)
  end

    

Then, you actually track stuff in controllers:

  #Logged in users come here first.
  def dashboard
    #Snipped business logic.
    #Note I track how many word lists users have
    #across all events/funnels as it makes
    #for a quick proxy on how heavy of a user they are.
    log_event("Dashboard")
    log_funnel("Basic Use", 1, "Dashboard", {}, {:word_list_count => @word_lists.size})
    log_funnel("Purchase", 1, "Dashboard", {}, {:word_list_count => @word_lists.size})
  end
  

Easy peasy! Adding a new event or funnel is a quick one-liner in the appropriate place(s).

Back to the Table of Contents

How Mixpanel Improved Bingo Card Creator

Alright, geek hats off, businessman hats on: how does that work make me more money? Well, I'll give you an example from my business: I recently released an online version of the application I sell.

<tangent>I am thrilled about selling online applications versus selling desktop software, for a few reasons. One is that you get opportunities to look at aggregate data from your users and everyone is pretty much OK with this, whereas phoning home from downloadable software is generally frowned on. It occurs to me, though, that I could just as easily have plugged Mixpanel into my downloadable application... I'll have to think that one over. </tangent>

Previous experience with customers over the last three years suggests that nothing sells my software as well as printed bingo cards actually in their hand. So I'm going to try to optimize the percentage of trial users who successfully complete that experience. If 1% more trial users get through this, I will get approximately 1% better sales, so there is clear motivation to get as many through this as possible.

Back to the Table of Contents

Define the Funnel

1. Sign up (or sign in).
2. Create a word list (or open an existing one).
3. Add at least 24 words and.
4. Save the word list.
5. Customize the bingo cards.
6. Tell the server to whip you up a batch.
7. Download the resulting PDF file.
8. Print it. (Happens within Adobe Reader so I can't see it.)

Back to the Table of Contents

Initial Funnel Results

I really like Mixpanel's presentation of funnel results, because it makes the multiplicative nature of funnel performance very explicit. However, I don't have their CSS skills so unless you want to read it in the screenshots ( one, two) you get to see a text version:

• Users signing in: 260
• ... who created or opened a word list: 218 (83%)
• ... and saved it at least once: 191 (87%)
• ... and proceeded to the customization page: 161 (84%)
• ... and customized and created cards: 131 (81%)
• ... and actually downloaded those cards: 126 (95%)

Back to the Table of Contents

Data Driven Improvements

This lets me prioritize on improvements. Clearly, optimizing the download step is a waste of my time: almost everyone already succeeds at it. The saving and proceeding customization steps are sort of difficult by nature -- they actually require the user to type significant amounts of text. I have ideas for improving that, but they are not the low hanging fruit.

The lowest hanging fruit is the very first step. I lose 20% of my trial accounts before they even open a word list. That is one freaking click into the funnel! So one thing for me to test this week is tweaking the dashboard and the introductory text on it to make it more obvious how to proceed -- probably with big, colorful buttons. Another option is to take people directly to the next step of the funnel on signup, and let them see the dashboard for the first time later. I may test that as well.

The other low hanging fruit is at customization. This asks the customer to look at a fairly involved form, but all they really HAVE to do is click a button. I should probably hide some of the complexity from the user, and make that button a graphic with an arrow on it rather than the current HTML form submit button. For the moment I've improved it via hiding the complexity for some users. (I use a quick heuristic to decide who is likely to be a power user. They get to see the whole form by default. Everyone else gets it hidden by default.) I'll have to get in touch with my design guy to get a decent button done up, but that will do for now.

You might be wondering "Hey, shouldn't this be A/B tested?", and if you're wondering that you are absolutely correct. However, my Rails A/B test framework isn't quite ready for pushing live yet. That will be my next little mini-project.

Back to the Table of Contents

Results of Improvements

I want to reiterate that this experiment is not optimal, because I did it before my A/B testing framework (A/Bingo) was ready. That out of the way: the experiment was a success. You can see the new funnel in screenshots: one, two) but the highlights are:

Completion on Create List went from 87% to 97%, probably due to improvements in my welcome page text. (Rather than letting people figuring out what to do next, I lead them by the freaking nose. For example, if they came from an ad for Baby Shower bingo cards, I make it very obvious how to get them.)

Completion of the Customization screen went from 81% to 88%, probably due to the simplification I described above.

These two changes taken together mean total funnel completion was lifted by 16.9%, from 48.5% to 56.7%. Not bad for an hour of work, huh? I will say it again: because these tests were run sequentially rather than A/B tested, I can't be positive that I'm not seeing the results of a different customer mix rather than the results of the tested experiments only. Additionally, smart readers might wonder "Is that improvement statistically significant?" If you consider the whole funnel as the experiment, then yes, the improvement is statistically significant at the 90% confidence level.

(If you aren't comfortable running your own Z-tests I recommend the SEOBook A/B test calculator for that sort of math. Full disclosure: I moderate in their forums.)

I'd love to hear what you think of this article. Either shoot me an email at patrick@bingocardcreator.com, post a comment on my blog, or post something about this and I'll find you from my referrer logs.

Regards,
Patrick McKenzie

P.S. Feel free to pass this article or code around to people you think will benefit from it.

Quick Links

• My Blog
• Bingo Card Creator (my business)
• Mixpanel API in Ruby

Analytics: The Rationale

• Why Analytics?
• What To Track?
• How To Track?

Mixpanel Implementation

• How Mixpanel is different
• The API
• OSS Code

Changes to Site

• Rationale
• My Funnel
• Initial Funnel Results
• Data Driven Improvements
• Results of Improvements