I’m working my way through an epic Rails 3.1 upgrade and some of my cucumber features were failing because I was using a custom RSpec matcher and the method wasn’t found.

My custom matcher looks something like this:

module CustomMatchers

  class XmlSubsetMatcher
      :
  end

  def be_xml_subset_of(expected)
    XmlSubsetMatcher.new(expected)
  end

and when I ran my feature I was getting this failure:

undefined method `xml_subset_of?' for # (NoMethodError)


As it turns out, in my zeal to make sure everything was using the latest and great new stuff, I had forgotten to move over this critical configuration line in cucumbers env.rb:


World(CustomMatchers)


Now, my cucumber feature is happily failing cuz my code doesn’t work. Whew. I couldn’t find this documented anywhere and I’m not even sure where this documentation would belong. I found a hint on the cucumber wiki rspec expectations page, but none of the code on that page is actually needed when using cucumber with Rails, so I decided not to touch it and just write this blog post.

I found that I needed to convert an m4a audio file (which is what QuickTime saves when I record audio) to a wav file, so I decided to use my favorite “can opener.” The versatile open source ffmpeg tool has always seemed to be able to convert anything to anything in audio-video formats.

I decided to pull the source from git:

$ git clone git://source.ffmpeg.org/ffmpeg.git
$ cd ffmpeg/


Stable versions are tagged (which I could see with “git tag -l”). I don’t need to live on the edge right now, so I switched to the tag “n0.9.1″ which I assume is for the latest stable build “harmony” 0.9.1 and made a local branch based on that.

$ git co n0.9.1
$ git checkout -b n0.9.1


Instructions for building ffmpeg are in the “INSTALL” file. I discovered I needed yasm, which I could install with brew. Here’s what I did:

$ brew install yasm
$ ./configure
$ make
CC libavdevice/alldevices.o
CC libavdevice/avdevice.o
CC libavdevice/lavfi.o
AR libavdevice/libavdevice.a
CC libavfilter/af_aconvert.o
libavfilter/af_aconvert.c:53: warning: function declaration isn’t a prototype
libavfilter/af_aconvert.c:105: warning: function declaration isn’t a prototype
CC libavfilter/af_aformat.o
CC libavfilter/af_anull.o
CC libavfilter/af_aresample.o
:
:
ffserver.c: In function ‘parse_ffconfig’:
ffserver.c:4236: warning: ‘avcodec_get_context_defaults2’ is deprecated (declared at ./libavcodec/avcodec.h:3948)
ffserver.c:4237: warning: ‘avcodec_get_context_defaults2’ is deprecated (declared at ./libavcodec/avcodec.h:3948)
LD ffserver_g
CP ffserver
STRIP ffserver

I saw a lot of warnings, but they didn’t seem to negatively affect what I was trying to do. I found a nice blog post from catswhocode to remind me of the usage, and was able to use this simple command:


$ ./ffmpeg -i frog.m4a frog.wav
ffmpeg version 0.9.1, Copyright (c) 2000-2012 the FFmpeg developers
built on Jan 7 2012 21:19:08 with llvm_gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
configuration:
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 42. 4 / 53. 42. 4
libavformat 53. 24. 2 / 53. 24. 2
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'frog.m4a':
Metadata:
major_brand : M4A
minor_version : 0
compatible_brands: M4V M4A mp42isom
creation_time : 2012-01-08 05:09:05
Duration: 00:00:07.22, start: 0.000000, bitrate: 206 kb/s
Stream #0:0(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, s16, 201 kb/s
Metadata:
creation_time : 2012-01-08 05:09:05
handler_name :
Output #0, wav, to 'frog.wav':
Metadata:
major_brand : M4A
minor_version : 0
compatible_brands: M4V M4A mp42isom
creation_time : 2012-01-08 05:09:05
encoder : Lavf53.24.2
Stream #0:0(und): Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Metadata:
creation_time : 2012-01-08 05:09:05
handler_name :
Stream mapping:
Stream #0:0 -> #0:0 (aac -> pcm_s16le)
Press [q] to stop, [?] for help
size= 1244kB time=00:00:07.22 bitrate=1411.3kbits/s
video:0kB audio:1244kB global headers:0kB muxing overhead 0.003611%

$ ls
frog.m4a frog.wav


Success!!

I really want to upgrade a Rails 3.0 project to Rails 3.1, but I’ve done a few spikes and it lacks test coverage, so I decided to pull in cucumber and write some features before moving forward.

I added cucumber-rails to my gemfile, and ran “bundle” and got this error:

/Users/sarah/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:289:in `load': uninitialized constant Psych::Syck (NameError)

What I really needed was to update my Ruby Gems (bundle update –system) but before I discovered that I did “bundle update” which moved me forward to rake 0.9.2, so I started getting these warnings:

/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:93: warning: already initialized constant VERSION
/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:96: warning: already initialized constant LIBYAML_VERSION
WARNING: Global access to Rake DSL methods is deprecated.  Please include
...  Rake::DSL into classes and modules which use the Rake DSL methods.
WARNING: DSL method Bakery::Application#task called at /Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/railties-3.0.0/lib/rails/application.rb:214:in `initialize_tasks'

So, I’ve learned from google, stackoverflow, various blogs and my twitter friend @excid3 that I need to update my Rakefile to include:

require 'rake/dsl_definition'
require 'rake'
include Rake::DSL

That lets me use rake (yay!). I still have the following two warnings:

/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:93: warning: already initialized constant VERSION
/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:96: warning: already initialized constant LIBYAML_VERSION

which I’m hoping will go away with the Rails 3.1 upgrade, but I thought I would write up the rest of it in case it helps other wayward souls on their journey.

$ rake spec
(in /Users/sarah/src/../my_app)
You have 1 pending migrations:
20110416135407 CreateCourses

The rake spec command reminds us that we need to run our migration before running
our tests. In fact, it does a whole lot more than that. There are a whole bunch of best practices rolled in that one
command. To see exactly what is going on, we can run rake spec with the –trace
option:


$ rake spec --trace
(in /Users/sarah/src/tfr/svn/Book/code/class_app_new_source)
** Invoke spec (first_time)
** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:abort_if_pending_migrations
** Execute db:test:prepare
** Invoke db:test:load (first_time)
** Invoke db:test:purge (first_time)
** Invoke environment
** Execute db:test:purge
** Execute db:test:load
** Invoke db:schema:load (first_time)
** Invoke environment
** Execute db:schema:load
** Execute spec


When it says invoke it is calling a particular rake task, but then it will call its dependencies. To really see what is happening in what order, check out the execute commands. The commands db:test:prepare and db:test:load don’t do much themselves, aside from setting up the environment and executing another task or two. We can see from the output that rake is actually executing the following steps:

1. Don’t run the specs if there are pending migrations in the development database. (db:abort_if_pending_migrations)

2. Drop the test database (db:test:purge)

3. Load the schema into the test database (db:schema:load in environment “test”)

These steps make sure that we are always testing in a clean environment, so we know exactly what we’re testing when we run our specs.

The code that makes this happen in Rails 3, can now be found in railties. (Thanks to @pixeltrix for pointing me to it.)/62206174505873408

REPL (Read-Eval-Print-Loop) is a great way to learn. With Ruby, the experience is enabled with irb. Sometimes, to do this we need to peek into the innards of things, which I find to be an extremely effective way to explain mocks and stubs. It’s a regular part of my Ruby curriculum, even though I have needed to figure out the syntax three times in the last couple of years. (Many thanks to Jen-Mei Wu for the most recent iteration.) I still think it is worth it, even though it seems to change with crazy frequency.

Just in case anyone else ever wants to do this with current or previous versions of RSpec, I thought I would write it down before old versions become lost in the mists of time:

RSpec 2.5

>> require 'rspec/mocks'
>> include RSpec::Mocks::Methods
>>Time.stub(:now).and_return(10,20)
>>Time.now
10
>>Time.now
20
>>Time.now
20

RSpec 2.0

>> require 'rspec/mocks'
>> require 'rspec/mocks/extensions/object'
>>Time.stub(:now).and_return(10,20)
>>Time.now
10
>>Time.now
20
>>Time.now
20

RSpec 1.3

>> require 'spec'
>> require 'spec/mocks'
>>Time.stub(:now).and_return(10,20)
>>Time.now
10
>>Time.now
20
>>Time.now
20

Not sure if this is a bug or a feature. I’d guess it is here for a reason, and maybe I’m late for noticing, but Rails 3 errors now provides an array for each attribute, whereas in Rails 2.3 it was just a string.  Here’s the output from two almost identical applications…

Loading development environment (Rails 2.3.8)
>> person = Person.new
=> #<Person id: nil, first_name: nil, last_name: nil, present: nil…
>> person.valid?
=> false
>> person.errors
=> #<ActiveRecord::Errors:0×1034d8f10 @errors=#<OrderedHash …
>> person.errors[:first_name]
=> “can’t be blank”

Loading development environment (Rails 3.0.0)
>> person = Person.new
=> #<Person id: nil, first_name: nil, last_name: nil, present: nil…
>> person.valid?
=> false
>> person.errors
=> {:first_name=>["can't be blank"]}
>> person.errors.class
=> ActiveModel::Errors
>> person.errors[:first_name]
=> ["can't be blank"]

I didn’t see that in the release notes, but it failed my tests for ActiveRecord class. Someone else must have a list of these details, yes?

I’m reviewing the security of a web app built with Ruby on Rails, so I put together a checklist for a security audit. This isn’t a bank or high security situation, but there were a number of engineers and quite a bit of open source code, so I thought a few checks were in order.

Here’s the list I came up with that I thought other folks might appreciate as a starting point (special thanks to the sfruby list, Mike Gunderloy, and Scott Bronson for feedback):

0) Make sure your Rails and gems are up to date for latest security patches (see rails security mailing list for recent advisory notes)

1) Active Record audit:
  A) SQL injection:
    (i) whole word search for “find”, “first”, and “all” then visually inspect all instances of ActiveRecord find calls for potential SQL injection vulnerability (also search for “sql” not whole work search to find find_by_sql and “execute” to find cases where raw sql is executed.
    (ii) search your models for “named_scope” and check :conditions
  B) check for mass assignment Either disable mass assignment as Eric suggests in his article, or audit its use. If doing an audit, check every model to make sure it declares which attributes are settable with attr_accessible. (While attr_protected may technically work, a white list approach is recommended by security experts and the rails security advisory on this topic)

2) Scripting attack: search all eRB files for <%= and ensure that if dynamically generated text was originally entered by the user, it is HTML escaped. Consider rails_xss

3) Secure Access: If some of the site does not have public access, check controllers and ensure that public actions are specifically allowed and that protected access is the default

4) search for “eval” (whole word) and verify that usages are safe (assume javascript eval is ok)

5) search for “forgery” (not whole word), make sure that
config.action_controller.allow_forgery_protection = false
is only disabled in test config
protect_from_forgery should be in the ApplicationController, unless there is a good reason for it not to be

6) check user auth and review that controller actions are limited to expected use

7) passwords: not saved as clear-text in the db, not logged

8) check that private data is not stored in cookies

So, I needed to change markdown to textile and google didn’t yield any handy scripts, so I sharpened my vim fu with Rubular, my favorite regular expression tester and came up with a few substitutions that took care of everything but lists and code blocks.

In vi, type ESC to go into command mode, then :%s/one/two/g will find every instance of “one” and replace it with “two”

First the easy stuff, headers. ^ finds the beginning of the line.

:%s/^# /h1. /g
:%s/^## /h2. /g
:%s/^### /h3. /g

To replace images, I needed to replace  with !link! so I needed to capture text. I suppose I didn’t really need the first capture, but I was working on the replace expression for a regular link when I realized it would be easier to do the images first. To understand the expression below, you need to know that \(stuff\) captures some text which can be inserted in the replacement text with \1 and \2, etc. So to get everything between square brackets, I use [\(.*\)]

:%s/!\[\(.*\)](\(.*\))/!\2!/g

All of my images appeared on a single line, so I didn’t catch a potential issue in the above expression until I got to replacing text links. I needed to use a non “greedy” capture so that I wouldn’t pull in text after the link that happened to include a parenthetical comment. Normally, in reg ex I would use (.*?) but in vim I needed to write \(.\{-}\) …wtf?

:%s/\[\(.*\)](\(.\{-}\))/"\1":\2/g

Special thanks to Adam Wolf’s tip via ShareGrove which helped me document these steps.

you can put VIM in a mode where the command history is just like another buffer. Not in insert mode, try q:

You should get a new buffer that you can edit with the command history in it, so “*yy would yank the current line into the system clipboard, etc.

There’s a nice Railscast introduction to rake for Rails, which goes into a number of other important details that aren’t covered in this post. Below is a little tutorial of creating a Rails rake task and getting it to run remotely on heroku.

Introduction to Rake

In lib/tasks, create a file called greet.rake

task :greet do
   puts "Hello world"
end

By naming the task .rake and putting it in this special place rails will automatically pick it up and make it available to you. You can see it listed if you type: rake -T on the command line. To run it:

rake greet

which will print “Hello world”

to run one task before another, specify a dependency like this (multiple tasks may be specified in the same file):

task :ask => :greet do
   puts "How are you?"
end

Writing a Practical Rake Task

Now for the task at hand, I’m going to create a rake task which creates a bunch of fake data for me to test with. First I’ll create a little experimental app:

rails rake_example
cd rake example
script/generate scaffold person first_name:string last_name:string
rake db:migrate

Here’s the rake task (lib/tasks/fake_people.rake):

require 'faker'

namespace :admin  do
  desc "create some fake data"
  task :fake_people => :environment do
    print "How many fake people do you want?"
    num_people = $stdin.gets.to_i
    num_people.times do
      Person.create(:first_name => Faker::Name.first_name,
                    :last_name => Faker::Name.last_name)
    end
    print "#{num_people} created.\n"
  end
end

Note that I’m using the faker gem (docs here) and I created a task dependency on loading the rails environment so I could access my Person model.

Now I can run

rake admin:fake_people

and it will prompt me to ask how many I want and then it will create them. Cool goodness, yes?

Running Remotely on Heroku

We’re not done yet. I want to deploy this on heroku and be able to run the task remotely. For this, there are two gotchas, first I can’t run an interactive script remotely; also I need to tell heroku that I am using the fake gem and make sure it is installed.

1) removing interactivity

Instead of an interactive script, we can set an environment variable or command line argument (thanks to a tip by Adam Wiggins).

My modified task looks like this:

require 'faker'

namespace :admin  do
  desc "create some fake data"
  task :fake_people => :environment do
    num_people = ENV['NUM_RECORDS'].to_i
    num_people.times do
      Person.create(:first_name => Faker::Name.first_name,
                    :last_name => Faker::Name.last_name)
    end
    print "#{num_people} created.\n"
  end
end

which I can call locally from the command line like this:

rake admin:fake_people NUM_RECORDS=1

2) adding gem to heroku

I need to create a gems manifest, which sounds fancy, but is simply creating a .gems file at the root of my app with contents similar to what I would put in my config environment.rb to specify that my app requires a gem:

faker --version ">=0.3.1"

3) Deploy and Run

So I can deploy my app to heroku with the usual steps

git init
git add .
git commit -m "example app for rake script testing"
heroku create
git push heroku master
heroku rake db:migrate

and run the task remotely:

heroku rake admin:fake_people NUM_RECORDS=1

We ran into an issue last week where our XML APIs were returning HTML under certain error conditions, rather than the expected XML. Our solution was to add the following code to the ApplicationController:

  rescue_from Exception do |exception|
    respond_to do |format|
      format.xml  { render :xml =>
           "<error>Internal Server Error #{exception.message}</error>",
           :status => 500 }
      format.html { render :html => {:file => 'public/500.html'}, :status => 500 }
      format.json { render :json =>
            {:error => "Internal Server Error #{exception.message}"}.to_json,
             :status => 500 }
    end
  end

We might have also declared a rescue_action, and I’m not sure of the benefits of one over the other, except that perhaps we needed to implement a general form of rescue_from since we had another more specific form already declared.

It seemed to me that this should be the default behavior in rails, so I decided to dig into it a little more and see what I could discover. I started by making a little test app to reproduce the exception. The particular case from last week was a database limit that wasn’t being caught in the app with a length validation. When I tried to re-create the error in MySql, I noticed that no exception is thrown since MySql will just truncate the data (although perhaps that is only because I am not running MySql in strict mode). In PostgreSQL, the database layer will throw an exception.

Test app setup:

rails -d postgresql test_postgresql
cd test_postgresql/
script/generate scaffold person first:string last:string present:boolean

Edit the migration to create a database limit:

class CreatePeople < ActiveRecord::Migration
  def self.up
    create_table :people do |t|
      t.string :first, :limit => 40
      t.string :last, :limit => 40
      t.boolean :present

      t.timestamps
    end
  end

  def self.down
    drop_table :people
  end
end

Create the postgres user. Note double-quotes around user, single quotes around password. It has to be that way. Go figure.

$ sudo su postgres -c psql
postgres=# create user "test_postgresql" with superuser password 'password';
CREATE ROLE
postgres=# \q

Finally create the database, run migration, and start the server:

rake db:create:all
rake db:migrate
./script/server

If you point your browser at localhost:3000/people and try to create a person with more that 40 characters in the first name, you will see the following error:

ActiveRecord::StatementInvalid in PeopleController#create
PGError: ERROR:  value too long for type character varying(40)

That is all well and good; however, if you do the same in XML, you will get the same error in HTML.

$ curl -X POST -d "<person><first>This is a first name that is too long for the database limit</first></person>" -H "Content-Type: application/xml" localhost:3000/people.xml
<html xmlns="www.w3.org/1999/xhtml">
<head>
  <title>Action Controller: Exception caught</title>
  <style>
    body { background-color: #fff; color: #333; }

    body, p, ol, ul, td {
      font-family: verdana, arial, helvetica, sans-serif;
      px;
      line-px;
    }

That seems like a bug to me. Perhaps this should be a lighthouse ticket rather than a blog post.. still not confident in identifying bugs in Rails, so I figured I’d post here first.