writing tests

Testling tests are just javascript. When you upload a test to testling, it gets browserified, fully-instrumented to support stack traces, and then run inside all the browsers requested.

an example test

var test = require('testling');
test('the test name', function (t) {
    t.equal(2 + 2, 4);
    
    t.log(window.navigator.appName);
    
    t.createWindow('substack.net', function (win, $) {
        t.equal(win.document.title, 'The Universe of Discord');
        t.end();
    });
});

This example test defines a new test with the name "the test name". The test asserts that 2 + 2 = 4, logs the window.navigator.appName, then navigates to substack.net with t.createWindow(). When substack.net loads, the test checks that the title is what is expected. Finally the test ends when t.end() fires.

In IE9, our test prints:

robot : ~ $ curl -sSNT test.js \
  testling.com/?browsers=iexplore/9.0
Bundling....  done

iexplore/9.0        2/2  100 % ok
  Log: Microsoft Internet Explorer

total               2/2  100 % ok
robot : ~ $

Check out the running tests section for more info on running these tests.

test methods

The testling API is heavily inspired by node's assert module and node-tap.

To load the test harness, use require() to load the testling module:

var test = require('testling')

Then just call test() with a name and test callback.

test(testName, cb)

Create a new test with an optional name and test callback.

test('the test name', function (t) { /* ... */ })

The test callback will get a new test object with the methods listed below. When your test is finished, make sure to call t.end().

test.browser

Get the browser name and version being used. This object looks like:

{"name":"firefox","version":"10.0"}

test control

t.plan(n)

Plan to run n tests. If more or less tests are run, an error is generated.

t.end()

Finish a test. No more assertions should fire once this method has been called.

t.createWindow(opts, cb)

Load the web page at opts.url. If opts.url starts with a file://, . or a /, load an HTML file from the multi-file upload.

Once the window object has loaded, cb(win, $) fires with the window object win and a jquery object, $, which is bound to the window object.

Returns the new window object, which may not be fully loaded yet.

If opts.url is a string, treat it as opts.url.

You can also set opts.method, opts.headers, and opts.data in order to make POST and other fancy requests.

Once the window has loaded you can submit forms and click links for the most part. HTTPS doesn't work yet.

To get XMLHttpRequests (AJAX) to work, try using absolute URLs when making requests. Instead of xhr.open("POST", "/", true), try xhr.open("POST", window.location.href, true).

t.submitForm(form, opts={}, cb)

Submit the HTML form object provided.

This method is deprecated. Now you can just form.submit()!

opts and cb(win, $) work mostly the same as in t.createWindow().

However if the form.method is "POST", opts.data is specified, and opts.headers['transfer-encoding'] isn't "chunked", then opts.headers['content-length'] with be automatically sent with the correct length for the form data.

test assertions

t.ok(value, [message])

Assert that value has a truthy (non-falsy) value.

t.notOk(value, [message])

Assert that value has a falsy value.

t.fail([message])

Generate an error.

t.equal(x, y, [message])

Assert that x shallowly equals y with coercive equality using the == operator.

t.notEqual(x, y, [message])

Assert that x does not shallowly equal y with coercive equality using the != operator.

t.deepEqual(x, y, [message])

Assert that x deeply equals y using a recursive traversal of both arguments.

t.notDeepEqual(x, y, [message])

Assert that x doesn't not deeply equal y using a recursive traversal of both arguments.

t.strictEqual(x, y, [message])

Assert that x strictly equals y using the === operator.

t.notStrictEqual(x, y, [message])

Assert that x does not strictly equal y using the !== operator.

t.throws(cb, [error], [message])

Assert that the callback cb() will raise an exception when run. If error is specified, the exception should match error. error can be a constructor, regexp, or validation function.

t.doesNotThrow(cb, [error], [message])

Assert that the callback cb() will not raise an exception when run.

require()

The test harness is made available via require() but you can also load other files. Require is provided by browserify, which emulates the require() in node.

relative requires

If your test upload spans multiple files, you'll be able to require() those other files with a relative path starting in a "./". Here's an example of a multi-file upload with relative requires:

package requires

Since testling uses browserify to bundle test uploads, you can install modules from npm into the node_modules/ directory. Modules in node_modules/ can then be loaded by just doing require('modulename').

See the using npm section for an example.

require.load()

If you have code that isn't written using node-style module.exports, you can use require.load() instead.

For instance, if you have a file foo.js that defines a function foo():

function foo (x) {
    return x + 1;
}

then in your test.js test file you can do:

var test = require('testling');
require.load('./foo.js');

test('foo', function (t) {
    t.deepEqual(foo(5), 6);
    t.end();
});

and after the require.load('./foo.js'), the function foo() will be defined in the global scope.

To run this example, just do:

robot : ~ $ tar -cf- foo.js test.js |
  curl -sSNT- localhost:8080/?browsers=chrome/14.0
Bundling...  done

chrome/14.0         1/1  100 % ok

total               1/1  100 % ok
robot : ~ $

See also the section on running multiple test files.

running tests

To run a test with testling, just send an HTTP PUT! The file that you PUT to testling.com can either be a single javascript file or a tar file with as many javascript files as you like.

authentication

Testling just uses basic auth over HTTP. Just use your browserling account email and password. Don't have an account? Create one!

Once you have an account, with curl you can just do:

curl -u me@example.com:mypassword -sSNT test.js testling.com

Or if you want to be prompted for your password you can do:

curl -u me@example.com -sSNT test.js testling.com

running a single test file

Running a single test file is super simple! Just send an HTTP PUT request to browserling with the contents of your test. Here's what that looks like using curl:

robot : ~ $ curl -sSNT test.js \
  testling.com/?browsers=iexplore/6.0,iexplore/7.0,safarai/5.0
Bundling....  done

iexplore/6.0        1/1  100 % ok
iexplore/7.0        1/1  100 % ok
safari/5.0          1/1  100 % ok

total               3/3  100 % ok
robot : ~ $

running multiple test files

You can upload a tar file containing multiple files to run. The tar file must either contain a test.js or the main parameter must be specified so testling knows where to start executing.

Any other files should be executed from the main file using require() or require.load(). Your tar file may also contain HTML files that you can load with t.createWindow().

Suppose we've got a test.js:

var test = require('testling');
var b = require('./b');

test('multifile', function (t) {
    t.equal(b({ a : 3, b : 4 }), 2);
    t.end();
});

and the other file it uses, b.js, looks like:

module.exports = function (n) {
    return Object.keys(n).length
};

You can just make tar use stdout for output so that curl can read from a pipe and forward along both tests to testling:

robot : ~ $ tar -cf- test.js b.js | curl -sSNT- \
  testling.com/?browsers=iexplore/9.0,chrome/13.0
Bundling....  done

iexplore/9.0        1/1  100 % ok
chrome/13.0         1/1  100 % ok

total               2/2  100 % ok
robot : ~ $

stack traces

When an error occurs in a test, you'll get a full stack trace no matter which browser is being used. We make this work by rewriting the AST at runtime using burrito and stackedy.

query parameters

When you run a test, you can specify parameters in the query string of the HTTP PUT request.

For example with curl you can do:

curl -sSNT test.js \
  'testling.com/?browsers=iexplore/7.0,firefox/3.5&output=json'

to run your test.js in internet explorer 7 and firefox 3.5 with json output.

browsers parameter

This query parameter controls which browsers your test will run under. Use a comma separator to specify multiple browsers.

Click here to see which browsers we're running or use curl:

curl -s testling.com/browsers.json

output parameter

Specify the format for test output. The default output is text.

output=text

Display output with human-readable lines of text as they become available. Differences in deepEqual comparisons are rendered using difflet.

output=json

Display output as streaming json. That is, when the output is complete, the result will be valid json, but the elements in the json will be populated in realtime as the results come back from the browsers.

Specifically, testling uses JSONStream to write streaming json output. You can use JSONStream's .parse() to read the output as new elements arrive.

output=tap

Display output with the Test Anything Protocol using node-tap. Like node-tap, differences in deepEqual comparisons are rendered using difflet!

output=log

Sometimes it's useful to just collect information from all the browsers. In output=log mode, a streaming json object is rendered populating each browser with json data from all the log messages. All error and assertion data is hidden.

Example output:

  {"iexplore/9.0":[
  "a"
  ,
  "b"
  ,
  "c"
  ]
  ,"chrome/15.0":[
  "a"
  ,
  "b"
  ,
  "c"
  ]
  }

script

For cases when you just want to inject some <script> tags into the page before everything else, you can use the script query parameter. You can specify script=... as many times as you like and each script will be included into the page.

curl -sSNT test.js \
  'testling.com/?script=code.jquery.com/jquery-1.7.1.min.js'

noinstrument

Sometimes instrumentation can cause problems. If you upload some big files and testling just hangs, the instrumenter could be hung up. You can disable the instrumentation with this field as a comma-separated list of files.

Or, you can disable instrumentation for all files by specifying ?noinstrument by itself without any files.

Here's an example:

tar -cf- beep.js boop.js | curl -sSNT- \
  testling.com/?noinstrument=boop.js

argv

Sometimes you might want to pass in variables to your tests so that you don't have to modify your test.js with constants. Just use the argv parameter!

In your tests you can access process.argv just like in node:

var test = require('testling');

test('argv test', function (t) {
    t.log(process.argv);
    t.end();
});

Now to run your test, just pass in argv on the query string.

robot : ~ $ curl -sSNT argv.js \
  'testling.com/?browsers=firefox/3.6&argv=HELLO&argv=WORLD!'
Bundling...  done

firefox/3.6         0/0    0 % ok
  Log: ["testling","/test.js","HELLO","WORLD!"]

total               0/0    0 % ok
robot : ~ $

Each argv will be another element in process.argv. The first two elements in process.argv are populated for compatability with node since you can also run testling tests locally.

main

If you want to specify a different entry point than "test.js" for multi-file uploads, use the main parameter on the query string.

running tests locally

You can run your testling tests locally too! With node.js and npm installed, just do:

npm install -g testling

Now to run a test, just use the testling command:

robot : ~ $ testling test.js
node/jsdom                      1/1  100 % ok
robot : ~ $

To run your test on real browsers, just add --browsers=... to the command:

robot : ~ $ testling test.js \
  --browsers=iexplore/7.0,iexplore/8.0,firefox/3.5
Bundling...  done

iexplore/7.0        0/1    0 % ok
  Error: 'JSON' is undefined
    at [anonymous]() in /test.js : line: 4, column: 5
    at [anonymous]() in /test.js : line: 3, column: 29
    at test() in /test.js : line: 3, column: 1

  > t.deepEqual(JSON.parse('[1,2]'), [1,2]);

iexplore/8.0        1/1  100 % ok
firefox/3.5         1/1  100 % ok

total               2/3   66 % ok
robot : ~ $

That was easy! The testling command will take care of remembering your account information for running tests on real testling.com browsers.

Check out the code on github!

using npm

Since testling uses browserify, you can use many node.js modules published to npm.

Suppose we'd like to use the traverse module in a new testling test. We can make a new test directory and install traverse into the node_modules directory with npm:

robot : ~ $ mkdir traverse-test
robot : ~ $ cd traverse-test/
robot : traverse-test $ npm install traverse
traverse@0.5.1 ./node_modules/traverse 
robot : traverse-test $

You might need to mkdir node_modules first to get the module to install into ./ if there is a node_modules in some parent directory.

Then in your test you can require('traverse'):

var test = require('testling');
var traverse = require('traverse');

test('scrub circular refs', function (t) {
    var obj =  { a : 1, b : 2, c : [ 3, 4 ] };
    obj.c.push(obj);
    
    var scrubbed = traverse(obj).map(function (x) {
        if (this.circular) this.remove()
    });
    
    t.deepEqual(scrubbed, { a : 1, b : 2, c : [ 3, 4 ] });
    t.end();
});

To run your test, just upload a tar file of the whole directory:

robot : traverse-test $ tar -cf- . | curl -sSNT- \
  testling.com/?browsers=iexplore/9.0
Bundling....  done

iexplore/9.0        1/1  100 % ok

total               1/1  100 % ok
robot : traverse-test $

checking your account usage

To see how many minutes you've used, just visit testling.com with curl:

robot : traverse-test $ curl -u me@example.com -s \
  testling.com/usage
Enter host password for user 'me@example.com':
You have used 51 out of 200 minutes this month.
robot : traverse-test $

Or if you prefer JSON output:

robot : traverse-test $ curl -u me@example.com -s \
  testling.com/usage.json
Enter host password for user 'me@example.com':
{"used":51.41045000000002,"limit":200}
robot : traverse-test $

tunnels

You can use ssh tunnels to write testling tests that point at a web server that you're running locally.

uploading your public key

First you'll need to upload your ssh public key. Those usually live in ~/.ssh and will probably be either ~/.ssh/id_rsa.pub or ~/.ssh/id_dsa.pub.

If you have id_rsa.pub:

curl -u me@example.com -sST ~/.ssh/id_rsa.pub \
  testling.com/tunnel

or if you have id_dsa.pub:

curl -u me@example.com -sST ~/.ssh/id_dsa.pub \
  testling.com/tunnel

open a tunnel

To open a new ssh tunnel just do:

curl -u me@example.com testling.com/tunnel/open

The command will output the ssh command you can use to proxy your local service into testling.

close a tunnel

To close your open ssh tunnel you can run:

curl -u me@example.com testling.com/tunnel/close

tunnel status

To get information about tunnels you can do:

curl -u me@example.com testling.com/tunnel

This command will display the ssh commands you can execute to start a tunnel.