meck and eunit best practices
David Reid
(@dreid)
on
erlang, testing, mochi
Posted on:
At this point, after 3 years at Mochi, I've written a significant amount of Erlang code with nearly 100% line coverage. I've learned over those 3 years that many Erlang programmers think mock libraries mean your code is being written wrong. However having not seen a significant amount of well tested code that doesn't use them I find them to be quite useful. So I'm not here to argue why you should use a mock library (in particular meck) in your Erlang project. I assume you've already made the decision to use it to test your code and I'm merely here to provide you with some useful tips for how to get the most out of it when using eunit.
Our example code
Here is a funciton called get_url/2 which takes a url as a string, and a list of options. It then makes an HTTP request, through a proxy using lhttpc. I wrote this code last week.
%% @spec get_url(string(), list()) ->
%% {ok, {{pos_integer(), string()},
%% [{string(), string()}], string()}} | {error, term()}.
get_url(Url, Options) ->
TimeoutMSec = proplists:get_value(
timeout_msec, Options, ?DEFAULT_TIMEOUT_MSEC),
{Host, Port, _Path, SSL} = lhttpc_lib:parse_url(URL),
{ok, {{Code, Reason}, Headers, Body}} =
lhttpc:request(
?PROXY_HOST,
?PROXY_PORT,
SSL,
URL,
"GET",
[?USER_AGENT,
{"Proxy-Connection", "Keep-Alive"},
{"Connection", "Keep-Alive"},
{"Host", lists:flatten([Host, $:, integer_to_list(Port)])}],
[],
TimeoutMSec,
[]),
case Code of
200 ->
{ok, {{Code, Reason}, Headers, binary_to_list(Body)}};
C when C >= 400, C < 600 ->
{error, {http_error, Code, Reason, URL}}
end.
- To test this code we want to:
- mock the lhttpc module
- set an expectation about the lhttpc:request/9 function.
- run get_url and assert on it's result.
- validate our expectations for the lhttpc module
- unload the mocked lhttpc module.
Our first test case
Here is a test that verifies that when we get a 404 from the lhttpc module we return an appropriate error tuple.
get_url_error_test() ->
meck:new(lhttpc),
meck:expect(lhttpc, request, 9,
{ok, {{404, "Not Found"}, [], <<"Not found">>}}),
?assertEqual({error, {http_error, 404, "Not Found", "foo.com"}},
get_url("foo.com", [])),
meck:validate(lhttpc),
meck:unload(lhttpc).
Now, you might be looking at that and saying "this is a bad test" and you'd be right. This test does just about everything wrong except the actual asserting the result.
Poorly isolated
If get_url is changed such that an exception is raised or the assertion fails the lhttpc module will not be unloaded and future tests calling meck:new will fail immediately and somewhat violently. And here is how most people would immediately rewrite it.
get_url_error_test() ->
meck:new(lhttpc),
meck:expect(lhttpc, request, 9,
{ok, {{404, "Not Found"}, [], <<"Not found">>}}),
try
?assertEqual({error, {http_error, 404, "Not Found", "foo.com"}},
get_url("foo.com", [])),
after
meck:validate(lhttpc),
meck:unload(lhttpc)
end.
This fixes the isolation problem, but it's still suboptimal, mostly because obviously this code is going to have other test cases and they're all going to have to mock lhttpc and unload lhttpc. What we actually want do here is use what eunit refers to as fixtures. Specifically the fixture we're interested in is called foreach and it takes the following form.
{foreach,
fun() -> %% A setup function
%% I'll be run once before each test case.
ok
end,
fun(SetupResult) -> %% A cleanup function
%% I'll be run once after each test case, even if it failed.
ok
end
[
%% I'm a list of simple test functions.
]}.
We could also use the setup fixture if we only wanted the setup and cleanup functions to be run once for all of our test cases. However foreach provides greater isolation and I think it should be preferred to setup almost always.
So let's rewrite our tests.
get_url_test_() ->
{foreach,
fun() ->
meck:new(lhttpc)
end,
fun(_) ->
meck:unload(lhttpc)
end,
[{"Handles error codes",
fun() ->
meck:expect(lhttpc, request, 9,
{ok, {{404, "Not Found"}, [], <<"Not found">>}}),
?assertEqual({error,
{http_error, 404,
"Not Found", "foo.com"}},
get_url("foo.com", [])),
meck:validate(lhttpc)
end}]}.
This is a much better test. First we've started to give ourselves a framework for adding more test cases which have the same setup and cleanup steps.
We start by converting our simple test function to a test generator by adding changing the test name from get_url_error_test() to get_url_test_(). This is required to use fixtures if you don't use a test generator you'll see a single test case that always passes because your test code isn't actually being run. The foreach tuple will just assume to be a successful result of your test case.
We've then moved creating of the mock module to our setup function. And unloading of the mock module to our cleanup function. We've also for good measure pattern matched on the return value of unload. So if the unload fails, this test case fails with an error. Otherwise we might just get a stranger failure later on when we try to create a new mock of the lhttpc module.
This example also uses a title, notice how the 4th element of the foreach tuple is actually a list of 2 tuples? That first string element is actually the title and will be displayed in the test output. Like this:
simple_client: get_url_test_ (Handles error codes)...[0.006 s] ok
Without that title you'd just see this:
simple_client: get_url_test_...[0.0006 s] ok
And once you have more than one test case you'll be stuck deducing which test failed from the error message or just a line number.
Doesn't assert validation
This is a very common mistake, especially with code ported from using effigy, and it is potentially the most dangerous problem because it can cause bad tests to pass.
Most people assume that meck:validate/1 will raise an exception if it fails, however it actually returns a boolean, and if you're not checking this value your test might seem to pass when they shouldn't. For the test here this isn't a problem, if the expectation wasn't met then there is almost no way that the return value of get_url would be correct. However for tests which assert that some event is triggered or when assertions are necessarily done in the body of expectations the potential for a missed assertion to cause a false pass is a very real danger.
Lets look at the fixed example:
get_url_test_() ->
{foreach,
fun() ->
meck:new(lhttpc)
end,
fun(_) ->
meck:unload(lhttpc)
end,
[{"Handles error codes",
fun() ->
meck:expect(lhttpc, request, 9,
{ok, {{404, "Not Found"}, [], <<"Not found">>}}),
?assertEqual({error,
{http_error, 404,
"Not Found", "foo.com"}},
get_url("foo.com", [])),
?assert(meck:validate(lhttpc))
end}]}.
Here we've used ?assert to check the return value of meck:validate/1. We could have used pattern matching but in general the ?assert* macros provide nicer error messages.
Complete Example
Here is a complete example which includes multiple tests and use of meck:expect/3 for specifying a fun which can pattern match it's arguments as well as assert the expected values.
get_url_test_() ->
{foreach,
fun() ->
meck:new(lhttpc)
end,
fun(_) ->
meck:unload(lhttpc)
end,
[{"sends user-agent",
fun() ->
meck:expect(lhttpc, request,
fun(_Host, _Port, _Ssl, _Url, _Method,
Headers, _Body, _Timeout, _Options) ->
?assert(lists:member(
?USER_AGENT, Headers)),
{ok, {{200, "OK"}, [], <<"OK">>}}
end),
?assertEqual({ok, {{200, "OK"}, [], "OK"}},
get_url("foo.com", [])),
?assert(meck:validate(lhttpc))
end},
{"uses proxy",
fun() ->
meck:expect(lhttpc, request,
fun(?PROXY_HOST, ?PROXY_PORT, _Ssl,
"foo.com", "GET", Headers,
[], ?DEFAULT_TIMEOUT_MSEC, []) ->
?assert(lists:member(
{"Host", "foo.com:80"}, Headers)),
{ok, {{200, "OK"}, [], <<"OK">>}}
end),
?assertEqual({ok, {{200, "OK"}, [], "OK"}},
get_url("foo.com", [])),
?assert(meck:validate(lhttpc))
end},
{"Handles error codes",
fun() ->
meck:expect(lhttpc, request, 9,
{ok, {{404, "Not Found"}, [], <<"Not found">>}}),
?assertEqual({error,
{http_error, 404,
"Not Found", "foo.com"}},
get_url("foo.com", [])),
?assert(meck:validate(lhttpc))
end}]}.
Conclusion
meck and eunit are very powerful but often poorly understood tools. There are not a lot of complete examples of how to write good tests for either of them or for erlang in general. So hopefully this has been helpful to you. As usual, the best recommendation is Read The Docs. If the docs fail you Read The Source. If the source fails you Ask The Internet.
You may also enjoy this fantastic presentation by meck's author Adam Lindberg speakerrate.com/talks/7749-meck-at-erlang-factory-london-2011 and another one by my boss Bob Ippolito etrepum.github.com/erl_testing_2011/.
Complaints?
This document represents an attempt to document my personal style of writing tests and some of the reasoning behind them. It is not complete, and many of my recommendations are probably a matter of personal taste. If you have a valid argument against them, or feel like I missed a very important recommendation. Please let me know in the comments or via twitter (I'm @dreid).
If you disagree with things I've said here strongly do not hesitate to tell me that. If you disagree with me very strongly, consider applying for a job writing erlang at Mochi: bit.ly/mochijobs