JUnit pain
JUnit
Why does a TestCase get instantiated as many times as it contains test methods?
Try this:
public class MyTest extends TestCase {
public MyTest(String name) {
super(name);
System.out.println(“Created a MyTest”);
}public void testA() {}
public void testB() {}
}
and you will see “Created a MyTest” displayed twice.
My initial reaction when I see code that puzzles me is to give the author the
benefit of the doubt, so my first thought was “Maybe they do this to make sure that test methods are independent from each other, that is, a test cannot fail because of a side effect created by another test method”.
Then I realized that this justification was stupid for two reasons:
- If I want two test methods to be truly independent, I will just put them in two different classes.
- The setUp() method is called before each test method, thus guaranteeing that the state of the object is restored before each invocation.
The more I thought about it, the more I realized that being able to share some state between test methods, such as a JDBC connection or a parsed XML file, was actually quite useful. The problem is that JUnit
makes this simply impossible: you have to put this code in setUp() and
therefore, rerun it before every test method.
At this point, and after finding all these deficiencies in JUnit’s design and implementation, I am beginning to think we are once more stuck with a de facto standard that is utterly broken.
Unless someone can explain to me this behavior?
This entry was posted on February 8, 2004, 5:45 pm and is filed under Java. You can follow any responses to this entry through RSS 2.0. Both comments and pings are currently closed.
#1 by Tim Downey on February 9, 2004 - 9:35 am
Whether or not it makes sense, if you want once-per-test-class initialization, you can implement on your test class.
public static Test suite(){
TestSuite suite = new TestSuite(MyTestClass.class);
return new TestSetup(suite) {
protected void setUp() {
// one-time setup
}
protected void tearDown() {
// one-time teardown
}
};
}
#2 by Anonymous on February 9, 2004 - 10:47 am
Why is this a problem? So what if it ititializes test# times?
#3 by Cedric on February 9, 2004 - 10:47 am
It creates unnecessary instances and can be very time consuming.
#4 by Vinny Carpenter on February 9, 2004 - 1:08 pm
Hi Cedric. I think a lot of it may just be defensive programming. You’re right that you can create fixtures such as setUp() and teadDown(), but they don’t have mandate the use of those fixture methods. So I think JUnit fires up x number of instances of the class if there are x number of testxx() methods and calls setup/teardown on each distinct instance and executes each of the testxx() methods separately. I think the goal is probably to create the cleanest possible test scenario and not worry about extra object creation and instantiation time.
#5 by Jon Mountjoy on February 9, 2004 - 2:13 pm
The reason is that the running of one test should never affect the running of another. That was the design decision behind it, I believe – they want them isolated.
So if one test breaks, it won’t cause another test to break simply by virtue of it breaking, and perhaps leaving some instance data in a bad state. Likewise, since they are independent you can select any subset and run that subset in any order. (Hmm, handy functionality that Eclipse doesn’t seem to offer. Darn.)
Hence the number of class instantiations.
So hints Kent Beck in “Test-Driven Development by Example”
#6 by Carlos Villela on February 9, 2004 - 3:17 pm
Indeed, this behaviour is just unexplainable to me – if the reason for the multiple instances is to avoid having stale data from failing tests hurting other sane ones, then the setUp and tearDown mechanism sounds quite a bit redundant to me…
Ok, being pragmatic: anyone tried Artima TestRunner?
#7 by J. Betancourt on February 9, 2004 - 3:46 pm
In the JUnit FAQ they do describe how to do a one-time setup by using a TestSetup wrapper. I’ve done it, works ok.
See: How can I run setUp() and tearDown() code once for all of my tests?
J. Betancourt
#8 by Brian Slesinsky on February 9, 2004 - 10:20 pm
It makes a little more sense if you look at how tests are used in a data structure. A TestSuite is basically a list of Test instances. Running a test suite looks like this:
for test in TestSuite:
test.setUp()
test.run()
test.tearDown()
Since a test suite can contain any number of tests in any order, and the tests need not be instances of the same class, it probably wasn’t immediately obvious how to optimize this.
I got around it with some custom subclasses: LazyTest and LazyTestSuite. A LazyTestSuite checks to see if two consecutive LazyTests have the same setup, and if so, calls the test’s clean() in the middle instead of tearDown()/setUp().
#9 by Joel Hockey on February 10, 2004 - 4:04 pm
I’ve also been finding junit (integrated with ant) to be quite frustrating. I wasn’t aware of the new class for each test quirk, but I found that the ant batchtest created a separate TestSuite for each class and ran each in a different classloader. This meant that my Hibernate (and other) config would need to load many times throughout my tests. At around 4s per load, this was really annoying.
My solution was to create a DynamicTestSuite class that creates a single testsuite by matching a classname suffix that you give it.
E.g. Give it “Test” to match all test classes, “LgoinActionTest” for a specific class, or “ActionTest” to test all actions.
More detail (including source code) at: marc.theaimsgroup.com/?l=ant-user&m=107645600632538&w=2
#10 by Hani Suleiman on February 10, 2004 - 6:57 pm
I’m surprised the TDD fanatics haven’t proposed putting you up against a wall and shooting you for daring to suggest that in the real world you might sometimes need fixtures that remain in place across tests.
#11 by Codito ergo sum on February 11, 2004 - 7:36 am
Junit: designed to be painful
Cedric blogs about his pain when using Junit :
I am beginning to think we are once more stuck with a de facto standard that is utterly broken.
Actually his problem is that Junit instantiated TestCase as many times as it contains test me
#12 by Fred Grott on February 11, 2004 - 8:06 am
Cedric are you sure you are not confusing unit tests with system tests?
would not maintaining states or connections or objects between tests denote a system test insteadof a unit test?
#13 by NatPryce on February 11, 2004 - 9:25 am
“if the reason for the multiple instances is to avoid having stale data from failing tests hurting other sane ones, then the setUp and tearDown mechanism sounds quite a bit redundant to me…”
If test cases shared the same fixture state and something failed during the tearDown of one test, the fixture would be left in an inconsistent state, and would cause confusing errors in subsequent tests running with the same fixture.
The alternative would be to require extremely defensive programming in the tearDown method, but that is impractical in practice: if you know where you should be defensive against programmer errors (RuntimeExceptions, for example) then you should just fix those errors!
#14 by Jason Marshall on February 11, 2004 - 12:32 pm
A lot of people seem to either be talking around Cedric’s problem, or simply apologizing for (as in, rationalizing) the behavior of JUnit, which I find to be a bit of a shame.
I’ll attempt a different tack, and see if perhaps this illuminates the problem for the folks that don’t see Cedric’s point.
When you write tests, you subclass a class called TestCase. Not TestCases, not SetOfTestCases… TestCase, as in One Test Case. My expectation would be that an instanceof TestCase constitutes an indivisible unit of testability, and therefore should only need one set up and one tear down method call.
This notion that any method begining with ‘test’ in its name constituted a single unit test is excessively foreign to Java, despite the fact that numerous programmers seem to think it’s a good idea to push things in that direction. The unit of design contract in Java is the Interface. Use it, instead of fighting to make the method name the unit of contract.
#15 by Jim Moore on February 12, 2004 - 5:48 am
When I absolutely MUST share state between invocations of different test* methods in a test class, the solution is trivial:
public class MyTest extends TestCase {
static {
// shared setup
}
//…
}
the class only gets loaded once, so the static block is only run once.
Of course, as has been brought up many times, it’s best to avoid doing so. But when you need to do it, it’s easy.
#16 by Phil Aston on February 19, 2004 - 11:28 am
I’ve always wondered this.
I’ve thought for a long time that JUnit design is v. poor – hard to stomouch considering its ubercool parentage, and that orginal design paper on its evolution.
Two vaguely relevant mini-rants:
1. I so frustrated with the ANT JUnit task having to fork a process for every testcase (as in class, sigh – Jason is so right about the stupid naming; there are two types of sets of test cases, test suites of test cases which are actually Java classes of things that you and I would call test cases, and all of these are tests. got that?), because forking a JVM is very slow when you’ve got clearcase hosting half your classpath, that I contributed a version that didn’t. I’m found I wasn’t the only one to find that the ANT development community is deaf to such useful contributions.
2. The previous version of The Grinder (The Grinder 2 – grinder.sourceforge.net) has a JUnit plugin. Once upon a time I wrote this simple monstrosity to produce a list of JUnit tests decorated with my wrapper:
private void getTests(junit.framework.Test test, Set tests) {
// Really hacky switch on type, but no obvious other way of
// doing this with the JUnit API.
if (test instanceof TestSuite) {
final Enumeration jUnitTestEnumeration = ((TestSuite)test).tests();
while (jUnitTestEnumeration.hasMoreElements()) {
junit.framework.Test jUnitTest = (junit.framework.Test)jUnitTestEnumeration.nextElement();
getTests(jUnitTest, tests); // Recurse.
}
}
else if (test instanceof TestCase) {
tests.add(new TestWrapper((TestCase)test, …));
}
else { /* Handle unknown type. */ }
}
which both sucks and blows. I don’t think this was totally my fault.
#17 by Gavin on February 22, 2004 - 9:31 pm
This problem is an absolute PITA to me and I had to do disgusting things to work around it in the Hibernate test suite (the initialization I need to perform is to read the mapping documents, create the SessionFactory, and export the db schema). The workaround? Keep the SessionFactory in a static variable and lazily initialize it from setUp(). Uggghhhh…..
#18 by Catherine on May 8, 2004 - 5:09 am
Learn Linux.
#19 by Stephane Bailliez on July 26, 2004 - 12:49 am
For people complaining about Ant JUnit task’s behavior..in Ant 1.6.2 there has been a new forkmode attribute introduced. More on this in the documentation: ant.apache.org/manual/OptionalTasks/junit.html
#20 by Sarah S on November 6, 2004 - 1:46 pm
Ciao
#21 by aarone on January 13, 2005 - 11:52 am
I thought jim moore a simple answer:
public class MyTest extends TestCase {
static {
// shared setup
}
//…
}
and then I quickly realized that doesn’t allow you to tear down the type of things you actually want to set up, like database connections.
Cedric is right. Junit is broken. It’s worth rewriting from scratch just for this feature alone.
#22 by d2dcc15c17cb352a9389fc29 on April 17, 2005 - 5:04 pm
d2dcc15c17cb352a9389fc29
f84626766e926c2d5a569eee5bf97636 3e6de3.
#23 by screaming penguin on May 16, 2005 - 8:01 am
No, not really. I am
No, not really.
I am entering this discussion admittedly late but it is still a useful dialog so . . .
#24 by j. betancourt on June 14, 2005 - 9:30 am
Easy solution? Leave in the existing per test setup and teardown, but add parallel one-time versions of these:
protected void setUpOnce() throws java.lang.Exception;
protected void tearDownOnce() throws java.lang.Exception
#25 by Michael Bushe on August 16, 2005 - 7:10 pm
I’m also annoyed about having instances for each test method, but one-time setup is trivial, no? There is still one test class, right?
public class TestMyClass {
static hasSetup = false;
public void setUp() {
if (hasSetUp) {
doMySetup();
hasSetUp = false;
}
}
}
Granted, I’d rather not do this.
#26 by Steven on August 17, 2005 - 11:34 am
Lots of discussion about JUnit and there have been some harsh edicts, like:
Cedric: “I am beginning to think we are once more stuck with a de facto standard that is utterly broken.”
Utterly broken? I have produced enterprise applications for companies that have made millions of dollars and used JUnit for testing 80% of the code. Utterly is an extreme adjective here.
Cedric: “It creates unnecessary instances and can be very time consuming.”
I guess I would like to see some metrics on this. Object creation, with todays hardward and OS’s, is incredibly fast. I don’t see what the problem is with unnecessary instances. For instance, I have ten unit tests that each have 3-6 test methods, and the suite runs in about 2 seconds. This is super fast. In my experience, the most JUnit tests I’ve had for a project was about 400, with probably (on average) 5 tests per class and it took about 15 minutes to run. And we once calculated that 5 minutes of that was dedicated to starting up and tearing down Hibernate.
Carlos: “then the setUp and tearDown mechanism sounds quite a bit redundant to me…”
In some cases that is true, but I write unit tests with the same design principles as my production code. So I use the setUp() method to prevent code duplication in the tests. By the way, I rarely override the tearDown() method. I try to write all tests such that they are responsible for setting up their but not tearing it down. That has another benefit too, if the test fails you can see the current state of the test. If tearDown() runs then your broken test data gets cleaned up.
Jason: “This notion that any method begining with ‘test’ in its name constituted a single unit test is excessively foreign to Java, despite the fact that numerous programmers seem to think it’s a good idea to push things in that direction. The unit of design contract in Java is the Interface. Use it, instead of fighting to make the method name the unit of contract.”
Well, a good argument in theory I suppose. I prefix all test methods with “test” because 1) it’s easy to find the test and non-test methods in the class, and 2) it works well with the underlying JUnitRunner (it finds all test methods with the prefix “test”). I think pragmatically this argument is not that powerful, in reality the use of the prefix “test” is idempotent.
Also, comments about the Ant JUnit task are well founded, but I have managed to build successful software projects with this.
My last comment, I overall believe