Otaku, Cedric's blog

Imagine I am trying to test a server.  In order to do this, my test
class will contain the following test methods:

• Check that we are running the correct JVM.
• Check that the server started correctly.
• About twenty methods making various calls to the server.

Obviously, the first two test methods listed above should be run before
everything else.  The way you would do this with JUnit is to put the first
two methods in the initialization code, which will probably have to be static
since JUnit instantiates a new object before each test method invocation, and we
can’t use setUp() either since it’s invoked before each test method. 
This initialization code will set two booleans and the twenty test methods will
have to test for these two booleans before proceeding, and if one of them is
false, it will fail.

Now let’s assume that on a test run, I used the wrong JVM.  In this
case, JUnit will probably report this in the initialization code and it will
then report twenty more failures for each test method.

This is very bad for a variety of reasons:

• The initialization code is not in a test method per itself.  This
  is bad because report of its success/failure will be done in a separate
  track.
• When QA reads the result of this test run, they will see "1 SUCCESS, 21
  FAILS" and they will, rightfully, get really scared.
• Upon reading of this result, QA will have to provision for fixing 20
  tests, while the reality is that only one test failed.

This is why I believe that a test framework needs to provide support for
"dependent test methods", where you can mark test method b() as depending on the
successful run of test method a().  If a() failed, then b() will be marked
as a SKIP, and not a FAIL.

With such a feature, the test run will be marked "1 SUCCESS, 1 FAIL, 20
SKIPS", which is much more accurate.

Here is how I would write this test using TestNG:

@Test
public correctVM() {}
@Test
public serverStartedOk() {}
@Test(dependsOnMethods = { "correctVM", "serverStartedOk" })
public method1() {}
...

This is a good start, but having to list all the methods that we depend on for each new
test method on is error-prone, so instead, let’s use TestNG’s groups:

@Test(groups = { "init" })
public correctVM() {}
@Test(groups = { "init" })
public serverStartedOk() {}
@Test(dependsOnGroups = { "init.* })
public method1() {}
...

We have gained a lot of flexibility with groups.  For example, imagine
that I want to add another init test method, such as "firewall is on".  All
I need to do is add this test method and declare that it is part of the group
"init".

Also, note that I used a regular expression in the "dependsOnGroups"
declaration, as a reminder that you can actually define several init groups
(such as "initOS", "initJVM", etc…) and they will automatically be run before
any test method is invoked.

But we can do even better.

In the above example, I don’t like the fact
that whenever I add a new "real" test method, I need to remember to specify that
it depends on the group "init.*".  In TestNG, the traditional way to
indicate that an annotation should apply to all test methods is to move this
annotation at the class level.

Also, I don’t like the fact that "init" methods and "real" test methods are
in the same class, so I’d like to use inheritance to provide a cleaner
separation of roles.

Therefore, I will restructure my tests like
this:

@Test(groups = "init")
public class BaseTest {
public correctVM() {}
public serverStartedOk() {}
}

This is now the base class for my tests.  The @Test annotation is now on
the class, which means it applies to all the public methods inside that class
(so there is no need to repeat it on each individual method).  Therefore,
each public method automatically becomes part of the group "init".

Next, I
write my test class as a subclass of BaseTest:

@Test(dependsOnGroups = { "init.*" })
public class TestServer extends BaseTest {
public method1() { ... }
public method2() { ... }
...
}

Here again, the @Test annotation is now on the class, which means that it
applies to all the public test methods, making it easier to add new testing
methods.  Also, since this class extends BaseTest, it "sees" not only the
methods that are being inherited, but their annotations as well, so TestNG will
include all the methods from the base class that belong to the "init.*" group to
determine which methods need to be run first.

What’s the overall result?

• Two classes that have responsibilities that are very clearly delineated.
   
• Very easy maintenance and evolution, since adding test methods (whether
  they are init or "real" test methods) boils down to just adding these
  methods to the right class (no need to annotate them).  A newcomer
  doesn’t even need to know about what annotations that are needed to get all
  this to work.
   
• A report that will accurately reflect the result of the test runs and
  will correctly identify the real failures from those that are caused by a
  cascade effect, and whose resolution should therefore be postponed until all
  the FAILs have been resolved.