Unit testing our server code - List of tools we're using...

So we're starting the push to get our unit / system test cases as automated as possible... We've already got a set of test plans - but as the system has grown in complexity, testing is now taking over 2 man days per release of manual testing...

We're driving the running of testcases from Ant... Ant 1.6.1 - we've already got our build / packaging process in place here, and it's working well ...

Obviously we're using Junit (3.8.1) ... for our unit test framework...

We've got the surprisingly excellent and free (commercial option available too) JCoverage 1.0.5 to provide code coverage...

And we're using Http Unit 1.5.4 (the stable version ... looks like a new version is coming soon, though) ... I can't say enough about the HttpUnit toolset ... I love it... I'll get a big gushing mini-review out when I have a moment ... This really does everything I'm looking for on the system-test front...

A couple of guidelines for unit test cases:

1. They must be repeatable / reproducible - they should always pass or always fail (until you make them pass...) ... none of this failing sometime, but not others...
   
2. They should leave the system in a known state, and not count on the order in which they're run, the success of previous test cases, etc...
   

Our server reads (several) config files on startup, and there are many many different states that the system could be starting in... We’ve created data-sets of config files that the system could be starting in… We’ve created a BaseTest that extends junit’s TestCase – it implements a consistent set-up / teardown so we meet criteria #2 of unit test cases… We had initially been using ant to setup the directories / copy datafiles around / etc – because ant has such strong file manipulation support… But we couldn’t always count on the test cases being run via ant – and required separate test cases files for each test case… Too many dependencies for our liking, so we flipped things, and had the BaseTest make calls directly to Ant’s tasks… And this has worked great…

Another tricky area that we ran into… We’ve got hooks to call directly into the framework for our unit tests, but I also wanted to use HttpUnit for system tests, to take advantage of HttpUnit’s javascript support (which I’ve been SO happy with…)… This communicates over http, and once again our testcase criteria demand that the system be started in a known state, and left in a known state… So we created a helper thread which we can use to start and stop our server… It took some tweaking, but it looks like things are working really well now…

Over time we’ll build up our unit and system test cases, and our library of startup configurations… We’ve put hooks into our email notification so that we can even test that notification is working correctly, without actually sending / receiving email…