Friday, July 23, 2004

Unit testing our server code - List of tools we're using... - Part 2

I missed a tool in our list of tools we're using for unit testing ... We also use Xalan J - A little 15M library for XSLT stuff, that the ant junitreport tool needs to covert the xml output generated by the testcases into nice html output...

The process for checking code coverage:

ant clean - start from a clean slate :)
ant [jarfile] - compile everything, obfuscate the library and generate a nice little jar file... all of the tests need this jar, so it's an important task...
ant coverage - run through our unit test cases... this task depends on our ant "instrument" task (in which class files are run through the jcoverage instrumenation process), and the ant "allTests" task which sets up a temp test directory and starts batch running our unit test cases... all of the output from the testcases goes in a reports directory... and the code coverage information builds up in the temporary directory...
ant junitreport - calls the junitreport to convert the xml output in the reports directory into human friendly html output in a reports/html directory...

Once the process if running well for everyone we'll create a ant runAllCleanCoverage that will do everything we need above...

At the end of the process, we have two useful reports:
Testcase results in reports/html
Codecoverage results in build/instrumented/reports

Monday, July 19, 2004

100% code coverage no guarentee...

Note: I have edited this entry on July 19, 2004 to clean-up the code after someone pointed out some of the errors below.

A bit of reading lately about code coverage, and aiming for 100% and whether that is realistic or not...

It's important for people to step back and not just a code coverage, but the quality of the unit test cases that they are writing...

We'll look at a function to take 2 strings, convert the strings to numbers, add them together and return the result as a string (yes - we all know that when coverting strings to numbers to add them together, you should take a Vector... just kidding... for demonstration purposes only folks...)


Ex:

/**
* Add 2 numbers two numbers together...
*/
public String addNumbers(String value1, String value2) {
int i1 = Integer.parseInt(value1);
int i2 = Integer.parseInt(value2);
return("" + (i1+i2));
}


We could write a testcase:


public void testAddNumbers() {
int i1 = 5;
int i2 = 10;
assertTrue((""+(i1+i2)).equals(addNumbers(""+i1,""+i2)));
}


Bingo ... I've tested that my function works, and I've got 100% code coverage...

But what happens when I take user input and pass it directly into my function ... And the user doesn't type a number... Let's simulate in our testcase...


public void testAddNumbers() {
int i1 = 5;
int i2 = 10;
assertTrue((""+(i1+i2)).equals(addNumbers(""+i1,""+i2)));
assertTrue(("NaN".equals(addNumbers("bob","doug")));
}


Hmmm... My function is throws a NumberFormat exception... and my test case is clearly failing... We'll update our function:


/**
* Add 2 numbers two numbers together...
* @return sum of two numers, or "NaN" if either string is not a number...
*/
public String addNumbers(String value1, String value2) {
try {
int i1 = Integer.parseInt(value1);
int i2 = Integer.parseInt(value2);
return("" + (i1+i2));
} catch (NumberFormatException nfe) {
// debug: nfe.printStackTrace();
return("NaN");
}
}


Now my testcase is passing, I'm back at 100% code coverage, and all is wonderful... Until the guy in the next cubicle start's passing null's into my function...

We update the testcase:


public void testAddNumbers() {
int i1 = 5;
int i2 = 10;
assertTrue((""+(i1+i2)).equals(addNumbers(""+i1,""+i2)));
assertTrue("NaN".equals(addNumbers("bob","doug")));
assertTrue("NaN".equals(addNumbers("bob",null)));
}


Now I'm getting NullPointerExceptions...

Update the function:


/**
* Add 2 numbers two numbers together...
* @return sum of two numers, or "NaN" if either string is not a number...
*/
public String addNumbers(String value1, String value2) {
try {
int i1 = Integer.parseInt(value1);
int i2 = Integer.parseInt(value2);
return("" + (i1+i2));
} catch (NumberFormatException nfe) {
// debug: nfe.printStackTrace();
return("NaN");
} catch (NullPointerException npe) {
// debug: npe.printStackTrace();
return("NaN");
}
}


... You get the point... This can go on for quite a while ... Someone might pass in "4.5" and "3" and expect "7.5" - but won't get it because our implementation is using Integers... We've had 100% code coverage all along, but found that some input throws Runtime exceptions, that if uncaught, would bring our program down entirely...

While code coverage is a great and noble goal we should all strive for, it is no guarentee that the code won't fail...

Wednesday, July 14, 2004

Integrating our bug tracking system with CVS...

Well...

We've had a SOAP interface built into our bug tracking software (FBT) for some time, but we've had very little time to play with it ourselves... I had been intending to use it to (among other things), integrate our CVS source code repository with our bug tracking system...

I spent the past week creating a perl command line tool that uses the soap interface of FBT to:
1) Get the details of a single bug
2) Get an overview of a list of bugs (based on various criteria)
3) Create a new bug
4) Update an existing bug (This is what we use when integrating with CVS) ...
[These operations map to the 4 primary operations of our soap interface, which in turn map to the 4 basic CRU(not D, but another type of higher level Read) operations that you reguarly perform in FBT...]

Creating the perl command line tool, and working with SOAP::Lite a perl library for soap calls, has been really enjoyable... The tool can take tons of command line arguments for all of the various fields within the system... I've started playing around as just a handy was to read the details about an issue from the command line without needing to use a browser to connect to our system ("fbtcmd.pl getId 123") ...

We wrote another small perl script that takes some details from CVS looks for "bug XYZ", and then calls the fbtcmd tool, and sends a link to our cvsweb web interface to CVS that can display the differences that were checked in... fbtcmd also automatically marks the bug(s) "Ready for Retest" (this is configurable) and assigns them back to the creator (and activates the various notification within FBT to notify the creator, etc...)

We'll be using the tool internally for the next couple of weeks, and will be releasing it to our customers once we're happy with how it's working... This integration will greatly speed up the code reviews that we perform on each other's code as it gets commited...