Code Coverage’s Mixed Message

I’m in agreement with Robert Glass when he says “100% test coverage is insufficient. 35% of the faults are missing logic paths.” It’s not controversial, but I’d like to give my perspective on it.

If you have an automated unit test suite, low code coverage is an indication that you need more tests. Unfortunately, high code coverage does not tell you if you have enough tests or the right. Adding to Robert Glass’ observation, executed code is not necessarily tested code. Imagine a test case that runs through many lines of code, but never checks that they are doing the right thing. At best this is the “I don’t have any bad pointers” test.

Here are Gerard Meszaros’ steps of the Four Phase test pattern

1. Setup preconditions
2. Exercise the code under test
3. Check the desired behavior
4. Restore the system to its original state

You can get 100% test coverage (executing every line of code) and never do step 3. The code is initialized and exercised, but we are not sure if it is correct. Low test coverage is telling, while high test coverage can be meaningless. Here is an example test case:

TEST(MyComplexThing, ScenarioA)
{
	// init for scenario A
 
	result = MyComplexOperation(a,b,c,d);
 
	CHECK(result == TRUE);
}
 
TEST(MyComplexThing, ScenarioB)
{
	// init for scenario B
 
	result = MyComplexOperation(a,b,c,d);
 
	CHECK(result == TRUE);
}
 
TEST(MyComplexThing, ScenarioC)
{
	// init for scenario C
 
	result = MyComplexOperation(a,b,c,d);
 
	CHECK(result == FALSE);
}

If MyComplexOperation() operation has a cyclomatic complexity of 3, I should be able to test it with three test cases, one for each execution scenario. If all MyComplexOperation() does is return a result, this is adequate. If it interacts with some other part of the system, it is not adequate. There are no checks to see if it did the right thing.

Metrics are important, and it is equally important to understand what they do and do not tell us.

©2013 James Grenning's Blog. All Rights Reserved.