Importance of independent testing

In this age of TDD, people are getting an impression that TDD or other forms of developer-testing virtually guarantee absence of bugs. They think that 100% code coverage means that bugs have nowhere to hide.

The problem is that developer-tests are made with the same assumptions and logical errors as the code itself. If you don’t think about a particular scenario that would cause your code to malfunction, you are not going to write a test for it.

Of course, modules/classes/services can and should be unit and integration-tested. That provides guarantee against basic errors and allows conducting safe code-refactoring by verifying that existing functionality is not broken. Personally I’m a big fan of integration-testing and intend to blog on it later.

People should also keep in mind that despite wide-spread beliefs in extereme effectiveness of developer-testing, empirical studies show moderate rate of bugs detection though developer-tests. In Code Complete 2 Steve McConnel reports that “Individual testing steps (unit test, component test, and integration test) typically find less than 50 percent of the errors present each. The combination of testing steps often finds less than 60 percent of the errors present (Jones 1998).”[McConnell, 2004]

That compares with about 80% detection rate achieved by modeling and prototyping, and 75% detection rate achieved by formal code reviews. You might not agree with the conclusions but the numbers are accumulated over a large number of projects and chances are that they are at least somewhat reflective of the reality.

I truly believe that in addition to all developer testing (include unit-testing, reviews and prototyping), independent QA/Functional testing is indispensable in finding real bugs that would otherwise make it into the production.

First of all, a person or a team of testers are free from the assumptions that led to the bugs in the first place. Second, a fresh pair of eyes and an experience (and talent) in finding faults of the system are what is needed to uncover the dark corners of the software.

Third, I’m a firm believer that a division of labor leads to a higher productivity. Without going into long economic theories, it is known that people that specialize in a particular skill tend to be more productive in it. From this stems the overall increased productivity of the team and the company. A tester is a specialist, well…in testing. That’s what he knows to do best. Just as a developer, he accumulated a bag of techniques and tricks that help him solve testing puzzles. Because of this he can find more bugs, more boundary-conditions, more unexpected inputs than the developer.

For these reasons I believe that rigorous testing by an independent team of testers is absolutely necessary in addition to all forms of developer-testing.