I'm a big proponent of unit tests and Test Driven Design (TDD). Unfortunately something I see relatively often is developers who don't understand that the quality of the unit tests themselves is important. In this post I'm focusing on a particular area of concern: unit tests that will intermittently fail for reasons not due to errors in the code under test.

Failures of this kind are generally due to two causes. Firstly an external resource that the test depends on is unavailable or not in the expected state. This can be dealt with but the effort required to do so can be excessive and make testing difficult and expensive. The external resource may be another unit test, where the order the tests are run in can adversely affect the success of the test run. This causes test fragility and is to be avoided.

Secondly unit tests may fail due to timing issues. This can occur when operations are time sensitive. Such a test is subject to timing conditions beyond its control such as when the operating system schedules its thread to run. I've seen tests with Sleep() statements intended to ensure the timing of the test is valid. This is a fragile and unreliable hack that does not address the underlying problems that cause test failure. This kind of attempt to play with the timing of operations is a sign that the design of your code is inadequate, properly factored code should allow the time sensitive elements to be appropriately controlled or mock to allow reliable testing.

The net outcome of unreliable tests is that they provide negative value. When the test fails you have two choices. Investigate to see if there is an actual bug or ignore the failure. Investigation, even if it's running the test again, is time consuming, especially if the test happens to fail multiple times. Ignoring the test means that it might as well not be there. Indeed this would be preferable as its presence suggests that there is testing where in practice there is not. This false confidence runs the risk of introducing errors that may not be detected.

I've been burnt by test runs that required complex setup and specific ordering to be effective. Doing this damages the value of testing and tends to lead to team members ignoring the unit tests. This is not TDD. Accept no substitutes, demand unit tests that pass consistently and independently.