Strong Testing Does Not Obsolete Strong Typing

Great holy wars are waged at the battlegrounds of statically typed languages versus dynamically typed languages or object oriented programming versus functional programming today. I am in the it depends camp in both subjects, there is no clear winner. This ambiguity annoys the hell out of me. That is why this post is about a different controversial topic.

Testing versus type checking has come up in a couple different places recently. I have found a well stated argument in the article titled Strong Typing vs. Strong Testing.[1]:

The only guarantee of correctness, regardless of whether your language is strongly or weakly typed, is whether it passes all the tests that define the correctness of your program.

I agree that type checking is not a complete guarantee of correctness. It is a partial guarantee. A partial guarantee is better than nothing.

The quote above implies testing is a complete guarantee of correctness. That if all your tests pass, your program is correct. If we define correctness as adherence to the specification, this proposition would be valid only if the tests express the specification perfectly.

In my experience tests can express specifications very very well but never perfectly. It is quite easy to come up with happy path tests. It is almost as easy to come up with most likely scenarios of error conditions. Writing tests for less likely errors is more difficult but not impossible. But 100% coverage of all possibilities (not to be confused with 100% code coverage) is practically impossible. That is why I say tests can do a very very good, but not perfect, job at expressing specification.

You may argue that properly written unit tests can cover pretty much all possibilities. I agree. But a proper unit test must mock all its dependencies and run only the unit under test[2]. If we relied only on unit tests, testing would be an even smaller guarantee of correctness. Real life programs consist of more than one unit. So the guarantee of correctness here is more about integration tests and integration tests are hard.

Integration tests are harder to write, harder to maintain and most importantly it is harder to cover all possible scenarios. That is not to say integration tests are too expensive so we should give up on them. On the contrary they are very valuable for the reason I mentioned above; unit testing is (relatively) easy but limited by design, whereas integration tests have a multiplier effect:

But those syntax tests can only go so far. The compiler cannot know how you expect the program to behave, so you must “extend” the compiler by adding unit tests (regardless of the language you’re using). If you do this, you can make sweeping changes (refactoring code or modifying design) in a rapid manner because you know that your suite of tests will back you up, and immediately fail if there’s a problem – just like a compilation fails when there’s a syntax problem.

It is true that a good test suite can catch all issues type checkers can and then some. But type checking comes free (if we discount annotations in our code) and runs faster. A good test suite must include a significant amount of integration tests to provide the confidence mentioned in the quoted text. They are hard to write, hard to maintain. Most importantly having integration tests is an incomplete guarantee of correctness, just like type checking.

Although I think it contains some overgeneralizations and is biased against statically typed languages, the article is an interesting read. If you are interested in such comparisons I would suggest reading it. In closing I would like to summarize my position on this topic:

• Unit tests are the foundation[3].
• Type checking is a low-cost way to catch a good deal of integration errors.
• Integration tests act as a multiplier to the value of unit tests.
• These three combined still cannot provide complete guarantee of correctness.
• Feel free to prove your programs if that’s your thing.