Risk/Reward Analysis of Test-Driven Development

Abstract

Test Driven Development (TDD) is a software development practice advocated by practitioners of the Agile development approach. Many developers believe that TDD increases productivity and product quality, while others have claimed that it delivers cleaner, more maintainable code. Many studies have attempted to evaluate these claims through various empirical methods. Most recently, Fucci, et al. have used the Besouro tool produced by Becker et al. to measure TDD conformance over a series of closely replicated experiments. Our approach used an originally-written analysis tool run against a cumulative series of homework assignments, providing measurements for the three dimensions identified by Fucci, et al: granularity, uniformity, and sequencing [Fucci, Nov 2016], along with other characteristics associated with TDD. In addition to Test Driven Development, our study incorporates elements of the Transformation Priority Premise (TPP) set forth by Robert Martin. It proposes that as developers write their production code to make their test code pass (the green light phase of TDD), they are transforming their code (in contrast to refactoring). His premise is that there are a finite number of transformations that a developer can choose, and that there is a priority whereby some transformations should be preferred over others. This priority, in an isolated example, has been demonstrated to produce an algorithm with better performance than a comparable solution that ignored the priority order and produced a different algorithm. A post-development analysis tool was written to examine student submissions for both TDD and TPP conformance. Submissions were committed using a special Eclipse plug-in to inject specific comments into a local git repository. The analysis tool then used those comments as indicators of the TDD phase in the student’s process and evaluated how closely students followed the prescribed recommendations. Our findings concur with recent studies, in that we did not find quantifiable proof that following the TDD process delivers a better product than not following the process. We also find that a concise yet meaningful definition of TDD is elusive and contributes to the difficulty in drawing definitive conclusions about its value as a software development practice.