One of the most fundamental concerns of developers testing code is how to determine if a test suite strikes a good balance between the cost of undetected faults and the cost of further testing. The most common approach may be to use code coverage as a measure for test suite quality, with diminishing returns in increased coverage or high absolute coverage as a stopping rule. In testing research, suite quality is often evaluated by measuring the ability of a suite to kill mutations, artificially seeded code changes. Mutation testing is effective but expensive and complex, thus seldom used by practitioners. Determining which criteria best predict mutation kills is therefore critical to practical estimation of suite quality. Previous work uses only a small set of programs, and usually compares multiple suites for a single program. Practitioners, however, seldom compare suites— they evaluate one suite. Using suites (both manual and automatically generated) from a large set of real-world open-source projects shows that results for evaluation differ from those for suite-comparison: statement (not block, branch, or path) coverage predicts mutation kills best.
The full paper is available here
