Mutation analysis is the primary means of evaluating the quality of test suites, though it suffers from inadequate standardization. Mutation analysis tools vary based on language, when mutants are generated (phase of compilation), and target audience. Mutation tools rarely implement the complete set of operators proposed in the literature, and most implement at least a few domain-specific mutation operators. Thus different tools may not always agree on the mutant kills of a test suite, and few criteria exist to guide a practitioner in choosing a tool, or a researcher in comparing previous results. We investigate an ensemble of measures such as traditional difficulty of detection, strength of minimal sets, diversity of mutants, as well as the information carried by the mutants produced , to evaluate the efficacy of mutant sets. By these measures, mutation tools rarely agree, often with large differences, and the variation due to project, even after accounting for difference due to test suites, is significant. However, the mean difference between tools is very small indicating that no single tool consistently skews mutation scores high or low for all projects. These results suggest that research using a single tool, a small number of projects, or small increments in mutation score may not yield reliable results. There is a clear need for greater standardization of mutation analysis; we propose one approach for such a standardization.
Login
Links
-
Recent Posts
- ICST 2017: The Theory of Composite Faults 10/12/2016
- FSE 2016: Can Testedness be Effectively Measured? 29/05/2016
- Software Quality Journal 2016: Does The Choice of Mutation Tool Matter? 08/05/2016
- ICSTW 2016: Measuring Effectiveness of Mutant Sets 08/05/2016
- ICSE 2016: On the limits of mutation reduction strategies 15/12/2015
- ISSRE 2015: How hard does mutation analysis have to be, anyway? 20/08/2015
- ASE 2015: How Verified is My Code? Falsification-Driven Verification 20/07/2015
- ESEM 2015: An empirical study of design degradation: how software projects get worse over time 20/05/2015
- ISSRE 2014: Mutations How close are they to real faults? 06/08/2014
- Sunbelt 2014: Temporal Visualization of Dynamic Collaboration Graphs of OSS Software Forks 28/01/2014
- ICSE 2014: Code Coverage for Suite Evaluation by Developers 28/01/2014
- OSS 2014: An Exploration of Factors Affecting Code Quality in FOSS Projects 20/01/2014
- OSS 2014: Drawing the Big Picture: Temporal Visualization of Dynamic Collaboration Graphs of OSS Software Forks 10/01/2014
- CHI 2014: Abandonment of Social Networks: Shift from Use to Non-Use and Experiences of Technology Non-Use 01/01/2014
- Entries 31/12/2013