Study of Equivalent and Stubborn Mutation Operators Using Human Analysis of Equivalence

Overview

Though mutation testing has been widely studied for more than thirty years, the prevalence and properties of equivalent mutants remain largely unknown. We report on the causes and prevalence of equivalent mutants and their relationship to stubborn mutants (those that remain undetected by a high quality test suite, yet are non-equivalent). Our results, based on manual analysis of 1,194 mutants from 18 programs, reveal a highly uneven distribution of equivalence and stubbornness. For example, the ABS class and half UOI class generate many equivalent and almost no stubborn mutants, while the LCR class generates many stubborn and few equivalent mutants. We conclude that previous test effectiveness studies based on fault seeding could be skewed, while developers of mutation testing tools should prioritise those operators that we found generate disproportionately many stubborn (and few equivalent) mutants.

Equivalent Mutants Categories

A Summary of the Human-Evaluated Decision Procedure for Mutant Equivalence

Case 1: Mutant cannot be reached by any test input

Case 1.1 (normal): Mutated statement or predicate cannot be reached
Case 1.2 (short-circuit): Mutated sub-expression never evaluated though its containing predicate is

Case 2: Mutant is reached by at least one test input, but no test causes state infection

Case 2.1 (context free): Infection can never occur in any state
Case 2.2 (context sensitive): Not context free, but infection cannot occur in any reaching state
Case 2.3 (subpath equivalence): Mutation changes path executed, but all paths are equivalent

Case 3: Mutant is reached and infects the state, but no infection propagates to an output

Case 3.1 (unobservable): No output statement mentions an infected variable
Case 3.2 (observable): Outputs mention infected variable(s), but infection fails to reach any

Equivalent Mutants Benchmark

Subjects	C1.2	C2.2	C2.3	C3.1	C3.2
Min	0	1	0	8	0
Bubble Sort	0	0	0	3	6
Profit	0	17	5	24	0
Mid	0	0	5	14	0
Prime_num	0	5	0	4	0
Triangle	0	13	0	15	12
Insert	0	2	0	9	8
Day	0	6	0	8	0
Calendar	0	25	0	16	0
Carsimulator	0	13	7	24	0
Tcas	44	26	0	20	0
Defroster	54	41	0	48	0
Schedule	0	28	0	20	0
Hashmap	0	47	0	20	0
Replace	0	135	0	80	0
Space	0	72	1	14	0
Flex	0	40	0	0	0
Make	0	32	0	28	0

Stubborn mutants

All stubborn mutants: download

Paper

A preprint version of the paper can be download from here.

@inproceedings{YaoHJ14,
 author = {Yao, Xiangjuan and Harman, Mark and Jia, Yue},
 title = {A Study of Equivalent and Stubborn Mutation Operators Using Human Analysis of Equivalence},
 booktitle = {Proceedings of the 36th International Conference on Software Engineering},
 series = {ICSE 2014},
 year = {2014},
 isbn = {978-1-4503-2756-5},
 location = {Hyderabad, India},
 pages = {919--930},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/2568225.2568265},
 doi = {10.1145/2568225.2568265},
 publisher = {ACM},
 keywords = {Equivalent Mutant, Mutation Testing, Stubborn Mutant},
}

Contact

yue.jia@ucl.ac.uk