This page is supplementary to the paper entitled "Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection" submitted to TSE.

Abstract

Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. Simulated Annealing (SA) and Greedy Algorithms have been widely used to find CIT test suites. From the literature, there is a widely-held belief that SA is slower, but produces more effective tests suites than Greedy and that SA cannot scale to higher strength coverage. We evaluated both algorithms on 7 real-world subjects for the well-studied 2-way up to the rarely-studied 6-way interaction strengths. Our findings present evidence to challenge this current orthodoxy: real-world constraints allow SA to achieve higher strengths. Furthermore, there was no evidence that Greedy was less effective (in terms of time to fault revelation) compared to SA; the results for the greedy algorithm are actually slightly superior. However, the results are critically dependent on the approach adopted to constraint handling. Moreover, we have also evaluated a genetic algorithm for constrained CIT test suite generation. This is the first time strengths higher than 3 and constraint handling have been used to evaluate GA. Our results show that GA is competitive only for pairwise testing for subjects with a small number of constraints.

Experimental Setup

Subject Programs

We use seven Unix utilities from the Software Infrastructure Repository (SIR), maintained by University of Nebraska Lincoln: flex, grep, gzip, sed, make, nanoxml and siena.

Test Suites

The test suites we use are subsets of those provided by SIR. These can be downloaded from here. For each of the subjects we provide:

  1. An annotated TSL file (from SIR) : [subject_name]_v0_#_small.tsl

    Description: The v0_# matches the associated TSL name from SIR for that subject. The leftmost column in this file is a mapping to the covering array values given as sequential numbers starting at 0. For instance, the first parameter if binary will be 0,1 and the next parameter's values will start at 2. The single and error parameters (and any other parameters that we did not include in our model) are not numbered and not used in our experiments.

  2. An input model file for CASA: [subject_name]_v0_#_small.model

    Description: The first row is the strength (t) and can be changed to 3,4 etc. to generate higher strength arrays. The second row is the number of factors (TSL parameters in the model). The last row lists the number of values (TSL choices) for each of the parameters. This model matches the associated TSL file.

  3. A constraint file in CASA input format: [subject_name]_v0_#_small.constraints

    Description: The first row of this file is the number of constraints. For each of the constraints there are two rows. The first row gives the arity of the constraint (i.e. how many values will be included in this constraint). The next row is list of the constraints between values. The - signs indicate excludes. For instance, a row with a 2, followed by a row that has -10 -12 encodes the binary constraint that values 10 and 12 cannot appear together in a test. Please see the CASA tool for full documentation.

  4. An input model file for ACTS: [subject_name]_v0_#_small.xml

    Description: Parameters and values are described explicitly. Constraints are described in propositional logic.

  5. An input model file for the genetic alrogithm, which we call GAcit: [subject_name]_v0_#_small.csv

    Description: Parameters and values are described explicitly. Constraints are represented as a set of disallowe parameter-value combinations.

  6. Mapping
    In order to use data from fault matrices available in SIR, we created a mapping between test cases generated by the CIT tools and test cases provided in SIR, as follows:

    For each row of array generated by CASA, GAcit or ACTS we used a mapping to the 'Key' in *tsl.tsl files available from SIR (note that every test case which has values for all parameters has such a key). We also enclose the mapping we used (*mapping file). The 'Key' is essentially the first array in the *mapping file. A positive number in that array indicates the value of the single-valued parameters, while -1 represents parameters where the value might change. The 'canumber' column represents the parameter-value in the *tsl file, the 'parameter' value points to the parameter number (order defined in *tsl file) while the 'value' column points to the value number for that parameter which should be put in place of -1 in the 'Key' if that parameter-value is chosen.

Combinatorial Interaction Testing Tool

We use CASA (Covering Arrays by Simulated Annealing) and ACTS (Advanced Combinatorial Testing System) tools to generate combinatorial interaction test suites from the TSL files. The genetic-based CIT tool, which we call GAcit, has been obtained from Tatsuhiro Tsuchiya, one of the authors of the paper describing the algorithm ("Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing.").

Full Results

This section contains the full APCC data for all experiments and plots for APCC and fault detection. (Please wait a few minutes to allow all the data to load.)

Subject Generation Strength Prioritsation Strengths Subject Generation Strength Prioritsation Strengths
flex 2 2, 3, 4, 5, 6 sed 5 2, 3, 4, 5, 6
flex 3 2, 3, 4, 5, 6 sed 6 2, 3, 4, 5, 6
flex 4 2, 3, 4, 5, 6 gzip 2 2, 3, 4, 5, 6
flex 5 2, 3, 4, 5, 6 gzip 3 2, 3, 4, 5, 6
flex 6 2, 3, 4, 5, 6 gzip 4 2, 3, 4, 5, 6
make 2 2, 3, 4, 5, 6 gzip 5 2, 3, 4, 5, 6
make 3 2, 3, 4, 5, 6 gzip 6 2, 3, 4, 5, 6
make 4 2, 3, 4, 5, 6 nanoxml 2 2, 3, 4, 5, 6
make 5 2, 3, 4, 5, 6 nanoxml 3 2, 3, 4, 5, 6
make 6 2, 3, 4, 5, 6 nanoxml 4 2, 3, 4, 5, 6
grep 2 2, 3, 4, 5, 6 nanoxml 5 2, 3, 4, 5, 6
grep 3 2, 3, 4, 5, 6 nanoxml 6 2, 3, 4, 5, 6
grep 4 2, 3, 4, 5, 6 siena 2 2, 3, 4, 5, 6
grep 5 2, 3, 4, 5, 6 siena 3 2, 3, 4, 5, 6
grep 6 2, 3, 4, 5, 6 siena 4 2, 3, 4, 5, 6
sed 2 2, 3, 4, 5, 6 siena 5 2, 3, 4, 5, 6
sed 3 2, 3, 4, 5, 6 siena 6 2, 3, 4, 5, 6
sed 4 2, 3, 4, 5, 6

flex

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.357 72.021 60.344 50.040 41.263
5 97.674 88.528 77.338 66.196 55.977 46.982
6 100.000 92.352 82.027 71.283 61.128 51.974
7 100.000 93.743 84.741 74.893 65.204 56.180
8 100.000 94.786 86.873 77.848 68.662 59.863
9 100.000 95.597 88.618 80.367 71.712 63.204
10 100.000 96.176 89.837 82.114 73.835 65.558
11 100.000 96.640 90.870 83.652 75.747 67.703
12 100.000 97.103 91.923 85.246 77.760 69.996
13 100.000 97.567 92.883 86.605 79.397 71.800
14 100.000 97.914 93.649 87.747 80.834 73.440
15 100.000 98.262 94.452 88.974 82.391 75.219
16 100.000 98.494 95.024 89.900 83.627 76.694
17 100.000 98.725 95.541 90.687 84.639 77.881
18 100.000 98.957 96.077 91.523 85.738 79.189
19 100.000 99.189 96.585 92.281 86.698 80.298
20 100.000 99.305 96.871 92.751 87.339 81.081
21 100.000 99.421 97.129 93.145 87.848 81.682
22 100.000 99.537 97.415 93.608 88.468 82.427
23 100.000 99.652 97.692 94.049 89.053 83.126
24 100.000 99.768 97.951 94.448 89.579 83.761
25 100.000 99.884 98.265 94.982 90.314 84.657
26 100.000 100.000 98.588 95.552 91.127 85.679

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.820 72.833 61.307 51.003 42.134
5 100.000 91.194 79.692 68.083 57.418 48.050
6 100.000 92.352 81.990 71.189 60.978 51.779
7 100.000 93.511 84.344 74.430 64.744 55.764
8 100.000 94.554 86.430 77.274 68.035 59.243
9 100.000 95.365 88.092 79.588 70.765 62.179
10 100.000 95.944 89.246 81.156 72.574 64.090
11 100.000 96.524 90.464 82.889 74.654 66.359
12 100.000 97.103 91.692 84.646 76.772 68.681
13 100.000 97.567 92.652 86.007 78.416 70.497
14 100.000 98.030 93.686 87.545 80.327 72.641
15 100.000 98.262 94.212 88.349 81.359 73.842
16 100.000 98.494 94.748 89.180 82.437 75.107
17 100.000 98.725 95.329 90.120 83.683 76.578
18 100.000 98.957 95.864 90.942 84.734 77.788
19 100.000 99.073 96.123 91.340 85.256 78.414
20 100.000 99.189 96.428 91.862 85.990 79.334
21 100.000 99.305 96.714 92.331 86.630 80.117
22 100.000 99.421 97.028 92.876 87.400 81.083
23 100.000 99.537 97.314 93.339 88.021 81.828
24 100.000 99.652 97.572 93.737 88.543 82.456
25 100.000 99.768 97.849 94.178 89.128 83.155
26 100.000 99.884 98.209 94.851 90.121 84.433
27 100.000 100.000 98.458 95.226 90.607 85.014

GAcit Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 95.349 85.168 73.442 62.030 51.725 42.789
5 100.000 91.773 80.670 69.201 58.492 48.986
6 100.000 93.048 83.218 72.658 62.459 53.135
7 100.000 94.090 85.332 75.564 65.835 56.707
8 100.000 95.017 87.289 78.345 69.155 60.299
9 100.000 95.944 89.181 80.970 72.239 63.600
10 100.000 96.640 90.630 83.027 74.712 66.312
11 100.000 97.103 91.674 84.599 76.693 68.566
12 100.000 97.567 92.652 86.003 78.398 70.455
13 100.000 97.914 93.474 87.278 80.034 72.338
14 100.000 98.262 94.304 88.576 81.706 74.267
15 100.000 98.494 94.886 89.532 83.001 75.832
16 100.000 98.725 95.421 90.371 84.110 77.157
17 100.000 98.957 95.929 91.137 85.094 78.314
18 100.000 99.073 96.206 91.577 85.678 79.013
19 100.000 99.189 96.465 91.973 86.194 79.627
20 100.000 99.305 96.751 92.436 86.815 80.372
21 100.000 99.421 97.055 92.943 87.497 81.189
22 100.000 99.537 97.332 93.385 88.089 81.903
23 100.000 99.652 97.609 93.830 88.687 82.628
24 100.000 99.768 97.969 94.493 89.650 83.850
25 100.000 99.884 98.265 94.979 90.304 84.634
26 100.000 100.000 98.514 95.353 90.786 85.207

CASA Coverage

ACTS Coverage

GAcit Coverage

CASA Fault Detection

ACTS Fault Detection

GAcit Fault Detection

flex

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 88.372 75.087 62.467 51.349 41.924 34.106
4 93.023 82.271 70.728 59.769 49.962 41.475
5 93.023 84.125 74.310 64.477 55.206 46.808
6 95.349 87.949 79.009 69.580 60.374 51.808
7 100.000 94.322 85.766 76.114 66.423 57.277
8 100.000 95.249 87.677 78.782 69.569 60.652
9 100.000 96.060 89.357 81.144 72.377 63.692
10 100.000 96.408 90.252 82.609 74.321 65.979
11 100.000 97.103 91.664 84.564 76.617 68.444
12 100.000 97.335 92.311 85.674 78.145 70.294
13 100.000 97.683 93.114 86.906 79.719 72.103
14 100.000 98.030 93.908 88.123 81.283 73.921
15 100.000 98.262 94.480 89.034 82.468 75.295
16 100.000 98.494 95.043 89.933 83.655 76.699
17 100.000 98.725 95.588 90.793 84.793 78.058
18 100.000 98.957 96.114 91.602 85.842 79.292
19 100.000 99.073 96.446 92.190 86.674 80.329
20 100.000 99.305 96.954 92.952 87.647 81.464
21 100.000 99.421 97.277 93.522 88.459 82.486
22 100.000 99.537 97.563 93.991 89.100 83.270
23 100.000 99.652 97.821 94.389 89.622 83.896
24 100.000 99.768 98.080 94.783 90.131 84.497
25 100.000 99.884 98.338 95.182 90.657 85.132
26 100.000 100.000 98.588 95.552 91.127 85.679

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.636 64.599 52.953 43.063 34.884
4 93.023 82.155 70.498 59.463 49.624 41.140
5 95.349 86.211 75.612 65.075 55.321 46.643
6 95.349 87.717 78.584 69.059 59.842 51.322
7 100.000 94.206 85.600 75.965 66.327 57.243
8 100.000 94.902 87.113 78.173 69.023 60.219
9 100.000 95.944 89.144 80.874 72.081 63.392
10 100.000 96.756 90.778 83.119 74.700 66.185
11 100.000 97.335 92.015 84.908 76.884 68.610
12 100.000 97.451 92.477 85.829 78.260 70.364
13 100.000 97.798 93.271 87.049 79.831 72.193
14 100.000 98.030 93.871 88.036 81.152 73.762
15 100.000 98.262 94.434 88.936 82.342 75.173
16 100.000 98.494 94.978 89.783 83.437 76.447
17 100.000 98.725 95.495 90.561 84.422 77.578
18 100.000 98.841 95.818 91.132 85.241 78.616
19 100.000 99.073 96.317 91.865 86.155 79.659
20 100.000 99.189 96.612 92.357 86.833 80.494
21 100.000 99.305 96.898 92.821 87.453 81.238
22 100.000 99.421 97.185 93.290 88.094 82.022
23 100.000 99.537 97.443 93.684 88.603 82.623
24 100.000 99.652 97.701 94.079 89.116 83.231
25 100.000 99.768 97.960 94.477 89.638 83.860
26 100.000 99.884 98.209 94.852 90.125 84.440
27 100.000 100.000 98.458 95.226 90.607 85.014

GAcit Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.705 72.667 61.154 50.894 42.076
5 100.000 91.541 80.310 68.829 58.176 48.751
6 100.000 93.048 83.292 72.837 62.734 53.475
7 100.000 94.206 85.646 76.079 66.507 57.472
8 100.000 94.902 87.169 78.313 69.249 60.512
9 100.000 95.597 88.627 80.388 71.742 63.238
10 100.000 96.524 90.437 82.806 74.499 66.123
11 100.000 97.219 91.858 84.791 76.851 68.673
12 100.000 97.567 92.671 86.060 78.507 70.617
13 100.000 97.914 93.483 87.309 80.097 72.433
14 100.000 98.146 94.083 88.296 81.417 74.002
15 100.000 98.378 94.664 89.249 82.703 75.551
16 100.000 98.725 95.421 90.371 84.110 77.157
17 100.000 98.841 95.800 91.081 85.149 78.477
18 100.000 98.957 96.141 91.681 85.983 79.492
19 100.000 99.189 96.649 92.447 86.967 80.649
20 100.000 99.305 96.945 92.938 87.641 81.475
21 100.000 99.421 97.221 93.381 88.232 82.189
22 100.000 99.537 97.498 93.826 88.830 82.914
23 100.000 99.652 97.757 94.217 89.332 83.501
24 100.000 99.768 98.015 94.613 89.848 84.115
25 100.000 99.884 98.265 94.987 90.331 84.688
26 100.000 100.000 98.514 95.353 90.786 85.207

CASA Coverage

ACTS Coverage

GAcit Coverage

CASA Fault Detection

ACTS Fault Detection

GAcit Fault Detection

flex

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 90.698 77.752 64.802 53.194 43.303 35.100
4 93.023 82.271 70.701 59.703 49.864 41.356
5 95.349 86.327 75.833 65.358 55.623 46.932
6 100.000 92.700 82.683 72.113 62.008 52.814
7 100.000 93.859 85.009 75.286 65.670 56.667
8 100.000 94.786 86.920 77.955 68.819 60.048
9 100.000 95.365 88.240 79.944 71.308 62.847
10 100.000 96.060 89.698 82.021 73.808 65.586
11 100.000 96.524 90.741 83.594 75.792 67.846
12 100.000 96.871 91.544 84.837 77.402 69.726
13 100.000 97.335 92.523 86.248 79.127 71.652
14 100.000 97.914 93.741 87.980 81.203 73.916
15 100.000 98.146 94.332 88.949 82.505 75.471
16 100.000 98.378 94.877 89.813 83.653 76.849
17 100.000 98.610 95.440 90.713 84.842 78.257
18 100.000 98.957 96.224 91.892 86.321 79.931
19 100.000 99.189 96.751 92.703 87.376 81.177
20 100.000 99.305 97.037 93.173 88.017 81.961
21 100.000 99.421 97.305 93.597 88.587 82.661
22 100.000 99.537 97.563 93.996 89.113 83.296
23 100.000 99.652 97.821 94.393 89.635 83.922
24 100.000 99.768 98.080 94.788 90.148 84.531
25 100.000 99.884 98.338 95.182 90.657 85.132
26 100.000 100.000 98.588 95.552 91.127 85.679

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 93.023 82.271 70.701 59.703 49.864 41.356
5 95.349 86.674 76.415 66.015 56.247 47.468
6 97.674 90.614 81.316 71.363 61.668 52.709
7 97.674 91.309 82.895 73.728 64.603 55.981
8 100.000 94.902 87.132 78.221 69.102 60.323
9 100.000 95.481 88.369 80.014 71.299 62.773
10 100.000 96.408 90.206 82.503 74.171 65.808
11 100.000 96.640 90.806 83.508 75.547 67.483
12 100.000 96.987 91.627 84.794 77.217 69.430
13 100.000 97.683 93.021 86.704 79.444 71.810
14 100.000 97.914 93.612 87.668 80.729 73.334
15 100.000 98.146 94.212 88.654 82.043 74.889
16 100.000 98.378 94.757 89.500 83.138 76.163
17 100.000 98.610 95.264 90.263 84.112 77.301
18 100.000 98.841 95.781 91.045 85.111 78.462
19 100.000 99.073 96.308 91.845 86.129 79.633
20 100.000 99.189 96.594 92.315 86.770 80.417
21 100.000 99.305 96.871 92.761 87.374 81.155
22 100.000 99.421 97.148 93.205 87.970 81.877
23 100.000 99.537 97.406 93.603 88.493 82.506
24 100.000 99.652 97.674 94.022 89.044 83.165
25 100.000 99.768 97.941 94.439 89.593 83.819
26 100.000 99.884 98.200 94.834 90.105 84.427
27 100.000 100.000 98.458 95.226 90.607 85.014

GAcit Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 88.372 75.087 62.467 51.349 41.924 34.106
4 93.023 82.619 71.310 60.426 50.586 42.011
5 97.674 89.224 78.575 67.685 57.484 48.365
6 97.674 90.267 80.744 70.729 61.080 52.219
7 100.000 93.975 85.230 75.571 65.978 56.969
8 100.000 95.017 87.335 78.460 69.338 60.532
9 100.000 95.597 88.609 80.345 71.680 63.165
10 100.000 95.944 89.440 81.658 73.398 65.182
11 100.000 96.640 90.861 83.642 75.750 67.732
12 100.000 97.567 92.671 86.060 78.507 70.617
13 100.000 97.798 93.261 87.035 79.825 72.204
14 100.000 98.030 93.843 87.987 81.107 73.747
15 100.000 98.378 94.655 89.236 82.697 75.564
16 100.000 98.725 95.412 90.358 84.104 77.170
17 100.000 98.841 95.800 91.081 85.149 78.477
18 100.000 99.073 96.308 91.847 86.133 79.634
19 100.000 99.189 96.649 92.447 86.967 80.649
20 100.000 99.305 96.945 92.938 87.641 81.475
21 100.000 99.421 97.221 93.383 88.239 82.200
22 100.000 99.537 97.498 93.826 88.830 82.914
23 100.000 99.652 97.757 94.222 89.346 83.528
24 100.000 99.768 98.006 94.596 89.829 84.102
25 100.000 99.884 98.265 94.987 90.331 84.688
26 100.000 100.000 98.514 95.353 90.786 85.207

CASA Coverage

ACTS Coverage

GAcit Coverage

CASA Fault Detection

ACTS Fault Detection

GAcit Fault Detection

flex

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 88.372 74.855 62.070 50.890 41.477 33.712
4 90.698 79.722 68.577 58.122 48.761 40.623
5 93.023 83.546 73.304 63.297 54.043 45.772
6 95.349 87.138 77.578 67.879 58.679 50.285
7 95.349 88.297 79.950 71.160 62.502 54.330
8 95.349 89.108 81.686 73.654 65.506 57.607
9 100.000 95.481 88.433 80.164 71.519 63.031
10 100.000 96.408 90.326 82.790 74.603 66.332
11 100.000 96.871 91.341 84.297 76.488 68.474
12 100.000 97.335 92.357 85.801 78.364 70.598
13 100.000 97.683 93.151 87.015 79.919 72.396
14 100.000 97.914 93.741 87.984 81.217 73.944
15 100.000 98.378 94.720 89.390 82.929 75.846
16 100.000 98.610 95.246 90.201 83.985 77.093
17 100.000 98.841 95.791 91.054 85.100 78.406
18 100.000 98.957 96.132 91.669 85.981 79.513
19 100.000 99.189 96.668 92.502 87.069 80.798
20 100.000 99.305 96.945 92.951 87.682 81.556
21 100.000 99.421 97.277 93.532 88.493 82.550
22 100.000 99.537 97.554 93.976 89.087 83.270
23 100.000 99.652 97.821 94.395 89.639 83.929
24 100.000 99.768 98.080 94.788 90.148 84.531
25 100.000 99.884 98.338 95.182 90.657 85.132
26 100.000 100.000 98.588 95.552 91.127 85.679

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 74.855 62.088 50.933 41.540 33.785
4 95.349 84.705 72.667 61.154 50.894 42.076
5 97.674 88.876 77.984 67.007 56.830 47.795
6 97.674 90.035 80.366 70.315 60.704 51.916
7 100.000 93.859 84.990 75.248 65.624 56.625
8 100.000 94.670 86.689 77.656 68.504 59.756
9 100.000 95.597 88.618 80.369 71.719 63.216
10 100.000 95.829 89.237 81.417 73.158 64.966
11 100.000 96.640 90.861 83.642 75.750 67.732
12 100.000 96.756 91.230 84.331 76.760 69.018
13 100.000 97.219 92.246 85.825 78.603 71.078
14 100.000 97.335 92.587 86.441 79.491 72.201
15 100.000 97.798 93.547 87.797 81.118 73.987
16 100.000 98.146 94.295 88.889 82.466 75.503
17 100.000 98.378 94.812 89.671 83.465 76.665
18 100.000 98.610 95.357 90.518 84.560 77.939
19 100.000 99.073 96.308 91.845 86.129 79.633
20 100.000 99.189 96.585 92.289 86.725 80.355
21 100.000 99.305 96.871 92.758 87.365 81.139
22 100.000 99.421 97.148 93.205 87.970 81.877
23 100.000 99.537 97.406 93.603 88.493 82.506
24 100.000 99.652 97.674 94.022 89.044 83.165
25 100.000 99.768 97.941 94.439 89.593 83.819
26 100.000 99.884 98.200 94.834 90.105 84.427
27 100.000 100.000 98.458 95.226 90.607 85.014

GAcit Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 79.070 62.109 48.777 38.407 30.360 24.107
3 86.047 71.958 59.356 48.629 39.714 32.397
4 93.023 81.576 69.556 58.434 48.684 40.368
5 93.023 84.241 74.522 64.740 55.482 47.070
6 95.349 87.949 79.027 69.629 60.456 51.918
7 95.349 88.876 81.012 72.476 63.874 55.628
8 95.349 89.340 82.092 74.140 66.002 58.071
9 97.674 93.048 86.486 78.764 70.577 62.432
10 97.674 93.395 87.326 80.102 72.337 64.506
11 97.674 93.743 88.157 81.406 74.029 66.474
12 100.000 96.987 91.701 84.969 77.478 69.745
13 100.000 97.335 92.495 86.179 79.020 71.518
14 100.000 97.567 93.086 87.151 80.328 73.085
15 100.000 98.262 94.471 89.039 82.523 75.427
16 100.000 98.610 95.218 90.139 83.897 76.994
17 100.000 98.957 96.031 91.388 85.487 78.811
18 100.000 99.073 96.326 91.888 86.189 79.694
19 100.000 99.189 96.621 92.379 86.863 80.521
20 100.000 99.305 96.917 92.870 87.533 81.338
21 100.000 99.421 97.194 93.314 88.131 82.063
22 100.000 99.537 97.471 93.759 88.729 82.788
23 100.000 99.652 97.748 94.200 89.313 83.487
24 100.000 99.768 98.006 94.596 89.829 84.102
25 100.000 99.884 98.265 94.987 90.331 84.688
26 100.000 100.000 98.514 95.353 90.786 85.207

CASA Coverage

ACTS Coverage

GAcit Coverage

CASA Fault Detection

ACTS Fault Detection

GAcit Fault Detection

flex

Generation Strength: 2, Prioritisation Strength: 6

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 90.698 77.520 64.405 52.735 42.856 34.707
4 90.698 79.374 67.950 57.355 47.970 39.883
5 95.349 85.863 75.002 64.350 54.592 45.976
6 95.349 87.370 77.993 68.379 59.182 50.740
7 97.674 91.309 82.913 73.772 64.672 56.066
8 97.674 92.121 84.603 76.158 67.519 59.157
9 97.674 92.584 85.673 77.800 69.611 61.554
10 97.674 93.279 87.104 79.814 72.022 64.192
11 100.000 96.524 90.658 83.403 75.517 67.527
12 100.000 96.756 91.240 84.356 76.802 69.075
13 100.000 97.335 92.440 86.053 78.840 71.312
14 100.000 97.683 93.252 87.304 80.437 73.144
15 100.000 98.030 94.074 88.579 82.073 75.027
16 100.000 98.494 95.071 90.026 83.844 76.995
17 100.000 98.725 95.643 90.948 85.066 78.445
18 100.000 98.841 95.948 91.468 85.794 79.353
19 100.000 99.073 96.501 92.345 86.947 80.716
20 100.000 99.305 97.009 93.107 87.921 81.851
21 100.000 99.421 97.295 93.577 88.561 82.635
22 100.000 99.537 97.554 93.976 89.087 83.270
23 100.000 99.652 97.821 94.395 89.639 83.929
24 100.000 99.768 98.080 94.788 90.148 84.531
25 100.000 99.884 98.338 95.182 90.657 85.132
26 100.000 100.000 98.588 95.552 91.127 85.679

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 79.070 62.109 48.777 38.407 30.360 24.107
3 86.047 71.842 59.143 48.365 39.439 32.136
4 90.698 79.143 67.571 56.938 47.585 39.563
5 93.023 83.546 73.267 63.207 53.906 45.602
6 97.674 90.151 80.522 70.445 60.774 51.923
7 100.000 93.975 85.193 75.487 65.857 56.829
8 100.000 94.670 86.707 77.700 68.569 59.835
9 100.000 95.249 88.037 79.712 71.094 62.680
10 100.000 95.829 89.246 81.434 73.175 64.975
11 100.000 96.060 89.827 82.387 74.460 66.523
12 100.000 96.871 91.461 84.632 77.082 69.322
13 100.000 97.798 93.271 87.052 79.841 72.211
14 100.000 97.914 93.612 87.668 80.729 73.334
15 100.000 98.146 94.175 88.565 81.910 74.727
16 100.000 98.378 94.720 89.416 83.017 76.027
17 100.000 98.610 95.264 90.263 84.112 77.301
18 100.000 98.841 95.781 91.045 85.111 78.462
19 100.000 98.957 96.058 91.489 85.707 79.184
20 100.000 99.189 96.585 92.289 86.725 80.355
21 100.000 99.305 96.861 92.735 87.329 81.093
22 100.000 99.421 97.138 93.182 87.934 81.832
23 100.000 99.537 97.406 93.603 88.493 82.506
24 100.000 99.652 97.674 94.021 89.041 83.159
25 100.000 99.768 97.941 94.438 89.590 83.813
26 100.000 99.884 98.200 94.834 90.105 84.427
27 100.000 100.000 98.458 95.226 90.607 85.014

GAcit Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 76.744 59.328 46.229 36.299 28.706 22.850
3 88.372 74.623 61.673 50.429 41.024 33.307
4 90.698 78.795 66.990 56.281 46.961 39.027
5 90.698 81.228 71.559 62.128 53.312 45.336
6 93.023 84.936 76.064 67.017 58.286 50.184
7 97.674 91.425 83.061 73.884 64.722 56.057
8 97.674 92.121 84.584 76.118 67.464 59.097
9 97.674 92.468 85.424 77.451 69.211 61.148
10 100.000 96.060 89.643 81.900 73.644 65.412
11 100.000 96.756 91.055 83.862 75.961 67.916
12 100.000 97.219 92.052 85.318 77.759 69.936
13 100.000 97.798 93.261 87.036 79.828 72.211
14 100.000 98.146 94.074 88.289 81.432 74.055
15 100.000 98.494 94.831 89.411 82.839 75.661
16 100.000 98.725 95.348 90.198 83.855 76.857
17 100.000 98.841 95.735 90.922 84.900 78.164
18 100.000 98.957 96.114 91.622 85.906 79.419
19 100.000 99.189 96.640 92.423 86.927 80.596
20 100.000 99.305 96.917 92.866 87.519 81.309
21 100.000 99.421 97.194 93.310 88.117 82.035
22 100.000 99.537 97.471 93.751 88.701 82.734
23 100.000 99.652 97.748 94.191 89.286 83.433
24 100.000 99.768 98.006 94.587 89.801 84.047
25 100.000 99.884 98.255 94.961 90.284 84.620
26 100.000 100.000 98.514 95.353 90.786 85.207

CASA Coverage

ACTS Coverage

GAcit Coverage

CASA Fault Detection

ACTS Fault Detection

GAcit Fault Detection

flex

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.705 72.630 61.067 50.763 41.918
5 100.000 91.194 79.682 68.062 57.385 48.011
6 100.000 92.816 82.839 72.239 62.068 52.803
7 100.000 94.090 85.360 75.630 65.937 56.832
8 100.000 95.017 87.243 78.234 68.988 60.095
9 100.000 95.944 89.181 80.972 72.245 63.614
10 100.000 96.640 90.584 82.908 74.515 66.051
11 100.000 97.103 91.600 84.403 76.365 68.122
12 100.000 97.451 92.477 85.819 78.227 70.299
13 100.000 97.798 93.252 86.980 79.686 71.959
14 100.000 98.146 94.009 88.088 81.048 73.476
15 100.000 98.378 94.526 88.877 82.068 74.677
16 100.000 98.494 94.868 89.476 82.901 75.691
17 100.000 98.610 95.218 90.109 83.802 76.813
18 100.000 98.725 95.532 90.651 84.565 77.764
19 100.000 98.841 95.855 91.215 85.357 78.749
20 100.000 98.957 96.104 91.580 85.813 79.267
21 100.000 99.073 96.354 91.946 86.268 79.786
22 100.000 99.189 96.640 92.416 86.909 80.570
23 100.000 99.305 96.954 92.967 87.698 81.573
24 100.000 99.421 97.305 93.613 88.646 82.787
25 100.000 99.537 97.563 94.010 89.165 83.409
26 100.000 99.652 97.812 94.376 89.621 83.927
27 100.000 99.768 98.061 94.741 90.076 84.446
28 100.000 99.884 98.366 95.256 90.785 85.316
29 100.000 100.000 98.689 95.828 91.605 86.355
30 100.000 100.000 98.818 96.174 92.197 87.176
31 100.000 100.000 98.892 96.374 92.542 87.659
32 100.000 100.000 98.902 96.403 92.597 87.746
33 100.000 100.000 98.985 96.646 93.046 88.409
34 100.000 100.000 99.012 96.724 93.186 88.614
35 100.000 100.000 99.022 96.753 93.240 88.699
36 100.000 100.000 99.114 97.014 93.708 89.376
37 100.000 100.000 99.123 97.045 93.773 89.481
38 100.000 100.000 99.132 97.073 93.827 89.566
39 100.000 100.000 99.206 97.277 94.185 90.075
40 100.000 100.000 99.215 97.306 94.240 90.161
41 100.000 100.000 99.225 97.334 94.294 90.246
42 100.000 100.000 99.234 97.364 94.356 90.346
43 100.000 100.000 99.298 97.566 94.746 90.944
44 100.000 100.000 99.308 97.595 94.804 91.037
45 100.000 100.000 99.391 97.827 95.216 91.628
46 100.000 100.000 99.455 98.017 95.566 92.149
47 100.000 100.000 99.548 98.274 96.021 92.803
48 100.000 100.000 99.686 98.660 96.699 93.766
49 100.000 100.000 99.751 98.841 97.024 94.239
50 100.000 100.000 99.797 98.968 97.247 94.560
51 100.000 100.000 99.843 99.106 97.509 94.958
52 100.000 100.000 99.852 99.133 97.560 95.036
53 100.000 100.000 99.862 99.177 97.669 95.235
54 100.000 100.000 99.926 99.357 97.989 95.697
55 100.000 100.000 99.945 99.421 98.123 95.923
56 100.000 100.000 100.000 99.583 98.423 96.367

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 95.349 85.052 73.230 61.767 51.449 42.528
5 100.000 91.657 80.467 68.959 58.249 48.764
6 100.000 93.163 83.412 72.876 62.663 53.305
7 100.000 94.206 85.553 75.854 66.160 57.040
8 100.000 95.249 87.621 78.650 69.369 60.409
9 100.000 96.060 89.329 81.078 72.276 63.568
10 100.000 96.756 90.824 83.242 74.906 66.465
11 100.000 97.219 91.812 84.669 76.648 68.399
12 100.000 97.567 92.587 85.832 78.113 70.072
13 100.000 97.914 93.409 87.124 79.802 72.055
14 100.000 98.262 94.203 88.332 81.336 73.813
15 100.000 98.494 94.840 89.415 82.814 75.589
16 100.000 98.725 95.394 90.299 83.991 76.998
17 100.000 98.957 95.883 91.010 84.873 78.004
18 100.000 99.073 96.215 91.598 85.709 79.049
19 100.000 99.189 96.621 92.370 86.832 80.461
20 100.000 99.305 96.880 92.765 87.345 81.068
21 100.000 99.421 97.129 93.139 87.831 81.648
22 100.000 99.537 97.434 93.654 88.540 82.517
23 100.000 99.652 97.720 94.120 89.167 83.276
24 100.000 99.768 97.969 94.487 89.630 83.811
25 100.000 99.884 98.274 95.012 90.374 84.749
26 100.000 100.000 98.523 95.382 90.843 85.297
27 100.000 100.000 98.625 95.671 91.365 86.054
28 100.000 100.000 98.717 95.939 91.854 86.773
29 100.000 100.000 98.735 95.997 91.969 86.956
30 100.000 100.000 98.846 96.296 92.482 87.671
31 100.000 100.000 98.865 96.359 92.612 87.882
32 100.000 100.000 98.948 96.591 93.024 88.475
33 100.000 100.000 98.957 96.618 93.075 88.554
34 100.000 100.000 98.966 96.645 93.126 88.631
35 100.000 100.000 98.975 96.675 93.187 88.731
36 100.000 100.000 98.985 96.708 93.255 88.842
37 100.000 100.000 99.012 96.795 93.426 89.110
38 100.000 100.000 99.022 96.823 93.481 89.197
39 100.000 100.000 99.058 96.930 93.675 89.485
40 100.000 100.000 99.077 96.983 93.773 89.630
41 100.000 100.000 99.105 97.071 93.949 89.909
42 100.000 100.000 99.160 97.233 94.247 90.353
43 100.000 100.000 99.206 97.366 94.491 90.716
44 100.000 100.000 99.252 97.498 94.731 91.068
45 100.000 100.000 99.289 97.612 94.950 91.408
46 100.000 100.000 99.317 97.696 95.112 91.657
47 100.000 100.000 99.363 97.835 95.373 92.055
48 100.000 100.000 99.418 97.992 95.657 92.471
49 100.000 100.000 99.502 98.226 96.077 93.079
50 100.000 100.000 99.511 98.254 96.131 93.165
51 100.000 100.000 99.520 98.284 96.189 93.258
52 100.000 100.000 99.548 98.369 96.355 93.517
53 100.000 100.000 99.585 98.477 96.557 93.820
54 100.000 100.000 99.603 98.530 96.654 93.965
55 100.000 100.000 99.649 98.669 96.916 94.363
56 100.000 100.000 99.732 98.902 97.331 94.959
57 100.000 100.000 99.742 98.938 97.410 95.097
58 100.000 100.000 99.797 99.095 97.695 95.516
59 100.000 100.000 99.825 99.177 97.849 95.751
60 100.000 100.000 99.834 99.214 97.933 95.900
61 100.000 100.000 99.880 99.345 98.170 96.246
62 100.000 100.000 99.889 99.372 98.221 96.324
63 100.000 100.000 99.935 99.498 98.441 96.639
64 100.000 100.000 99.963 99.582 98.604 96.892
65 100.000 100.000 100.000 99.691 98.806 97.195
66 100.000 100.000 100.000 99.699 98.835 97.259

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 95.349 84.936 73.064 61.614 51.341 42.469
5 95.349 86.674 76.461 66.123 56.407 47.660
6 100.000 93.163 83.504 73.097 63.000 53.719
7 100.000 94.206 85.664 76.119 66.560 57.527
8 100.000 94.786 87.012 78.176 69.153 60.456
9 100.000 95.365 88.286 80.056 71.478 63.056
10 100.000 96.060 89.726 82.089 73.912 65.715
11 100.000 96.871 91.314 84.224 76.367 68.310
12 100.000 97.335 92.338 85.748 78.269 70.460
13 100.000 97.567 92.947 86.766 79.653 72.127
14 100.000 97.914 93.732 87.958 81.171 73.877
15 100.000 98.146 94.323 88.923 82.459 75.406
16 100.000 98.262 94.711 89.656 83.531 76.764
17 100.000 98.378 95.089 90.362 84.556 78.058
18 100.000 98.494 95.449 91.011 85.472 79.182
19 100.000 98.494 95.606 91.432 86.192 80.180
20 100.000 98.610 95.938 92.030 87.058 81.283
21 100.000 98.841 96.455 92.820 88.080 82.488
22 100.000 98.957 96.760 93.349 88.838 83.456
23 100.000 99.073 97.055 93.858 89.569 84.395
24 100.000 99.073 97.166 94.170 90.123 85.190
25 100.000 99.189 97.443 94.621 90.741 85.956
26 100.000 99.189 97.526 94.860 91.177 86.595
27 100.000 99.305 97.794 95.289 91.762 87.319
28 100.000 99.421 98.052 95.685 92.278 87.933
29 100.000 99.537 98.311 96.086 92.810 88.579
30 100.000 99.652 98.569 96.490 93.353 89.244
31 100.000 99.768 98.818 96.864 93.836 89.819
32 100.000 99.768 98.883 97.052 94.183 90.334
33 100.000 99.884 99.132 97.420 94.646 90.871
34 100.000 99.884 99.197 97.603 94.977 91.356
35 100.000 100.000 99.446 97.969 95.433 91.878
36 100.000 100.000 99.511 98.149 95.753 92.340
37 100.000 100.000 99.566 98.312 96.052 92.783
38 100.000 100.000 99.622 98.466 96.325 93.175
39 100.000 100.000 99.668 98.593 96.549 93.497
40 100.000 100.000 99.714 98.723 96.783 93.838
41 100.000 100.000 99.760 98.851 97.010 94.166
42 100.000 100.000 99.806 98.990 97.272 94.565
43 100.000 100.000 99.843 99.092 97.452 94.822
44 100.000 100.000 99.880 99.195 97.635 95.087
45 100.000 100.000 99.898 99.262 97.780 95.332
46 100.000 100.000 99.908 99.302 97.878 95.511
47 100.000 100.000 99.917 99.330 97.933 95.597
48 100.000 100.000 99.926 99.357 97.984 95.676
49 100.000 100.000 99.935 99.385 98.038 95.761
50 100.000 100.000 99.945 99.414 98.093 95.848
51 100.000 100.000 99.954 99.442 98.147 95.933
52 100.000 100.000 99.963 99.471 98.206 96.027
53 100.000 100.000 99.972 99.499 98.260 96.112
54 100.000 100.000 99.982 99.528 98.318 96.205
55 100.000 100.000 99.991 99.556 98.372 96.290
56 100.000 100.000 100.000 99.583 98.423 96.367

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 95.349 84.936 73.064 61.614 51.341 42.469
5 100.000 91.657 80.532 69.113 58.478 49.040
6 100.000 93.279 83.744 73.424 63.368 54.087
7 100.000 94.322 85.876 76.379 66.825 57.770
8 100.000 95.133 87.593 78.835 69.784 61.004
9 100.000 95.944 89.264 81.175 72.559 64.005
10 100.000 96.408 90.344 82.844 74.700 66.473
11 100.000 96.871 91.378 84.395 76.652 68.694
12 100.000 97.103 92.015 85.486 78.157 70.523
13 100.000 97.567 93.012 86.933 79.924 72.485
14 100.000 97.914 93.797 88.120 81.430 74.210
15 100.000 98.262 94.572 89.287 82.905 75.902
16 100.000 98.610 95.338 90.428 84.338 77.535
17 100.000 98.725 95.698 91.097 85.318 78.788
18 100.000 98.841 96.040 91.707 86.186 79.870
19 100.000 98.957 96.381 92.319 87.058 80.961
20 100.000 99.073 96.723 92.934 87.942 82.076
21 100.000 99.189 97.037 93.476 88.702 83.021
22 100.000 99.189 97.166 93.829 89.314 83.882
23 100.000 99.305 97.452 94.292 89.935 84.627
24 100.000 99.421 97.738 94.768 90.597 85.455
25 100.000 99.421 97.831 95.026 91.051 86.105
26 100.000 99.421 97.914 95.264 91.484 86.737
27 100.000 99.421 97.988 95.481 91.883 87.330
28 100.000 99.537 98.246 95.882 92.416 87.977
29 100.000 99.652 98.495 96.253 92.889 88.534
30 100.000 99.768 98.745 96.620 93.352 89.069
31 100.000 99.884 98.994 96.992 93.829 89.634
32 100.000 100.000 99.243 97.368 94.319 90.220
33 100.000 100.000 99.298 97.534 94.630 90.690
34 100.000 100.000 99.354 97.693 94.922 91.123
35 100.000 100.000 99.400 97.832 95.183 91.519
36 100.000 100.000 99.446 97.968 95.437 91.901
37 100.000 100.000 99.492 98.102 95.686 92.275
38 100.000 100.000 99.538 98.241 95.947 92.669
39 100.000 100.000 99.585 98.383 96.217 93.080
40 100.000 100.000 99.622 98.499 96.446 93.437
41 100.000 100.000 99.658 98.613 96.664 93.775
42 100.000 100.000 99.695 98.723 96.873 94.095
43 100.000 100.000 99.723 98.810 97.043 94.361
44 100.000 100.000 99.751 98.895 97.208 94.617
45 100.000 100.000 99.778 98.980 97.374 94.877
46 100.000 100.000 99.806 99.063 97.529 95.113
47 100.000 100.000 99.825 99.120 97.642 95.295
48 100.000 100.000 99.843 99.181 97.768 95.500
49 100.000 100.000 99.862 99.242 97.890 95.697
50 100.000 100.000 99.871 99.273 97.957 95.809
51 100.000 100.000 99.880 99.301 98.011 95.895
52 100.000 100.000 99.889 99.328 98.063 95.976
53 100.000 100.000 99.898 99.356 98.117 96.061
54 100.000 100.000 99.908 99.383 98.168 96.141
55 100.000 100.000 99.917 99.412 98.223 96.226
56 100.000 100.000 99.926 99.445 98.296 96.351
57 100.000 100.000 99.935 99.479 98.372 96.484
58 100.000 100.000 99.945 99.506 98.423 96.563
59 100.000 100.000 99.954 99.540 98.496 96.686
60 100.000 100.000 99.963 99.567 98.546 96.763
61 100.000 100.000 99.972 99.594 98.597 96.841
62 100.000 100.000 99.982 99.630 98.681 96.989
63 100.000 100.000 99.991 99.657 98.732 97.067
64 100.000 100.000 100.000 99.684 98.783 97.145
65 100.000 100.000 100.000 99.691 98.806 97.195
66 100.000 100.000 100.000 99.699 98.835 97.259

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 93.023 82.851 71.707 60.886 51.033 42.404
5 97.674 89.455 78.981 68.166 57.964 48.798
6 100.000 93.279 83.689 73.295 63.177 53.862
7 100.000 94.322 85.858 76.341 66.776 57.721
8 100.000 94.786 86.984 78.113 69.065 60.355
9 100.000 95.713 88.867 80.718 72.119 63.623
10 100.000 96.292 90.095 82.490 74.284 66.034
11 100.000 97.103 91.720 84.715 76.876 68.800
12 100.000 97.335 92.338 85.758 78.302 70.524
13 100.000 97.451 92.726 86.496 79.390 71.914
14 100.000 97.683 93.307 87.443 80.659 73.432
15 100.000 97.683 93.474 87.889 81.420 74.484
16 100.000 97.798 93.806 88.488 82.288 75.592
17 100.000 98.146 94.572 89.626 83.712 77.207
18 100.000 98.262 94.923 90.264 84.630 78.364
19 100.000 98.378 95.255 90.860 85.489 79.456
20 100.000 98.494 95.597 91.474 86.370 80.563
21 100.000 98.494 95.726 91.824 86.973 81.407
22 100.000 98.494 95.837 92.129 87.506 82.162
23 100.000 98.610 96.141 92.653 88.246 83.092
24 100.000 98.841 96.658 93.436 89.247 84.253
25 100.000 98.957 96.945 93.913 89.913 85.090
26 100.000 99.073 97.249 94.430 90.629 85.971
27 100.000 99.305 97.738 95.141 91.513 86.982
28 100.000 99.305 97.803 95.332 91.867 87.509
29 100.000 99.305 97.868 95.521 92.217 88.031
30 100.000 99.305 97.923 95.687 92.527 88.493
31 100.000 99.305 97.978 95.853 92.836 88.955
32 100.000 99.305 98.034 96.018 93.146 89.422
33 100.000 99.305 98.098 96.201 93.474 89.903
34 100.000 99.305 98.154 96.358 93.760 90.326
35 100.000 99.305 98.191 96.471 93.980 90.667
36 100.000 99.305 98.237 96.606 94.226 91.031
37 100.000 99.305 98.274 96.718 94.442 91.363
38 100.000 99.421 98.523 97.088 94.912 91.913
39 100.000 99.537 98.772 97.457 95.378 92.456
40 100.000 99.537 98.818 97.586 95.608 92.788
41 100.000 99.652 99.068 97.953 96.071 93.324
42 100.000 99.768 99.317 98.321 96.534 93.859
43 100.000 99.884 99.566 98.689 96.997 94.395
44 100.000 99.884 99.612 98.816 97.221 94.716
45 100.000 100.000 99.862 99.182 97.680 95.243
46 100.000 100.000 99.908 99.309 97.904 95.565
47 100.000 100.000 99.917 99.338 97.958 95.652
48 100.000 100.000 99.926 99.366 98.013 95.739
49 100.000 100.000 99.935 99.394 98.068 95.824
50 100.000 100.000 99.945 99.421 98.118 95.901
51 100.000 100.000 99.954 99.448 98.169 95.978
52 100.000 100.000 99.963 99.475 98.220 96.055
53 100.000 100.000 99.972 99.502 98.270 96.132
54 100.000 100.000 99.982 99.529 98.321 96.209
55 100.000 100.000 99.991 99.556 98.372 96.288
56 100.000 100.000 100.000 99.583 98.423 96.367

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 90.698 77.752 64.802 53.194 43.303 35.100
4 95.349 84.936 73.064 61.614 51.341 42.469
5 95.349 86.790 76.673 66.387 56.683 47.922
6 95.349 87.717 78.713 69.364 60.294 51.863
7 100.000 94.090 85.489 75.942 66.411 57.416
8 100.000 94.902 87.187 78.353 69.301 60.566
9 100.000 95.713 88.849 80.676 72.057 63.552
10 100.000 96.176 89.901 82.275 74.090 65.882
11 100.000 96.640 90.935 83.825 76.038 68.098
12 100.000 97.335 92.347 85.786 78.355 70.604
13 100.000 97.683 93.151 87.022 79.940 72.436
14 100.000 97.914 93.714 87.933 81.166 73.917
15 100.000 98.146 94.304 88.902 82.464 75.465
16 100.000 98.262 94.637 89.502 83.340 76.588
17 100.000 98.378 94.960 90.081 84.180 77.667
18 100.000 98.610 95.523 90.979 85.367 79.073
19 100.000 98.610 95.652 91.335 85.987 79.950
20 100.000 98.725 95.994 91.945 86.855 81.035
21 100.000 98.841 96.317 92.511 87.658 82.041
22 100.000 99.073 96.815 93.254 88.605 83.149
23 100.000 99.073 96.945 93.600 89.195 83.966
24 100.000 99.073 97.037 93.860 89.660 84.637
25 100.000 99.189 97.323 94.334 90.315 85.451
26 100.000 99.189 97.397 94.546 90.696 86.004
27 100.000 99.189 97.480 94.778 91.108 86.599
28 100.000 99.305 97.775 95.269 91.782 87.423
29 100.000 99.305 97.840 95.457 92.124 87.927
30 100.000 99.305 97.886 95.599 92.396 88.343
31 100.000 99.305 97.941 95.760 92.693 88.786
32 100.000 99.421 98.200 96.159 93.219 89.420
33 100.000 99.537 98.458 96.558 93.746 90.056
34 100.000 99.537 98.486 96.653 93.941 90.371
35 100.000 99.537 98.541 96.810 94.225 90.787
36 100.000 99.537 98.588 96.943 94.469 91.148
37 100.000 99.537 98.634 97.076 94.713 91.509
38 100.000 99.652 98.883 97.448 95.189 92.071
39 100.000 99.652 98.920 97.560 95.401 92.396
40 100.000 99.652 98.948 97.646 95.570 92.661
41 100.000 99.768 99.197 98.014 96.034 93.201
42 100.000 99.884 99.446 98.380 96.494 93.730
43 100.000 100.000 99.695 98.747 96.954 94.259
44 100.000 100.000 99.742 98.874 97.178 94.581
45 100.000 100.000 99.769 98.958 97.340 94.834
46 100.000 100.000 99.806 99.064 97.531 95.115
47 100.000 100.000 99.825 99.126 97.661 95.329
48 100.000 100.000 99.852 99.207 97.812 95.557
49 100.000 100.000 99.862 99.244 97.896 95.705
50 100.000 100.000 99.871 99.280 97.980 95.854
51 100.000 100.000 99.880 99.314 98.053 95.981
52 100.000 100.000 99.889 99.347 98.128 96.112
53 100.000 100.000 99.898 99.380 98.200 96.235
54 100.000 100.000 99.908 99.413 98.270 96.353
55 100.000 100.000 99.908 99.421 98.299 96.417
56 100.000 100.000 99.917 99.449 98.353 96.502
57 100.000 100.000 99.926 99.476 98.404 96.581
58 100.000 100.000 99.935 99.503 98.456 96.660
59 100.000 100.000 99.945 99.530 98.507 96.740
60 100.000 100.000 99.954 99.557 98.558 96.819
61 100.000 100.000 99.963 99.584 98.609 96.897
62 100.000 100.000 99.972 99.612 98.660 96.976
63 100.000 100.000 99.982 99.639 98.711 97.054
64 100.000 100.000 99.991 99.666 98.762 97.132
65 100.000 100.000 100.000 99.693 98.812 97.209
66 100.000 100.000 100.000 99.699 98.835 97.259

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 79.070 62.109 48.777 38.407 30.360 24.107
3 86.047 71.958 59.356 48.629 39.714 32.397
4 90.698 79.374 68.014 57.508 48.199 40.160
5 90.698 80.997 71.199 61.756 52.996 45.100
6 90.698 82.155 73.608 65.129 56.963 49.328
7 93.023 85.747 77.818 69.558 61.369 53.563
8 97.674 92.121 84.575 76.091 67.414 59.026
9 97.674 92.932 86.255 78.455 70.229 62.078
10 100.000 96.292 90.049 82.374 74.101 65.801
11 100.000 96.524 90.704 83.507 75.665 67.695
12 100.000 97.103 91.914 85.233 77.757 70.014
13 100.000 97.451 92.744 86.541 79.463 72.007
14 100.000 97.451 92.957 87.105 80.412 73.305
15 100.000 97.451 93.151 87.623 81.290 74.511
16 100.000 97.567 93.492 88.240 82.182 75.642
17 100.000 97.798 94.055 89.144 83.385 77.078
18 100.000 97.914 94.424 89.826 84.372 78.321
19 100.000 98.030 94.720 90.333 85.094 79.242
20 100.000 98.030 94.803 90.575 85.540 79.902
21 100.000 98.262 95.320 91.360 86.548 81.082
22 100.000 98.494 95.864 92.213 87.662 82.396
23 100.000 98.725 96.381 93.000 88.678 83.588
24 100.000 98.725 96.474 93.256 89.130 84.234
25 100.000 98.841 96.806 93.845 89.964 85.274
26 100.000 98.841 96.880 94.054 90.341 85.824
27 100.000 98.841 96.926 94.199 90.623 86.260
28 100.000 98.841 97.000 94.404 90.986 86.784
29 100.000 98.841 97.046 94.539 91.239 87.166
30 100.000 98.957 97.351 95.061 91.970 88.077
31 100.000 99.073 97.600 95.440 92.471 88.688
32 100.000 99.073 97.637 95.557 92.699 89.043
33 100.000 99.189 97.905 95.983 93.275 89.749
34 100.000 99.305 98.172 96.410 93.853 90.461
35 100.000 99.305 98.228 96.572 94.153 90.906
36 100.000 99.305 98.237 96.613 94.251 91.085
37 100.000 99.421 98.495 97.009 94.768 91.701
38 100.000 99.421 98.541 97.137 94.995 92.030
39 100.000 99.537 98.791 97.507 95.465 92.580
40 100.000 99.537 98.837 97.634 95.689 92.901
41 100.000 99.652 99.086 98.001 96.152 93.437
42 100.000 99.768 99.335 98.369 96.615 93.972
43 100.000 99.884 99.585 98.736 97.074 94.499
44 100.000 100.000 99.834 99.102 97.533 95.026
45 100.000 100.000 99.871 99.205 97.716 95.291
46 100.000 100.000 99.908 99.307 97.896 95.548
47 100.000 100.000 99.917 99.337 97.954 95.642
48 100.000 100.000 99.926 99.365 98.009 95.729
49 100.000 100.000 99.935 99.393 98.064 95.816
50 100.000 100.000 99.945 99.421 98.118 95.902
51 100.000 100.000 99.954 99.448 98.170 95.981
52 100.000 100.000 99.963 99.475 98.220 96.058
53 100.000 100.000 99.972 99.502 98.271 96.136
54 100.000 100.000 99.982 99.529 98.322 96.213
55 100.000 100.000 99.991 99.556 98.372 96.290
56 100.000 100.000 100.000 99.583 98.423 96.367

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 74.623 61.691 50.473 41.093 33.392
4 93.023 82.155 70.516 59.508 49.693 41.225
5 93.023 83.778 73.710 63.777 54.520 46.199
6 97.674 90.151 80.550 70.510 60.872 52.042
7 97.674 90.962 82.276 72.987 63.861 55.310
8 100.000 94.786 86.938 78.005 68.904 60.164
9 100.000 95.481 88.433 80.171 71.541 63.073
10 100.000 95.944 89.514 81.835 73.666 65.510
11 100.000 96.756 91.194 84.197 76.474 68.550
12 100.000 97.103 92.006 85.458 78.104 70.446
13 100.000 97.335 92.578 86.397 79.380 71.997
14 100.000 97.567 93.132 87.284 80.567 73.428
15 100.000 97.798 93.686 88.168 81.745 74.839
16 100.000 97.914 94.018 88.766 82.610 75.943
17 100.000 98.146 94.554 89.605 83.716 77.262
18 100.000 98.378 95.098 90.466 84.856 78.625
19 100.000 98.610 95.661 91.366 86.047 80.038
20 100.000 98.725 96.031 92.047 87.030 81.272
21 100.000 98.725 96.151 92.376 87.603 82.080
22 100.000 98.841 96.455 92.899 88.341 83.010
23 100.000 98.841 96.538 93.132 88.758 83.615
24 100.000 98.841 96.603 93.320 89.103 84.127
25 100.000 99.073 97.101 94.059 90.040 85.217
26 100.000 99.073 97.166 94.243 90.375 85.709
27 100.000 99.189 97.471 94.762 91.098 86.607
28 100.000 99.189 97.554 94.994 91.510 87.199
29 100.000 99.305 97.831 95.442 92.119 87.947
30 100.000 99.305 97.895 95.629 92.458 88.445
31 100.000 99.305 97.960 95.812 92.788 88.925
32 100.000 99.305 98.006 95.952 93.052 89.325
33 100.000 99.305 98.052 96.083 93.290 89.679
34 100.000 99.305 98.108 96.240 93.574 90.095
35 100.000 99.305 98.145 96.350 93.782 90.411
36 100.000 99.305 98.181 96.463 93.997 90.739
37 100.000 99.305 98.209 96.550 94.169 91.009
38 100.000 99.305 98.237 96.635 94.331 91.261
39 100.000 99.305 98.255 96.696 94.458 91.468
40 100.000 99.421 98.505 97.065 94.925 92.014
41 100.000 99.421 98.532 97.151 95.093 92.276
42 100.000 99.537 98.782 97.519 95.557 92.815
43 100.000 99.537 98.800 97.581 95.683 93.020
44 100.000 99.537 98.809 97.615 95.761 93.157
45 100.000 99.537 98.828 97.676 95.886 93.360
46 100.000 99.652 99.077 98.044 96.350 93.896
47 100.000 99.768 99.326 98.411 96.809 94.426
48 100.000 99.884 99.575 98.777 97.269 94.955
49 100.000 99.884 99.585 98.812 97.346 95.088
50 100.000 100.000 99.834 99.177 97.802 95.611
51 100.000 100.000 99.852 99.236 97.917 95.794
52 100.000 100.000 99.862 99.271 97.997 95.932
53 100.000 100.000 99.889 99.352 98.148 96.159
54 100.000 100.000 99.898 99.385 98.221 96.284
55 100.000 100.000 99.908 99.419 98.293 96.407
56 100.000 100.000 99.908 99.427 98.323 96.471
57 100.000 100.000 99.917 99.454 98.374 96.550
58 100.000 100.000 99.926 99.481 98.425 96.630
59 100.000 100.000 99.935 99.509 98.480 96.715
60 100.000 100.000 99.945 99.537 98.530 96.793
61 100.000 100.000 99.954 99.564 98.581 96.871
62 100.000 100.000 99.963 99.591 98.632 96.950
63 100.000 100.000 99.972 99.618 98.683 97.027
64 100.000 100.000 99.982 99.645 98.734 97.105
65 100.000 100.000 99.991 99.672 98.785 97.182
66 100.000 100.000 100.000 99.699 98.835 97.259

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 3, Prioritisation Strength: 6

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 79.070 62.109 48.777 38.407 30.360 24.107
3 88.372 74.855 62.088 50.933 41.540 33.785
4 93.023 82.039 70.350 59.352 49.577 41.154
5 93.023 83.430 73.138 63.141 53.928 45.702
6 93.023 84.241 74.919 65.748 57.109 49.201
7 97.674 90.614 81.667 72.260 63.125 54.630
8 97.674 91.425 83.347 74.624 65.940 57.682
9 97.674 91.657 83.984 75.719 67.454 59.528
10 97.674 92.352 85.443 77.797 69.957 62.274
11 97.674 92.816 86.513 79.437 72.046 64.665
12 97.674 93.163 87.307 80.655 73.610 66.482
13 97.674 93.395 87.907 81.655 74.971 68.127
14 100.000 96.640 91.434 85.172 78.348 71.307
15 100.000 97.103 92.403 86.557 80.031 73.178
16 100.000 97.335 92.957 87.436 81.192 74.559
17 100.000 97.683 93.704 88.534 82.560 76.116
18 100.000 97.914 94.240 89.370 83.659 77.422
19 100.000 98.030 94.563 89.938 84.466 78.436
20 100.000 98.030 94.637 90.153 84.862 79.023
21 100.000 98.146 94.960 90.721 85.669 80.038
22 100.000 98.494 95.744 91.907 87.171 81.756
23 100.000 98.725 96.289 92.763 88.295 83.088
24 100.000 98.725 96.391 93.045 88.790 83.793
25 100.000 98.725 96.465 93.255 89.170 84.350
26 100.000 98.725 96.566 93.536 89.664 85.054
27 100.000 98.725 96.612 93.678 89.936 85.471
28 100.000 98.957 97.157 94.527 91.037 86.757
29 100.000 98.957 97.203 94.671 91.315 87.185
30 100.000 98.957 97.249 94.805 91.564 87.561
31 100.000 98.957 97.323 95.012 91.934 88.095
32 100.000 99.073 97.572 95.389 92.428 88.694
33 100.000 99.073 97.609 95.504 92.650 89.036
34 100.000 99.189 97.877 95.932 93.231 89.754
35 100.000 99.189 97.932 96.094 93.531 90.198
36 100.000 99.305 98.237 96.616 94.263 91.109
37 100.000 99.305 98.283 96.744 94.490 91.438
38 100.000 99.305 98.329 96.874 94.724 91.779
39 100.000 99.305 98.375 97.001 94.948 92.100
40 100.000 99.421 98.625 97.369 95.411 92.637
41 100.000 99.537 98.874 97.737 95.874 93.172
42 100.000 99.652 99.123 98.104 96.336 93.707
43 100.000 99.768 99.372 98.472 96.799 94.243
44 100.000 99.768 99.409 98.578 96.993 94.528
45 100.000 99.884 99.658 98.945 97.453 95.058
46 100.000 100.000 99.908 99.312 97.912 95.585
47 100.000 100.000 99.917 99.340 97.967 95.671
48 100.000 100.000 99.926 99.367 98.018 95.749
49 100.000 100.000 99.935 99.394 98.068 95.827
50 100.000 100.000 99.945 99.421 98.119 95.905
51 100.000 100.000 99.954 99.448 98.170 95.982
52 100.000 100.000 99.963 99.475 98.220 96.059
53 100.000 100.000 99.972 99.502 98.271 96.136
54 100.000 100.000 99.982 99.529 98.322 96.213
55 100.000 100.000 99.991 99.556 98.372 96.290
56 100.000 100.000 100.000 99.583 98.423 96.367

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 76.744 59.328 46.229 36.299 28.706 22.850
3 86.047 71.611 58.728 47.861 38.923 31.658
4 90.698 78.679 66.759 55.975 46.623 38.692
5 93.023 82.851 72.058 61.784 52.497 44.338
6 95.349 87.138 77.559 67.835 58.611 50.200
7 95.349 87.833 79.092 70.089 61.382 53.274
8 95.349 88.992 81.409 73.244 65.024 57.113
9 95.349 89.571 82.692 75.149 67.393 59.774
10 97.674 93.048 86.689 79.310 71.508 63.718
11 100.000 96.408 90.492 83.255 75.425 67.504
12 100.000 96.987 91.720 85.025 77.584 69.903
13 100.000 97.567 92.947 86.793 79.740 72.296
14 100.000 97.683 93.307 87.461 80.717 73.545
15 100.000 98.030 94.101 88.673 82.265 75.331
16 100.000 98.146 94.434 89.271 83.131 76.436
17 100.000 98.262 94.784 89.910 84.053 77.601
18 100.000 98.494 95.311 90.721 85.109 78.849
19 100.000 98.494 95.431 91.050 85.682 79.658
20 100.000 98.494 95.551 91.378 86.252 80.460
21 100.000 98.725 96.058 92.142 87.229 81.603
22 100.000 98.725 96.169 92.446 87.760 82.354
23 100.000 98.957 96.741 93.365 88.975 83.790
24 100.000 98.957 96.806 93.555 89.326 84.312
25 100.000 98.957 96.889 93.785 89.732 84.894
26 100.000 99.073 97.166 94.240 90.365 85.690
27 100.000 99.073 97.240 94.449 90.738 86.232
28 100.000 99.073 97.332 94.705 91.189 86.874
29 100.000 99.073 97.388 94.867 91.489 87.322
30 100.000 99.073 97.415 94.954 91.659 87.591
31 100.000 99.073 97.471 95.112 91.947 88.015
32 100.000 99.073 97.480 95.155 92.051 88.203
33 100.000 99.189 97.748 95.580 92.623 88.904
34 100.000 99.189 97.794 95.713 92.868 89.268
35 100.000 99.305 98.052 96.117 93.411 89.936
36 100.000 99.305 98.098 96.249 93.652 90.292
37 100.000 99.305 98.145 96.383 93.896 90.653
38 100.000 99.305 98.163 96.443 94.019 90.853
39 100.000 99.421 98.421 96.840 94.538 91.475
40 100.000 99.421 98.458 96.949 94.743 91.784
41 100.000 99.421 98.495 97.059 94.947 92.092
42 100.000 99.421 98.505 97.093 95.024 92.225
43 100.000 99.421 98.514 97.130 95.107 92.370
44 100.000 99.421 98.541 97.218 95.282 92.643
45 100.000 99.537 98.791 97.587 95.748 93.187
46 100.000 99.652 99.040 97.958 96.221 93.743
47 100.000 99.652 99.086 98.084 96.442 94.057
48 100.000 99.768 99.335 98.449 96.898 94.581
49 100.000 99.884 99.585 98.816 97.358 95.110
50 100.000 100.000 99.834 99.182 97.818 95.640
51 100.000 100.000 99.843 99.218 97.897 95.778
52 100.000 100.000 99.871 99.299 98.049 96.007
53 100.000 100.000 99.880 99.330 98.115 96.119
54 100.000 100.000 99.908 99.412 98.266 96.346
55 100.000 100.000 99.908 99.420 98.296 96.410
56 100.000 100.000 99.908 99.426 98.318 96.460
57 100.000 100.000 99.917 99.454 98.373 96.547
58 100.000 100.000 99.926 99.481 98.424 96.626
59 100.000 100.000 99.935 99.509 98.479 96.712
60 100.000 100.000 99.945 99.537 98.530 96.791
61 100.000 100.000 99.954 99.564 98.581 96.871
62 100.000 100.000 99.963 99.591 98.632 96.949
63 100.000 100.000 99.972 99.618 98.683 97.027
64 100.000 100.000 99.982 99.645 98.734 97.105
65 100.000 100.000 99.991 99.672 98.785 97.182
66 100.000 100.000 100.000 99.699 98.835 97.259

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 93.023 82.039 70.313 59.265 49.446 40.996
5 97.674 88.528 77.301 66.105 55.833 46.800
6 100.000 92.468 82.092 71.178 60.821 51.490
7 100.000 93.743 84.676 74.726 64.932 55.821
8 100.000 94.902 86.993 77.877 68.561 59.635
9 100.000 95.944 89.135 80.856 72.058 63.370
10 100.000 96.871 91.046 83.530 75.224 66.781
11 100.000 97.335 92.034 84.961 76.978 68.741
12 100.000 97.798 93.086 86.552 78.982 71.017
13 100.000 98.146 93.926 87.885 80.730 73.068
14 100.000 98.378 94.526 88.880 82.075 74.683
15 100.000 98.610 95.071 89.731 83.182 75.982
16 100.000 98.725 95.403 90.325 84.037 77.064
17 100.000 98.841 95.671 90.746 84.596 77.738
18 100.000 98.957 95.948 91.192 85.198 78.471
19 100.000 99.073 96.224 91.640 85.807 79.220
20 100.000 99.189 96.548 92.205 86.606 80.219
21 100.000 99.305 96.880 92.791 87.432 81.246
22 100.000 99.421 97.148 93.216 88.004 81.945
23 100.000 99.537 97.415 93.634 88.556 82.605
24 100.000 99.652 97.665 94.000 89.011 83.123
25 100.000 99.768 97.951 94.463 89.631 83.868
26 100.000 99.884 98.200 94.837 90.114 84.442
27 100.000 100.000 98.458 95.231 90.623 85.043
28 100.000 100.000 98.551 95.491 91.088 85.714
29 100.000 100.000 98.588 95.593 91.268 85.971
30 100.000 100.000 98.625 95.700 91.462 86.258
31 100.000 100.000 98.708 95.932 91.874 86.852
32 100.000 100.000 98.828 96.257 92.434 87.636
33 100.000 100.000 98.855 96.342 92.599 87.890
34 100.000 100.000 98.911 96.507 92.906 88.352
35 100.000 100.000 98.957 96.644 93.165 88.743
36 100.000 100.000 99.003 96.779 93.412 89.111
37 100.000 100.000 99.012 96.809 93.474 89.210
38 100.000 100.000 99.049 96.921 93.688 89.537
39 100.000 100.000 99.068 96.976 93.792 89.697
40 100.000 100.000 99.086 97.040 93.926 89.918
41 100.000 100.000 99.142 97.195 94.203 90.320
42 100.000 100.000 99.188 97.327 94.443 90.672
43 100.000 100.000 99.234 97.461 94.687 91.034
44 100.000 100.000 99.252 97.520 94.806 91.226
45 100.000 100.000 99.289 97.629 95.012 91.540
46 100.000 100.000 99.335 97.768 95.274 91.940
47 100.000 100.000 99.382 97.905 95.533 92.333
48 100.000 100.000 99.391 97.934 95.588 92.420
49 100.000 100.000 99.418 98.022 95.764 92.700
50 100.000 100.000 99.465 98.155 96.009 93.064
51 100.000 100.000 99.511 98.285 96.239 93.396
52 100.000 100.000 99.538 98.368 96.398 93.642
53 100.000 100.000 99.575 98.478 96.607 93.961
54 100.000 100.000 99.575 98.491 96.651 94.055
55 100.000 100.000 99.585 98.523 96.720 94.172
56 100.000 100.000 99.612 98.609 96.886 94.432
57 100.000 100.000 99.658 98.738 97.116 94.766
58 100.000 100.000 99.658 98.745 97.142 94.823
59 100.000 100.000 99.668 98.778 97.215 94.948
60 100.000 100.000 99.677 98.812 97.289 95.077
61 100.000 100.000 99.705 98.894 97.444 95.315
62 100.000 100.000 99.705 98.907 97.492 95.416
63 100.000 100.000 99.714 98.935 97.542 95.493
64 100.000 100.000 99.723 98.965 97.605 95.598
65 100.000 100.000 99.751 99.054 97.784 95.882
66 100.000 100.000 99.751 99.062 97.810 95.940
67 100.000 100.000 99.751 99.067 97.829 95.980
68 100.000 100.000 99.751 99.079 97.872 96.072
69 100.000 100.000 99.760 99.107 97.927 96.159
70 100.000 100.000 99.760 99.116 97.957 96.223
71 100.000 100.000 99.788 99.195 98.102 96.442
72 100.000 100.000 99.797 99.226 98.168 96.550
73 100.000 100.000 99.815 99.279 98.267 96.702
74 100.000 100.000 99.843 99.365 98.431 96.957
75 100.000 100.000 99.889 99.500 98.682 97.332
76 100.000 100.000 99.889 99.509 98.715 97.402
77 100.000 100.000 99.908 99.565 98.820 97.565
78 100.000 100.000 99.908 99.571 98.843 97.614
79 100.000 100.000 99.926 99.626 98.947 97.774
80 100.000 100.000 99.935 99.656 99.010 97.879
81 100.000 100.000 99.945 99.683 99.061 97.959
82 100.000 100.000 99.954 99.715 99.127 98.070
83 100.000 100.000 99.963 99.746 99.193 98.182
84 100.000 100.000 99.963 99.750 99.209 98.216
85 100.000 100.000 99.963 99.759 99.242 98.288
86 100.000 100.000 99.972 99.786 99.293 98.365
87 100.000 100.000 99.972 99.795 99.323 98.432
88 100.000 100.000 99.982 99.822 99.374 98.509
89 100.000 100.000 99.982 99.826 99.390 98.549
90 100.000 100.000 99.982 99.836 99.429 98.635
91 100.000 100.000 99.982 99.847 99.466 98.717
92 100.000 100.000 99.982 99.851 99.482 98.752
93 100.000 100.000 99.982 99.863 99.523 98.839
94 100.000 100.000 99.991 99.892 99.582 98.936
95 100.000 100.000 99.991 99.899 99.608 98.995
96 100.000 100.000 99.991 99.905 99.631 99.048
97 100.000 100.000 99.991 99.909 99.648 99.087
98 100.000 100.000 99.991 99.913 99.660 99.117
99 100.000 100.000 99.991 99.919 99.684 99.171
100 100.000 100.000 99.991 99.925 99.708 99.225
101 100.000 100.000 99.991 99.933 99.738 99.293
102 100.000 100.000 99.991 99.936 99.750 99.320
103 100.000 100.000 99.991 99.941 99.766 99.356
104 100.000 100.000 99.991 99.944 99.778 99.383
105 100.000 100.000 99.991 99.949 99.797 99.429
106 100.000 100.000 99.991 99.952 99.809 99.456
107 100.000 100.000 99.991 99.957 99.829 99.500
108 100.000 100.000 100.000 99.984 99.879 99.577
109 100.000 100.000 100.000 99.990 99.898 99.620
110 100.000 100.000 100.000 99.997 99.925 99.680
111 100.000 100.000 100.000 100.000 99.937 99.706

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.705 72.630 61.067 50.763 41.918
5 97.674 88.876 77.947 66.919 56.699 47.637
6 100.000 92.700 82.664 72.069 61.943 52.735
7 100.000 94.090 85.443 75.837 66.261 57.243
8 100.000 95.133 87.492 78.583 69.388 60.500
9 100.000 96.060 89.338 81.089 72.275 63.541
10 100.000 96.871 90.963 83.313 74.863 66.298
11 100.000 97.335 91.997 84.858 76.798 68.490
12 100.000 97.798 93.067 86.500 78.890 70.887
13 100.000 98.146 93.880 87.754 80.497 72.736
14 100.000 98.378 94.461 88.699 81.759 74.238
15 100.000 98.610 94.960 89.435 82.683 75.302
16 100.000 98.725 95.301 90.035 83.517 76.317
17 100.000 98.841 95.643 90.651 84.404 77.438
18 100.000 98.957 95.901 91.043 84.907 78.025
19 100.000 99.073 96.261 91.701 85.853 79.211
20 100.000 99.189 96.575 92.235 86.588 80.107
21 100.000 99.305 96.880 92.750 87.298 80.977
22 100.000 99.421 97.129 93.124 87.781 81.550
23 100.000 99.537 97.461 93.715 88.626 82.615
24 100.000 99.652 97.711 94.092 89.120 83.211
25 100.000 99.768 97.969 94.488 89.635 83.824
26 100.000 99.884 98.218 94.854 90.091 84.343
27 100.000 100.000 98.468 95.227 90.570 84.911
28 100.000 100.000 98.477 95.255 90.625 84.996
29 100.000 100.000 98.505 95.345 90.806 85.284
30 100.000 100.000 98.634 95.697 91.415 86.138
31 100.000 100.000 98.689 95.861 91.719 86.593
32 100.000 100.000 98.754 96.055 92.083 87.140
33 100.000 100.000 98.800 96.191 92.338 87.525
34 100.000 100.000 98.828 96.271 92.486 87.745
35 100.000 100.000 98.837 96.298 92.536 87.822
36 100.000 100.000 98.837 96.306 92.562 87.879
37 100.000 100.000 98.865 96.395 92.741 88.161
38 100.000 100.000 98.911 96.524 92.971 88.493
39 100.000 100.000 99.012 96.807 93.469 89.205
40 100.000 100.000 99.068 96.970 93.773 89.662
41 100.000 100.000 99.132 97.159 94.121 90.179
42 100.000 100.000 99.188 97.324 94.431 90.648
43 100.000 100.000 99.271 97.556 94.841 91.236
44 100.000 100.000 99.308 97.658 95.021 91.493
45 100.000 100.000 99.317 97.688 95.083 91.596
46 100.000 100.000 99.326 97.719 95.148 91.703
47 100.000 100.000 99.345 97.777 95.264 91.888
48 100.000 100.000 99.363 97.834 95.372 92.057
49 100.000 100.000 99.382 97.889 95.477 92.219
50 100.000 100.000 99.409 97.975 95.647 92.487
51 100.000 100.000 99.428 98.033 95.760 92.667
52 100.000 100.000 99.437 98.064 95.826 92.779
53 100.000 100.000 99.437 98.077 95.872 92.876
54 100.000 100.000 99.520 98.310 96.285 93.469
55 100.000 100.000 99.548 98.388 96.425 93.673
56 100.000 100.000 99.557 98.419 96.490 93.782
57 100.000 100.000 99.566 98.447 96.545 93.867
58 100.000 100.000 99.585 98.506 96.665 94.064
59 100.000 100.000 99.594 98.534 96.716 94.141
60 100.000 100.000 99.594 98.542 96.745 94.204
61 100.000 100.000 99.594 98.551 96.778 94.273
62 100.000 100.000 99.622 98.635 96.938 94.523
63 100.000 100.000 99.631 98.672 97.025 94.678
64 100.000 100.000 99.686 98.831 97.315 95.103
65 100.000 100.000 99.695 98.861 97.374 95.199
66 100.000 100.000 99.714 98.921 97.496 95.396
67 100.000 100.000 99.732 98.977 97.606 95.571
68 100.000 100.000 99.742 99.005 97.661 95.661
69 100.000 100.000 99.760 99.066 97.784 95.860
70 100.000 100.000 99.760 99.066 97.785 95.863
71 100.000 100.000 99.769 99.102 97.869 96.011
72 100.000 100.000 99.769 99.111 97.898 96.076
73 100.000 100.000 99.788 99.170 98.018 96.269
74 100.000 100.000 99.825 99.284 98.240 96.613
75 100.000 100.000 99.834 99.319 98.317 96.747
76 100.000 100.000 99.843 99.353 98.393 96.879
77 100.000 100.000 99.843 99.359 98.415 96.927
78 100.000 100.000 99.843 99.362 98.423 96.946
79 100.000 100.000 99.843 99.370 98.453 97.008
80 100.000 100.000 99.843 99.373 98.466 97.040
81 100.000 100.000 99.852 99.406 98.538 97.165
82 100.000 100.000 99.862 99.437 98.600 97.267
83 100.000 100.000 99.862 99.447 98.638 97.348
84 100.000 100.000 99.862 99.450 98.650 97.376
85 100.000 100.000 99.862 99.457 98.675 97.430
86 100.000 100.000 99.862 99.462 98.691 97.466
87 100.000 100.000 99.862 99.468 98.713 97.516
88 100.000 100.000 99.862 99.478 98.750 97.594
89 100.000 100.000 99.871 99.506 98.805 97.682
90 100.000 100.000 99.871 99.510 98.821 97.718
91 100.000 100.000 99.871 99.516 98.840 97.762
92 100.000 100.000 99.871 99.518 98.848 97.779
93 100.000 100.000 99.880 99.552 98.924 97.912
94 100.000 100.000 99.880 99.553 98.928 97.921
95 100.000 100.000 99.880 99.557 98.944 97.956
96 100.000 100.000 99.880 99.563 98.962 97.995
97 100.000 100.000 99.889 99.591 99.017 98.082
98 100.000 100.000 99.889 99.593 99.025 98.101
99 100.000 100.000 99.898 99.620 99.076 98.180
100 100.000 100.000 99.898 99.621 99.081 98.192
101 100.000 100.000 99.898 99.625 99.097 98.230
102 100.000 100.000 99.898 99.626 99.101 98.241
103 100.000 100.000 99.898 99.633 99.129 98.304
104 100.000 100.000 99.898 99.640 99.152 98.355
105 100.000 100.000 99.898 99.645 99.171 98.398
106 100.000 100.000 99.908 99.672 99.222 98.475
107 100.000 100.000 99.908 99.678 99.245 98.526
108 100.000 100.000 99.908 99.680 99.252 98.543
109 100.000 100.000 99.917 99.707 99.303 98.621
110 100.000 100.000 99.917 99.708 99.308 98.633
111 100.000 100.000 99.926 99.738 99.366 98.728
112 100.000 100.000 99.935 99.765 99.417 98.805
113 100.000 100.000 99.935 99.768 99.429 98.833
114 100.000 100.000 99.935 99.774 99.452 98.886
115 100.000 100.000 99.935 99.777 99.464 98.914
116 100.000 100.000 99.935 99.783 99.487 98.965
117 100.000 100.000 99.963 99.863 99.632 99.181
118 100.000 100.000 99.963 99.867 99.648 99.217
119 100.000 100.000 99.963 99.871 99.664 99.254
120 100.000 100.000 99.963 99.874 99.676 99.281
121 100.000 100.000 99.972 99.901 99.726 99.358
122 100.000 100.000 99.982 99.928 99.777 99.437
123 100.000 100.000 99.982 99.930 99.785 99.455
124 100.000 100.000 99.991 99.957 99.836 99.532
125 100.000 100.000 99.991 99.960 99.847 99.557
126 100.000 100.000 99.991 99.964 99.859 99.586
127 100.000 100.000 100.000 99.991 99.910 99.663
128 100.000 100.000 100.000 99.994 99.921 99.687
129 100.000 100.000 100.000 99.996 99.929 99.708
130 100.000 100.000 100.000 100.000 99.945 99.742

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 93.023 82.503 71.125 60.228 50.409 41.868
5 97.674 89.108 78.381 67.466 57.277 48.188
6 100.000 92.932 83.107 72.640 62.560 53.337
7 100.000 93.859 85.064 75.424 65.889 56.946
8 100.000 94.786 86.966 78.073 69.009 60.294
9 100.000 95.829 89.043 80.894 72.264 63.727
10 100.000 96.756 90.889 83.407 75.174 66.815
11 100.000 97.219 91.960 85.045 77.254 69.189
12 100.000 97.567 92.818 86.422 79.067 71.318
13 100.000 97.914 93.631 87.680 80.686 73.192
14 100.000 98.262 94.443 88.935 82.296 75.048
15 100.000 98.494 95.024 89.886 83.575 76.584
16 100.000 98.610 95.384 90.553 84.552 77.831
17 100.000 98.841 95.911 91.365 85.608 79.077
18 100.000 98.957 96.243 91.946 86.419 80.071
19 100.000 99.073 96.575 92.540 87.271 81.150
20 100.000 99.189 96.908 93.123 88.089 82.156
21 100.000 99.189 97.046 93.501 88.744 83.078
22 100.000 99.305 97.360 94.051 89.532 84.079
23 100.000 99.305 97.471 94.355 90.062 84.828
24 100.000 99.421 97.766 94.845 90.732 85.646
25 100.000 99.421 97.849 95.074 91.131 86.211
26 100.000 99.537 98.108 95.472 91.657 86.846
27 100.000 99.652 98.366 95.868 92.172 87.459
28 100.000 99.768 98.615 96.243 92.659 88.041
29 100.000 99.884 98.865 96.616 93.139 88.612
30 100.000 100.000 99.114 96.990 93.622 89.187
31 100.000 100.000 99.178 97.181 93.975 89.712
32 100.000 100.000 99.234 97.344 94.278 90.161
33 100.000 100.000 99.289 97.511 94.592 90.635
34 100.000 100.000 99.335 97.652 94.864 91.054
35 100.000 100.000 99.382 97.794 95.135 91.471
36 100.000 100.000 99.428 97.934 95.401 91.876
37 100.000 100.000 99.465 98.045 95.612 92.197
38 100.000 100.000 99.502 98.152 95.810 92.494
39 100.000 100.000 99.529 98.243 95.992 92.785
40 100.000 100.000 99.557 98.323 96.140 93.008
41 100.000 100.000 99.585 98.415 96.327 93.309
42 100.000 100.000 99.612 98.497 96.482 93.545
43 100.000 100.000 99.640 98.589 96.669 93.849
44 100.000 100.000 99.668 98.675 96.840 94.118
45 100.000 100.000 99.695 98.760 97.002 94.370
46 100.000 100.000 99.723 98.841 97.154 94.602
47 100.000 100.000 99.751 98.920 97.298 94.815
48 100.000 100.000 99.769 98.978 97.414 95.000
49 100.000 100.000 99.788 99.035 97.523 95.172
50 100.000 100.000 99.806 99.096 97.650 95.382
51 100.000 100.000 99.815 99.125 97.708 95.475
52 100.000 100.000 99.825 99.152 97.759 95.552
53 100.000 100.000 99.834 99.187 97.837 95.688
54 100.000 100.000 99.843 99.231 97.948 95.894
55 100.000 100.000 99.852 99.258 97.999 95.972
56 100.000 100.000 99.862 99.290 98.066 96.087
57 100.000 100.000 99.871 99.317 98.117 96.164
58 100.000 100.000 99.880 99.348 98.182 96.272
59 100.000 100.000 99.889 99.387 98.272 96.433
60 100.000 100.000 99.898 99.424 98.360 96.588
61 100.000 100.000 99.908 99.457 98.433 96.713
62 100.000 100.000 99.917 99.484 98.484 96.792
63 100.000 100.000 99.926 99.523 98.574 96.953
64 100.000 100.000 99.935 99.551 98.628 97.038
65 100.000 100.000 99.945 99.582 98.694 97.147
66 100.000 100.000 99.954 99.614 98.760 97.257
67 100.000 100.000 99.963 99.641 98.810 97.334
68 100.000 100.000 99.972 99.670 98.869 97.429
69 100.000 100.000 99.982 99.700 98.930 97.529
70 100.000 100.000 99.991 99.727 98.981 97.608
71 100.000 100.000 100.000 99.754 99.032 97.686
72 100.000 100.000 100.000 99.765 99.070 97.768
73 100.000 100.000 100.000 99.775 99.107 97.850
74 100.000 100.000 100.000 99.780 99.127 97.895
75 100.000 100.000 100.000 99.789 99.157 97.961
76 100.000 100.000 100.000 99.796 99.184 98.020
77 100.000 100.000 100.000 99.801 99.203 98.064
78 100.000 100.000 100.000 99.808 99.230 98.123
79 100.000 100.000 100.000 99.816 99.256 98.180
80 100.000 100.000 100.000 99.822 99.278 98.229
81 100.000 100.000 100.000 99.828 99.300 98.277
82 100.000 100.000 100.000 99.832 99.316 98.312
83 100.000 100.000 100.000 99.840 99.343 98.373
84 100.000 100.000 100.000 99.847 99.370 98.433
85 100.000 100.000 100.000 99.850 99.381 98.459
86 100.000 100.000 100.000 99.863 99.427 98.557
87 100.000 100.000 100.000 99.868 99.446 98.603
88 100.000 100.000 100.000 99.876 99.477 98.670
89 100.000 100.000 100.000 99.883 99.504 98.731
90 100.000 100.000 100.000 99.890 99.527 98.782
91 100.000 100.000 100.000 99.894 99.543 98.818
92 100.000 100.000 100.000 99.897 99.555 98.846
93 100.000 100.000 100.000 99.902 99.574 98.890
94 100.000 100.000 100.000 99.909 99.601 98.949
95 100.000 100.000 100.000 99.916 99.623 98.999
96 100.000 100.000 100.000 99.922 99.645 99.047
97 100.000 100.000 100.000 99.926 99.661 99.083
98 100.000 100.000 100.000 99.930 99.677 99.120
99 100.000 100.000 100.000 99.936 99.701 99.174
100 100.000 100.000 100.000 99.944 99.727 99.233
101 100.000 100.000 100.000 99.948 99.743 99.267
102 100.000 100.000 100.000 99.954 99.765 99.316
103 100.000 100.000 100.000 99.957 99.778 99.346
104 100.000 100.000 100.000 99.960 99.790 99.376
105 100.000 100.000 100.000 99.968 99.817 99.435
106 100.000 100.000 100.000 99.972 99.832 99.469
107 100.000 100.000 100.000 99.975 99.844 99.496
108 100.000 100.000 100.000 99.981 99.867 99.548
109 100.000 100.000 100.000 99.989 99.894 99.608
110 100.000 100.000 100.000 99.995 99.917 99.660
111 100.000 100.000 100.000 100.000 99.937 99.706

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.820 72.851 61.352 51.071 42.219
5 97.674 89.108 78.381 67.466 57.277 48.188
6 97.674 90.498 81.187 71.298 61.690 52.809
7 100.000 94.322 85.867 76.363 66.809 57.761
8 100.000 95.249 87.769 79.009 69.922 61.096
9 100.000 96.060 89.449 81.369 72.724 64.124
10 100.000 96.640 90.714 83.234 75.036 66.722
11 100.000 96.987 91.572 84.606 76.836 68.826
12 100.000 97.335 92.403 85.913 78.538 70.813
13 100.000 97.798 93.418 87.414 80.404 72.918
14 100.000 98.146 94.231 88.669 82.014 74.773
15 100.000 98.378 94.840 89.694 83.417 76.478
16 100.000 98.494 95.209 90.386 84.433 77.776
17 100.000 98.610 95.569 91.035 85.349 78.901
18 100.000 98.725 95.920 91.672 86.266 80.055
19 100.000 98.841 96.261 92.284 87.141 81.151
20 100.000 99.073 96.769 93.045 88.107 82.273
21 100.000 99.189 97.074 93.572 88.858 83.224
22 100.000 99.305 97.369 94.076 89.573 84.133
23 100.000 99.305 97.480 94.382 90.110 84.896
24 100.000 99.305 97.563 94.618 90.536 85.516
25 100.000 99.421 97.821 95.014 91.052 86.130
26 100.000 99.421 97.886 95.205 91.406 86.660
27 100.000 99.537 98.135 95.578 91.886 87.230
28 100.000 99.652 98.385 95.953 92.372 87.811
29 100.000 99.768 98.634 96.325 92.852 88.379
30 100.000 99.884 98.883 96.695 93.322 88.929
31 100.000 99.884 98.948 96.885 93.673 89.452
32 100.000 100.000 99.197 97.256 94.147 90.010
33 100.000 100.000 99.262 97.444 94.494 90.526
34 100.000 100.000 99.317 97.608 94.798 90.981
35 100.000 100.000 99.372 97.773 95.109 91.451
36 100.000 100.000 99.418 97.914 95.377 91.861
37 100.000 100.000 99.455 98.023 95.582 92.172
38 100.000 100.000 99.492 98.137 95.801 92.508
39 100.000 100.000 99.529 98.250 96.019 92.846
40 100.000 100.000 99.566 98.363 96.234 93.172
41 100.000 100.000 99.594 98.446 96.391 93.413
42 100.000 100.000 99.622 98.530 96.554 93.666
43 100.000 100.000 99.649 98.622 96.740 93.963
44 100.000 100.000 99.677 98.709 96.909 94.228
45 100.000 100.000 99.705 98.799 97.092 94.524
46 100.000 100.000 99.732 98.892 97.282 94.831
47 100.000 100.000 99.760 98.974 97.438 95.069
48 100.000 100.000 99.788 99.057 97.596 95.311
49 100.000 100.000 99.806 99.118 97.720 95.516
50 100.000 100.000 99.825 99.177 97.840 95.709
51 100.000 100.000 99.834 99.212 97.916 95.841
52 100.000 100.000 99.843 99.242 97.979 95.945
53 100.000 100.000 99.852 99.279 98.065 96.096
54 100.000 100.000 99.862 99.306 98.116 96.175
55 100.000 100.000 99.871 99.336 98.175 96.274
56 100.000 100.000 99.880 99.373 98.262 96.425
57 100.000 100.000 99.889 99.402 98.320 96.519
58 100.000 100.000 99.898 99.429 98.371 96.598
59 100.000 100.000 99.908 99.456 98.422 96.678
60 100.000 100.000 99.917 99.483 98.473 96.756
61 100.000 100.000 99.926 99.514 98.535 96.858
62 100.000 100.000 99.935 99.541 98.586 96.937
63 100.000 100.000 99.945 99.568 98.637 97.014
64 100.000 100.000 99.954 99.601 98.710 97.140
65 100.000 100.000 99.963 99.628 98.761 97.217
66 100.000 100.000 99.972 99.657 98.820 97.314
67 100.000 100.000 99.982 99.684 98.871 97.393
68 100.000 100.000 99.991 99.721 98.954 97.539
69 100.000 100.000 100.000 99.748 99.005 97.616
70 100.000 100.000 100.000 99.752 99.021 97.653
71 100.000 100.000 100.000 99.759 99.047 97.710
72 100.000 100.000 100.000 99.768 99.077 97.775
73 100.000 100.000 100.000 99.778 99.114 97.856
74 100.000 100.000 100.000 99.781 99.125 97.880
75 100.000 100.000 100.000 99.790 99.155 97.945
76 100.000 100.000 100.000 99.794 99.171 97.982
77 100.000 100.000 100.000 99.800 99.194 98.033
78 100.000 100.000 100.000 99.808 99.225 98.103
79 100.000 100.000 100.000 99.816 99.252 98.162
80 100.000 100.000 100.000 99.819 99.264 98.192
81 100.000 100.000 100.000 99.827 99.294 98.258
82 100.000 100.000 100.000 99.831 99.310 98.294
83 100.000 100.000 100.000 99.833 99.317 98.311
84 100.000 100.000 100.000 99.836 99.329 98.336
85 100.000 100.000 100.000 99.849 99.373 98.429
86 100.000 100.000 100.000 99.854 99.392 98.472
87 100.000 100.000 100.000 99.856 99.400 98.492
88 100.000 100.000 100.000 99.860 99.416 98.527
89 100.000 100.000 100.000 99.864 99.429 98.559
90 100.000 100.000 100.000 99.869 99.448 98.603
91 100.000 100.000 100.000 99.872 99.461 98.635
92 100.000 100.000 100.000 99.874 99.471 98.661
93 100.000 100.000 100.000 99.875 99.475 98.673
94 100.000 100.000 100.000 99.878 99.487 98.701
95 100.000 100.000 100.000 99.883 99.506 98.743
96 100.000 100.000 100.000 99.891 99.532 98.799
97 100.000 100.000 100.000 99.896 99.552 98.844
98 100.000 100.000 100.000 99.900 99.567 98.878
99 100.000 100.000 100.000 99.905 99.587 98.924
100 100.000 100.000 100.000 99.913 99.614 98.985
101 100.000 100.000 100.000 99.920 99.641 99.044
102 100.000 100.000 100.000 99.927 99.667 99.101
103 100.000 100.000 100.000 99.928 99.671 99.111
104 100.000 100.000 100.000 99.932 99.686 99.144
105 100.000 100.000 100.000 99.933 99.690 99.153
106 100.000 100.000 100.000 99.938 99.706 99.190
107 100.000 100.000 100.000 99.940 99.714 99.208
108 100.000 100.000 100.000 99.944 99.730 99.245
109 100.000 100.000 100.000 99.949 99.748 99.285
110 100.000 100.000 100.000 99.951 99.757 99.304
111 100.000 100.000 100.000 99.953 99.765 99.325
112 100.000 100.000 100.000 99.957 99.782 99.366
113 100.000 100.000 100.000 99.957 99.783 99.370
114 100.000 100.000 100.000 99.964 99.805 99.419
115 100.000 100.000 100.000 99.968 99.821 99.454
116 100.000 100.000 100.000 99.971 99.832 99.480
117 100.000 100.000 100.000 99.974 99.844 99.508
118 100.000 100.000 100.000 99.975 99.849 99.521
119 100.000 100.000 100.000 99.977 99.858 99.541
120 100.000 100.000 100.000 99.980 99.869 99.568
121 100.000 100.000 100.000 99.981 99.873 99.577
122 100.000 100.000 100.000 99.983 99.881 99.594
123 100.000 100.000 100.000 99.985 99.889 99.615
124 100.000 100.000 100.000 99.989 99.901 99.642
125 100.000 100.000 100.000 99.992 99.913 99.670
126 100.000 100.000 100.000 99.994 99.921 99.688
127 100.000 100.000 100.000 99.995 99.925 99.697
128 100.000 100.000 100.000 99.998 99.936 99.722
129 100.000 100.000 100.000 100.000 99.944 99.740
130 100.000 100.000 100.000 100.000 99.945 99.742

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 74.855 62.088 50.933 41.540 33.785
4 93.023 82.271 70.747 59.812 50.024 41.548
5 97.674 89.108 78.418 67.553 57.408 48.346
6 97.674 90.382 80.993 71.079 61.483 52.632
7 100.000 94.090 85.461 75.883 66.336 57.340
8 100.000 95.017 87.381 78.580 69.534 60.791
9 100.000 95.481 88.470 80.265 71.691 63.267
10 100.000 96.292 90.132 82.584 74.434 66.228
11 100.000 97.103 91.775 84.856 77.105 69.099
12 100.000 97.335 92.366 85.839 78.449 70.736
13 100.000 97.567 92.966 86.838 79.807 72.377
14 100.000 97.798 93.538 87.772 81.070 73.904
15 100.000 97.914 93.917 88.486 82.119 75.244
16 100.000 98.030 94.286 89.173 83.122 76.519
17 100.000 98.378 95.043 90.293 84.523 78.113
18 100.000 98.378 95.181 90.670 85.174 79.026
19 100.000 98.494 95.532 91.307 86.092 80.183
20 100.000 98.494 95.652 91.638 86.670 81.002
21 100.000 98.610 95.984 92.228 87.512 82.059
22 100.000 98.841 96.492 92.993 88.492 83.209
23 100.000 98.957 96.825 93.579 89.321 84.240
24 100.000 99.073 97.120 94.076 90.012 85.100
25 100.000 99.189 97.388 94.507 90.606 85.846
26 100.000 99.305 97.655 94.933 91.183 86.558
27 100.000 99.305 97.711 95.096 91.487 87.015
28 100.000 99.305 97.785 95.302 91.850 87.536
29 100.000 99.305 97.840 95.465 92.155 87.994
30 100.000 99.305 97.905 95.647 92.481 88.466
31 100.000 99.421 98.154 96.024 92.974 89.062
32 100.000 99.421 98.191 96.139 93.196 89.405
33 100.000 99.537 98.449 96.535 93.713 90.022
34 100.000 99.537 98.477 96.628 93.902 90.328
35 100.000 99.537 98.514 96.741 94.120 90.661
36 100.000 99.652 98.763 97.113 94.596 91.223
37 100.000 99.652 98.791 97.202 94.776 91.511
38 100.000 99.768 99.040 97.571 95.244 92.057
39 100.000 99.884 99.289 97.940 95.710 92.602
40 100.000 99.884 99.317 98.026 95.879 92.865
41 100.000 99.884 99.335 98.091 96.017 93.097
42 100.000 99.884 99.363 98.176 96.182 93.354
43 100.000 99.884 99.400 98.283 96.377 93.643
44 100.000 99.884 99.418 98.347 96.513 93.870
45 100.000 100.000 99.668 98.713 96.970 94.393
46 100.000 100.000 99.695 98.797 97.132 94.644
47 100.000 100.000 99.714 98.860 97.263 94.861
48 100.000 100.000 99.723 98.901 97.363 95.041
49 100.000 100.000 99.742 98.961 97.484 95.238
50 100.000 100.000 99.751 98.999 97.574 95.399
51 100.000 100.000 99.769 99.058 97.694 95.593
52 100.000 100.000 99.788 99.118 97.813 95.785
53 100.000 100.000 99.815 99.198 97.961 96.008
54 100.000 100.000 99.825 99.236 98.048 96.162
55 100.000 100.000 99.843 99.294 98.164 96.345
56 100.000 100.000 99.852 99.330 98.247 96.489
57 100.000 100.000 99.852 99.342 98.289 96.584
58 100.000 100.000 99.862 99.374 98.359 96.704
59 100.000 100.000 99.862 99.385 98.400 96.791
60 100.000 100.000 99.871 99.418 98.470 96.912
61 100.000 100.000 99.871 99.428 98.507 96.993
62 100.000 100.000 99.880 99.459 98.574 97.109
63 100.000 100.000 99.889 99.491 98.640 97.220
64 100.000 100.000 99.898 99.521 98.702 97.323
65 100.000 100.000 99.898 99.530 98.736 97.397
66 100.000 100.000 99.898 99.540 98.770 97.472
67 100.000 100.000 99.898 99.548 98.801 97.541
68 100.000 100.000 99.908 99.577 98.860 97.637
69 100.000 100.000 99.908 99.585 98.891 97.709
70 100.000 100.000 99.908 99.594 98.921 97.773
71 100.000 100.000 99.908 99.601 98.948 97.832
72 100.000 100.000 99.908 99.607 98.970 97.880
73 100.000 100.000 99.908 99.614 98.993 97.933
74 100.000 100.000 99.917 99.641 99.044 98.010
75 100.000 100.000 99.926 99.668 99.094 98.087
76 100.000 100.000 99.926 99.674 99.117 98.137
77 100.000 100.000 99.926 99.680 99.140 98.189
78 100.000 100.000 99.935 99.707 99.191 98.266
79 100.000 100.000 99.945 99.734 99.241 98.343
80 100.000 100.000 99.954 99.761 99.292 98.420
81 100.000 100.000 99.963 99.789 99.343 98.497
82 100.000 100.000 99.972 99.816 99.394 98.575
83 100.000 100.000 99.982 99.843 99.444 98.653
84 100.000 100.000 99.991 99.870 99.496 98.733
85 100.000 100.000 100.000 99.897 99.547 98.811
86 100.000 100.000 100.000 99.902 99.566 98.855
87 100.000 100.000 100.000 99.907 99.587 98.904
88 100.000 100.000 100.000 99.913 99.607 98.952
89 100.000 100.000 100.000 99.918 99.626 98.993
90 100.000 100.000 100.000 99.923 99.645 99.037
91 100.000 100.000 100.000 99.928 99.665 99.082
92 100.000 100.000 100.000 99.933 99.684 99.125
93 100.000 100.000 100.000 99.938 99.700 99.163
94 100.000 100.000 100.000 99.942 99.717 99.202
95 100.000 100.000 100.000 99.946 99.732 99.234
96 100.000 100.000 100.000 99.950 99.747 99.268
97 100.000 100.000 100.000 99.954 99.763 99.305
98 100.000 100.000 100.000 99.958 99.778 99.340
99 100.000 100.000 100.000 99.963 99.794 99.374
100 100.000 100.000 100.000 99.967 99.809 99.410
101 100.000 100.000 100.000 99.970 99.822 99.440
102 100.000 100.000 100.000 99.973 99.834 99.470
103 100.000 100.000 100.000 99.976 99.846 99.496
104 100.000 100.000 100.000 99.979 99.859 99.527
105 100.000 100.000 100.000 99.982 99.871 99.555
106 100.000 100.000 100.000 99.985 99.882 99.581
107 100.000 100.000 100.000 99.989 99.894 99.606
108 100.000 100.000 100.000 99.992 99.905 99.632
109 100.000 100.000 100.000 99.995 99.917 99.660
110 100.000 100.000 100.000 99.998 99.929 99.687
111 100.000 100.000 100.000 100.000 99.937 99.706

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 93.023 82.503 71.125 60.228 50.409 41.868
5 97.674 89.340 78.806 67.991 57.822 48.700
6 97.674 90.614 81.427 71.625 62.058 53.177
7 100.000 94.554 86.255 76.800 67.223 58.115
8 100.000 95.017 87.381 78.575 69.518 60.762
9 100.000 95.713 88.849 80.677 72.060 63.558
10 100.000 96.060 89.717 82.080 73.919 65.751
11 100.000 96.640 90.944 83.850 76.080 68.154
12 100.000 97.219 92.181 85.642 78.272 70.593
13 100.000 97.683 93.234 87.234 80.283 72.882
14 100.000 97.914 93.824 88.208 81.598 74.464
15 100.000 98.146 94.397 89.137 82.844 75.961
16 100.000 98.378 94.988 90.103 84.135 77.496
17 100.000 98.494 95.329 90.722 85.030 78.633
18 100.000 98.610 95.680 91.361 85.951 79.795
19 100.000 98.725 95.994 91.908 86.732 80.784
20 100.000 98.841 96.308 92.456 87.513 81.770
21 100.000 98.841 96.428 92.785 88.086 82.580
22 100.000 98.841 96.548 93.109 88.642 83.356
23 100.000 99.073 97.046 93.849 89.580 84.447
24 100.000 99.189 97.332 94.326 90.246 85.283
25 100.000 99.305 97.600 94.754 90.829 86.007
26 100.000 99.305 97.655 94.919 91.141 86.478
27 100.000 99.305 97.692 95.043 91.393 86.881
28 100.000 99.305 97.748 95.208 91.700 87.342
29 100.000 99.421 97.997 95.588 92.203 87.956
30 100.000 99.421 98.052 95.748 92.497 88.390
31 100.000 99.421 98.108 95.909 92.790 88.825
32 100.000 99.421 98.154 96.047 93.052 89.221
33 100.000 99.421 98.191 96.164 93.280 89.577
34 100.000 99.421 98.228 96.280 93.504 89.924
35 100.000 99.537 98.486 96.676 94.024 90.545
36 100.000 99.537 98.514 96.768 94.211 90.848
37 100.000 99.537 98.560 96.901 94.456 91.213
38 100.000 99.537 98.606 97.034 94.697 91.569
39 100.000 99.537 98.643 97.144 94.906 91.888
40 100.000 99.652 98.892 97.515 95.380 92.447
41 100.000 99.768 99.142 97.884 95.846 92.989
42 100.000 99.768 99.169 97.971 96.019 93.262
43 100.000 99.884 99.418 98.339 96.482 93.798
44 100.000 99.884 99.446 98.424 96.648 94.058
45 100.000 99.884 99.455 98.468 96.755 94.251
46 100.000 99.884 99.474 98.532 96.892 94.480
47 100.000 100.000 99.723 98.898 97.348 94.999
48 100.000 100.000 99.751 98.981 97.507 95.245
49 100.000 100.000 99.769 99.042 97.630 95.447
50 100.000 100.000 99.788 99.101 97.750 95.639
51 100.000 100.000 99.797 99.139 97.836 95.792
52 100.000 100.000 99.815 99.196 97.949 95.970
53 100.000 100.000 99.825 99.231 98.030 96.116
54 100.000 100.000 99.825 99.245 98.077 96.216
55 100.000 100.000 99.843 99.300 98.182 96.378
56 100.000 100.000 99.852 99.333 98.256 96.506
57 100.000 100.000 99.871 99.388 98.358 96.665
58 100.000 100.000 99.880 99.421 98.432 96.792
59 100.000 100.000 99.880 99.432 98.474 96.884
60 100.000 100.000 99.889 99.465 98.543 97.002
61 100.000 100.000 99.898 99.496 98.608 97.110
62 100.000 100.000 99.898 99.505 98.641 97.181
63 100.000 100.000 99.898 99.514 98.672 97.248
64 100.000 100.000 99.908 99.543 98.730 97.343
65 100.000 100.000 99.908 99.551 98.761 97.411
66 100.000 100.000 99.908 99.559 98.790 97.475
67 100.000 100.000 99.908 99.568 98.822 97.546
68 100.000 100.000 99.908 99.575 98.848 97.603
69 100.000 100.000 99.917 99.602 98.898 97.680
70 100.000 100.000 99.926 99.629 98.950 97.760
71 100.000 100.000 99.926 99.635 98.972 97.810
72 100.000 100.000 99.926 99.642 98.995 97.860
73 100.000 100.000 99.935 99.669 99.046 97.939
74 100.000 100.000 99.945 99.696 99.098 98.019
75 100.000 100.000 99.954 99.723 99.149 98.097
76 100.000 100.000 99.963 99.750 99.199 98.174
77 100.000 100.000 99.972 99.777 99.251 98.254
78 100.000 100.000 99.972 99.783 99.274 98.309
79 100.000 100.000 99.982 99.810 99.326 98.388
80 100.000 100.000 99.991 99.838 99.377 98.467
81 100.000 100.000 99.991 99.844 99.399 98.514
82 100.000 100.000 100.000 99.871 99.449 98.591
83 100.000 100.000 100.000 99.876 99.468 98.634
84 100.000 100.000 100.000 99.881 99.488 98.678
85 100.000 100.000 100.000 99.886 99.508 98.726
86 100.000 100.000 100.000 99.892 99.528 98.771
87 100.000 100.000 100.000 99.896 99.544 98.807
88 100.000 100.000 100.000 99.900 99.560 98.843
89 100.000 100.000 100.000 99.904 99.575 98.879
90 100.000 100.000 100.000 99.908 99.591 98.913
91 100.000 100.000 100.000 99.913 99.606 98.949
92 100.000 100.000 100.000 99.917 99.622 98.987
93 100.000 100.000 100.000 99.921 99.638 99.023
94 100.000 100.000 100.000 99.925 99.654 99.059
95 100.000 100.000 100.000 99.929 99.669 99.094
96 100.000 100.000 100.000 99.932 99.681 99.122
97 100.000 100.000 100.000 99.935 99.693 99.147
98 100.000 100.000 100.000 99.939 99.704 99.173
99 100.000 100.000 100.000 99.942 99.716 99.198
100 100.000 100.000 100.000 99.945 99.727 99.223
101 100.000 100.000 100.000 99.948 99.739 99.250
102 100.000 100.000 100.000 99.951 99.750 99.276
103 100.000 100.000 100.000 99.954 99.762 99.304
104 100.000 100.000 100.000 99.957 99.775 99.333
105 100.000 100.000 100.000 99.960 99.786 99.360
106 100.000 100.000 100.000 99.964 99.798 99.387
107 100.000 100.000 100.000 99.967 99.809 99.412
108 100.000 100.000 100.000 99.970 99.821 99.439
109 100.000 100.000 100.000 99.973 99.834 99.468
110 100.000 100.000 100.000 99.975 99.841 99.485
111 100.000 100.000 100.000 99.977 99.849 99.504
112 100.000 100.000 100.000 99.979 99.857 99.523
113 100.000 100.000 100.000 99.981 99.866 99.544
114 100.000 100.000 100.000 99.983 99.874 99.564
115 100.000 100.000 100.000 99.985 99.883 99.587
116 100.000 100.000 100.000 99.988 99.891 99.604
117 100.000 100.000 100.000 99.990 99.899 99.625
118 100.000 100.000 100.000 99.992 99.908 99.647
119 100.000 100.000 100.000 99.993 99.913 99.659
120 100.000 100.000 100.000 99.994 99.917 99.669
121 100.000 100.000 100.000 99.995 99.921 99.679
122 100.000 100.000 100.000 99.996 99.925 99.690
123 100.000 100.000 100.000 99.997 99.930 99.701
124 100.000 100.000 100.000 99.998 99.934 99.711
125 100.000 100.000 100.000 99.999 99.938 99.720
126 100.000 100.000 100.000 100.000 99.942 99.732
127 100.000 100.000 100.000 100.000 99.943 99.734
128 100.000 100.000 100.000 100.000 99.943 99.736
129 100.000 100.000 100.000 100.000 99.944 99.741
130 100.000 100.000 100.000 100.000 99.945 99.742

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 86.047 72.190 59.753 49.088 40.161 32.791
4 90.698 79.490 68.208 57.727 48.406 40.337
5 93.023 84.009 74.135 64.302 55.065 46.711
6 97.674 90.382 80.975 71.035 61.418 52.553
7 97.674 91.541 83.329 74.275 65.184 56.538
8 97.674 92.352 85.036 76.710 68.113 59.739
9 97.674 92.700 85.895 78.094 69.946 61.905
10 97.674 93.395 87.353 80.174 72.455 64.662
11 97.674 93.743 88.175 81.464 74.141 66.640
12 100.000 97.335 92.357 85.819 78.420 70.704
13 100.000 97.683 93.169 87.080 80.050 72.600
14 100.000 97.798 93.557 87.817 81.139 73.990
15 100.000 97.914 93.926 88.505 82.142 75.266
16 100.000 97.914 94.074 88.906 82.834 76.234
17 100.000 98.030 94.397 89.484 83.674 77.311
18 100.000 98.146 94.748 90.125 84.605 78.494
19 100.000 98.378 95.301 91.002 85.759 79.861
20 100.000 98.494 95.643 91.614 86.631 80.952
21 100.000 98.725 96.188 92.465 87.739 82.253
22 100.000 98.841 96.465 92.919 88.369 83.043
23 100.000 99.073 96.981 93.704 89.378 84.223
24 100.000 99.073 97.046 93.894 89.729 84.745
25 100.000 99.073 97.111 94.081 90.073 85.256
26 100.000 99.189 97.378 94.509 90.656 85.981
27 100.000 99.305 97.637 94.913 91.201 86.655
28 100.000 99.305 97.711 95.123 91.576 87.200
29 100.000 99.305 97.775 95.311 91.922 87.711
30 100.000 99.305 97.821 95.449 92.181 88.103
31 100.000 99.305 97.840 95.512 92.316 88.331
32 100.000 99.305 97.941 95.786 92.786 88.984
33 100.000 99.305 97.988 95.920 93.034 89.355
34 100.000 99.305 98.025 96.032 93.246 89.682
35 100.000 99.305 98.052 96.122 93.429 89.974
36 100.000 99.305 98.061 96.165 93.533 90.163
37 100.000 99.305 98.098 96.273 93.736 90.470
38 100.000 99.305 98.117 96.339 93.876 90.703
39 100.000 99.305 98.145 96.428 94.051 90.981
40 100.000 99.305 98.163 96.492 94.188 91.207
41 100.000 99.305 98.191 96.582 94.366 91.489
42 100.000 99.421 98.440 96.951 94.837 92.043
43 100.000 99.421 98.440 96.968 94.896 92.167
44 100.000 99.421 98.468 97.056 95.067 92.436
45 100.000 99.421 98.486 97.119 95.200 92.655
46 100.000 99.421 98.505 97.179 95.320 92.850
47 100.000 99.421 98.523 97.240 95.446 93.054
48 100.000 99.421 98.541 97.297 95.559 93.235
49 100.000 99.421 98.551 97.331 95.633 93.365
50 100.000 99.537 98.800 97.698 96.097 93.901
51 100.000 99.537 98.809 97.734 96.177 94.042
52 100.000 99.537 98.828 97.794 96.299 94.238
53 100.000 99.537 98.837 97.829 96.379 94.377
54 100.000 99.537 98.846 97.863 96.452 94.504
55 100.000 99.652 99.095 98.228 96.908 95.026
56 100.000 99.652 99.095 98.237 96.939 95.097
57 100.000 99.768 99.345 98.602 97.396 95.619
58 100.000 99.884 99.594 98.968 97.852 96.141
59 100.000 99.884 99.594 98.980 97.895 96.233
60 100.000 99.884 99.594 98.989 97.926 96.301
61 100.000 99.884 99.594 98.996 97.953 96.364
62 100.000 99.884 99.603 99.027 98.020 96.476
63 100.000 99.884 99.603 99.033 98.044 96.532
64 100.000 100.000 99.852 99.399 98.499 97.051
65 100.000 100.000 99.862 99.429 98.562 97.157
66 100.000 100.000 99.871 99.459 98.625 97.263
67 100.000 100.000 99.880 99.491 98.691 97.374
68 100.000 100.000 99.889 99.520 98.751 97.474
69 100.000 100.000 99.889 99.526 98.775 97.529
70 100.000 100.000 99.889 99.533 98.802 97.589
71 100.000 100.000 99.889 99.543 98.835 97.660
72 100.000 100.000 99.889 99.550 98.861 97.719
73 100.000 100.000 99.889 99.556 98.885 97.773
74 100.000 100.000 99.889 99.562 98.905 97.820
75 100.000 100.000 99.898 99.591 98.964 97.918
76 100.000 100.000 99.898 99.597 98.988 97.970
77 100.000 100.000 99.898 99.602 99.008 98.016
78 100.000 100.000 99.898 99.608 99.031 98.068
79 100.000 100.000 99.898 99.615 99.054 98.120
80 100.000 100.000 99.898 99.620 99.074 98.166
81 100.000 100.000 99.898 99.626 99.097 98.217
82 100.000 100.000 99.898 99.633 99.123 98.273
83 100.000 100.000 99.898 99.638 99.139 98.312
84 100.000 100.000 99.908 99.668 99.201 98.412
85 100.000 100.000 99.908 99.672 99.217 98.449
86 100.000 100.000 99.908 99.677 99.236 98.493
87 100.000 100.000 99.908 99.681 99.252 98.530
88 100.000 100.000 99.908 99.687 99.271 98.573
89 100.000 100.000 99.908 99.690 99.284 98.603
90 100.000 100.000 99.908 99.695 99.303 98.644
91 100.000 100.000 99.908 99.699 99.318 98.679
92 100.000 100.000 99.908 99.702 99.330 98.707
93 100.000 100.000 99.917 99.729 99.381 98.786
94 100.000 100.000 99.926 99.756 99.432 98.863
95 100.000 100.000 99.926 99.759 99.444 98.890
96 100.000 100.000 99.935 99.786 99.494 98.968
97 100.000 100.000 99.945 99.814 99.545 99.045
98 100.000 100.000 99.945 99.817 99.557 99.072
99 100.000 100.000 99.954 99.844 99.607 99.149
100 100.000 100.000 99.963 99.871 99.658 99.225
101 100.000 100.000 99.972 99.898 99.708 99.302
102 100.000 100.000 99.982 99.925 99.759 99.379
103 100.000 100.000 99.991 99.952 99.810 99.456
104 100.000 100.000 100.000 99.979 99.860 99.533
105 100.000 100.000 100.000 99.982 99.872 99.559
106 100.000 100.000 100.000 99.985 99.883 99.585
107 100.000 100.000 100.000 99.989 99.895 99.611
108 100.000 100.000 100.000 99.992 99.906 99.637
109 100.000 100.000 100.000 99.995 99.918 99.662
110 100.000 100.000 100.000 99.998 99.929 99.687
111 100.000 100.000 100.000 100.000 99.937 99.706

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 86.047 72.190 59.753 49.088 40.161 32.791
4 90.698 79.606 68.411 57.967 48.646 40.553
5 95.349 86.095 75.455 64.943 55.244 46.624
6 95.349 87.601 78.445 68.973 59.835 51.387
7 100.000 94.090 85.461 75.886 66.342 57.351
8 100.000 95.133 87.547 78.734 69.646 60.855
9 100.000 95.713 88.840 80.662 72.050 63.561
10 100.000 96.176 89.910 82.308 74.155 65.983
11 100.000 96.756 91.147 84.101 76.353 68.433
12 100.000 96.987 91.747 85.100 77.711 70.075
13 100.000 97.103 92.117 85.795 78.737 71.393
14 100.000 97.451 92.901 86.989 80.266 73.165
15 100.000 97.451 93.021 87.327 80.867 74.028
16 100.000 97.567 93.400 88.033 81.893 75.323
17 100.000 97.567 93.529 88.388 82.512 76.199
18 100.000 97.683 93.880 89.025 83.430 77.357
19 100.000 97.798 94.184 89.547 84.166 78.283
20 100.000 98.378 95.366 91.192 86.111 80.388
21 100.000 98.610 95.911 92.046 87.229 81.708
22 100.000 98.725 96.215 92.570 87.970 82.642
23 100.000 98.725 96.261 92.718 88.262 83.100
24 100.000 99.073 97.009 93.811 89.616 84.629
25 100.000 99.073 97.083 94.022 89.997 85.185
26 100.000 99.189 97.388 94.537 90.711 86.068
27 100.000 99.305 97.674 95.011 91.366 86.882
28 100.000 99.305 97.729 95.174 91.667 87.333
29 100.000 99.305 97.840 95.472 92.180 88.048
30 100.000 99.305 97.877 95.590 92.413 88.413
31 100.000 99.305 97.914 95.712 92.658 88.800
32 100.000 99.305 97.951 95.828 92.883 89.152
33 100.000 99.305 97.978 95.920 93.073 89.458
34 100.000 99.305 98.006 96.012 93.259 89.758
35 100.000 99.305 98.043 96.124 93.475 90.091
36 100.000 99.305 98.071 96.215 93.657 90.382
37 100.000 99.305 98.117 96.348 93.902 90.747
38 100.000 99.305 98.145 96.439 94.084 91.036
39 100.000 99.305 98.172 96.528 94.260 91.314
40 100.000 99.305 98.191 96.586 94.377 91.506
41 100.000 99.305 98.218 96.671 94.543 91.765
42 100.000 99.305 98.246 96.755 94.703 92.012
43 100.000 99.305 98.255 96.795 94.800 92.186
44 100.000 99.421 98.505 97.162 95.261 92.720
45 100.000 99.421 98.514 97.198 95.344 92.868
46 100.000 99.421 98.523 97.234 95.425 93.011
47 100.000 99.421 98.541 97.294 95.548 93.209
48 100.000 99.421 98.551 97.328 95.625 93.345
49 100.000 99.421 98.560 97.368 95.718 93.509
50 100.000 99.421 98.569 97.401 95.792 93.640
51 100.000 99.421 98.588 97.460 95.909 93.827
52 100.000 99.421 98.597 97.494 95.985 93.961
53 100.000 99.537 98.846 97.861 96.445 94.490
54 100.000 99.652 99.095 98.226 96.902 95.014
55 100.000 99.652 99.105 98.260 96.975 95.141
56 100.000 99.768 99.354 98.625 97.431 95.664
57 100.000 99.884 99.603 98.991 97.887 96.185
58 100.000 99.884 99.603 99.002 97.928 96.271
59 100.000 100.000 99.852 99.368 98.384 96.793
60 100.000 100.000 99.852 99.374 98.408 96.850
61 100.000 100.000 99.852 99.385 98.448 96.935
62 100.000 100.000 99.862 99.418 98.518 97.054
63 100.000 100.000 99.862 99.426 98.548 97.121
64 100.000 100.000 99.862 99.434 98.578 97.187
65 100.000 100.000 99.871 99.464 98.638 97.287
66 100.000 100.000 99.871 99.472 98.668 97.352
67 100.000 100.000 99.880 99.502 98.730 97.456
68 100.000 100.000 99.880 99.508 98.754 97.509
69 100.000 100.000 99.889 99.539 98.816 97.613
70 100.000 100.000 99.889 99.546 98.843 97.671
71 100.000 100.000 99.889 99.550 98.859 97.710
72 100.000 100.000 99.898 99.579 98.918 97.806
73 100.000 100.000 99.898 99.584 98.938 97.851
74 100.000 100.000 99.908 99.613 98.993 97.940
75 100.000 100.000 99.908 99.618 99.012 97.983
76 100.000 100.000 99.908 99.622 99.028 98.020
77 100.000 100.000 99.908 99.627 99.047 98.062
78 100.000 100.000 99.908 99.631 99.063 98.099
79 100.000 100.000 99.908 99.635 99.079 98.135
80 100.000 100.000 99.908 99.638 99.088 98.160
81 100.000 100.000 99.908 99.642 99.104 98.196
82 100.000 100.000 99.908 99.647 99.123 98.237
83 100.000 100.000 99.908 99.651 99.138 98.273
84 100.000 100.000 99.908 99.654 99.151 98.302
85 100.000 100.000 99.908 99.658 99.166 98.336
86 100.000 100.000 99.908 99.663 99.182 98.371
87 100.000 100.000 99.908 99.667 99.197 98.406
88 100.000 100.000 99.908 99.671 99.212 98.440
89 100.000 100.000 99.908 99.676 99.231 98.480
90 100.000 100.000 99.908 99.679 99.243 98.509
91 100.000 100.000 99.908 99.683 99.258 98.543
92 100.000 100.000 99.908 99.685 99.267 98.566
93 100.000 100.000 99.917 99.713 99.318 98.644
94 100.000 100.000 99.926 99.740 99.369 98.721
95 100.000 100.000 99.935 99.767 99.420 98.799
96 100.000 100.000 99.935 99.771 99.435 98.832
97 100.000 100.000 99.945 99.798 99.485 98.909
98 100.000 100.000 99.954 99.825 99.536 98.986
99 100.000 100.000 99.954 99.828 99.547 99.012
100 100.000 100.000 99.963 99.855 99.598 99.089
101 100.000 100.000 99.972 99.882 99.649 99.166
102 100.000 100.000 99.982 99.909 99.699 99.243
103 100.000 100.000 99.982 99.911 99.708 99.264
104 100.000 100.000 99.991 99.939 99.758 99.341
105 100.000 100.000 99.991 99.942 99.770 99.367
106 100.000 100.000 100.000 99.969 99.821 99.444
107 100.000 100.000 100.000 99.972 99.832 99.469
108 100.000 100.000 100.000 99.975 99.843 99.495
109 100.000 100.000 100.000 99.978 99.855 99.520
110 100.000 100.000 100.000 99.980 99.863 99.539
111 100.000 100.000 100.000 99.983 99.874 99.563
112 100.000 100.000 100.000 99.985 99.882 99.582
113 100.000 100.000 100.000 99.986 99.887 99.595
114 100.000 100.000 100.000 99.989 99.895 99.613
115 100.000 100.000 100.000 99.990 99.899 99.625
116 100.000 100.000 100.000 99.991 99.904 99.638
117 100.000 100.000 100.000 99.992 99.909 99.649
118 100.000 100.000 100.000 99.994 99.916 99.666
119 100.000 100.000 100.000 99.994 99.918 99.672
120 100.000 100.000 100.000 99.996 99.925 99.689
121 100.000 100.000 100.000 99.997 99.929 99.700
122 100.000 100.000 100.000 99.998 99.934 99.710
123 100.000 100.000 100.000 99.999 99.938 99.719
124 100.000 100.000 100.000 99.999 99.939 99.723
125 100.000 100.000 100.000 100.000 99.943 99.733
126 100.000 100.000 100.000 100.000 99.943 99.737
127 100.000 100.000 100.000 100.000 99.944 99.739
128 100.000 100.000 100.000 100.000 99.944 99.740
129 100.000 100.000 100.000 100.000 99.944 99.741
130 100.000 100.000 100.000 100.000 99.945 99.742

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 4, Prioritisation Strength: 6

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 74.855 62.088 50.933 41.540 33.785
4 90.698 79.490 68.181 57.661 48.308 40.218
5 93.023 82.966 72.261 62.025 52.737 44.554
6 93.023 84.009 74.393 64.983 56.204 48.255
7 95.349 87.717 78.879 69.829 61.116 53.030
8 95.349 88.528 80.569 72.215 63.964 56.123
9 95.349 89.340 82.276 74.650 66.893 59.324
10 97.674 92.700 86.070 78.573 70.774 63.064
11 97.674 93.279 87.298 80.342 72.933 65.463
12 100.000 96.524 90.898 84.044 76.601 69.014
13 100.000 97.103 92.107 85.771 78.698 71.340
14 100.000 97.451 92.892 86.967 80.233 73.125
15 100.000 97.567 93.280 87.703 81.315 74.502
16 100.000 97.683 93.640 88.366 82.279 75.726
17 100.000 98.030 94.406 89.508 83.713 77.364
18 100.000 98.030 94.517 89.816 84.257 78.142
19 100.000 98.030 94.628 90.120 84.789 78.894
20 100.000 98.030 94.701 90.330 85.169 79.453
21 100.000 98.378 95.458 91.450 86.569 81.047
22 100.000 98.494 95.772 91.993 87.334 82.003
23 100.000 98.610 96.058 92.467 87.990 82.821
24 100.000 98.725 96.354 92.968 88.694 83.709
25 100.000 98.841 96.640 93.443 89.354 84.532
26 100.000 98.957 96.908 93.867 89.928 85.240
27 100.000 98.957 96.954 94.006 90.190 85.640
28 100.000 98.957 96.991 94.120 90.409 85.981
29 100.000 99.189 97.498 94.883 91.386 87.122
30 100.000 99.305 97.757 95.285 91.923 87.780
31 100.000 99.305 97.794 95.400 92.145 88.124
32 100.000 99.305 97.840 95.534 92.393 88.496
33 100.000 99.305 97.886 95.670 92.648 88.881
34 100.000 99.305 97.923 95.785 92.871 89.226
35 100.000 99.305 97.969 95.920 93.122 89.603
36 100.000 99.305 97.997 96.010 93.301 89.890
37 100.000 99.305 98.015 96.070 93.425 90.093
38 100.000 99.305 98.043 96.157 93.595 90.360
39 100.000 99.305 98.061 96.221 93.732 90.587
40 100.000 99.305 98.089 96.304 93.888 90.829
41 100.000 99.305 98.126 96.415 94.100 91.155
42 100.000 99.305 98.135 96.455 94.194 91.322
43 100.000 99.305 98.145 96.493 94.284 91.483
44 100.000 99.305 98.181 96.598 94.476 91.767
45 100.000 99.305 98.181 96.605 94.500 91.823
46 100.000 99.305 98.200 96.666 94.626 92.027
47 100.000 99.305 98.228 96.753 94.794 92.290
48 100.000 99.305 98.228 96.762 94.828 92.364
49 100.000 99.305 98.246 96.818 94.937 92.536
50 100.000 99.305 98.265 96.882 95.070 92.753
51 100.000 99.305 98.283 96.941 95.189 92.945
52 100.000 99.305 98.283 96.950 95.223 93.021
53 100.000 99.305 98.292 96.982 95.289 93.133
54 100.000 99.305 98.292 96.993 95.330 93.219
55 100.000 99.305 98.301 97.027 95.407 93.354
56 100.000 99.305 98.301 97.042 95.457 93.458
57 100.000 99.305 98.320 97.096 95.559 93.616
58 100.000 99.305 98.329 97.127 95.625 93.728
59 100.000 99.305 98.348 97.183 95.731 93.895
60 100.000 99.305 98.348 97.191 95.761 93.960
61 100.000 99.421 98.597 97.557 96.218 94.483
62 100.000 99.537 98.846 97.923 96.677 95.012
63 100.000 99.537 98.846 97.935 96.718 95.099
64 100.000 99.537 98.855 97.966 96.784 95.212
65 100.000 99.652 99.105 98.331 97.241 95.735
66 100.000 99.652 99.114 98.363 97.307 95.848
67 100.000 99.652 99.114 98.369 97.330 95.901
68 100.000 99.652 99.114 98.376 97.357 95.962
69 100.000 99.652 99.132 98.430 97.460 96.120
70 100.000 99.652 99.142 98.460 97.520 96.220
71 100.000 99.768 99.391 98.825 97.976 96.742
72 100.000 99.768 99.400 98.855 98.038 96.846
73 100.000 99.884 99.649 99.221 98.494 97.366
74 100.000 100.000 99.898 99.587 98.950 97.886
75 100.000 100.000 99.898 99.594 98.976 97.944
76 100.000 100.000 99.898 99.600 98.999 97.996
77 100.000 100.000 99.898 99.605 99.018 98.038
78 100.000 100.000 99.898 99.612 99.041 98.089
79 100.000 100.000 99.898 99.618 99.064 98.140
80 100.000 100.000 99.898 99.623 99.084 98.185
81 100.000 100.000 99.898 99.628 99.103 98.229
82 100.000 100.000 99.898 99.632 99.119 98.266
83 100.000 100.000 99.898 99.637 99.135 98.303
84 100.000 100.000 99.898 99.641 99.151 98.339
85 100.000 100.000 99.898 99.646 99.170 98.384
86 100.000 100.000 99.898 99.650 99.186 98.420
87 100.000 100.000 99.898 99.655 99.205 98.463
88 100.000 100.000 99.898 99.659 99.222 98.501
89 100.000 100.000 99.898 99.664 99.238 98.538
90 100.000 100.000 99.908 99.694 99.299 98.639
91 100.000 100.000 99.908 99.697 99.311 98.666
92 100.000 100.000 99.908 99.700 99.323 98.694
93 100.000 100.000 99.908 99.703 99.335 98.721
94 100.000 100.000 99.908 99.707 99.350 98.754
95 100.000 100.000 99.908 99.710 99.362 98.780
96 100.000 100.000 99.908 99.714 99.373 98.807
97 100.000 100.000 99.908 99.717 99.385 98.834
98 100.000 100.000 99.908 99.720 99.396 98.860
99 100.000 100.000 99.908 99.723 99.408 98.885
100 100.000 100.000 99.917 99.750 99.459 98.963
101 100.000 100.000 99.917 99.753 99.470 98.988
102 100.000 100.000 99.926 99.780 99.521 99.065
103 100.000 100.000 99.935 99.807 99.571 99.142
104 100.000 100.000 99.945 99.834 99.622 99.219
105 100.000 100.000 99.945 99.838 99.633 99.244
106 100.000 100.000 99.954 99.865 99.684 99.321
107 100.000 100.000 99.963 99.892 99.734 99.398
108 100.000 100.000 99.972 99.919 99.785 99.475
109 100.000 100.000 99.982 99.946 99.836 99.552
110 100.000 100.000 99.991 99.973 99.886 99.629
111 100.000 100.000 100.000 100.000 99.937 99.706

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 86.047 71.842 59.143 48.365 39.439 32.136
4 90.698 79.143 67.571 56.938 47.585 39.563
5 95.349 85.747 74.790 64.088 54.322 45.726
6 97.674 89.803 79.876 69.634 59.921 51.110
7 97.674 90.846 82.027 72.634 63.448 54.879
8 97.674 91.889 84.178 75.637 66.984 58.664
9 97.674 92.700 85.886 78.071 69.910 61.860
10 97.674 93.163 86.984 79.780 72.106 64.387
11 97.674 93.511 87.778 81.002 73.684 66.229
12 97.674 93.743 88.387 82.027 75.090 67.938
13 97.674 93.859 88.757 82.720 76.110 69.243
14 97.674 93.975 89.098 83.340 77.009 70.389
15 97.674 94.322 89.901 84.572 78.584 72.203
16 100.000 97.567 93.427 88.089 81.961 75.383
17 100.000 97.914 94.194 89.237 83.414 77.057
18 100.000 98.146 94.711 90.024 84.432 78.257
19 100.000 98.262 95.061 90.660 85.344 79.403
20 100.000 98.262 95.163 90.942 85.842 80.115
21 100.000 98.378 95.458 91.445 86.553 81.017
22 100.000 98.378 95.560 91.726 87.048 81.723
23 100.000 98.841 96.520 93.087 88.692 83.539
24 100.000 98.841 96.575 93.256 89.013 84.029
25 100.000 98.841 96.677 93.533 89.494 84.709
26 100.000 98.841 96.741 93.719 89.832 85.206
27 100.000 98.957 97.000 94.126 90.386 85.898
28 100.000 99.073 97.268 94.549 90.953 86.592
29 100.000 99.073 97.314 94.688 91.218 86.997
30 100.000 99.305 97.803 95.401 92.107 88.016
31 100.000 99.305 97.840 95.515 92.329 88.362
32 100.000 99.305 97.886 95.655 92.594 88.765
33 100.000 99.305 97.923 95.765 92.803 89.085
34 100.000 99.305 97.941 95.830 92.940 89.314
35 100.000 99.305 97.988 95.964 93.188 89.684
36 100.000 99.305 98.034 96.099 93.439 90.060
37 100.000 99.305 98.052 96.164 93.575 90.286
38 100.000 99.305 98.061 96.201 93.660 90.437
39 100.000 99.305 98.089 96.293 93.848 90.739
40 100.000 99.305 98.117 96.385 94.034 91.035
41 100.000 99.305 98.145 96.471 94.203 91.300
42 100.000 99.305 98.172 96.558 94.372 91.566
43 100.000 99.305 98.209 96.665 94.571 91.865
44 100.000 99.305 98.218 96.706 94.668 92.039
45 100.000 99.305 98.237 96.765 94.788 92.235
46 100.000 99.305 98.255 96.822 94.902 92.418
47 100.000 99.305 98.255 96.832 94.936 92.493
48 100.000 99.305 98.274 96.890 95.052 92.680
49 100.000 99.305 98.283 96.922 95.123 92.802
50 100.000 99.305 98.301 96.988 95.262 93.032
51 100.000 99.421 98.551 97.360 95.739 93.595
52 100.000 99.421 98.551 97.373 95.786 93.695
53 100.000 99.421 98.560 97.405 95.852 93.809
54 100.000 99.537 98.809 97.770 96.309 94.334
55 100.000 99.537 98.818 97.800 96.372 94.440
56 100.000 99.537 98.837 97.855 96.478 94.604
57 100.000 99.537 98.846 97.890 96.554 94.737
58 100.000 99.537 98.846 97.895 96.575 94.784
59 100.000 99.537 98.846 97.901 96.598 94.836
60 100.000 99.537 98.846 97.910 96.628 94.904
61 100.000 99.537 98.855 97.939 96.687 95.001
62 100.000 99.537 98.855 97.944 96.707 95.048
63 100.000 99.537 98.865 97.976 96.777 95.166
64 100.000 99.537 98.865 97.982 96.797 95.214
65 100.000 99.537 98.883 98.036 96.899 95.370
66 100.000 99.537 98.892 98.066 96.961 95.472
67 100.000 99.537 98.902 98.096 97.024 95.578
68 100.000 99.537 98.902 98.105 97.057 95.648
69 100.000 99.652 99.151 98.471 97.512 96.168
70 100.000 99.768 99.400 98.837 97.968 96.689
71 100.000 99.768 99.400 98.843 97.992 96.741
72 100.000 99.768 99.400 98.850 98.018 96.800
73 100.000 99.884 99.649 99.216 98.474 97.320
74 100.000 99.884 99.649 99.222 98.497 97.372
75 100.000 99.884 99.649 99.227 98.517 97.417
76 100.000 99.884 99.649 99.232 98.536 97.461
77 100.000 100.000 99.898 99.598 98.992 97.980
78 100.000 100.000 99.898 99.602 99.008 98.018
79 100.000 100.000 99.898 99.606 99.024 98.055
80 100.000 100.000 99.898 99.612 99.043 98.096
81 100.000 100.000 99.898 99.615 99.055 98.125
82 100.000 100.000 99.898 99.619 99.071 98.162
83 100.000 100.000 99.898 99.623 99.086 98.198
84 100.000 100.000 99.908 99.652 99.145 98.293
85 100.000 100.000 99.908 99.656 99.161 98.329
86 100.000 100.000 99.908 99.658 99.169 98.350
87 100.000 100.000 99.908 99.662 99.181 98.379
88 100.000 100.000 99.908 99.664 99.190 98.400
89 100.000 100.000 99.908 99.668 99.206 98.435
90 100.000 100.000 99.908 99.675 99.231 98.490
91 100.000 100.000 99.908 99.678 99.243 98.517
92 100.000 100.000 99.908 99.682 99.258 98.550
93 100.000 100.000 99.908 99.685 99.270 98.578
94 100.000 100.000 99.908 99.689 99.282 98.605
95 100.000 100.000 99.908 99.694 99.300 98.645
96 100.000 100.000 99.908 99.698 99.315 98.678
97 100.000 100.000 99.917 99.725 99.366 98.756
98 100.000 100.000 99.917 99.728 99.377 98.781
99 100.000 100.000 99.917 99.730 99.386 98.801
100 100.000 100.000 99.917 99.732 99.394 98.820
101 100.000 100.000 99.917 99.733 99.399 98.833
102 100.000 100.000 99.926 99.760 99.449 98.911
103 100.000 100.000 99.926 99.764 99.461 98.936
104 100.000 100.000 99.935 99.791 99.511 99.013
105 100.000 100.000 99.935 99.792 99.516 99.025
106 100.000 100.000 99.945 99.819 99.567 99.102
107 100.000 100.000 99.954 99.846 99.617 99.179
108 100.000 100.000 99.963 99.873 99.668 99.256
109 100.000 100.000 99.972 99.900 99.719 99.333
110 100.000 100.000 99.972 99.902 99.726 99.351
111 100.000 100.000 99.982 99.929 99.777 99.428
112 100.000 100.000 99.982 99.932 99.788 99.452
113 100.000 100.000 99.982 99.933 99.793 99.464
114 100.000 100.000 99.991 99.960 99.843 99.541
115 100.000 100.000 100.000 99.988 99.894 99.617
116 100.000 100.000 100.000 99.989 99.898 99.628
117 100.000 100.000 100.000 99.990 99.903 99.639
118 100.000 100.000 100.000 99.991 99.907 99.650
119 100.000 100.000 100.000 99.993 99.914 99.667
120 100.000 100.000 100.000 99.993 99.915 99.670
121 100.000 100.000 100.000 99.995 99.923 99.687
122 100.000 100.000 100.000 99.997 99.930 99.704
123 100.000 100.000 100.000 99.997 99.931 99.708
124 100.000 100.000 100.000 99.998 99.935 99.718
125 100.000 100.000 100.000 99.999 99.939 99.727
126 100.000 100.000 100.000 100.000 99.943 99.737
127 100.000 100.000 100.000 100.000 99.944 99.739
128 100.000 100.000 100.000 100.000 99.944 99.740
129 100.000 100.000 100.000 100.000 99.944 99.741
130 100.000 100.000 100.000 100.000 99.945 99.742

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 93.023 82.155 70.498 59.463 49.624 41.140
5 97.674 88.760 77.707 66.587 56.315 47.238
6 100.000 92.700 82.581 71.866 61.628 52.342
7 100.000 94.090 85.332 75.568 65.848 56.732
8 100.000 95.249 87.686 78.810 69.621 60.728
9 100.000 96.176 89.597 81.481 72.774 64.115
10 100.000 96.987 91.267 83.823 75.556 67.128
11 100.000 97.335 92.098 85.132 77.264 69.126
12 100.000 97.683 92.864 86.274 78.697 70.759
13 100.000 98.030 93.695 87.571 80.369 72.688
14 100.000 98.378 94.498 88.800 81.933 74.479
15 100.000 98.610 95.043 89.651 83.040 75.778
16 100.000 98.725 95.421 90.364 84.086 77.109
17 100.000 98.841 95.708 90.834 84.726 77.893
18 100.000 98.957 96.021 91.373 85.479 78.826
19 100.000 99.073 96.326 91.885 86.182 79.683
20 100.000 99.189 96.668 92.485 87.016 80.697
21 100.000 99.305 96.954 92.961 87.678 81.528
22 100.000 99.421 97.203 93.330 88.145 82.069
23 100.000 99.537 97.461 93.728 88.667 82.697
24 100.000 99.652 97.711 94.106 89.164 83.298
25 100.000 99.768 98.025 94.656 89.950 84.295
26 100.000 99.884 98.283 95.055 90.476 84.930
27 100.000 100.000 98.532 95.424 90.942 85.471
28 100.000 100.000 98.625 95.685 91.411 86.151
29 100.000 100.000 98.726 95.961 91.889 86.823
30 100.000 100.000 98.892 96.408 92.652 87.880
31 100.000 100.000 98.948 96.571 92.956 88.335
32 100.000 100.000 99.049 96.854 93.454 89.046
33 100.000 100.000 99.086 96.966 93.670 89.381
34 100.000 100.000 99.132 97.096 93.904 89.720
35 100.000 100.000 99.142 97.135 93.994 89.879
36 100.000 100.000 99.169 97.218 94.152 90.123
37 100.000 100.000 99.206 97.323 94.343 90.404
38 100.000 100.000 99.234 97.406 94.499 90.645
39 100.000 100.000 99.252 97.467 94.624 90.845
40 100.000 100.000 99.271 97.527 94.746 91.042
41 100.000 100.000 99.335 97.703 95.052 91.472
42 100.000 100.000 99.372 97.811 95.251 91.774
43 100.000 100.000 99.382 97.849 95.340 91.930
44 100.000 100.000 99.418 97.959 95.545 92.241
45 100.000 100.000 99.465 98.092 95.790 92.604
46 100.000 100.000 99.492 98.183 95.972 92.896
47 100.000 100.000 99.492 98.193 96.010 92.979
48 100.000 100.000 99.529 98.302 96.215 93.290
49 100.000 100.000 99.548 98.360 96.327 93.466
50 100.000 100.000 99.548 98.364 96.342 93.500
51 100.000 100.000 99.566 98.418 96.444 93.654
52 100.000 100.000 99.575 98.454 96.527 93.799
53 100.000 100.000 99.585 98.489 96.603 93.931
54 100.000 100.000 99.603 98.548 96.722 94.121
55 100.000 100.000 99.622 98.607 96.841 94.314
56 100.000 100.000 99.631 98.635 96.892 94.391
57 100.000 100.000 99.640 98.673 96.981 94.548
58 100.000 100.000 99.658 98.737 97.113 94.761
59 100.000 100.000 99.677 98.797 97.236 94.962
60 100.000 100.000 99.677 98.802 97.255 95.004
61 100.000 100.000 99.686 98.830 97.310 95.090
62 100.000 100.000 99.695 98.868 97.398 95.248
63 100.000 100.000 99.714 98.929 97.523 95.452
64 100.000 100.000 99.732 98.982 97.621 95.597
65 100.000 100.000 99.742 99.020 97.707 95.747
66 100.000 100.000 99.742 99.025 97.726 95.791
67 100.000 100.000 99.742 99.033 97.757 95.860
68 100.000 100.000 99.760 99.089 97.862 96.023
69 100.000 100.000 99.769 99.118 97.921 96.118
70 100.000 100.000 99.769 99.121 97.933 96.147
71 100.000 100.000 99.769 99.133 97.976 96.236
72 100.000 100.000 99.797 99.219 98.141 96.493
73 100.000 100.000 99.797 99.223 98.158 96.533
74 100.000 100.000 99.806 99.251 98.213 96.621
75 100.000 100.000 99.806 99.255 98.229 96.657
76 100.000 100.000 99.806 99.259 98.244 96.692
77 100.000 100.000 99.806 99.265 98.263 96.734
78 100.000 100.000 99.815 99.295 98.326 96.839
79 100.000 100.000 99.815 99.295 98.326 96.840
80 100.000 100.000 99.825 99.323 98.382 96.932
81 100.000 100.000 99.825 99.326 98.394 96.959
82 100.000 100.000 99.834 99.359 98.467 97.084
83 100.000 100.000 99.843 99.389 98.525 97.177
84 100.000 100.000 99.862 99.443 98.627 97.334
85 100.000 100.000 99.862 99.446 98.638 97.361
86 100.000 100.000 99.871 99.475 98.698 97.461
87 100.000 100.000 99.880 99.504 98.757 97.557
88 100.000 100.000 99.880 99.512 98.784 97.617
89 100.000 100.000 99.880 99.514 98.792 97.638
90 100.000 100.000 99.880 99.517 98.805 97.667
91 100.000 100.000 99.880 99.519 98.813 97.688
92 100.000 100.000 99.889 99.547 98.868 97.773
93 100.000 100.000 99.889 99.555 98.898 97.838
94 100.000 100.000 99.889 99.558 98.910 97.870
95 100.000 100.000 99.898 99.585 98.961 97.947
96 100.000 100.000 99.898 99.588 98.970 97.970
97 100.000 100.000 99.898 99.593 98.989 98.013
98 100.000 100.000 99.898 99.598 99.009 98.057
99 100.000 100.000 99.898 99.599 99.013 98.068
100 100.000 100.000 99.908 99.626 99.064 98.148
101 100.000 100.000 99.908 99.626 99.065 98.152
102 100.000 100.000 99.908 99.629 99.077 98.177
103 100.000 100.000 99.908 99.634 99.096 98.219
104 100.000 100.000 99.917 99.663 99.150 98.305
105 100.000 100.000 99.917 99.666 99.162 98.333
106 100.000 100.000 99.917 99.668 99.170 98.352
107 100.000 100.000 99.917 99.672 99.186 98.387
108 100.000 100.000 99.917 99.675 99.197 98.413
109 100.000 100.000 99.935 99.729 99.299 98.569
110 100.000 100.000 99.935 99.734 99.318 98.612
111 100.000 100.000 99.935 99.738 99.330 98.638
112 100.000 100.000 99.935 99.743 99.349 98.682
113 100.000 100.000 99.935 99.745 99.358 98.703
114 100.000 100.000 99.935 99.747 99.366 98.722
115 100.000 100.000 99.935 99.750 99.378 98.751
116 100.000 100.000 99.935 99.753 99.390 98.780
117 100.000 100.000 99.935 99.756 99.402 98.808
118 100.000 100.000 99.935 99.759 99.414 98.836
119 100.000 100.000 99.935 99.760 99.419 98.848
120 100.000 100.000 99.935 99.765 99.435 98.885
121 100.000 100.000 99.935 99.767 99.443 98.905
122 100.000 100.000 99.935 99.768 99.449 98.920
123 100.000 100.000 99.935 99.772 99.464 98.957
124 100.000 100.000 99.935 99.775 99.476 98.984
125 100.000 100.000 99.935 99.776 99.482 98.999
126 100.000 100.000 99.935 99.778 99.489 99.017
127 100.000 100.000 99.935 99.779 99.494 99.028
128 100.000 100.000 99.935 99.780 99.499 99.041
129 100.000 100.000 99.945 99.807 99.549 99.118
130 100.000 100.000 99.945 99.807 99.550 99.122
131 100.000 100.000 99.945 99.809 99.559 99.144
132 100.000 100.000 99.954 99.836 99.609 99.220
133 100.000 100.000 99.954 99.838 99.614 99.233
134 100.000 100.000 99.954 99.838 99.615 99.236
135 100.000 100.000 99.954 99.840 99.623 99.255
136 100.000 100.000 99.954 99.842 99.631 99.274
137 100.000 100.000 99.954 99.842 99.632 99.277
138 100.000 100.000 99.954 99.842 99.632 99.278
139 100.000 100.000 99.954 99.844 99.640 99.299
140 100.000 100.000 99.954 99.845 99.645 99.312
141 100.000 100.000 99.954 99.847 99.654 99.333
142 100.000 100.000 99.954 99.849 99.662 99.352
143 100.000 100.000 99.954 99.851 99.670 99.371
144 100.000 100.000 99.954 99.851 99.671 99.375
145 100.000 100.000 99.954 99.852 99.676 99.389
146 100.000 100.000 99.954 99.852 99.676 99.391
147 100.000 100.000 99.954 99.853 99.681 99.403
148 100.000 100.000 99.963 99.880 99.731 99.480
149 100.000 100.000 99.972 99.907 99.782 99.557
150 100.000 100.000 99.972 99.908 99.787 99.569
151 100.000 100.000 99.972 99.908 99.788 99.575
152 100.000 100.000 99.972 99.908 99.789 99.578
153 100.000 100.000 99.972 99.908 99.789 99.580
154 100.000 100.000 99.972 99.910 99.797 99.599
155 100.000 100.000 99.972 99.910 99.798 99.602
156 100.000 100.000 99.972 99.911 99.802 99.612
157 100.000 100.000 99.972 99.913 99.807 99.624
158 100.000 100.000 99.972 99.914 99.811 99.636
159 100.000 100.000 99.972 99.914 99.812 99.640
160 100.000 100.000 99.972 99.915 99.817 99.653
161 100.000 100.000 99.972 99.915 99.818 99.658
162 100.000 100.000 99.982 99.942 99.869 99.735
163 100.000 100.000 99.982 99.943 99.873 99.747
164 100.000 100.000 99.982 99.944 99.878 99.760
165 100.000 100.000 99.982 99.944 99.879 99.764
166 100.000 100.000 99.982 99.944 99.880 99.769
167 100.000 100.000 99.982 99.944 99.881 99.772
168 100.000 100.000 99.982 99.945 99.885 99.783
169 100.000 100.000 99.991 99.972 99.936 99.860
170 100.000 100.000 99.991 99.972 99.937 99.864
171 100.000 100.000 99.991 99.972 99.938 99.868
172 100.000 100.000 99.991 99.972 99.939 99.872
173 100.000 100.000 99.991 99.972 99.939 99.876
174 100.000 100.000 99.991 99.972 99.940 99.880
175 100.000 100.000 99.991 99.972 99.941 99.884
176 100.000 100.000 99.991 99.973 99.946 99.898
177 100.000 100.000 99.991 99.973 99.948 99.905
178 100.000 100.000 100.000 100.000 99.998 99.982
179 100.000 100.000 100.000 100.000 99.999 99.985
180 100.000 100.000 100.000 100.000 100.000 99.989

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 88.372 74.739 61.885 50.692 41.300 33.569
4 93.023 82.039 70.341 59.331 49.545 41.115
5 97.674 88.528 77.365 66.262 56.075 47.101
6 97.674 90.035 80.356 70.294 60.672 51.877
7 100.000 93.859 84.953 75.148 65.453 56.390
8 100.000 94.902 87.086 78.106 68.921 60.091
9 100.000 95.944 89.181 80.975 72.254 63.628
10 100.000 96.871 91.009 83.437 75.073 66.584
11 100.000 97.335 92.024 84.930 76.916 68.644
12 100.000 97.798 93.049 86.447 78.795 70.750
13 100.000 98.146 93.889 87.776 80.529 72.776
14 100.000 98.378 94.443 88.654 81.687 74.147
15 100.000 98.610 94.951 89.416 82.661 75.284
16 100.000 98.725 95.200 89.791 83.147 75.864
17 100.000 98.841 95.560 90.450 84.097 77.059
18 100.000 98.957 95.883 91.026 84.930 78.123
19 100.000 99.073 96.206 91.590 85.722 79.107
20 100.000 99.189 96.474 92.010 86.281 79.781
21 100.000 99.305 96.732 92.405 86.793 80.388
22 100.000 99.421 97.129 93.158 87.895 81.780
23 100.000 99.537 97.397 93.578 88.449 82.445
24 100.000 99.652 97.665 93.995 88.994 83.092
25 100.000 99.768 97.960 94.486 89.668 83.915
26 100.000 99.884 98.209 94.858 90.144 84.477
27 100.000 100.000 98.468 95.256 90.666 85.103
28 100.000 100.000 98.532 95.444 91.014 85.621
29 100.000 100.000 98.597 95.628 91.345 86.105
30 100.000 100.000 98.689 95.883 91.793 86.743
31 100.000 100.000 98.745 96.030 92.041 87.085
32 100.000 100.000 98.772 96.121 92.228 87.387
33 100.000 100.000 98.809 96.230 92.428 87.685
34 100.000 100.000 98.846 96.335 92.619 87.965
35 100.000 100.000 98.920 96.541 92.984 88.487
36 100.000 100.000 99.105 97.032 93.812 89.622
37 100.000 100.000 99.132 97.115 93.970 89.863
38 100.000 100.000 99.215 97.355 94.407 90.505
39 100.000 100.000 99.215 97.365 94.444 90.588
40 100.000 100.000 99.215 97.369 94.460 90.624
41 100.000 100.000 99.234 97.430 94.582 90.821
42 100.000 100.000 99.234 97.434 94.598 90.856
43 100.000 100.000 99.262 97.528 94.794 91.174
44 100.000 100.000 99.271 97.562 94.866 91.294
45 100.000 100.000 99.289 97.617 94.970 91.455
46 100.000 100.000 99.298 97.645 95.025 91.540
47 100.000 100.000 99.298 97.647 95.033 91.562
48 100.000 100.000 99.335 97.762 95.256 91.908
49 100.000 100.000 99.345 97.791 95.314 92.003
50 100.000 100.000 99.363 97.851 95.437 92.200
51 100.000 100.000 99.391 97.936 95.599 92.453
52 100.000 100.000 99.400 97.970 95.675 92.584
53 100.000 100.000 99.400 97.977 95.701 92.641
54 100.000 100.000 99.446 98.109 95.938 92.986
55 100.000 100.000 99.465 98.170 96.063 93.188
56 100.000 100.000 99.474 98.209 96.155 93.353
57 100.000 100.000 99.511 98.316 96.352 93.647
58 100.000 100.000 99.520 98.343 96.403 93.726
59 100.000 100.000 99.529 98.375 96.473 93.843
60 100.000 100.000 99.529 98.381 96.495 93.890
61 100.000 100.000 99.557 98.474 96.684 94.196
62 100.000 100.000 99.566 98.505 96.750 94.307
63 100.000 100.000 99.566 98.511 96.769 94.347
64 100.000 100.000 99.575 98.547 96.851 94.490
65 100.000 100.000 99.594 98.606 96.970 94.681
66 100.000 100.000 99.612 98.661 97.071 94.836
67 100.000 100.000 99.612 98.670 97.106 94.911
68 100.000 100.000 99.612 98.675 97.125 94.954
69 100.000 100.000 99.622 98.713 97.211 95.107
70 100.000 100.000 99.622 98.715 97.221 95.131
71 100.000 100.000 99.631 98.747 97.291 95.250
72 100.000 100.000 99.631 98.753 97.313 95.298
73 100.000 100.000 99.640 98.780 97.363 95.375
74 100.000 100.000 99.658 98.833 97.462 95.524
75 100.000 100.000 99.658 98.837 97.474 95.552
76 100.000 100.000 99.677 98.892 97.579 95.717
77 100.000 100.000 99.677 98.893 97.583 95.726
78 100.000 100.000 99.686 98.920 97.634 95.803
79 100.000 100.000 99.686 98.923 97.646 95.832
80 100.000 100.000 99.705 98.979 97.755 96.003
81 100.000 100.000 99.714 99.006 97.806 96.082
82 100.000 100.000 99.732 99.064 97.918 96.259
83 100.000 100.000 99.732 99.065 97.924 96.276
84 100.000 100.000 99.732 99.067 97.932 96.296
85 100.000 100.000 99.732 99.068 97.937 96.308
86 100.000 100.000 99.751 99.121 98.035 96.454
87 100.000 100.000 99.760 99.149 98.090 96.541
88 100.000 100.000 99.769 99.176 98.140 96.618
89 100.000 100.000 99.769 99.176 98.141 96.621
90 100.000 100.000 99.778 99.204 98.195 96.706
91 100.000 100.000 99.778 99.211 98.218 96.758
92 100.000 100.000 99.778 99.214 98.230 96.784
93 100.000 100.000 99.788 99.246 98.300 96.904
94 100.000 100.000 99.788 99.247 98.304 96.916
95 100.000 100.000 99.788 99.253 98.327 96.964
96 100.000 100.000 99.788 99.255 98.336 96.986
97 100.000 100.000 99.788 99.262 98.358 97.035
98 100.000 100.000 99.788 99.264 98.366 97.056
99 100.000 100.000 99.797 99.292 98.421 97.141
100 100.000 100.000 99.797 99.292 98.421 97.143
101 100.000 100.000 99.797 99.294 98.429 97.161
102 100.000 100.000 99.806 99.321 98.480 97.239
103 100.000 100.000 99.806 99.324 98.491 97.265
104 100.000 100.000 99.806 99.325 98.496 97.275
105 100.000 100.000 99.806 99.326 98.500 97.285
106 100.000 100.000 99.806 99.328 98.508 97.305
107 100.000 100.000 99.806 99.331 98.520 97.331
108 100.000 100.000 99.806 99.331 98.520 97.334
109 100.000 100.000 99.806 99.332 98.524 97.343
110 100.000 100.000 99.806 99.336 98.537 97.373
111 100.000 100.000 99.806 99.336 98.537 97.373
112 100.000 100.000 99.815 99.363 98.588 97.452
113 100.000 100.000 99.815 99.364 98.592 97.461
114 100.000 100.000 99.825 99.391 98.642 97.539
115 100.000 100.000 99.825 99.391 98.643 97.542
116 100.000 100.000 99.825 99.392 98.648 97.553
117 100.000 100.000 99.825 99.392 98.648 97.556
118 100.000 100.000 99.825 99.395 98.660 97.581
119 100.000 100.000 99.825 99.398 98.672 97.610
120 100.000 100.000 99.825 99.398 98.672 97.612
121 100.000 100.000 99.843 99.453 98.778 97.775
122 100.000 100.000 99.871 99.533 98.926 97.998
123 100.000 100.000 99.880 99.563 98.984 98.093
124 100.000 100.000 99.880 99.564 98.989 98.106
125 100.000 100.000 99.880 99.565 98.993 98.116
126 100.000 100.000 99.880 99.569 99.008 98.149
127 100.000 100.000 99.889 99.596 99.059 98.226
128 100.000 100.000 99.889 99.596 99.059 98.228
129 100.000 100.000 99.898 99.625 99.117 98.322
130 100.000 100.000 99.898 99.629 99.133 98.358
131 100.000 100.000 99.898 99.630 99.137 98.367
132 100.000 100.000 99.908 99.658 99.192 98.453
133 100.000 100.000 99.926 99.714 99.297 98.616
134 100.000 100.000 99.926 99.717 99.309 98.644
135 100.000 100.000 99.926 99.719 99.317 98.663
136 100.000 100.000 99.926 99.719 99.317 98.665
137 100.000 100.000 99.926 99.719 99.319 98.670
138 100.000 100.000 99.926 99.723 99.334 98.704
139 100.000 100.000 99.926 99.724 99.338 98.714
140 100.000 100.000 99.926 99.727 99.350 98.743
141 100.000 100.000 99.935 99.754 99.401 98.822
142 100.000 100.000 99.935 99.757 99.413 98.849
143 100.000 100.000 99.935 99.758 99.417 98.859
144 100.000 100.000 99.945 99.786 99.473 98.948
145 100.000 100.000 99.945 99.788 99.477 98.959
146 100.000 100.000 99.945 99.788 99.477 98.960
147 100.000 100.000 99.945 99.788 99.477 98.961
148 100.000 100.000 99.954 99.817 99.536 99.055
149 100.000 100.000 99.963 99.844 99.586 99.132
150 100.000 100.000 99.963 99.845 99.590 99.142
151 100.000 100.000 99.963 99.847 99.598 99.161
152 100.000 100.000 99.963 99.849 99.607 99.181
153 100.000 100.000 99.963 99.851 99.615 99.199
154 100.000 100.000 99.963 99.853 99.623 99.219
155 100.000 100.000 99.963 99.859 99.646 99.270
156 100.000 100.000 99.963 99.859 99.646 99.271
157 100.000 100.000 99.963 99.859 99.646 99.273
158 100.000 100.000 99.963 99.860 99.650 99.282
159 100.000 100.000 99.963 99.863 99.659 99.303
160 100.000 100.000 99.963 99.863 99.659 99.304
161 100.000 100.000 99.963 99.863 99.661 99.310
162 100.000 100.000 99.963 99.865 99.668 99.327
163 100.000 100.000 99.963 99.865 99.669 99.329
164 100.000 100.000 99.963 99.865 99.669 99.330
165 100.000 100.000 99.963 99.867 99.677 99.351
166 100.000 100.000 99.963 99.867 99.678 99.353
167 100.000 100.000 99.972 99.894 99.729 99.432
168 100.000 100.000 99.972 99.896 99.737 99.449
169 100.000 100.000 99.982 99.923 99.787 99.526
170 100.000 100.000 99.982 99.923 99.787 99.528
171 100.000 100.000 99.982 99.923 99.788 99.531
172 100.000 100.000 99.982 99.924 99.792 99.541
173 100.000 100.000 99.982 99.924 99.793 99.542
174 100.000 100.000 99.982 99.926 99.800 99.560
175 100.000 100.000 99.991 99.953 99.851 99.637
176 100.000 100.000 99.991 99.953 99.852 99.640
177 100.000 100.000 99.991 99.953 99.852 99.642
178 100.000 100.000 99.991 99.955 99.860 99.660
179 100.000 100.000 99.991 99.955 99.861 99.666
180 100.000 100.000 99.991 99.957 99.869 99.684
181 100.000 100.000 99.991 99.957 99.869 99.685
182 100.000 100.000 99.991 99.957 99.870 99.688
183 100.000 100.000 99.991 99.958 99.874 99.698
184 100.000 100.000 99.991 99.958 99.875 99.699
185 100.000 100.000 99.991 99.958 99.875 99.700
186 100.000 100.000 99.991 99.958 99.875 99.702
187 100.000 100.000 99.991 99.958 99.876 99.704
188 100.000 100.000 99.991 99.958 99.876 99.705
189 100.000 100.000 99.991 99.958 99.876 99.706
190 100.000 100.000 99.991 99.958 99.877 99.709
191 100.000 100.000 99.991 99.958 99.877 99.710
192 100.000 100.000 99.991 99.960 99.885 99.728
193 100.000 100.000 99.991 99.964 99.897 99.756
194 100.000 100.000 99.991 99.964 99.897 99.758
195 100.000 100.000 99.991 99.964 99.898 99.760
196 100.000 100.000 99.991 99.964 99.898 99.763
197 100.000 100.000 99.991 99.964 99.899 99.766
198 100.000 100.000 99.991 99.966 99.907 99.784
199 100.000 100.000 99.991 99.966 99.907 99.786
200 100.000 100.000 99.991 99.967 99.911 99.795
201 100.000 100.000 99.991 99.967 99.912 99.797
202 100.000 100.000 99.991 99.967 99.912 99.799
203 100.000 100.000 100.000 99.994 99.963 99.876
204 100.000 100.000 100.000 99.994 99.963 99.876
205 100.000 100.000 100.000 99.995 99.967 99.888
206 100.000 100.000 100.000 99.995 99.968 99.890
207 100.000 100.000 100.000 99.995 99.968 99.892
208 100.000 100.000 100.000 99.996 99.973 99.905
209 100.000 100.000 100.000 99.996 99.973 99.906
210 100.000 100.000 100.000 99.996 99.974 99.908
211 100.000 100.000 100.000 99.996 99.974 99.908
212 100.000 100.000 100.000 99.997 99.978 99.919
213 100.000 100.000 100.000 99.998 99.982 99.929
214 100.000 100.000 100.000 99.998 99.983 99.931
215 100.000 100.000 100.000 99.998 99.983 99.934
216 100.000 100.000 100.000 99.998 99.983 99.935
217 100.000 100.000 100.000 99.998 99.984 99.936
218 100.000 100.000 100.000 99.998 99.984 99.937
219 100.000 100.000 100.000 99.998 99.984 99.938
220 100.000 100.000 100.000 99.999 99.989 99.952
221 100.000 100.000 100.000 99.999 99.990 99.955
222 100.000 100.000 100.000 99.999 99.990 99.956
223 100.000 100.000 100.000 99.999 99.990 99.957
224 100.000 100.000 100.000 99.999 99.991 99.958
225 100.000 100.000 100.000 99.999 99.991 99.960
226 100.000 100.000 100.000 99.999 99.991 99.961
227 100.000 100.000 100.000 99.999 99.992 99.963
228 100.000 100.000 100.000 99.999 99.992 99.965
229 100.000 100.000 100.000 99.999 99.992 99.966
230 100.000 100.000 100.000 99.999 99.993 99.969
231 100.000 100.000 100.000 99.999 99.994 99.972
232 100.000 100.000 100.000 99.999 99.994 99.973
233 100.000 100.000 100.000 100.000 99.998 99.982
234 100.000 100.000 100.000 100.000 99.998 99.985
235 100.000 100.000 100.000 100.000 99.999 99.987
236 100.000 100.000 100.000 100.000 100.000 99.990
237 100.000 100.000 100.000 100.000 100.000 99.992
238 100.000 100.000 100.000 100.000 100.000 99.992

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 95.349 84.936 73.064 61.614 51.341 42.469
5 95.349 86.790 76.673 66.387 56.683 47.922
6 100.000 93.279 83.763 73.469 63.436 54.172
7 100.000 94.438 86.098 76.667 67.141 58.084
8 100.000 95.365 88.009 79.336 70.290 61.465
9 100.000 96.060 89.486 81.464 72.878 64.326
10 100.000 96.640 90.769 83.365 75.232 66.960
11 100.000 97.103 91.803 84.909 77.164 69.143
12 100.000 97.451 92.643 86.239 78.902 71.175
13 100.000 97.798 93.474 87.549 80.611 73.175
14 100.000 98.146 94.286 88.808 82.234 75.058
15 100.000 98.378 94.868 89.764 83.529 76.624
16 100.000 98.610 95.431 90.666 84.726 78.047
17 100.000 98.725 95.800 91.345 85.702 79.269
18 100.000 98.957 96.326 92.153 86.747 80.495
19 100.000 99.073 96.658 92.751 87.612 81.596
20 100.000 99.073 96.797 93.128 88.264 82.509
21 100.000 99.189 97.101 93.651 89.003 83.440
22 100.000 99.189 97.212 93.957 89.540 84.202
23 100.000 99.305 97.498 94.430 90.193 85.015
24 100.000 99.421 97.775 94.874 90.787 85.734
25 100.000 99.421 97.868 95.130 91.238 86.378
26 100.000 99.537 98.126 95.528 91.761 87.004
27 100.000 99.652 98.385 95.931 92.300 87.663
28 100.000 99.768 98.643 96.332 92.833 88.311
29 100.000 99.884 98.892 96.704 93.309 88.873
30 100.000 100.000 99.142 97.077 93.792 89.448
31 100.000 100.000 99.197 97.234 94.071 89.852
32 100.000 100.000 99.252 97.400 94.386 90.328
33 100.000 100.000 99.298 97.538 94.644 90.717
34 100.000 100.000 99.345 97.676 94.905 91.113
35 100.000 100.000 99.391 97.812 95.156 91.490
36 100.000 100.000 99.437 97.951 95.421 91.892
37 100.000 100.000 99.483 98.089 95.680 92.287
38 100.000 100.000 99.520 98.197 95.882 92.592
39 100.000 100.000 99.557 98.301 96.069 92.868
40 100.000 100.000 99.594 98.412 96.277 93.181
41 100.000 100.000 99.631 98.517 96.468 93.462
42 100.000 100.000 99.668 98.631 96.688 93.801
43 100.000 100.000 99.705 98.737 96.879 94.081
44 100.000 100.000 99.732 98.827 97.062 94.375
45 100.000 100.000 99.760 98.908 97.213 94.604
46 100.000 100.000 99.788 98.992 97.372 94.847
47 100.000 100.000 99.815 99.076 97.534 95.096
48 100.000 100.000 99.834 99.138 97.660 95.300
49 100.000 100.000 99.852 99.199 97.787 95.511
50 100.000 100.000 99.871 99.253 97.889 95.669
51 100.000 100.000 99.889 99.312 98.005 95.852
52 100.000 100.000 99.898 99.340 98.059 95.937
53 100.000 100.000 99.908 99.370 98.122 96.042
54 100.000 100.000 99.917 99.397 98.173 96.122
55 100.000 100.000 99.926 99.424 98.224 96.201
56 100.000 100.000 99.935 99.464 98.319 96.374
57 100.000 100.000 99.945 99.491 98.370 96.452
58 100.000 100.000 99.954 99.518 98.421 96.531
59 100.000 100.000 99.963 99.545 98.472 96.608
60 100.000 100.000 99.972 99.572 98.523 96.686
61 100.000 100.000 99.982 99.599 98.574 96.765
62 100.000 100.000 99.991 99.626 98.624 96.842
63 100.000 100.000 100.000 99.653 98.675 96.919
64 100.000 100.000 100.000 99.659 98.698 96.968
65 100.000 100.000 100.000 99.672 98.743 97.067
66 100.000 100.000 100.000 99.685 98.791 97.173
67 100.000 100.000 100.000 99.688 98.801 97.198
68 100.000 100.000 100.000 99.688 98.802 97.203
69 100.000 100.000 100.000 99.693 98.821 97.247
70 100.000 100.000 100.000 99.703 98.858 97.327
71 100.000 100.000 100.000 99.709 98.881 97.377
72 100.000 100.000 100.000 99.719 98.915 97.450
73 100.000 100.000 100.000 99.725 98.937 97.500
74 100.000 100.000 100.000 99.729 98.953 97.535
75 100.000 100.000 100.000 99.736 98.980 97.597
76 100.000 100.000 100.000 99.742 98.999 97.640
77 100.000 100.000 100.000 99.747 99.019 97.684
78 100.000 100.000 100.000 99.754 99.045 97.743
79 100.000 100.000 100.000 99.763 99.076 97.809
80 100.000 100.000 100.000 99.769 99.098 97.858
81 100.000 100.000 100.000 99.776 99.124 97.913
82 100.000 100.000 100.000 99.782 99.147 97.965
83 100.000 100.000 100.000 99.784 99.155 97.986
84 100.000 100.000 100.000 99.789 99.170 98.018
85 100.000 100.000 100.000 99.792 99.183 98.048
86 100.000 100.000 100.000 99.795 99.195 98.076
87 100.000 100.000 100.000 99.798 99.207 98.105
88 100.000 100.000 100.000 99.802 99.223 98.142
89 100.000 100.000 100.000 99.810 99.253 98.210
90 100.000 100.000 100.000 99.816 99.273 98.254
91 100.000 100.000 100.000 99.819 99.285 98.282
92 100.000 100.000 100.000 99.821 99.293 98.301
93 100.000 100.000 100.000 99.825 99.309 98.337
94 100.000 100.000 100.000 99.832 99.335 98.393
95 100.000 100.000 100.000 99.839 99.357 98.443
96 100.000 100.000 100.000 99.843 99.373 98.477
97 100.000 100.000 100.000 99.848 99.392 98.519
98 100.000 100.000 100.000 99.853 99.411 98.563
99 100.000 100.000 100.000 99.856 99.423 98.590
100 100.000 100.000 100.000 99.858 99.431 98.610
101 100.000 100.000 100.000 99.863 99.447 98.645
102 100.000 100.000 100.000 99.868 99.466 98.689
103 100.000 100.000 100.000 99.872 99.482 98.724
104 100.000 100.000 100.000 99.875 99.493 98.750
105 100.000 100.000 100.000 99.883 99.523 98.817
106 100.000 100.000 100.000 99.886 99.535 98.844
107 100.000 100.000 100.000 99.890 99.546 98.869
108 100.000 100.000 100.000 99.893 99.558 98.897
109 100.000 100.000 100.000 99.896 99.571 98.926
110 100.000 100.000 100.000 99.900 99.587 98.964
111 100.000 100.000 100.000 99.904 99.603 99.000
112 100.000 100.000 100.000 99.906 99.611 99.022
113 100.000 100.000 100.000 99.911 99.631 99.065
114 100.000 100.000 100.000 99.917 99.650 99.108
115 100.000 100.000 100.000 99.918 99.654 99.120
116 100.000 100.000 100.000 99.920 99.663 99.141
117 100.000 100.000 100.000 99.928 99.693 99.206
118 100.000 100.000 100.000 99.932 99.708 99.241
119 100.000 100.000 100.000 99.933 99.713 99.254
120 100.000 100.000 100.000 99.935 99.721 99.274
121 100.000 100.000 100.000 99.941 99.741 99.319
122 100.000 100.000 100.000 99.945 99.756 99.352
123 100.000 100.000 100.000 99.947 99.765 99.376
124 100.000 100.000 100.000 99.948 99.770 99.386
125 100.000 100.000 100.000 99.951 99.781 99.412
126 100.000 100.000 100.000 99.952 99.785 99.423
127 100.000 100.000 100.000 99.955 99.797 99.451
128 100.000 100.000 100.000 99.957 99.805 99.470
129 100.000 100.000 100.000 99.958 99.810 99.482
130 100.000 100.000 100.000 99.959 99.815 99.494
131 100.000 100.000 100.000 99.963 99.826 99.520
132 100.000 100.000 100.000 99.967 99.842 99.556
133 100.000 100.000 100.000 99.968 99.846 99.568
134 100.000 100.000 100.000 99.969 99.852 99.583
135 100.000 100.000 100.000 99.970 99.856 99.594
136 100.000 100.000 100.000 99.970 99.858 99.600
137 100.000 100.000 100.000 99.971 99.862 99.612
138 100.000 100.000 100.000 99.972 99.867 99.624
139 100.000 100.000 100.000 99.974 99.875 99.642
140 100.000 100.000 100.000 99.975 99.879 99.655
141 100.000 100.000 100.000 99.976 99.884 99.665
142 100.000 100.000 100.000 99.977 99.888 99.677
143 100.000 100.000 100.000 99.979 99.896 99.694
144 100.000 100.000 100.000 99.980 99.900 99.705
145 100.000 100.000 100.000 99.981 99.904 99.715
146 100.000 100.000 100.000 99.982 99.910 99.730
147 100.000 100.000 100.000 99.983 99.915 99.745
148 100.000 100.000 100.000 99.983 99.916 99.748
149 100.000 100.000 100.000 99.985 99.923 99.766
150 100.000 100.000 100.000 99.986 99.928 99.777
151 100.000 100.000 100.000 99.988 99.932 99.788
152 100.000 100.000 100.000 99.988 99.933 99.792
153 100.000 100.000 100.000 99.989 99.938 99.804
154 100.000 100.000 100.000 99.991 99.946 99.823
155 100.000 100.000 100.000 99.991 99.947 99.827
156 100.000 100.000 100.000 99.992 99.951 99.839
157 100.000 100.000 100.000 99.992 99.953 99.846
158 100.000 100.000 100.000 99.993 99.957 99.857
159 100.000 100.000 100.000 99.993 99.957 99.858
160 100.000 100.000 100.000 99.993 99.958 99.861
161 100.000 100.000 100.000 99.994 99.962 99.871
162 100.000 100.000 100.000 99.995 99.966 99.881
163 100.000 100.000 100.000 99.995 99.967 99.885
164 100.000 100.000 100.000 99.997 99.975 99.904
165 100.000 100.000 100.000 99.997 99.976 99.907
166 100.000 100.000 100.000 99.997 99.977 99.911
167 100.000 100.000 100.000 99.997 99.978 99.916
168 100.000 100.000 100.000 99.998 99.982 99.927
169 100.000 100.000 100.000 99.998 99.983 99.930
170 100.000 100.000 100.000 99.998 99.984 99.933
171 100.000 100.000 100.000 99.999 99.988 99.946
172 100.000 100.000 100.000 99.999 99.990 99.953
173 100.000 100.000 100.000 99.999 99.991 99.957
174 100.000 100.000 100.000 99.999 99.991 99.958
175 100.000 100.000 100.000 100.000 99.995 99.969
176 100.000 100.000 100.000 100.000 99.996 99.973
177 100.000 100.000 100.000 100.000 99.998 99.979
178 100.000 100.000 100.000 100.000 99.999 99.983
179 100.000 100.000 100.000 100.000 99.999 99.986
180 100.000 100.000 100.000 100.000 100.000 99.989

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 90.698 77.752 64.802 53.194 43.303 35.100
4 95.349 84.936 73.064 61.614 51.341 42.469
5 95.349 86.790 76.673 66.387 56.683 47.922
6 97.674 90.614 81.427 71.625 62.058 53.177
7 100.000 94.322 85.913 76.471 66.969 57.952
8 100.000 95.017 87.464 78.771 69.810 61.111
9 100.000 95.713 88.941 80.898 72.398 63.972
10 100.000 96.524 90.594 83.199 75.121 66.919
11 100.000 97.219 92.024 85.208 77.516 69.526
12 100.000 97.683 93.077 86.800 79.523 71.808
13 100.000 98.030 93.898 88.077 81.165 73.703
14 100.000 98.378 94.701 89.316 82.762 75.559
15 100.000 98.494 95.089 90.045 83.824 76.897
16 100.000 98.725 95.652 90.945 85.014 78.308
17 100.000 98.841 96.003 91.583 85.935 79.470
18 100.000 98.957 96.345 92.194 86.803 80.554
19 100.000 98.957 96.492 92.579 87.443 81.417
20 100.000 99.073 96.825 93.160 88.253 82.411
21 100.000 99.189 97.138 93.701 89.010 83.350
22 100.000 99.305 97.443 94.232 89.774 84.329
23 100.000 99.305 97.545 94.522 90.295 85.086
24 100.000 99.421 97.812 94.949 90.874 85.796
25 100.000 99.421 97.895 95.184 91.295 86.406
26 100.000 99.421 97.969 95.402 91.698 87.003
27 100.000 99.537 98.218 95.775 92.177 87.571
28 100.000 99.652 98.468 96.148 92.660 88.146
29 100.000 99.768 98.717 96.522 93.144 88.722
30 100.000 99.768 98.782 96.711 93.491 89.238
31 100.000 99.884 99.031 97.081 93.961 89.788
32 100.000 100.000 99.280 97.449 94.428 90.333
33 100.000 100.000 99.326 97.586 94.682 90.712
34 100.000 100.000 99.372 97.721 94.934 91.092
35 100.000 100.000 99.418 97.865 95.211 91.519
36 100.000 100.000 99.455 97.971 95.406 91.808
37 100.000 100.000 99.492 98.088 95.634 92.165
38 100.000 100.000 99.529 98.205 95.867 92.530
39 100.000 100.000 99.566 98.311 96.057 92.810
40 100.000 100.000 99.603 98.420 96.263 93.123
41 100.000 100.000 99.640 98.535 96.484 93.464
42 100.000 100.000 99.677 98.647 96.699 93.792
43 100.000 100.000 99.714 98.767 96.938 94.169
44 100.000 100.000 99.742 98.854 97.112 94.448
45 100.000 100.000 99.769 98.943 97.287 94.723
46 100.000 100.000 99.797 99.026 97.445 94.965
47 100.000 100.000 99.815 99.088 97.572 95.174
48 100.000 100.000 99.834 99.146 97.689 95.364
49 100.000 100.000 99.852 99.215 97.841 95.625
50 100.000 100.000 99.862 99.245 97.902 95.724
51 100.000 100.000 99.871 99.288 98.007 95.913
52 100.000 100.000 99.880 99.319 98.072 96.020
53 100.000 100.000 99.889 99.351 98.141 96.139
54 100.000 100.000 99.898 99.379 98.196 96.224
55 100.000 100.000 99.908 99.418 98.288 96.391
56 100.000 100.000 99.917 99.445 98.339 96.470
57 100.000 100.000 99.926 99.473 98.393 96.556
58 100.000 100.000 99.935 99.500 98.444 96.635
59 100.000 100.000 99.945 99.529 98.503 96.730
60 100.000 100.000 99.954 99.565 98.582 96.865
61 100.000 100.000 99.963 99.592 98.633 96.944
62 100.000 100.000 99.972 99.619 98.684 97.023
63 100.000 100.000 99.982 99.655 98.767 97.168
64 100.000 100.000 99.991 99.682 98.817 97.244
65 100.000 100.000 100.000 99.709 98.868 97.323
66 100.000 100.000 100.000 99.719 98.902 97.396
67 100.000 100.000 100.000 99.730 98.943 97.484
68 100.000 100.000 100.000 99.733 98.955 97.512
69 100.000 100.000 100.000 99.741 98.981 97.571
70 100.000 100.000 100.000 99.746 99.001 97.614
71 100.000 100.000 100.000 99.749 99.012 97.641
72 100.000 100.000 100.000 99.756 99.040 97.704
73 100.000 100.000 100.000 99.761 99.059 97.745
74 100.000 100.000 100.000 99.768 99.081 97.793
75 100.000 100.000 100.000 99.770 99.089 97.812
76 100.000 100.000 100.000 99.773 99.100 97.836
77 100.000 100.000 100.000 99.780 99.125 97.889
78 100.000 100.000 100.000 99.790 99.159 97.962
79 100.000 100.000 100.000 99.791 99.163 97.971
80 100.000 100.000 100.000 99.796 99.182 98.015
81 100.000 100.000 100.000 99.803 99.209 98.076
82 100.000 100.000 100.000 99.808 99.228 98.119
83 100.000 100.000 100.000 99.814 99.247 98.160
84 100.000 100.000 100.000 99.817 99.258 98.185
85 100.000 100.000 100.000 99.822 99.277 98.225
86 100.000 100.000 100.000 99.825 99.289 98.254
87 100.000 100.000 100.000 99.834 99.322 98.323
88 100.000 100.000 100.000 99.835 99.326 98.335
89 100.000 100.000 100.000 99.836 99.331 98.347
90 100.000 100.000 100.000 99.836 99.332 98.350
91 100.000 100.000 100.000 99.840 99.344 98.379
92 100.000 100.000 100.000 99.842 99.352 98.399
93 100.000 100.000 100.000 99.847 99.372 98.442
94 100.000 100.000 100.000 99.850 99.384 98.472
95 100.000 100.000 100.000 99.851 99.390 98.487
96 100.000 100.000 100.000 99.852 99.396 98.505
97 100.000 100.000 100.000 99.859 99.422 98.564
98 100.000 100.000 100.000 99.864 99.439 98.604
99 100.000 100.000 100.000 99.868 99.455 98.642
100 100.000 100.000 100.000 99.869 99.460 98.653
101 100.000 100.000 100.000 99.870 99.464 98.665
102 100.000 100.000 100.000 99.874 99.480 98.701
103 100.000 100.000 100.000 99.876 99.488 98.719
104 100.000 100.000 100.000 99.877 99.492 98.731
105 100.000 100.000 100.000 99.881 99.508 98.765
106 100.000 100.000 100.000 99.885 99.524 98.802
107 100.000 100.000 100.000 99.891 99.543 98.847
108 100.000 100.000 100.000 99.891 99.544 98.850
109 100.000 100.000 100.000 99.894 99.556 98.879
110 100.000 100.000 100.000 99.897 99.568 98.907
111 100.000 100.000 100.000 99.898 99.572 98.916
112 100.000 100.000 100.000 99.898 99.572 98.917
113 100.000 100.000 100.000 99.901 99.584 98.944
114 100.000 100.000 100.000 99.903 99.592 98.963
115 100.000 100.000 100.000 99.906 99.604 98.990
116 100.000 100.000 100.000 99.907 99.608 99.000
117 100.000 100.000 100.000 99.908 99.612 99.011
118 100.000 100.000 100.000 99.909 99.617 99.022
119 100.000 100.000 100.000 99.910 99.621 99.033
120 100.000 100.000 100.000 99.914 99.633 99.059
121 100.000 100.000 100.000 99.918 99.648 99.093
122 100.000 100.000 100.000 99.919 99.653 99.106
123 100.000 100.000 100.000 99.921 99.661 99.124
124 100.000 100.000 100.000 99.921 99.661 99.126
125 100.000 100.000 100.000 99.921 99.662 99.130
126 100.000 100.000 100.000 99.922 99.666 99.139
127 100.000 100.000 100.000 99.925 99.678 99.168
128 100.000 100.000 100.000 99.932 99.705 99.229
129 100.000 100.000 100.000 99.932 99.705 99.229
130 100.000 100.000 100.000 99.932 99.706 99.231
131 100.000 100.000 100.000 99.935 99.717 99.257
132 100.000 100.000 100.000 99.940 99.733 99.292
133 100.000 100.000 100.000 99.942 99.741 99.313
134 100.000 100.000 100.000 99.942 99.742 99.315
135 100.000 100.000 100.000 99.943 99.746 99.327
136 100.000 100.000 100.000 99.945 99.755 99.347
137 100.000 100.000 100.000 99.950 99.774 99.389
138 100.000 100.000 100.000 99.952 99.781 99.407
139 100.000 100.000 100.000 99.954 99.789 99.424
140 100.000 100.000 100.000 99.956 99.797 99.442
141 100.000 100.000 100.000 99.959 99.809 99.470
142 100.000 100.000 100.000 99.960 99.813 99.479
143 100.000 100.000 100.000 99.960 99.813 99.482
144 100.000 100.000 100.000 99.961 99.818 99.493
145 100.000 100.000 100.000 99.961 99.819 99.497
146 100.000 100.000 100.000 99.961 99.819 99.498
147 100.000 100.000 100.000 99.964 99.827 99.516
148 100.000 100.000 100.000 99.964 99.827 99.518
149 100.000 100.000 100.000 99.965 99.832 99.532
150 100.000 100.000 100.000 99.967 99.840 99.551
151 100.000 100.000 100.000 99.969 99.848 99.569
152 100.000 100.000 100.000 99.969 99.849 99.570
153 100.000 100.000 100.000 99.969 99.849 99.573
154 100.000 100.000 100.000 99.969 99.849 99.573
155 100.000 100.000 100.000 99.969 99.849 99.574
156 100.000 100.000 100.000 99.969 99.850 99.578
157 100.000 100.000 100.000 99.971 99.858 99.596
158 100.000 100.000 100.000 99.971 99.859 99.599
159 100.000 100.000 100.000 99.971 99.859 99.600
160 100.000 100.000 100.000 99.971 99.859 99.601
161 100.000 100.000 100.000 99.973 99.867 99.618
162 100.000 100.000 100.000 99.974 99.871 99.629
163 100.000 100.000 100.000 99.974 99.871 99.630
164 100.000 100.000 100.000 99.976 99.879 99.649
165 100.000 100.000 100.000 99.977 99.884 99.660
166 100.000 100.000 100.000 99.978 99.888 99.671
167 100.000 100.000 100.000 99.978 99.889 99.675
168 100.000 100.000 100.000 99.980 99.897 99.693
169 100.000 100.000 100.000 99.980 99.898 99.695
170 100.000 100.000 100.000 99.980 99.898 99.699
171 100.000 100.000 100.000 99.980 99.899 99.701
172 100.000 100.000 100.000 99.981 99.903 99.711
173 100.000 100.000 100.000 99.981 99.904 99.717
174 100.000 100.000 100.000 99.981 99.905 99.718
175 100.000 100.000 100.000 99.981 99.905 99.719
176 100.000 100.000 100.000 99.981 99.906 99.723
177 100.000 100.000 100.000 99.981 99.906 99.724
178 100.000 100.000 100.000 99.981 99.906 99.726
179 100.000 100.000 100.000 99.982 99.911 99.737
180 100.000 100.000 100.000 99.982 99.912 99.741
181 100.000 100.000 100.000 99.986 99.927 99.776
182 100.000 100.000 100.000 99.989 99.935 99.793
183 100.000 100.000 100.000 99.990 99.939 99.803
184 100.000 100.000 100.000 99.991 99.943 99.813
185 100.000 100.000 100.000 99.992 99.947 99.823
186 100.000 100.000 100.000 99.992 99.948 99.825
187 100.000 100.000 100.000 99.994 99.955 99.843
188 100.000 100.000 100.000 99.994 99.956 99.845
189 100.000 100.000 100.000 99.995 99.960 99.857
190 100.000 100.000 100.000 99.995 99.961 99.860
191 100.000 100.000 100.000 99.995 99.962 99.864
192 100.000 100.000 100.000 99.995 99.962 99.865
193 100.000 100.000 100.000 99.995 99.962 99.865
194 100.000 100.000 100.000 99.995 99.962 99.866
195 100.000 100.000 100.000 99.995 99.963 99.868
196 100.000 100.000 100.000 99.995 99.964 99.872
197 100.000 100.000 100.000 99.995 99.964 99.874
198 100.000 100.000 100.000 99.995 99.964 99.874
199 100.000 100.000 100.000 99.996 99.969 99.886
200 100.000 100.000 100.000 99.996 99.969 99.886
201 100.000 100.000 100.000 99.997 99.973 99.895
202 100.000 100.000 100.000 99.997 99.973 99.896
203 100.000 100.000 100.000 99.998 99.977 99.905
204 100.000 100.000 100.000 99.998 99.977 99.908
205 100.000 100.000 100.000 99.998 99.978 99.910
206 100.000 100.000 100.000 99.999 99.982 99.919
207 100.000 100.000 100.000 99.999 99.982 99.920
208 100.000 100.000 100.000 99.999 99.983 99.922
209 100.000 100.000 100.000 99.999 99.983 99.926
210 100.000 100.000 100.000 100.000 99.988 99.938
211 100.000 100.000 100.000 100.000 99.988 99.939
212 100.000 100.000 100.000 100.000 99.989 99.942
213 100.000 100.000 100.000 100.000 99.989 99.944
214 100.000 100.000 100.000 100.000 99.990 99.946
215 100.000 100.000 100.000 100.000 99.990 99.947
216 100.000 100.000 100.000 100.000 99.990 99.948
217 100.000 100.000 100.000 100.000 99.990 99.949
218 100.000 100.000 100.000 100.000 99.991 99.954
219 100.000 100.000 100.000 100.000 99.992 99.955
220 100.000 100.000 100.000 100.000 99.993 99.960
221 100.000 100.000 100.000 100.000 99.993 99.961
222 100.000 100.000 100.000 100.000 99.993 99.963
223 100.000 100.000 100.000 100.000 99.994 99.964
224 100.000 100.000 100.000 100.000 99.995 99.968
225 100.000 100.000 100.000 100.000 99.995 99.969
226 100.000 100.000 100.000 100.000 99.995 99.971
227 100.000 100.000 100.000 100.000 99.996 99.973
228 100.000 100.000 100.000 100.000 99.997 99.977
229 100.000 100.000 100.000 100.000 99.997 99.978
230 100.000 100.000 100.000 100.000 99.997 99.978
231 100.000 100.000 100.000 100.000 99.998 99.981
232 100.000 100.000 100.000 100.000 99.998 99.983
233 100.000 100.000 100.000 100.000 99.998 99.985
234 100.000 100.000 100.000 100.000 99.998 99.986
235 100.000 100.000 100.000 100.000 99.999 99.989
236 100.000 100.000 100.000 100.000 99.999 99.990
237 100.000 100.000 100.000 100.000 100.000 99.992
238 100.000 100.000 100.000 100.000 100.000 99.992

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 93.023 81.924 70.119 59.046 49.239 40.820
5 93.023 83.778 73.701 63.754 54.484 46.153
6 93.023 85.168 76.498 67.564 58.864 50.735
7 97.674 91.657 83.504 74.452 65.332 56.647
8 97.674 92.468 85.221 76.908 68.291 59.882
9 97.674 93.163 86.726 79.099 70.967 62.842
10 100.000 96.408 90.335 82.828 74.687 66.469
11 100.000 96.871 91.415 84.491 76.809 68.900
12 100.000 97.219 92.246 85.803 78.527 70.918
13 100.000 97.335 92.634 86.542 79.622 72.320
14 100.000 98.030 94.018 88.432 81.820 74.668
15 100.000 98.146 94.388 89.122 82.833 75.962
16 100.000 98.262 94.766 89.827 83.857 77.252
17 100.000 98.494 95.338 90.750 85.083 78.709
18 100.000 98.610 95.698 91.412 86.040 79.919
19 100.000 98.725 96.040 92.031 86.939 81.064
20 100.000 98.957 96.585 92.884 88.054 82.378
21 100.000 99.073 96.898 93.431 88.830 83.355
22 100.000 99.073 96.972 93.652 89.243 83.974
23 100.000 99.073 97.055 93.887 89.667 84.591
24 100.000 99.073 97.129 94.101 90.056 85.164
25 100.000 99.189 97.425 94.594 90.737 86.005
26 100.000 99.305 97.692 95.025 91.327 86.743
27 100.000 99.305 97.757 95.214 91.678 87.266
28 100.000 99.421 98.015 95.618 92.220 87.931
29 100.000 99.421 98.080 95.804 92.557 88.425
30 100.000 99.537 98.338 96.207 93.097 89.086
31 100.000 99.537 98.385 96.347 93.365 89.496
32 100.000 99.537 98.431 96.486 93.626 89.893
33 100.000 99.537 98.468 96.600 93.847 90.234
34 100.000 99.652 98.726 96.995 94.360 90.844
35 100.000 99.652 98.772 97.129 94.605 91.209
36 100.000 99.652 98.818 97.260 94.844 91.561
37 100.000 99.652 98.846 97.348 95.019 91.838
38 100.000 99.652 98.892 97.478 95.253 92.178
39 100.000 99.652 98.929 97.588 95.459 92.492
40 100.000 99.652 98.948 97.654 95.602 92.732
41 100.000 99.768 99.197 98.023 96.068 93.275
42 100.000 99.884 99.446 98.391 96.532 93.813
43 100.000 99.884 99.474 98.476 96.698 94.073
44 100.000 99.884 99.492 98.540 96.831 94.293
45 100.000 100.000 99.742 98.905 97.288 94.816
46 100.000 100.000 99.778 99.010 97.475 95.091
47 100.000 100.000 99.815 99.114 97.664 95.370
48 100.000 100.000 99.834 99.175 97.790 95.574
49 100.000 100.000 99.843 99.214 97.881 95.739
50 100.000 100.000 99.852 99.251 97.969 95.897
51 100.000 100.000 99.871 99.307 98.079 96.072
52 100.000 100.000 99.871 99.321 98.127 96.175
53 100.000 100.000 99.880 99.355 98.203 96.307
54 100.000 100.000 99.889 99.389 98.276 96.434
55 100.000 100.000 99.889 99.400 98.317 96.522
56 100.000 100.000 99.908 99.453 98.415 96.670
57 100.000 100.000 99.908 99.465 98.455 96.754
58 100.000 100.000 99.908 99.475 98.492 96.835
59 100.000 100.000 99.908 99.485 98.529 96.913
60 100.000 100.000 99.908 99.495 98.563 96.988
61 100.000 100.000 99.908 99.504 98.596 97.059
62 100.000 100.000 99.908 99.514 98.629 97.131
63 100.000 100.000 99.908 99.522 98.659 97.195
64 100.000 100.000 99.908 99.529 98.686 97.254
65 100.000 100.000 99.908 99.537 98.712 97.311
66 100.000 100.000 99.908 99.544 98.739 97.373
67 100.000 100.000 99.908 99.551 98.766 97.433
68 100.000 100.000 99.908 99.558 98.792 97.491
69 100.000 100.000 99.908 99.566 98.818 97.548
70 100.000 100.000 99.917 99.593 98.870 97.627
71 100.000 100.000 99.917 99.599 98.893 97.682
72 100.000 100.000 99.926 99.626 98.944 97.760
73 100.000 100.000 99.935 99.653 98.995 97.837
74 100.000 100.000 99.945 99.680 99.045 97.914
75 100.000 100.000 99.954 99.707 99.097 97.994
76 100.000 100.000 99.963 99.734 99.149 98.075
77 100.000 100.000 99.972 99.761 99.199 98.153
78 100.000 100.000 99.982 99.789 99.250 98.231
79 100.000 100.000 99.991 99.816 99.301 98.309
80 100.000 100.000 100.000 99.843 99.352 98.389
81 100.000 100.000 100.000 99.848 99.372 98.435
82 100.000 100.000 100.000 99.853 99.391 98.477
83 100.000 100.000 100.000 99.858 99.411 98.522
84 100.000 100.000 100.000 99.864 99.430 98.566
85 100.000 100.000 100.000 99.868 99.447 98.603
86 100.000 100.000 100.000 99.872 99.462 98.639
87 100.000 100.000 100.000 99.876 99.478 98.677
88 100.000 100.000 100.000 99.880 99.493 98.710
89 100.000 100.000 100.000 99.884 99.509 98.746
90 100.000 100.000 100.000 99.889 99.525 98.781
91 100.000 100.000 100.000 99.893 99.540 98.816
92 100.000 100.000 100.000 99.897 99.555 98.848
93 100.000 100.000 100.000 99.901 99.571 98.885
94 100.000 100.000 100.000 99.905 99.586 98.920
95 100.000 100.000 100.000 99.909 99.603 98.957
96 100.000 100.000 100.000 99.914 99.619 98.995
97 100.000 100.000 100.000 99.918 99.635 99.034
98 100.000 100.000 100.000 99.922 99.650 99.066
99 100.000 100.000 100.000 99.925 99.662 99.096
100 100.000 100.000 100.000 99.928 99.674 99.123
101 100.000 100.000 100.000 99.931 99.686 99.151
102 100.000 100.000 100.000 99.934 99.699 99.181
103 100.000 100.000 100.000 99.938 99.711 99.209
104 100.000 100.000 100.000 99.941 99.723 99.238
105 100.000 100.000 100.000 99.944 99.735 99.266
106 100.000 100.000 100.000 99.947 99.747 99.293
107 100.000 100.000 100.000 99.949 99.755 99.311
108 100.000 100.000 100.000 99.951 99.763 99.330
109 100.000 100.000 100.000 99.953 99.771 99.350
110 100.000 100.000 100.000 99.955 99.779 99.371
111 100.000 100.000 100.000 99.957 99.788 99.391
112 100.000 100.000 100.000 99.959 99.796 99.409
113 100.000 100.000 100.000 99.961 99.804 99.429
114 100.000 100.000 100.000 99.964 99.812 99.447
115 100.000 100.000 100.000 99.966 99.820 99.468
116 100.000 100.000 100.000 99.968 99.829 99.488
117 100.000 100.000 100.000 99.970 99.837 99.506
118 100.000 100.000 100.000 99.972 99.844 99.524
119 100.000 100.000 100.000 99.974 99.852 99.542
120 100.000 100.000 100.000 99.976 99.860 99.561
121 100.000 100.000 100.000 99.978 99.868 99.579
122 100.000 100.000 100.000 99.980 99.876 99.599
123 100.000 100.000 100.000 99.982 99.886 99.623
124 100.000 100.000 100.000 99.984 99.894 99.642
125 100.000 100.000 100.000 99.986 99.902 99.664
126 100.000 100.000 100.000 99.988 99.907 99.677
127 100.000 100.000 100.000 99.989 99.912 99.690
128 100.000 100.000 100.000 99.990 99.917 99.703
129 100.000 100.000 100.000 99.991 99.921 99.714
130 100.000 100.000 100.000 99.992 99.926 99.727
131 100.000 100.000 100.000 99.993 99.931 99.739
132 100.000 100.000 100.000 99.994 99.935 99.751
133 100.000 100.000 100.000 99.995 99.940 99.763
134 100.000 100.000 100.000 99.996 99.945 99.778
135 100.000 100.000 100.000 99.997 99.950 99.791
136 100.000 100.000 100.000 99.998 99.955 99.806
137 100.000 100.000 100.000 99.999 99.961 99.821
138 100.000 100.000 100.000 100.000 99.965 99.832
139 100.000 100.000 100.000 100.000 99.966 99.838
140 100.000 100.000 100.000 100.000 99.967 99.842
141 100.000 100.000 100.000 100.000 99.968 99.847
142 100.000 100.000 100.000 100.000 99.969 99.851
143 100.000 100.000 100.000 100.000 99.971 99.857
144 100.000 100.000 100.000 100.000 99.972 99.863
145 100.000 100.000 100.000 100.000 99.973 99.868
146 100.000 100.000 100.000 100.000 99.974 99.872
147 100.000 100.000 100.000 100.000 99.975 99.877
148 100.000 100.000 100.000 100.000 99.976 99.882
149 100.000 100.000 100.000 100.000 99.976 99.883
150 100.000 100.000 100.000 100.000 99.977 99.886
151 100.000 100.000 100.000 100.000 99.978 99.889
152 100.000 100.000 100.000 100.000 99.978 99.890
153 100.000 100.000 100.000 100.000 99.979 99.895
154 100.000 100.000 100.000 100.000 99.980 99.898
155 100.000 100.000 100.000 100.000 99.980 99.901
156 100.000 100.000 100.000 100.000 99.981 99.905
157 100.000 100.000 100.000 100.000 99.982 99.907
158 100.000 100.000 100.000 100.000 99.982 99.910
159 100.000 100.000 100.000 100.000 99.983 99.913
160 100.000 100.000 100.000 100.000 99.984 99.918
161 100.000 100.000 100.000 100.000 99.985 99.923
162 100.000 100.000 100.000 100.000 99.986 99.927
163 100.000 100.000 100.000 100.000 99.987 99.930
164 100.000 100.000 100.000 100.000 99.988 99.934
165 100.000 100.000 100.000 100.000 99.988 99.937
166 100.000 100.000 100.000 100.000 99.989 99.940
167 100.000 100.000 100.000 100.000 99.990 99.943
168 100.000 100.000 100.000 100.000 99.990 99.946
169 100.000 100.000 100.000 100.000 99.991 99.948
170 100.000 100.000 100.000 100.000 99.992 99.953
171 100.000 100.000 100.000 100.000 99.993 99.956
172 100.000 100.000 100.000 100.000 99.994 99.962
173 100.000 100.000 100.000 100.000 99.994 99.963
174 100.000 100.000 100.000 100.000 99.995 99.967
175 100.000 100.000 100.000 100.000 99.996 99.971
176 100.000 100.000 100.000 100.000 99.997 99.975
177 100.000 100.000 100.000 100.000 99.998 99.978
178 100.000 100.000 100.000 100.000 99.998 99.980
179 100.000 100.000 100.000 100.000 99.999 99.985
180 100.000 100.000 100.000 100.000 100.000 99.989

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 90.698 77.752 64.802 53.194 43.303 35.100
4 93.023 82.503 71.125 60.228 50.409 41.868
5 95.349 86.327 75.852 65.403 55.691 47.017
6 97.674 90.267 80.753 70.751 61.112 52.258
7 97.674 90.962 82.341 73.139 64.087 55.580
8 100.000 94.670 86.818 77.962 68.962 60.310
9 100.000 95.597 88.738 80.659 72.161 63.761
10 100.000 96.408 90.409 83.003 74.949 66.785
11 100.000 96.871 91.452 84.580 76.946 69.070
12 100.000 97.335 92.477 86.106 78.855 71.235
13 100.000 97.567 93.067 87.088 80.196 72.866
14 100.000 98.030 94.092 88.607 82.083 74.987
15 100.000 98.146 94.471 89.322 83.138 76.338
16 100.000 98.378 95.052 90.275 84.424 77.888
17 100.000 98.494 95.412 90.939 85.388 79.109
18 100.000 98.610 95.763 91.574 86.299 80.251
19 100.000 98.841 96.298 92.405 87.380 81.524
20 100.000 98.841 96.418 92.734 87.954 82.334
21 100.000 99.073 96.926 93.496 88.924 83.463
22 100.000 99.073 97.009 93.735 89.360 84.102
23 100.000 99.189 97.295 94.211 90.021 84.929
24 100.000 99.189 97.388 94.468 90.476 85.578
25 100.000 99.189 97.461 94.680 90.858 86.136
26 100.000 99.189 97.517 94.845 91.168 86.602
27 100.000 99.189 97.572 95.010 91.475 87.063
28 100.000 99.305 97.831 95.414 92.020 87.739
29 100.000 99.305 97.877 95.557 92.293 88.154
30 100.000 99.305 97.932 95.717 92.588 88.592
31 100.000 99.305 97.978 95.856 92.849 88.987
32 100.000 99.421 98.237 96.255 93.375 89.621
33 100.000 99.421 98.274 96.370 93.600 89.972
34 100.000 99.421 98.311 96.485 93.822 90.317
35 100.000 99.421 98.348 96.599 94.043 90.656
36 100.000 99.537 98.606 96.994 94.556 91.266
37 100.000 99.537 98.634 97.086 94.743 91.567
38 100.000 99.537 98.661 97.175 94.921 91.850
39 100.000 99.652 98.911 97.546 95.395 92.409
40 100.000 99.652 98.948 97.657 95.604 92.728
41 100.000 99.768 99.197 98.026 96.075 93.280
42 100.000 99.768 99.215 98.092 96.217 93.521
43 100.000 99.768 99.243 98.178 96.387 93.792
44 100.000 99.884 99.492 98.546 96.850 94.326
45 100.000 100.000 99.742 98.913 97.310 94.857
46 100.000 100.000 99.760 98.976 97.444 95.079
47 100.000 100.000 99.778 99.039 97.573 95.289
48 100.000 100.000 99.806 99.121 97.728 95.528
49 100.000 100.000 99.825 99.179 97.844 95.712
50 100.000 100.000 99.834 99.217 97.932 95.869
51 100.000 100.000 99.843 99.252 98.012 96.012
52 100.000 100.000 99.852 99.288 98.093 96.153
53 100.000 100.000 99.862 99.323 98.173 96.294
54 100.000 100.000 99.871 99.357 98.249 96.426
55 100.000 100.000 99.871 99.370 98.293 96.519
56 100.000 100.000 99.880 99.403 98.366 96.643
57 100.000 100.000 99.889 99.435 98.435 96.761
58 100.000 100.000 99.889 99.446 98.472 96.842
59 100.000 100.000 99.898 99.476 98.534 96.945
60 100.000 100.000 99.898 99.485 98.569 97.021
61 100.000 100.000 99.898 99.495 98.603 97.094
62 100.000 100.000 99.908 99.525 98.665 97.200
63 100.000 100.000 99.908 99.534 98.698 97.270
64 100.000 100.000 99.908 99.543 98.730 97.342
65 100.000 100.000 99.908 99.551 98.761 97.411
66 100.000 100.000 99.908 99.559 98.791 97.479
67 100.000 100.000 99.908 99.568 98.823 97.550
68 100.000 100.000 99.908 99.575 98.850 97.612
69 100.000 100.000 99.908 99.582 98.876 97.668
70 100.000 100.000 99.917 99.609 98.926 97.745
71 100.000 100.000 99.917 99.616 98.949 97.796
72 100.000 100.000 99.917 99.622 98.973 97.850
73 100.000 100.000 99.926 99.649 99.024 97.929
74 100.000 100.000 99.935 99.676 99.075 98.008
75 100.000 100.000 99.935 99.682 99.099 98.061
76 100.000 100.000 99.945 99.709 99.149 98.138
77 100.000 100.000 99.954 99.736 99.200 98.216
78 100.000 100.000 99.963 99.764 99.251 98.293
79 100.000 100.000 99.972 99.791 99.301 98.370
80 100.000 100.000 99.972 99.797 99.324 98.421
81 100.000 100.000 99.982 99.824 99.376 98.501
82 100.000 100.000 99.991 99.851 99.426 98.578
83 100.000 100.000 100.000 99.878 99.477 98.657
84 100.000 100.000 100.000 99.883 99.497 98.703
85 100.000 100.000 100.000 99.889 99.517 98.747
86 100.000 100.000 100.000 99.894 99.536 98.790
87 100.000 100.000 100.000 99.899 99.555 98.831
88 100.000 100.000 100.000 99.904 99.575 98.878
89 100.000 100.000 100.000 99.908 99.590 98.911
90 100.000 100.000 100.000 99.913 99.606 98.947
91 100.000 100.000 100.000 99.917 99.622 98.985
92 100.000 100.000 100.000 99.921 99.638 99.024
93 100.000 100.000 100.000 99.925 99.654 99.060
94 100.000 100.000 100.000 99.929 99.670 99.097
95 100.000 100.000 100.000 99.933 99.685 99.130
96 100.000 100.000 100.000 99.938 99.700 99.163
97 100.000 100.000 100.000 99.941 99.712 99.191
98 100.000 100.000 100.000 99.944 99.725 99.223
99 100.000 100.000 100.000 99.947 99.738 99.254
100 100.000 100.000 100.000 99.950 99.749 99.281
101 100.000 100.000 100.000 99.953 99.762 99.311
102 100.000 100.000 100.000 99.956 99.774 99.338
103 100.000 100.000 100.000 99.958 99.783 99.361
104 100.000 100.000 100.000 99.960 99.791 99.381
105 100.000 100.000 100.000 99.963 99.800 99.403
106 100.000 100.000 100.000 99.965 99.808 99.422
107 100.000 100.000 100.000 99.967 99.816 99.441
108 100.000 100.000 100.000 99.969 99.825 99.462
109 100.000 100.000 100.000 99.971 99.833 99.481
110 100.000 100.000 100.000 99.973 99.841 99.502
111 100.000 100.000 100.000 99.975 99.849 99.521
112 100.000 100.000 100.000 99.977 99.857 99.539
113 100.000 100.000 100.000 99.979 99.865 99.558
114 100.000 100.000 100.000 99.981 99.874 99.579
115 100.000 100.000 100.000 99.983 99.882 99.599
116 100.000 100.000 100.000 99.985 99.891 99.620
117 100.000 100.000 100.000 99.986 99.895 99.631
118 100.000 100.000 100.000 99.988 99.900 99.645
119 100.000 100.000 100.000 99.989 99.904 99.655
120 100.000 100.000 100.000 99.990 99.908 99.665
121 100.000 100.000 100.000 99.991 99.913 99.678
122 100.000 100.000 100.000 99.992 99.917 99.688
123 100.000 100.000 100.000 99.993 99.922 99.702
124 100.000 100.000 100.000 99.994 99.927 99.713
125 100.000 100.000 100.000 99.995 99.931 99.724
126 100.000 100.000 100.000 99.996 99.935 99.733
127 100.000 100.000 100.000 99.997 99.940 99.745
128 100.000 100.000 100.000 99.998 99.945 99.759
129 100.000 100.000 100.000 99.999 99.949 99.769
130 100.000 100.000 100.000 100.000 99.954 99.781
131 100.000 100.000 100.000 100.000 99.954 99.784
132 100.000 100.000 100.000 100.000 99.955 99.787
133 100.000 100.000 100.000 100.000 99.955 99.790
134 100.000 100.000 100.000 100.000 99.956 99.793
135 100.000 100.000 100.000 100.000 99.957 99.796
136 100.000 100.000 100.000 100.000 99.957 99.799
137 100.000 100.000 100.000 100.000 99.958 99.801
138 100.000 100.000 100.000 100.000 99.958 99.803
139 100.000 100.000 100.000 100.000 99.959 99.807
140 100.000 100.000 100.000 100.000 99.960 99.810
141 100.000 100.000 100.000 100.000 99.961 99.814
142 100.000 100.000 100.000 100.000 99.961 99.816
143 100.000 100.000 100.000 100.000 99.962 99.820
144 100.000 100.000 100.000 100.000 99.963 99.823
145 100.000 100.000 100.000 100.000 99.963 99.825
146 100.000 100.000 100.000 100.000 99.964 99.826
147 100.000 100.000 100.000 100.000 99.964 99.827
148 100.000 100.000 100.000 100.000 99.964 99.830
149 100.000 100.000 100.000 100.000 99.965 99.833
150 100.000 100.000 100.000 100.000 99.966 99.836
151 100.000 100.000 100.000 100.000 99.966 99.836
152 100.000 100.000 100.000 100.000 99.966 99.839
153 100.000 100.000 100.000 100.000 99.967 99.841
154 100.000 100.000 100.000 100.000 99.968 99.846
155 100.000 100.000 100.000 100.000 99.968 99.847
156 100.000 100.000 100.000 100.000 99.969 99.852
157 100.000 100.000 100.000 100.000 99.970 99.854
158 100.000 100.000 100.000 100.000 99.971 99.858
159 100.000 100.000 100.000 100.000 99.971 99.859
160 100.000 100.000 100.000 100.000 99.971 99.862
161 100.000 100.000 100.000 100.000 99.972 99.862
162 100.000 100.000 100.000 100.000 99.972 99.864
163 100.000 100.000 100.000 100.000 99.972 99.865
164 100.000 100.000 100.000 100.000 99.972 99.866
165 100.000 100.000 100.000 100.000 99.973 99.868
166 100.000 100.000 100.000 100.000 99.973 99.869
167 100.000 100.000 100.000 100.000 99.973 99.871
168 100.000 100.000 100.000 100.000 99.974 99.875
169 100.000 100.000 100.000 100.000 99.975 99.878
170 100.000 100.000 100.000 100.000 99.975 99.880
171 100.000 100.000 100.000 100.000 99.976 99.881
172 100.000 100.000 100.000 100.000 99.976 99.884
173 100.000 100.000 100.000 100.000 99.976 99.885
174 100.000 100.000 100.000 100.000 99.976 99.885
175 100.000 100.000 100.000 100.000 99.977 99.886
176 100.000 100.000 100.000 100.000 99.977 99.888
177 100.000 100.000 100.000 100.000 99.978 99.890
178 100.000 100.000 100.000 100.000 99.978 99.892
179 100.000 100.000 100.000 100.000 99.978 99.893
180 100.000 100.000 100.000 100.000 99.979 99.896
181 100.000 100.000 100.000 100.000 99.980 99.900
182 100.000 100.000 100.000 100.000 99.980 99.902
183 100.000 100.000 100.000 100.000 99.980 99.902
184 100.000 100.000 100.000 100.000 99.981 99.904
185 100.000 100.000 100.000 100.000 99.981 99.907
186 100.000 100.000 100.000 100.000 99.982 99.911
187 100.000 100.000 100.000 100.000 99.983 99.913
188 100.000 100.000 100.000 100.000 99.983 99.913
189 100.000 100.000 100.000 100.000 99.983 99.914
190 100.000 100.000 100.000 100.000 99.983 99.917
191 100.000 100.000 100.000 100.000 99.983 99.917
192 100.000 100.000 100.000 100.000 99.984 99.920
193 100.000 100.000 100.000 100.000 99.985 99.925
194 100.000 100.000 100.000 100.000 99.986 99.927
195 100.000 100.000 100.000 100.000 99.986 99.927
196 100.000 100.000 100.000 100.000 99.986 99.929
197 100.000 100.000 100.000 100.000 99.986 99.929
198 100.000 100.000 100.000 100.000 99.987 99.932
199 100.000 100.000 100.000 100.000 99.987 99.934
200 100.000 100.000 100.000 100.000 99.988 99.936
201 100.000 100.000 100.000 100.000 99.988 99.936
202 100.000 100.000 100.000 100.000 99.988 99.939
203 100.000 100.000 100.000 100.000 99.988 99.940
204 100.000 100.000 100.000 100.000 99.988 99.940
205 100.000 100.000 100.000 100.000 99.989 99.941
206 100.000 100.000 100.000 100.000 99.989 99.942
207 100.000 100.000 100.000 100.000 99.989 99.944
208 100.000 100.000 100.000 100.000 99.990 99.946
209 100.000 100.000 100.000 100.000 99.991 99.950
210 100.000 100.000 100.000 100.000 99.991 99.952
211 100.000 100.000 100.000 100.000 99.991 99.953
212 100.000 100.000 100.000 100.000 99.991 99.953
213 100.000 100.000 100.000 100.000 99.992 99.955
214 100.000 100.000 100.000 100.000 99.992 99.956
215 100.000 100.000 100.000 100.000 99.993 99.958
216 100.000 100.000 100.000 100.000 99.993 99.959
217 100.000 100.000 100.000 100.000 99.993 99.961
218 100.000 100.000 100.000 100.000 99.993 99.961
219 100.000 100.000 100.000 100.000 99.993 99.961
220 100.000 100.000 100.000 100.000 99.994 99.965
221 100.000 100.000 100.000 100.000 99.995 99.969
222 100.000 100.000 100.000 100.000 99.995 99.969
223 100.000 100.000 100.000 100.000 99.995 99.970
224 100.000 100.000 100.000 100.000 99.995 99.970
225 100.000 100.000 100.000 100.000 99.996 99.973
226 100.000 100.000 100.000 100.000 99.996 99.973
227 100.000 100.000 100.000 100.000 99.996 99.975
228 100.000 100.000 100.000 100.000 99.996 99.976
229 100.000 100.000 100.000 100.000 99.997 99.979
230 100.000 100.000 100.000 100.000 99.997 99.980
231 100.000 100.000 100.000 100.000 99.997 99.981
232 100.000 100.000 100.000 100.000 99.998 99.983
233 100.000 100.000 100.000 100.000 99.998 99.985
234 100.000 100.000 100.000 100.000 99.999 99.987
235 100.000 100.000 100.000 100.000 100.000 99.990
236 100.000 100.000 100.000 100.000 100.000 99.991
237 100.000 100.000 100.000 100.000 100.000 99.992
238 100.000 100.000 100.000 100.000 100.000 99.992

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 79.070 62.109 48.777 38.407 30.360 24.107
3 88.372 74.739 61.885 50.692 41.300 33.569
4 90.698 79.606 68.393 57.924 48.583 40.480
5 93.023 83.778 73.728 63.822 54.588 46.284
6 95.349 87.717 78.658 69.239 60.117 51.661
7 97.674 91.309 82.969 73.912 64.894 56.349
8 97.674 92.352 85.064 76.782 68.230 59.892
9 100.000 95.829 89.117 81.078 72.551 64.091
10 100.000 96.408 90.400 82.979 74.910 66.733
11 100.000 96.871 91.406 84.468 76.773 68.854
12 100.000 96.871 91.600 84.992 77.670 70.097
13 100.000 97.335 92.615 86.495 79.543 72.215
14 100.000 97.683 93.400 87.689 81.071 73.986
15 100.000 98.262 94.609 89.407 83.139 76.257
16 100.000 98.378 94.988 90.117 84.178 77.578
17 100.000 98.378 95.135 90.516 84.863 78.532
18 100.000 98.494 95.468 91.110 85.716 79.613
19 100.000 98.725 95.975 91.874 86.697 80.764
20 100.000 98.725 96.068 92.132 87.156 81.424
21 100.000 98.957 96.575 92.895 88.130 82.563
22 100.000 99.073 96.871 93.394 88.829 83.438
23 100.000 99.189 97.157 93.869 89.488 84.262
24 100.000 99.189 97.203 94.011 89.764 84.689
25 100.000 99.189 97.314 94.310 90.278 85.408
26 100.000 99.189 97.369 94.474 90.583 85.867
27 100.000 99.189 97.443 94.683 90.959 86.414
28 100.000 99.189 97.498 94.844 91.257 86.858
29 100.000 99.189 97.545 94.984 91.522 87.264
30 100.000 99.189 97.572 95.078 91.716 87.579
31 100.000 99.189 97.628 95.238 92.010 88.017
32 100.000 99.189 97.674 95.376 92.268 88.408
33 100.000 99.305 97.932 95.773 92.792 89.040
34 100.000 99.305 97.960 95.863 92.972 89.327
35 100.000 99.305 97.997 95.976 93.187 89.658
36 100.000 99.305 98.015 96.042 93.330 89.898
37 100.000 99.305 98.052 96.154 93.542 90.222
38 100.000 99.305 98.089 96.265 93.754 90.545
39 100.000 99.305 98.126 96.373 93.956 90.851
40 100.000 99.305 98.172 96.504 94.190 91.194
41 100.000 99.421 98.421 96.876 94.670 91.765
42 100.000 99.421 98.468 97.008 94.908 92.113
43 100.000 99.421 98.486 97.071 95.041 92.332
44 100.000 99.537 98.745 97.465 95.550 92.934
45 100.000 99.537 98.763 97.524 95.670 93.129
46 100.000 99.652 99.012 97.894 96.140 93.679
47 100.000 99.652 99.058 98.025 96.377 94.025
48 100.000 99.768 99.308 98.394 96.844 94.569
49 100.000 99.768 99.308 98.405 96.885 94.660
50 100.000 99.768 99.326 98.466 97.007 94.857
51 100.000 99.768 99.335 98.500 97.084 94.991
52 100.000 99.884 99.585 98.867 97.544 95.520
53 100.000 99.884 99.603 98.926 97.663 95.710
54 100.000 99.884 99.622 98.982 97.772 95.882
55 100.000 99.884 99.622 98.993 97.809 95.963
56 100.000 99.884 99.622 99.004 97.850 96.049
57 100.000 100.000 99.871 99.370 98.305 96.570
58 100.000 100.000 99.871 99.378 98.336 96.638
59 100.000 100.000 99.889 99.434 98.445 96.808
60 100.000 100.000 99.889 99.444 98.478 96.881
61 100.000 100.000 99.889 99.455 98.518 96.965
62 100.000 100.000 99.898 99.487 98.584 97.077
63 100.000 100.000 99.898 99.495 98.614 97.143
64 100.000 100.000 99.898 99.502 98.641 97.204
65 100.000 100.000 99.898 99.510 98.671 97.269
66 100.000 100.000 99.898 99.518 98.698 97.329
67 100.000 100.000 99.898 99.525 98.725 97.388
68 100.000 100.000 99.908 99.554 98.784 97.485
69 100.000 100.000 99.908 99.560 98.807 97.537
70 100.000 100.000 99.908 99.567 98.830 97.588
71 100.000 100.000 99.908 99.572 98.850 97.634
72 100.000 100.000 99.908 99.577 98.870 97.679
73 100.000 100.000 99.908 99.582 98.889 97.724
74 100.000 100.000 99.908 99.588 98.909 97.769
75 100.000 100.000 99.908 99.593 98.929 97.813
76 100.000 100.000 99.908 99.595 98.939 97.841
77 100.000 100.000 99.908 99.600 98.958 97.885
78 100.000 100.000 99.908 99.604 98.974 97.923
79 100.000 100.000 99.908 99.608 98.991 97.961
80 100.000 100.000 99.908 99.613 99.007 97.998
81 100.000 100.000 99.908 99.617 99.023 98.035
82 100.000 100.000 99.908 99.621 99.038 98.071
83 100.000 100.000 99.908 99.625 99.054 98.107
84 100.000 100.000 99.908 99.628 99.067 98.137
85 100.000 100.000 99.908 99.631 99.079 98.166
86 100.000 100.000 99.908 99.634 99.091 98.195
87 100.000 100.000 99.908 99.638 99.104 98.224
88 100.000 100.000 99.908 99.641 99.116 98.254
89 100.000 100.000 99.908 99.643 99.125 98.277
90 100.000 100.000 99.908 99.647 99.140 98.312
91 100.000 100.000 99.908 99.651 99.156 98.346
92 100.000 100.000 99.908 99.655 99.171 98.380
93 100.000 100.000 99.908 99.658 99.183 98.409
94 100.000 100.000 99.908 99.663 99.198 98.442
95 100.000 100.000 99.908 99.666 99.210 98.470
96 100.000 100.000 99.917 99.693 99.261 98.548
97 100.000 100.000 99.917 99.697 99.276 98.581
98 100.000 100.000 99.917 99.699 99.285 98.602
99 100.000 100.000 99.917 99.701 99.293 98.624
100 100.000 100.000 99.926 99.728 99.344 98.701
101 100.000 100.000 99.935 99.755 99.395 98.778
102 100.000 100.000 99.935 99.757 99.403 98.799
103 100.000 100.000 99.945 99.784 99.454 98.876
104 100.000 100.000 99.954 99.811 99.504 98.953
105 100.000 100.000 99.963 99.839 99.555 99.030
106 100.000 100.000 99.972 99.866 99.606 99.107
107 100.000 100.000 99.982 99.893 99.656 99.184
108 100.000 100.000 99.991 99.920 99.707 99.261
109 100.000 100.000 100.000 99.947 99.757 99.338
110 100.000 100.000 100.000 99.949 99.766 99.358
111 100.000 100.000 100.000 99.951 99.774 99.378
112 100.000 100.000 100.000 99.953 99.783 99.399
113 100.000 100.000 100.000 99.955 99.791 99.419
114 100.000 100.000 100.000 99.957 99.799 99.440
115 100.000 100.000 100.000 99.959 99.808 99.460
116 100.000 100.000 100.000 99.960 99.813 99.474
117 100.000 100.000 100.000 99.961 99.818 99.489
118 100.000 100.000 100.000 99.964 99.826 99.509
119 100.000 100.000 100.000 99.966 99.835 99.528
120 100.000 100.000 100.000 99.968 99.843 99.547
121 100.000 100.000 100.000 99.970 99.851 99.566
122 100.000 100.000 100.000 99.972 99.859 99.584
123 100.000 100.000 100.000 99.974 99.867 99.603
124 100.000 100.000 100.000 99.976 99.875 99.622
125 100.000 100.000 100.000 99.978 99.882 99.640
126 100.000 100.000 100.000 99.980 99.890 99.659
127 100.000 100.000 100.000 99.982 99.898 99.678
128 100.000 100.000 100.000 99.983 99.903 99.690
129 100.000 100.000 100.000 99.985 99.911 99.708
130 100.000 100.000 100.000 99.988 99.919 99.726
131 100.000 100.000 100.000 99.990 99.927 99.744
132 100.000 100.000 100.000 99.992 99.934 99.762
133 100.000 100.000 100.000 99.994 99.942 99.780
134 100.000 100.000 100.000 99.996 99.950 99.798
135 100.000 100.000 100.000 99.998 99.958 99.815
136 100.000 100.000 100.000 99.999 99.962 99.826
137 100.000 100.000 100.000 99.999 99.963 99.832
138 100.000 100.000 100.000 99.999 99.964 99.837
139 100.000 100.000 100.000 99.999 99.965 99.842
140 100.000 100.000 100.000 99.999 99.967 99.847
141 100.000 100.000 100.000 99.999 99.968 99.852
142 100.000 100.000 100.000 99.999 99.969 99.857
143 100.000 100.000 100.000 99.999 99.970 99.862
144 100.000 100.000 100.000 99.999 99.971 99.866
145 100.000 100.000 100.000 99.999 99.972 99.871
146 100.000 100.000 100.000 99.999 99.973 99.876
147 100.000 100.000 100.000 99.999 99.974 99.880
148 100.000 100.000 100.000 99.999 99.975 99.885
149 100.000 100.000 100.000 99.999 99.976 99.889
150 100.000 100.000 100.000 99.999 99.977 99.893
151 100.000 100.000 100.000 99.999 99.978 99.897
152 100.000 100.000 100.000 99.999 99.979 99.901
153 100.000 100.000 100.000 99.999 99.980 99.906
154 100.000 100.000 100.000 99.999 99.981 99.910
155 100.000 100.000 100.000 99.999 99.981 99.914
156 100.000 100.000 100.000 99.999 99.982 99.918
157 100.000 100.000 100.000 100.000 99.986 99.927
158 100.000 100.000 100.000 100.000 99.987 99.930
159 100.000 100.000 100.000 100.000 99.988 99.934
160 100.000 100.000 100.000 100.000 99.989 99.937
161 100.000 100.000 100.000 100.000 99.989 99.941
162 100.000 100.000 100.000 100.000 99.990 99.944
163 100.000 100.000 100.000 100.000 99.991 99.947
164 100.000 100.000 100.000 100.000 99.991 99.950
165 100.000 100.000 100.000 100.000 99.992 99.953
166 100.000 100.000 100.000 100.000 99.993 99.955
167 100.000 100.000 100.000 100.000 99.993 99.958
168 100.000 100.000 100.000 100.000 99.994 99.961
169 100.000 100.000 100.000 100.000 99.995 99.964
170 100.000 100.000 100.000 100.000 99.995 99.967
171 100.000 100.000 100.000 100.000 99.996 99.969
172 100.000 100.000 100.000 100.000 99.996 99.972
173 100.000 100.000 100.000 100.000 99.997 99.975
174 100.000 100.000 100.000 100.000 99.998 99.978
175 100.000 100.000 100.000 100.000 99.998 99.981
176 100.000 100.000 100.000 100.000 99.999 99.984
177 100.000 100.000 100.000 100.000 100.000 99.986
178 100.000 100.000 100.000 100.000 100.000 99.987
179 100.000 100.000 100.000 100.000 100.000 99.988
180 100.000 100.000 100.000 100.000 100.000 99.989

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 79.070 62.109 48.777 38.407 30.360 24.107
3 86.047 71.958 59.356 48.629 39.714 32.397
4 90.698 79.027 67.405 56.787 47.483 39.516
5 95.349 86.327 75.861 65.426 55.727 47.062
6 97.674 90.151 80.606 70.645 61.078 52.296
7 97.674 91.194 82.766 73.671 64.654 56.133
8 97.674 91.889 84.261 75.843 67.307 59.072
9 97.674 92.584 85.710 77.899 69.777 61.778
10 100.000 96.176 89.920 82.322 74.165 65.979
11 100.000 96.640 90.917 83.786 75.986 68.043
12 100.000 97.103 91.941 85.312 77.898 70.213
13 100.000 97.451 92.763 86.599 79.570 72.166
14 100.000 97.798 93.575 87.858 81.194 74.050
15 100.000 98.146 94.351 89.030 82.687 75.775
16 100.000 98.262 94.692 89.651 83.592 76.934
17 100.000 98.378 95.052 90.315 84.556 78.156
18 100.000 98.494 95.412 90.973 85.505 79.349
19 100.000 98.494 95.532 91.304 86.085 80.173
20 100.000 98.610 95.837 91.829 86.830 81.116
21 100.000 98.610 95.929 92.093 87.305 81.807
22 100.000 98.725 96.206 92.548 87.938 82.607
23 100.000 98.725 96.298 92.807 88.400 83.271
24 100.000 98.957 96.815 93.593 89.409 84.451
25 100.000 99.189 97.332 94.374 90.406 85.608
26 100.000 99.189 97.406 94.589 90.801 86.190
27 100.000 99.189 97.471 94.772 91.129 86.670
28 100.000 99.189 97.498 94.866 91.326 86.993
29 100.000 99.189 97.572 95.076 91.702 87.539
30 100.000 99.189 97.628 95.235 91.994 87.973
31 100.000 99.305 97.886 95.636 92.529 88.628
32 100.000 99.305 97.914 95.731 92.725 88.947
33 100.000 99.305 97.969 95.890 93.016 89.376
34 100.000 99.305 97.997 95.982 93.202 89.675
35 100.000 99.305 98.034 96.093 93.414 89.999
36 100.000 99.305 98.061 96.181 93.587 90.273
37 100.000 99.305 98.098 96.291 93.795 90.592
38 100.000 99.305 98.126 96.378 93.965 90.859
39 100.000 99.305 98.163 96.486 94.167 91.164
40 100.000 99.305 98.191 96.574 94.343 91.442
41 100.000 99.305 98.218 96.661 94.512 91.709
42 100.000 99.305 98.246 96.748 94.684 91.979
43 100.000 99.305 98.255 96.788 94.778 92.148
44 100.000 99.305 98.274 96.850 94.908 92.362
45 100.000 99.305 98.283 96.891 95.005 92.535
46 100.000 99.421 98.532 97.263 95.482 93.099
47 100.000 99.421 98.541 97.300 95.569 93.254
48 100.000 99.421 98.551 97.338 95.656 93.408
49 100.000 99.537 98.800 97.703 96.114 93.936
50 100.000 99.537 98.809 97.741 96.201 94.089
51 100.000 99.537 98.818 97.778 96.287 94.242
52 100.000 99.652 99.068 98.145 96.747 94.773
53 100.000 99.652 99.077 98.178 96.821 94.901
54 100.000 99.652 99.086 98.211 96.891 95.023
55 100.000 99.652 99.086 98.222 96.932 95.110
56 100.000 99.768 99.335 98.588 97.388 95.633
57 100.000 99.768 99.345 98.620 97.458 95.754
58 100.000 99.768 99.354 98.654 97.535 95.885
59 100.000 99.884 99.603 99.020 97.991 96.406
60 100.000 99.884 99.612 99.050 98.054 96.515
61 100.000 99.884 99.622 99.080 98.117 96.622
62 100.000 100.000 99.871 99.446 98.573 97.142
63 100.000 100.000 99.880 99.478 98.642 97.259
64 100.000 100.000 99.880 99.485 98.669 97.319
65 100.000 100.000 99.880 99.491 98.690 97.368
66 100.000 100.000 99.880 99.498 98.716 97.427
67 100.000 100.000 99.889 99.527 98.776 97.525
68 100.000 100.000 99.889 99.532 98.796 97.573
69 100.000 100.000 99.889 99.538 98.816 97.620
70 100.000 100.000 99.898 99.567 98.875 97.717
71 100.000 100.000 99.898 99.574 98.902 97.775
72 100.000 100.000 99.898 99.579 98.922 97.821
73 100.000 100.000 99.898 99.583 98.938 97.861
74 100.000 100.000 99.908 99.613 98.997 97.957
75 100.000 100.000 99.908 99.618 99.017 98.002
76 100.000 100.000 99.908 99.623 99.036 98.046
77 100.000 100.000 99.908 99.627 99.052 98.084
78 100.000 100.000 99.908 99.631 99.068 98.121
79 100.000 100.000 99.908 99.634 99.081 98.153
80 100.000 100.000 99.908 99.641 99.104 98.201
81 100.000 100.000 99.908 99.645 99.120 98.238
82 100.000 100.000 99.908 99.649 99.135 98.275
83 100.000 100.000 99.908 99.653 99.151 98.311
84 100.000 100.000 99.908 99.656 99.164 98.342
85 100.000 100.000 99.908 99.660 99.180 98.377
86 100.000 100.000 99.908 99.666 99.198 98.418
87 100.000 100.000 99.908 99.670 99.214 98.454
88 100.000 100.000 99.908 99.674 99.229 98.489
89 100.000 100.000 99.908 99.678 99.245 98.524
90 100.000 100.000 99.908 99.681 99.257 98.552
91 100.000 100.000 99.908 99.683 99.266 98.575
92 100.000 100.000 99.908 99.685 99.275 98.598
93 100.000 100.000 99.908 99.689 99.287 98.626
94 100.000 100.000 99.908 99.692 99.299 98.654
95 100.000 100.000 99.917 99.719 99.350 98.731
96 100.000 100.000 99.926 99.746 99.400 98.809
97 100.000 100.000 99.926 99.748 99.409 98.830
98 100.000 100.000 99.935 99.775 99.460 98.907
99 100.000 100.000 99.935 99.777 99.468 98.928
100 100.000 100.000 99.945 99.804 99.519 99.005
101 100.000 100.000 99.954 99.831 99.569 99.082
102 100.000 100.000 99.963 99.858 99.620 99.159
103 100.000 100.000 99.972 99.885 99.671 99.236
104 100.000 100.000 99.982 99.913 99.721 99.313
105 100.000 100.000 99.991 99.940 99.772 99.390
106 100.000 100.000 100.000 99.967 99.822 99.467
107 100.000 100.000 100.000 99.970 99.834 99.493
108 100.000 100.000 100.000 99.972 99.842 99.513
109 100.000 100.000 100.000 99.973 99.848 99.528
110 100.000 100.000 100.000 99.976 99.859 99.553
111 100.000 100.000 100.000 99.978 99.867 99.573
112 100.000 100.000 100.000 99.981 99.879 99.598
113 100.000 100.000 100.000 99.983 99.887 99.617
114 100.000 100.000 100.000 99.983 99.888 99.624
115 100.000 100.000 100.000 99.983 99.890 99.631
116 100.000 100.000 100.000 99.985 99.897 99.649
117 100.000 100.000 100.000 99.985 99.899 99.655
118 100.000 100.000 100.000 99.986 99.904 99.667
119 100.000 100.000 100.000 99.989 99.911 99.685
120 100.000 100.000 100.000 99.990 99.916 99.696
121 100.000 100.000 100.000 99.990 99.917 99.702
122 100.000 100.000 100.000 99.991 99.922 99.714
123 100.000 100.000 100.000 99.992 99.926 99.726
124 100.000 100.000 100.000 99.992 99.927 99.731
125 100.000 100.000 100.000 99.992 99.929 99.737
126 100.000 100.000 100.000 99.993 99.933 99.748
127 100.000 100.000 100.000 99.993 99.934 99.754
128 100.000 100.000 100.000 99.994 99.939 99.764
129 100.000 100.000 100.000 99.994 99.940 99.770
130 100.000 100.000 100.000 99.995 99.944 99.780
131 100.000 100.000 100.000 99.996 99.948 99.791
132 100.000 100.000 100.000 99.996 99.949 99.795
133 100.000 100.000 100.000 99.997 99.954 99.806
134 100.000 100.000 100.000 99.998 99.958 99.815
135 100.000 100.000 100.000 99.998 99.959 99.819
136 100.000 100.000 100.000 99.999 99.963 99.829
137 100.000 100.000 100.000 99.999 99.964 99.833
138 100.000 100.000 100.000 99.999 99.965 99.837
139 100.000 100.000 100.000 99.999 99.965 99.842
140 100.000 100.000 100.000 99.999 99.966 99.845
141 100.000 100.000 100.000 99.999 99.967 99.849
142 100.000 100.000 100.000 99.999 99.968 99.852
143 100.000 100.000 100.000 100.000 99.972 99.861
144 100.000 100.000 100.000 100.000 99.972 99.864
145 100.000 100.000 100.000 100.000 99.973 99.868
146 100.000 100.000 100.000 100.000 99.974 99.871
147 100.000 100.000 100.000 100.000 99.975 99.875
148 100.000 100.000 100.000 100.000 99.975 99.878
149 100.000 100.000 100.000 100.000 99.976 99.881
150 100.000 100.000 100.000 100.000 99.977 99.884
151 100.000 100.000 100.000 100.000 99.977 99.886
152 100.000 100.000 100.000 100.000 99.978 99.889
153 100.000 100.000 100.000 100.000 99.979 99.892
154 100.000 100.000 100.000 100.000 99.979 99.895
155 100.000 100.000 100.000 100.000 99.980 99.897
156 100.000 100.000 100.000 100.000 99.980 99.900
157 100.000 100.000 100.000 100.000 99.981 99.903
158 100.000 100.000 100.000 100.000 99.981 99.905
159 100.000 100.000 100.000 100.000 99.982 99.907
160 100.000 100.000 100.000 100.000 99.982 99.909
161 100.000 100.000 100.000 100.000 99.983 99.911
162 100.000 100.000 100.000 100.000 99.983 99.913
163 100.000 100.000 100.000 100.000 99.984 99.916
164 100.000 100.000 100.000 100.000 99.984 99.918
165 100.000 100.000 100.000 100.000 99.985 99.920
166 100.000 100.000 100.000 100.000 99.985 99.922
167 100.000 100.000 100.000 100.000 99.986 99.924
168 100.000 100.000 100.000 100.000 99.986 99.926
169 100.000 100.000 100.000 100.000 99.986 99.928
170 100.000 100.000 100.000 100.000 99.987 99.930
171 100.000 100.000 100.000 100.000 99.987 99.933
172 100.000 100.000 100.000 100.000 99.988 99.935
173 100.000 100.000 100.000 100.000 99.988 99.937
174 100.000 100.000 100.000 100.000 99.989 99.939
175 100.000 100.000 100.000 100.000 99.989 99.941
176 100.000 100.000 100.000 100.000 99.990 99.943
177 100.000 100.000 100.000 100.000 99.990 99.944
178 100.000 100.000 100.000 100.000 99.990 99.946
179 100.000 100.000 100.000 100.000 99.991 99.947
180 100.000 100.000 100.000 100.000 99.991 99.949
181 100.000 100.000 100.000 100.000 99.991 99.950
182 100.000 100.000 100.000 100.000 99.991 99.951
183 100.000 100.000 100.000 100.000 99.992 99.953
184 100.000 100.000 100.000 100.000 99.992 99.954
185 100.000 100.000 100.000 100.000 99.992 99.956
186 100.000 100.000 100.000 100.000 99.993 99.957
187 100.000 100.000 100.000 100.000 99.993 99.958
188 100.000 100.000 100.000 100.000 99.993 99.960
189 100.000 100.000 100.000 100.000 99.994 99.961
190 100.000 100.000 100.000 100.000 99.994 99.963
191 100.000 100.000 100.000 100.000 99.994 99.964
192 100.000 100.000 100.000 100.000 99.995 99.966
193 100.000 100.000 100.000 100.000 99.995 99.967
194 100.000 100.000 100.000 100.000 99.995 99.967
195 100.000 100.000 100.000 100.000 99.995 99.968
196 100.000 100.000 100.000 100.000 99.995 99.969
197 100.000 100.000 100.000 100.000 99.995 99.970
198 100.000 100.000 100.000 100.000 99.995 99.970
199 100.000 100.000 100.000 100.000 99.996 99.971
200 100.000 100.000 100.000 100.000 99.996 99.972
201 100.000 100.000 100.000 100.000 99.996 99.973
202 100.000 100.000 100.000 100.000 99.996 99.973
203 100.000 100.000 100.000 100.000 99.996 99.974
204 100.000 100.000 100.000 100.000 99.996 99.975
205 100.000 100.000 100.000 100.000 99.997 99.976
206 100.000 100.000 100.000 100.000 99.997 99.976
207 100.000 100.000 100.000 100.000 99.997 99.977
208 100.000 100.000 100.000 100.000 99.997 99.978
209 100.000 100.000 100.000 100.000 99.997 99.979
210 100.000 100.000 100.000 100.000 99.997 99.979
211 100.000 100.000 100.000 100.000 99.998 99.980
212 100.000 100.000 100.000 100.000 99.998 99.981
213 100.000 100.000 100.000 100.000 99.998 99.981
214 100.000 100.000 100.000 100.000 99.998 99.982
215 100.000 100.000 100.000 100.000 99.998 99.983
216 100.000 100.000 100.000 100.000 99.998 99.984
217 100.000 100.000 100.000 100.000 99.998 99.984
218 100.000 100.000 100.000 100.000 99.999 99.985
219 100.000 100.000 100.000 100.000 99.999 99.986
220 100.000 100.000 100.000 100.000 99.999 99.987
221 100.000 100.000 100.000 100.000 99.999 99.988
222 100.000 100.000 100.000 100.000 99.999 99.988
223 100.000 100.000 100.000 100.000 99.999 99.989
224 100.000 100.000 100.000 100.000 100.000 99.990
225 100.000 100.000 100.000 100.000 100.000 99.990
226 100.000 100.000 100.000 100.000 100.000 99.991
227 100.000 100.000 100.000 100.000 100.000 99.992
228 100.000 100.000 100.000 100.000 100.000 99.992
229 100.000 100.000 100.000 100.000 100.000 99.992
230 100.000 100.000 100.000 100.000 100.000 99.992
231 100.000 100.000 100.000 100.000 100.000 99.992
232 100.000 100.000 100.000 100.000 100.000 99.992
233 100.000 100.000 100.000 100.000 100.000 99.992
234 100.000 100.000 100.000 100.000 100.000 99.992
235 100.000 100.000 100.000 100.000 100.000 99.992
236 100.000 100.000 100.000 100.000 100.000 99.992
237 100.000 100.000 100.000 100.000 100.000 99.992
238 100.000 100.000 100.000 100.000 100.000 99.992

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 5, Prioritisation Strength: 6

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 74.855 62.088 50.933 41.540 33.785
4 93.023 81.692 69.741 58.630 48.855 40.499
5 93.023 83.546 73.322 63.337 54.099 45.833
6 93.023 84.820 75.888 66.841 58.142 50.080
7 93.023 85.632 77.596 69.273 61.061 53.263
8 95.349 89.224 81.861 73.837 65.674 57.754
9 97.674 92.584 85.692 77.851 69.700 61.677
10 97.674 93.511 87.584 80.482 72.800 65.010
11 97.674 93.859 88.387 81.727 74.417 66.903
12 97.674 94.090 88.997 82.749 75.810 68.589
13 100.000 97.335 92.541 86.310 79.253 71.850
14 100.000 97.914 93.778 88.101 81.441 74.281
15 100.000 98.146 94.360 89.054 82.727 75.830
16 100.000 98.262 94.711 89.696 83.662 77.022
17 100.000 98.262 94.886 90.164 84.455 78.114
18 100.000 98.378 95.209 90.738 85.282 79.166
19 100.000 98.378 95.283 90.951 85.672 79.744
20 100.000 98.378 95.366 91.181 86.080 80.330
21 100.000 98.494 95.680 91.723 86.840 81.279
22 100.000 98.610 95.948 92.156 87.440 82.035
23 100.000 98.725 96.215 92.589 88.037 82.785
24 100.000 98.725 96.261 92.727 88.299 83.187
25 100.000 98.725 96.354 92.983 88.751 83.831
26 100.000 98.957 96.871 93.766 89.754 84.999
27 100.000 98.957 96.908 93.885 89.989 85.371
28 100.000 99.073 97.166 94.286 90.524 86.026
29 100.000 99.305 97.674 95.043 91.481 87.130
30 100.000 99.305 97.720 95.182 91.743 87.527
31 100.000 99.305 97.775 95.342 92.037 87.963
32 100.000 99.305 97.821 95.479 92.291 88.347
33 100.000 99.305 97.877 95.637 92.579 88.770
34 100.000 99.305 97.923 95.772 92.831 89.148
35 100.000 99.305 97.951 95.864 93.016 89.446
36 100.000 99.305 97.969 95.926 93.143 89.655
37 100.000 99.305 97.997 96.013 93.316 89.928
38 100.000 99.305 98.043 96.146 93.561 90.291
39 100.000 99.305 98.071 96.235 93.736 90.570
40 100.000 99.305 98.071 96.246 93.778 90.660
41 100.000 99.305 98.080 96.283 93.861 90.808
42 100.000 99.305 98.098 96.343 93.984 91.007
43 100.000 99.305 98.117 96.403 94.103 91.199
44 100.000 99.305 98.154 96.507 94.292 91.478
45 100.000 99.305 98.163 96.542 94.372 91.618
46 100.000 99.305 98.172 96.580 94.459 91.770
47 100.000 99.305 98.172 96.588 94.489 91.838
48 100.000 99.305 98.181 96.623 94.569 91.977
49 100.000 99.305 98.218 96.732 94.770 92.280
50 100.000 99.305 98.246 96.814 94.925 92.517
51 100.000 99.305 98.246 96.823 94.960 92.592
52 100.000 99.305 98.283 96.926 95.145 92.864
53 100.000 99.305 98.292 96.955 95.201 92.957
54 100.000 99.305 98.301 96.991 95.284 93.103
55 100.000 99.305 98.311 97.022 95.350 93.216
56 100.000 99.421 98.560 97.389 95.810 93.746
57 100.000 99.421 98.588 97.470 95.962 93.976
58 100.000 99.421 98.606 97.527 96.074 94.153
59 100.000 99.421 98.625 97.582 96.177 94.311
60 100.000 99.421 98.625 97.589 96.203 94.371
61 100.000 99.421 98.625 97.594 96.223 94.416
62 100.000 99.537 98.874 97.961 96.683 94.945
63 100.000 99.537 98.874 97.967 96.706 94.999
64 100.000 99.537 98.874 97.976 96.740 95.071
65 100.000 99.537 98.874 97.984 96.766 95.129
66 100.000 99.652 99.123 98.350 97.225 95.657
67 100.000 99.652 99.123 98.356 97.249 95.710
68 100.000 99.768 99.372 98.722 97.705 96.231
69 100.000 99.768 99.372 98.726 97.722 96.272
70 100.000 99.768 99.372 98.734 97.748 96.330
71 100.000 99.768 99.372 98.736 97.757 96.355
72 100.000 99.768 99.372 98.742 97.781 96.407
73 100.000 99.768 99.372 98.746 97.797 96.445
74 100.000 99.768 99.372 98.753 97.823 96.502
75 100.000 99.884 99.622 99.119 98.279 97.022
76 100.000 99.884 99.640 99.172 98.377 97.170
77 100.000 99.884 99.640 99.177 98.396 97.214
78 100.000 99.884 99.640 99.183 98.419 97.265
79 100.000 99.884 99.640 99.189 98.438 97.308
80 100.000 99.884 99.640 99.192 98.451 97.337
81 100.000 99.884 99.640 99.196 98.467 97.374
82 100.000 100.000 99.889 99.562 98.922 97.894
83 100.000 100.000 99.889 99.566 98.938 97.928
84 100.000 100.000 99.889 99.572 98.960 97.979
85 100.000 100.000 99.908 99.625 99.058 98.124
86 100.000 100.000 99.908 99.628 99.070 98.154
87 100.000 100.000 99.908 99.631 99.083 98.183
88 100.000 100.000 99.908 99.635 99.098 98.219
89 100.000 100.000 99.908 99.640 99.114 98.256
90 100.000 100.000 99.908 99.644 99.130 98.290
91 100.000 100.000 99.908 99.647 99.142 98.318
92 100.000 100.000 99.908 99.649 99.150 98.339
93 100.000 100.000 99.908 99.653 99.165 98.374
94 100.000 100.000 99.908 99.655 99.174 98.395
95 100.000 100.000 99.908 99.656 99.179 98.409
96 100.000 100.000 99.908 99.659 99.191 98.437
97 100.000 100.000 99.908 99.662 99.200 98.458
98 100.000 100.000 99.908 99.665 99.212 98.486
99 100.000 100.000 99.908 99.665 99.213 98.491
100 100.000 100.000 99.908 99.667 99.221 98.512
101 100.000 100.000 99.908 99.669 99.229 98.532
102 100.000 100.000 99.908 99.671 99.238 98.553
103 100.000 100.000 99.908 99.673 99.247 98.574
104 100.000 100.000 99.908 99.676 99.258 98.601
105 100.000 100.000 99.908 99.678 99.267 98.622
106 100.000 100.000 99.908 99.681 99.279 98.648
107 100.000 100.000 99.908 99.683 99.287 98.668
108 100.000 100.000 99.908 99.685 99.295 98.688
109 100.000 100.000 99.908 99.688 99.303 98.706
110 100.000 100.000 99.908 99.690 99.311 98.726
111 100.000 100.000 99.908 99.691 99.316 98.738
112 100.000 100.000 99.908 99.693 99.324 98.757
113 100.000 100.000 99.908 99.695 99.332 98.775
114 100.000 100.000 99.908 99.695 99.333 98.781
115 100.000 100.000 99.908 99.698 99.344 98.806
116 100.000 100.000 99.908 99.700 99.352 98.824
117 100.000 100.000 99.908 99.701 99.357 98.837
118 100.000 100.000 99.908 99.702 99.361 98.848
119 100.000 100.000 99.908 99.703 99.366 98.861
120 100.000 100.000 99.908 99.703 99.367 98.866
121 100.000 100.000 99.908 99.705 99.375 98.884
122 100.000 100.000 99.908 99.708 99.386 98.909
123 100.000 100.000 99.908 99.709 99.391 98.921
124 100.000 100.000 99.908 99.711 99.399 98.939
125 100.000 100.000 99.908 99.713 99.403 98.949
126 100.000 100.000 99.908 99.715 99.411 98.967
127 100.000 100.000 99.908 99.716 99.415 98.979
128 100.000 100.000 99.908 99.717 99.420 98.991
129 100.000 100.000 99.908 99.718 99.424 99.002
130 100.000 100.000 99.908 99.720 99.432 99.020
131 100.000 100.000 99.908 99.721 99.436 99.031
132 100.000 100.000 99.917 99.748 99.487 99.108
133 100.000 100.000 99.917 99.748 99.488 99.113
134 100.000 100.000 99.917 99.748 99.489 99.117
135 100.000 100.000 99.917 99.748 99.490 99.121
136 100.000 100.000 99.917 99.748 99.491 99.126
137 100.000 100.000 99.917 99.749 99.496 99.138
138 100.000 100.000 99.917 99.749 99.497 99.142
139 100.000 100.000 99.917 99.749 99.498 99.147
140 100.000 100.000 99.917 99.749 99.498 99.150
141 100.000 100.000 99.917 99.749 99.499 99.154
142 100.000 100.000 99.926 99.776 99.550 99.231
143 100.000 100.000 99.926 99.776 99.551 99.235
144 100.000 100.000 99.935 99.803 99.601 99.312
145 100.000 100.000 99.935 99.803 99.602 99.316
146 100.000 100.000 99.935 99.803 99.603 99.320
147 100.000 100.000 99.935 99.803 99.604 99.324
148 100.000 100.000 99.945 99.830 99.655 99.401
149 100.000 100.000 99.954 99.857 99.705 99.478
150 100.000 100.000 99.963 99.884 99.756 99.555
151 100.000 100.000 99.972 99.911 99.806 99.632
152 100.000 100.000 99.982 99.939 99.857 99.709
153 100.000 100.000 99.991 99.966 99.908 99.786
154 100.000 100.000 99.991 99.966 99.908 99.789
155 100.000 100.000 99.991 99.966 99.909 99.793
156 100.000 100.000 100.000 99.993 99.960 99.870
157 100.000 100.000 100.000 99.993 99.961 99.874
158 100.000 100.000 100.000 99.993 99.962 99.878
159 100.000 100.000 100.000 99.995 99.970 99.896
160 100.000 100.000 100.000 99.997 99.977 99.914
161 100.000 100.000 100.000 99.997 99.978 99.918
162 100.000 100.000 100.000 99.998 99.983 99.928
163 100.000 100.000 100.000 99.998 99.983 99.932
164 100.000 100.000 100.000 99.998 99.984 99.934
165 100.000 100.000 100.000 99.999 99.988 99.945
166 100.000 100.000 100.000 99.999 99.989 99.948
167 100.000 100.000 100.000 100.000 99.993 99.958
168 100.000 100.000 100.000 100.000 99.994 99.961
169 100.000 100.000 100.000 100.000 99.994 99.964
170 100.000 100.000 100.000 100.000 99.995 99.967
171 100.000 100.000 100.000 100.000 99.996 99.969
172 100.000 100.000 100.000 100.000 99.996 99.972
173 100.000 100.000 100.000 100.000 99.997 99.975
174 100.000 100.000 100.000 100.000 99.998 99.978
175 100.000 100.000 100.000 100.000 99.998 99.980
176 100.000 100.000 100.000 100.000 99.999 99.983
177 100.000 100.000 100.000 100.000 100.000 99.986
178 100.000 100.000 100.000 100.000 100.000 99.987
179 100.000 100.000 100.000 100.000 100.000 99.988
180 100.000 100.000 100.000 100.000 100.000 99.989

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 90.698 77.057 63.611 51.814 41.956 33.909
4 90.698 79.143 67.571 56.938 47.585 39.563
5 90.698 80.185 69.694 59.872 51.014 43.213
6 93.023 84.125 74.633 65.310 56.572 48.624
7 95.349 88.065 79.507 70.595 61.901 53.758
8 97.674 91.773 83.984 75.420 66.783 58.499
9 97.674 92.236 85.120 77.219 69.117 61.198
10 97.674 92.352 85.563 78.098 70.435 62.886
11 100.000 95.597 89.098 81.643 73.864 66.144
12 100.000 96.176 90.307 83.370 75.961 68.470
13 100.000 96.756 91.507 85.074 78.021 70.750
14 100.000 96.987 92.080 86.004 79.271 72.255
15 100.000 97.335 92.855 87.175 80.761 73.976
16 100.000 97.914 94.018 88.773 82.636 75.996
17 100.000 98.030 94.360 89.395 83.541 77.153
18 100.000 98.146 94.683 89.966 84.358 78.188
19 100.000 98.262 95.015 90.558 85.204 79.254
20 100.000 98.262 95.089 90.771 85.595 79.832
21 100.000 98.378 95.366 91.222 86.215 80.605
22 100.000 98.378 95.486 91.553 86.795 81.427
23 100.000 98.378 95.523 91.667 87.015 81.769
24 100.000 98.610 96.058 92.499 88.100 83.049
25 100.000 98.610 96.132 92.707 88.474 83.594
26 100.000 98.841 96.640 93.471 89.450 84.735
27 100.000 98.841 96.705 93.659 89.798 85.252
28 100.000 98.957 96.991 94.133 90.453 86.068
29 100.000 99.073 97.295 94.651 91.174 86.963
30 100.000 99.073 97.360 94.837 91.514 87.465
31 100.000 99.073 97.388 94.927 91.695 87.754
32 100.000 99.073 97.461 95.138 92.077 88.312
33 100.000 99.073 97.498 95.253 92.299 88.657
34 100.000 99.073 97.517 95.320 92.446 88.904
35 100.000 99.189 97.766 95.700 92.948 89.518
36 100.000 99.189 97.821 95.861 93.245 89.959
37 100.000 99.189 97.858 95.973 93.460 90.292
38 100.000 99.189 97.895 96.086 93.676 90.625
39 100.000 99.189 97.905 96.121 93.757 90.769
40 100.000 99.305 98.172 96.544 94.323 91.458
41 100.000 99.305 98.172 96.560 94.378 91.575
42 100.000 99.305 98.191 96.618 94.494 91.763
43 100.000 99.305 98.228 96.728 94.699 92.073
44 100.000 99.305 98.246 96.789 94.826 92.280
45 100.000 99.305 98.265 96.849 94.949 92.481
46 100.000 99.305 98.274 96.883 95.023 92.611
47 100.000 99.305 98.283 96.916 95.097 92.740
48 100.000 99.305 98.311 96.996 95.246 92.964
49 100.000 99.305 98.311 97.007 95.283 93.045
50 100.000 99.305 98.311 97.012 95.303 93.092
51 100.000 99.305 98.320 97.043 95.370 93.208
52 100.000 99.305 98.338 97.098 95.476 93.373
53 100.000 99.305 98.338 97.109 95.514 93.456
54 100.000 99.305 98.348 97.143 95.590 93.588
55 100.000 99.305 98.348 97.150 95.617 93.649
56 100.000 99.305 98.357 97.181 95.680 93.756
57 100.000 99.305 98.357 97.193 95.724 93.850
58 100.000 99.305 98.357 97.205 95.764 93.935
59 100.000 99.305 98.357 97.216 95.805 94.021
60 100.000 99.421 98.606 97.582 96.261 94.544
61 100.000 99.537 98.855 97.947 96.717 95.067
62 100.000 99.652 99.105 98.313 97.174 95.590
63 100.000 99.652 99.123 98.367 97.276 95.747
64 100.000 99.652 99.123 98.374 97.303 95.808
65 100.000 99.768 99.372 98.740 97.759 96.331
66 100.000 99.768 99.372 98.745 97.779 96.377
67 100.000 99.768 99.382 98.775 97.842 96.484
68 100.000 99.768 99.391 98.804 97.901 96.581
69 100.000 99.768 99.391 98.812 97.928 96.640
70 100.000 99.768 99.391 98.817 97.947 96.683
71 100.000 99.884 99.640 99.182 98.403 97.204
72 100.000 99.884 99.640 99.191 98.433 97.269
73 100.000 100.000 99.889 99.556 98.889 97.789
74 100.000 100.000 99.889 99.560 98.905 97.826
75 100.000 100.000 99.889 99.565 98.921 97.863
76 100.000 100.000 99.889 99.567 98.930 97.888
77 100.000 100.000 99.889 99.569 98.939 97.910
78 100.000 100.000 99.889 99.573 98.955 97.947
79 100.000 100.000 99.898 99.602 99.013 98.041
80 100.000 100.000 99.898 99.607 99.033 98.085
81 100.000 100.000 99.898 99.610 99.045 98.114
82 100.000 100.000 99.898 99.616 99.064 98.156
83 100.000 100.000 99.898 99.617 99.069 98.171
84 100.000 100.000 99.898 99.621 99.085 98.208
85 100.000 100.000 99.898 99.625 99.101 98.244
86 100.000 100.000 99.898 99.628 99.113 98.274
87 100.000 100.000 99.898 99.628 99.115 98.281
88 100.000 100.000 99.898 99.630 99.123 98.301
89 100.000 100.000 99.898 99.633 99.135 98.328
90 100.000 100.000 99.898 99.638 99.150 98.362
91 100.000 100.000 99.898 99.642 99.165 98.395
92 100.000 100.000 99.908 99.670 99.220 98.482
93 100.000 100.000 99.908 99.672 99.228 98.503
94 100.000 100.000 99.908 99.676 99.243 98.536
95 100.000 100.000 99.908 99.677 99.248 98.550
96 100.000 100.000 99.908 99.679 99.257 98.570
97 100.000 100.000 99.908 99.681 99.266 98.593
98 100.000 100.000 99.908 99.684 99.278 98.620
99 100.000 100.000 99.908 99.687 99.286 98.641
100 100.000 100.000 99.908 99.691 99.301 98.674
101 100.000 100.000 99.908 99.694 99.313 98.702
102 100.000 100.000 99.908 99.696 99.322 98.722
103 100.000 100.000 99.908 99.696 99.323 98.728
104 100.000 100.000 99.908 99.696 99.324 98.733
105 100.000 100.000 99.908 99.698 99.332 98.753
106 100.000 100.000 99.908 99.700 99.341 98.773
107 100.000 100.000 99.908 99.700 99.342 98.778
108 100.000 100.000 99.908 99.700 99.343 98.783
109 100.000 100.000 99.908 99.703 99.354 98.809
110 100.000 100.000 99.908 99.703 99.355 98.814
111 100.000 100.000 99.908 99.703 99.357 98.820
112 100.000 100.000 99.908 99.705 99.365 98.839
113 100.000 100.000 99.908 99.708 99.376 98.865
114 100.000 100.000 99.908 99.709 99.381 98.876
115 100.000 100.000 99.908 99.709 99.382 98.880
116 100.000 100.000 99.908 99.709 99.383 98.887
117 100.000 100.000 99.908 99.710 99.387 98.897
118 100.000 100.000 99.908 99.714 99.399 98.922
119 100.000 100.000 99.908 99.714 99.400 98.927
120 100.000 100.000 99.908 99.714 99.401 98.931
121 100.000 100.000 99.908 99.715 99.405 98.942
122 100.000 100.000 99.917 99.742 99.456 99.019
123 100.000 100.000 99.917 99.743 99.460 99.032
124 100.000 100.000 99.917 99.743 99.461 99.036
125 100.000 100.000 99.917 99.746 99.473 99.061
126 100.000 100.000 99.917 99.748 99.481 99.079
127 100.000 100.000 99.926 99.775 99.531 99.156
128 100.000 100.000 99.926 99.775 99.532 99.161
129 100.000 100.000 99.935 99.802 99.583 99.237
130 100.000 100.000 99.935 99.802 99.584 99.242
131 100.000 100.000 99.945 99.829 99.634 99.318
132 100.000 100.000 99.954 99.856 99.685 99.395
133 100.000 100.000 99.963 99.883 99.735 99.472
134 100.000 100.000 99.972 99.910 99.786 99.549
135 100.000 100.000 99.972 99.913 99.794 99.567
136 100.000 100.000 99.972 99.913 99.795 99.570
137 100.000 100.000 99.982 99.940 99.845 99.647
138 100.000 100.000 99.991 99.967 99.896 99.724
139 100.000 100.000 100.000 99.994 99.946 99.801
140 100.000 100.000 100.000 99.994 99.947 99.805
141 100.000 100.000 100.000 99.994 99.948 99.809
142 100.000 100.000 100.000 99.996 99.956 99.826
143 100.000 100.000 100.000 99.998 99.964 99.844
144 100.000 100.000 100.000 99.999 99.968 99.855
145 100.000 100.000 100.000 99.999 99.969 99.858
146 100.000 100.000 100.000 99.999 99.970 99.862
147 100.000 100.000 100.000 99.999 99.970 99.865
148 100.000 100.000 100.000 99.999 99.971 99.868
149 100.000 100.000 100.000 99.999 99.972 99.870
150 100.000 100.000 100.000 99.999 99.972 99.873
151 100.000 100.000 100.000 99.999 99.973 99.876
152 100.000 100.000 100.000 99.999 99.973 99.878
153 100.000 100.000 100.000 99.999 99.974 99.881
154 100.000 100.000 100.000 99.999 99.974 99.884
155 100.000 100.000 100.000 99.999 99.975 99.886
156 100.000 100.000 100.000 99.999 99.976 99.890
157 100.000 100.000 100.000 100.000 99.980 99.899
158 100.000 100.000 100.000 100.000 99.980 99.902
159 100.000 100.000 100.000 100.000 99.981 99.904
160 100.000 100.000 100.000 100.000 99.981 99.907
161 100.000 100.000 100.000 100.000 99.982 99.909
162 100.000 100.000 100.000 100.000 99.983 99.912
163 100.000 100.000 100.000 100.000 99.983 99.915
164 100.000 100.000 100.000 100.000 99.984 99.917
165 100.000 100.000 100.000 100.000 99.984 99.920
166 100.000 100.000 100.000 100.000 99.985 99.922
167 100.000 100.000 100.000 100.000 99.985 99.924
168 100.000 100.000 100.000 100.000 99.986 99.927
169 100.000 100.000 100.000 100.000 99.986 99.929
170 100.000 100.000 100.000 100.000 99.987 99.931
171 100.000 100.000 100.000 100.000 99.987 99.932
172 100.000 100.000 100.000 100.000 99.988 99.934
173 100.000 100.000 100.000 100.000 99.988 99.937
174 100.000 100.000 100.000 100.000 99.988 99.939
175 100.000 100.000 100.000 100.000 99.989 99.941
176 100.000 100.000 100.000 100.000 99.989 99.943
177 100.000 100.000 100.000 100.000 99.990 99.944
178 100.000 100.000 100.000 100.000 99.990 99.946
179 100.000 100.000 100.000 100.000 99.991 99.949
180 100.000 100.000 100.000 100.000 99.991 99.951
181 100.000 100.000 100.000 100.000 99.992 99.953
182 100.000 100.000 100.000 100.000 99.992 99.955
183 100.000 100.000 100.000 100.000 99.992 99.956
184 100.000 100.000 100.000 100.000 99.993 99.958
185 100.000 100.000 100.000 100.000 99.993 99.959
186 100.000 100.000 100.000 100.000 99.993 99.961
187 100.000 100.000 100.000 100.000 99.994 99.963
188 100.000 100.000 100.000 100.000 99.994 99.964
189 100.000 100.000 100.000 100.000 99.994 99.965
190 100.000 100.000 100.000 100.000 99.995 99.966
191 100.000 100.000 100.000 100.000 99.995 99.967
192 100.000 100.000 100.000 100.000 99.995 99.968
193 100.000 100.000 100.000 100.000 99.995 99.969
194 100.000 100.000 100.000 100.000 99.995 99.970
195 100.000 100.000 100.000 100.000 99.995 99.971
196 100.000 100.000 100.000 100.000 99.996 99.972
197 100.000 100.000 100.000 100.000 99.996 99.972
198 100.000 100.000 100.000 100.000 99.996 99.973
199 100.000 100.000 100.000 100.000 99.996 99.974
200 100.000 100.000 100.000 100.000 99.996 99.975
201 100.000 100.000 100.000 100.000 99.996 99.975
202 100.000 100.000 100.000 100.000 99.997 99.976
203 100.000 100.000 100.000 100.000 99.997 99.977
204 100.000 100.000 100.000 100.000 99.997 99.978
205 100.000 100.000 100.000 100.000 99.997 99.978
206 100.000 100.000 100.000 100.000 99.997 99.979
207 100.000 100.000 100.000 100.000 99.997 99.980
208 100.000 100.000 100.000 100.000 99.998 99.980
209 100.000 100.000 100.000 100.000 99.998 99.981
210 100.000 100.000 100.000 100.000 99.998 99.982
211 100.000 100.000 100.000 100.000 99.998 99.983
212 100.000 100.000 100.000 100.000 99.998 99.983
213 100.000 100.000 100.000 100.000 99.998 99.984
214 100.000 100.000 100.000 100.000 99.998 99.985
215 100.000 100.000 100.000 100.000 99.999 99.985
216 100.000 100.000 100.000 100.000 99.999 99.986
217 100.000 100.000 100.000 100.000 99.999 99.987
218 100.000 100.000 100.000 100.000 99.999 99.987
219 100.000 100.000 100.000 100.000 99.999 99.988
220 100.000 100.000 100.000 100.000 99.999 99.988
221 100.000 100.000 100.000 100.000 99.999 99.989
222 100.000 100.000 100.000 100.000 100.000 99.990
223 100.000 100.000 100.000 100.000 100.000 99.990
224 100.000 100.000 100.000 100.000 100.000 99.991
225 100.000 100.000 100.000 100.000 100.000 99.992
226 100.000 100.000 100.000 100.000 100.000 99.992
227 100.000 100.000 100.000 100.000 100.000 99.992
228 100.000 100.000 100.000 100.000 100.000 99.992
229 100.000 100.000 100.000 100.000 100.000 99.992
230 100.000 100.000 100.000 100.000 100.000 99.992
231 100.000 100.000 100.000 100.000 100.000 99.992
232 100.000 100.000 100.000 100.000 100.000 99.992
233 100.000 100.000 100.000 100.000 100.000 99.992
234 100.000 100.000 100.000 100.000 100.000 99.992
235 100.000 100.000 100.000 100.000 100.000 99.992
236 100.000 100.000 100.000 100.000 100.000 99.992
237 100.000 100.000 100.000 100.000 100.000 99.992
238 100.000 100.000 100.000 100.000 100.000 99.992

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 6, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.820 72.833 61.307 51.003 42.134
5 100.000 91.425 80.089 68.543 57.864 48.443
6 100.000 93.048 83.292 72.834 62.724 53.457
7 100.000 94.206 85.609 75.985 66.353 57.271
8 100.000 95.249 87.723 78.899 69.755 60.893
9 100.000 96.292 89.800 81.721 73.014 64.333
10 100.000 97.103 91.351 83.748 75.270 66.637
11 100.000 97.451 92.181 85.052 76.961 68.602
12 100.000 97.798 92.994 86.305 78.564 70.445
13 100.000 98.030 93.575 87.248 79.817 71.929
14 100.000 98.262 94.138 88.145 81.000 73.327
15 100.000 98.494 94.646 88.907 81.970 74.457
16 100.000 98.610 95.006 89.564 82.914 75.639
17 100.000 98.725 95.357 90.204 83.841 76.812
18 100.000 98.841 95.708 90.844 84.765 77.980
19 100.000 98.957 95.966 91.240 85.281 78.593
20 100.000 99.073 96.363 91.989 86.369 79.960
21 100.000 99.189 96.705 92.612 87.280 81.128
22 100.000 99.305 96.981 93.057 87.878 81.854
23 100.000 99.421 97.258 93.498 88.463 82.553
24 100.000 99.537 97.554 93.989 89.136 83.376
25 100.000 99.652 97.849 94.481 89.809 84.199
26 100.000 99.768 98.117 94.903 90.371 84.880
27 100.000 99.884 98.375 95.298 90.884 85.487
28 100.000 100.000 98.661 95.763 91.511 86.246
29 100.000 100.000 98.735 95.969 91.877 86.773
30 100.000 100.000 98.763 96.051 92.029 87.004
31 100.000 100.000 98.818 96.208 92.312 87.415
32 100.000 100.000 98.883 96.393 92.649 87.910
33 100.000 100.000 98.920 96.503 92.854 88.220
34 100.000 100.000 99.003 96.743 93.293 88.865
35 100.000 100.000 99.012 96.775 93.361 88.979
36 100.000 100.000 99.040 96.861 93.527 89.239
37 100.000 100.000 99.095 97.018 93.811 89.654
38 100.000 100.000 99.105 97.048 93.873 89.756
39 100.000 100.000 99.142 97.155 94.067 90.043
40 100.000 100.000 99.178 97.260 94.257 90.323
41 100.000 100.000 99.243 97.437 94.566 90.761
42 100.000 100.000 99.252 97.467 94.628 90.864
43 100.000 100.000 99.298 97.606 94.890 91.260
44 100.000 100.000 99.298 97.615 94.923 91.331
45 100.000 100.000 99.363 97.804 95.274 91.856
46 100.000 100.000 99.372 97.833 95.328 91.941
47 100.000 100.000 99.382 97.870 95.416 92.097
48 100.000 100.000 99.400 97.933 95.545 92.310
49 100.000 100.000 99.409 97.960 95.596 92.386
50 100.000 100.000 99.409 97.966 95.618 92.435
51 100.000 100.000 99.409 97.973 95.644 92.489
52 100.000 100.000 99.409 97.978 95.662 92.530
53 100.000 100.000 99.409 97.980 95.670 92.548
54 100.000 100.000 99.446 98.087 95.864 92.834
55 100.000 100.000 99.465 98.148 95.989 93.033
56 100.000 100.000 99.483 98.209 96.111 93.230
57 100.000 100.000 99.483 98.223 96.163 93.343
58 100.000 100.000 99.483 98.223 96.164 93.346
59 100.000 100.000 99.483 98.227 96.178 93.377
60 100.000 100.000 99.520 98.331 96.365 93.650
61 100.000 100.000 99.529 98.359 96.416 93.727
62 100.000 100.000 99.575 98.494 96.667 94.101
63 100.000 100.000 99.612 98.604 96.875 94.420
64 100.000 100.000 99.622 98.636 96.942 94.533
65 100.000 100.000 99.622 98.644 96.971 94.596
66 100.000 100.000 99.631 98.677 97.044 94.719
67 100.000 100.000 99.631 98.680 97.055 94.744
68 100.000 100.000 99.649 98.739 97.170 94.926
69 100.000 100.000 99.649 98.750 97.210 95.010
70 100.000 100.000 99.695 98.882 97.450 95.364
71 100.000 100.000 99.714 98.936 97.548 95.512
72 100.000 100.000 99.723 98.968 97.618 95.632
73 100.000 100.000 99.742 99.029 97.745 95.841
74 100.000 100.000 99.742 99.037 97.773 95.903
75 100.000 100.000 99.742 99.037 97.773 95.904
76 100.000 100.000 99.742 99.038 97.778 95.917
77 100.000 100.000 99.742 99.044 97.800 95.964
78 100.000 100.000 99.760 99.104 97.922 96.162
79 100.000 100.000 99.760 99.109 97.942 96.206
80 100.000 100.000 99.788 99.191 98.093 96.435
81 100.000 100.000 99.788 99.195 98.109 96.471
82 100.000 100.000 99.788 99.200 98.128 96.513
83 100.000 100.000 99.788 99.201 98.133 96.526
84 100.000 100.000 99.788 99.201 98.133 96.528
85 100.000 100.000 99.797 99.228 98.184 96.605
86 100.000 100.000 99.797 99.233 98.203 96.648
87 100.000 100.000 99.797 99.238 98.219 96.685
88 100.000 100.000 99.806 99.268 98.282 96.789
89 100.000 100.000 99.815 99.295 98.332 96.867
90 100.000 100.000 99.825 99.324 98.390 96.960
91 100.000 100.000 99.825 99.331 98.417 97.018
92 100.000 100.000 99.843 99.391 98.536 97.210
93 100.000 100.000 99.843 99.394 98.548 97.238
94 100.000 100.000 99.843 99.405 98.589 97.327
95 100.000 100.000 99.843 99.406 98.594 97.340
96 100.000 100.000 99.843 99.412 98.613 97.384
97 100.000 100.000 99.843 99.412 98.614 97.388
98 100.000 100.000 99.843 99.413 98.619 97.401
99 100.000 100.000 99.843 99.420 98.646 97.461
100 100.000 100.000 99.843 99.426 98.668 97.509
101 100.000 100.000 99.843 99.427 98.672 97.519
102 100.000 100.000 99.843 99.429 98.681 97.540
103 100.000 100.000 99.843 99.430 98.685 97.549
104 100.000 100.000 99.843 99.430 98.685 97.552
105 100.000 100.000 99.843 99.430 98.686 97.553
106 100.000 100.000 99.852 99.458 98.741 97.641
107 100.000 100.000 99.852 99.460 98.749 97.659
108 100.000 100.000 99.852 99.463 98.756 97.677
109 100.000 100.000 99.852 99.463 98.758 97.684
110 100.000 100.000 99.852 99.467 98.774 97.721
111 100.000 100.000 99.871 99.520 98.872 97.868
112 100.000 100.000 99.871 99.522 98.880 97.888
113 100.000 100.000 99.871 99.522 98.880 97.889
114 100.000 100.000 99.871 99.522 98.881 97.890
115 100.000 100.000 99.871 99.524 98.888 97.908
116 100.000 100.000 99.871 99.527 98.900 97.933
117 100.000 100.000 99.871 99.528 98.904 97.942
118 100.000 100.000 99.871 99.529 98.908 97.954
119 100.000 100.000 99.889 99.583 99.010 98.108
120 100.000 100.000 99.889 99.584 99.015 98.124
121 100.000 100.000 99.889 99.584 99.016 98.127
122 100.000 100.000 99.898 99.612 99.066 98.204
123 100.000 100.000 99.917 99.669 99.178 98.380
124 100.000 100.000 99.917 99.671 99.187 98.401
125 100.000 100.000 99.917 99.673 99.195 98.420
126 100.000 100.000 99.926 99.701 99.249 98.506
127 100.000 100.000 99.926 99.703 99.257 98.524
128 100.000 100.000 99.926 99.704 99.261 98.534
129 100.000 100.000 99.926 99.706 99.270 98.553
130 100.000 100.000 99.926 99.706 99.270 98.556
131 100.000 100.000 99.926 99.707 99.275 98.568
132 100.000 100.000 99.926 99.711 99.290 98.603
133 100.000 100.000 99.926 99.716 99.305 98.637
134 100.000 100.000 99.935 99.743 99.356 98.714
135 100.000 100.000 99.935 99.743 99.357 98.719
136 100.000 100.000 99.935 99.746 99.369 98.746
137 100.000 100.000 99.935 99.747 99.374 98.757
138 100.000 100.000 99.935 99.751 99.389 98.792
139 100.000 100.000 99.935 99.751 99.389 98.793
140 100.000 100.000 99.935 99.753 99.397 98.813
141 100.000 100.000 99.945 99.780 99.449 98.892
142 100.000 100.000 99.945 99.782 99.456 98.909
143 100.000 100.000 99.945 99.782 99.456 98.911
144 100.000 100.000 99.945 99.783 99.461 98.922
145 100.000 100.000 99.945 99.783 99.461 98.923
146 100.000 100.000 99.945 99.786 99.473 98.949
147 100.000 100.000 99.945 99.789 99.481 98.969
148 100.000 100.000 99.945 99.791 99.489 98.986
149 100.000 100.000 99.945 99.794 99.500 99.012
150 100.000 100.000 99.945 99.795 99.504 99.022
151 100.000 100.000 99.945 99.796 99.508 99.032
152 100.000 100.000 99.945 99.798 99.516 99.050
153 100.000 100.000 99.945 99.802 99.532 99.084
154 100.000 100.000 99.945 99.804 99.540 99.104
155 100.000 100.000 99.945 99.805 99.544 99.115
156 100.000 100.000 99.945 99.806 99.548 99.124
157 100.000 100.000 99.945 99.807 99.553 99.136
158 100.000 100.000 99.945 99.810 99.564 99.162
159 100.000 100.000 99.945 99.810 99.565 99.164
160 100.000 100.000 99.945 99.811 99.569 99.173
161 100.000 100.000 99.945 99.811 99.569 99.174
162 100.000 100.000 99.945 99.811 99.570 99.177
163 100.000 100.000 99.945 99.811 99.570 99.179
164 100.000 100.000 99.945 99.813 99.574 99.188
165 100.000 100.000 99.945 99.813 99.574 99.189
166 100.000 100.000 99.945 99.815 99.582 99.209
167 100.000 100.000 99.945 99.815 99.583 99.213
168 100.000 100.000 99.945 99.816 99.587 99.222
169 100.000 100.000 99.945 99.816 99.587 99.223
170 100.000 100.000 99.945 99.816 99.588 99.224
171 100.000 100.000 99.945 99.816 99.589 99.229
172 100.000 100.000 99.945 99.816 99.589 99.230
173 100.000 100.000 99.945 99.817 99.593 99.239
174 100.000 100.000 99.945 99.817 99.594 99.243
175 100.000 100.000 99.945 99.818 99.598 99.253
176 100.000 100.000 99.945 99.818 99.598 99.253
177 100.000 100.000 99.945 99.818 99.598 99.253
178 100.000 100.000 99.945 99.818 99.598 99.254
179 100.000 100.000 99.945 99.819 99.602 99.264
180 100.000 100.000 99.945 99.819 99.602 99.264
181 100.000 100.000 99.945 99.820 99.606 99.273
182 100.000 100.000 99.945 99.820 99.606 99.274
183 100.000 100.000 99.945 99.821 99.610 99.284
184 100.000 100.000 99.945 99.821 99.610 99.284
185 100.000 100.000 99.945 99.821 99.610 99.284
186 100.000 100.000 99.945 99.821 99.611 99.286
187 100.000 100.000 99.945 99.821 99.611 99.286
188 100.000 100.000 99.945 99.821 99.611 99.289
189 100.000 100.000 99.945 99.821 99.612 99.292
190 100.000 100.000 99.945 99.821 99.612 99.294
191 100.000 100.000 99.945 99.821 99.613 99.297
192 100.000 100.000 99.945 99.821 99.613 99.298
193 100.000 100.000 99.945 99.821 99.614 99.301
194 100.000 100.000 99.945 99.821 99.614 99.302
195 100.000 100.000 99.945 99.821 99.615 99.303
196 100.000 100.000 99.945 99.822 99.619 99.313
197 100.000 100.000 99.945 99.822 99.619 99.313
198 100.000 100.000 99.945 99.823 99.623 99.323
199 100.000 100.000 99.945 99.825 99.631 99.342
200 100.000 100.000 99.945 99.825 99.631 99.344
201 100.000 100.000 99.945 99.827 99.639 99.363
202 100.000 100.000 99.945 99.827 99.640 99.364
203 100.000 100.000 99.945 99.827 99.640 99.365
204 100.000 100.000 99.945 99.827 99.640 99.368
205 100.000 100.000 99.945 99.827 99.641 99.368
206 100.000 100.000 99.945 99.827 99.641 99.371
207 100.000 100.000 99.945 99.829 99.649 99.390
208 100.000 100.000 99.945 99.829 99.649 99.391
209 100.000 100.000 99.945 99.829 99.650 99.392
210 100.000 100.000 99.945 99.829 99.650 99.393
211 100.000 100.000 99.945 99.829 99.650 99.394
212 100.000 100.000 99.945 99.830 99.654 99.405
213 100.000 100.000 99.945 99.830 99.655 99.408
214 100.000 100.000 99.954 99.857 99.706 99.485
215 100.000 100.000 99.954 99.858 99.710 99.495
216 100.000 100.000 99.954 99.858 99.710 99.496
217 100.000 100.000 99.954 99.858 99.711 99.499
218 100.000 100.000 99.954 99.858 99.711 99.499
219 100.000 100.000 99.954 99.858 99.711 99.502
220 100.000 100.000 99.954 99.859 99.715 99.510
221 100.000 100.000 99.963 99.886 99.766 99.587
222 100.000 100.000 99.963 99.888 99.770 99.598
223 100.000 100.000 99.963 99.888 99.770 99.600
224 100.000 100.000 99.963 99.888 99.771 99.600
225 100.000 100.000 99.963 99.888 99.771 99.601
226 100.000 100.000 99.963 99.888 99.771 99.603
227 100.000 100.000 99.963 99.888 99.771 99.603
228 100.000 100.000 99.963 99.889 99.775 99.612
229 100.000 100.000 99.963 99.889 99.775 99.614
230 100.000 100.000 99.963 99.889 99.776 99.616
231 100.000 100.000 99.963 99.889 99.777 99.619
232 100.000 100.000 99.963 99.889 99.777 99.620
233 100.000 100.000 99.963 99.889 99.777 99.621
234 100.000 100.000 99.963 99.889 99.778 99.624
235 100.000 100.000 99.963 99.889 99.778 99.625
236 100.000 100.000 99.963 99.889 99.778 99.626
237 100.000 100.000 99.963 99.889 99.778 99.627
238 100.000 100.000 99.963 99.889 99.778 99.627
239 100.000 100.000 99.972 99.916 99.829 99.704
240 100.000 100.000 99.972 99.917 99.833 99.713
241 100.000 100.000 99.972 99.917 99.833 99.714
242 100.000 100.000 99.972 99.917 99.833 99.714
243 100.000 100.000 99.972 99.917 99.833 99.715
244 100.000 100.000 99.972 99.917 99.834 99.717
245 100.000 100.000 99.972 99.917 99.834 99.718
246 100.000 100.000 99.972 99.917 99.834 99.719
247 100.000 100.000 99.972 99.917 99.834 99.719
248 100.000 100.000 99.972 99.917 99.835 99.723
249 100.000 100.000 99.972 99.917 99.835 99.723
250 100.000 100.000 99.972 99.917 99.835 99.724
251 100.000 100.000 99.982 99.944 99.886 99.801
252 100.000 100.000 99.982 99.944 99.886 99.802
253 100.000 100.000 99.982 99.944 99.886 99.803
254 100.000 100.000 99.982 99.944 99.886 99.803
255 100.000 100.000 99.982 99.944 99.887 99.805
256 100.000 100.000 99.982 99.944 99.887 99.805
257 100.000 100.000 99.982 99.944 99.887 99.805
258 100.000 100.000 99.982 99.944 99.887 99.805
259 100.000 100.000 99.982 99.945 99.891 99.815
260 100.000 100.000 99.982 99.945 99.891 99.815
261 100.000 100.000 99.982 99.945 99.891 99.817
262 100.000 100.000 99.982 99.946 99.895 99.826
263 100.000 100.000 99.982 99.946 99.895 99.826
264 100.000 100.000 99.982 99.946 99.895 99.827
265 100.000 100.000 99.982 99.946 99.895 99.827
266 100.000 100.000 99.982 99.946 99.895 99.827
267 100.000 100.000 99.982 99.946 99.895 99.827
268 100.000 100.000 99.982 99.946 99.895 99.827
269 100.000 100.000 99.982 99.946 99.895 99.827
270 100.000 100.000 99.982 99.946 99.895 99.827
271 100.000 100.000 99.982 99.946 99.896 99.829
272 100.000 100.000 99.982 99.946 99.896 99.829
273 100.000 100.000 99.982 99.946 99.896 99.831
274 100.000 100.000 99.982 99.946 99.896 99.832
275 100.000 100.000 99.982 99.946 99.896 99.832
276 100.000 100.000 99.982 99.946 99.896 99.832
277 100.000 100.000 99.982 99.946 99.897 99.834
278 100.000 100.000 99.982 99.946 99.897 99.834
279 100.000 100.000 99.982 99.946 99.897 99.834
280 100.000 100.000 99.982 99.946 99.897 99.835
281 100.000 100.000 99.982 99.946 99.897 99.836
282 100.000 100.000 99.982 99.946 99.897 99.836
283 100.000 100.000 99.982 99.946 99.897 99.836
284 100.000 100.000 99.982 99.946 99.897 99.836
285 100.000 100.000 99.982 99.946 99.897 99.837
286 100.000 100.000 99.982 99.946 99.897 99.838
287 100.000 100.000 99.982 99.946 99.897 99.838
288 100.000 100.000 99.982 99.946 99.898 99.839
289 100.000 100.000 99.991 99.973 99.948 99.916
290 100.000 100.000 99.991 99.973 99.948 99.916
291 100.000 100.000 99.991 99.973 99.948 99.916
292 100.000 100.000 99.991 99.973 99.948 99.916
293 100.000 100.000 99.991 99.973 99.948 99.917
294 100.000 100.000 99.991 99.973 99.948 99.917
295 100.000 100.000 99.991 99.973 99.948 99.917
296 100.000 100.000 99.991 99.973 99.948 99.918
297 100.000 100.000 99.991 99.973 99.949 99.920
298 100.000 100.000 99.991 99.973 99.949 99.921
299 100.000 100.000 99.991 99.973 99.949 99.922
300 100.000 100.000 100.000 100.000 100.000 99.999
301 100.000 100.000 100.000 100.000 100.000 99.999
302 100.000 100.000 100.000 100.000 100.000 99.999
303 100.000 100.000 100.000 100.000 100.000 99.999
304 100.000 100.000 100.000 100.000 100.000 100.000
305 100.000 100.000 100.000 100.000 100.000 100.000
306 100.000 100.000 100.000 100.000 100.000 100.000
307 100.000 100.000 100.000 100.000 100.000 100.000
308 100.000 100.000 100.000 100.000 100.000 100.000
309 100.000 100.000 100.000 100.000 100.000 100.000
310 100.000 100.000 100.000 100.000 100.000 100.000

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 93.023 82.387 70.913 59.967 50.139 41.618
5 97.674 88.992 78.178 67.225 57.037 47.972
6 97.674 90.498 81.178 71.276 61.657 52.770
7 100.000 94.206 85.609 75.988 66.363 57.289
8 100.000 95.249 87.769 79.011 69.929 61.108
9 100.000 96.292 89.864 81.877 73.252 64.627
10 100.000 97.103 91.471 84.057 75.776 67.307
11 100.000 97.451 92.246 85.208 77.200 68.898
12 100.000 97.798 93.067 86.493 78.870 70.845
13 100.000 98.146 93.824 87.615 80.277 72.452
14 100.000 98.378 94.397 88.537 81.499 73.902
15 100.000 98.610 94.969 89.467 82.749 75.405
16 100.000 98.841 95.569 90.459 84.082 76.998
17 100.000 98.957 95.864 90.944 84.736 77.782
18 100.000 99.073 96.114 91.318 85.219 78.358
19 100.000 99.189 96.418 91.830 85.922 79.215
20 100.000 99.305 96.714 92.322 86.596 80.042
21 100.000 99.421 97.018 92.849 87.347 80.994
22 100.000 99.537 97.452 93.695 88.595 82.576
23 100.000 99.652 97.738 94.160 89.223 83.335
24 100.000 99.768 98.006 94.580 89.778 84.000
25 100.000 99.884 98.292 95.048 90.412 84.771
26 100.000 100.000 98.541 95.422 90.898 85.351
27 100.000 100.000 98.597 95.580 91.181 85.765
28 100.000 100.000 98.634 95.694 91.402 86.104
29 100.000 100.000 98.735 95.982 91.917 86.848
30 100.000 100.000 98.818 96.214 92.329 87.441
31 100.000 100.000 98.892 96.416 92.680 87.936
32 100.000 100.000 98.911 96.470 92.781 88.090
33 100.000 100.000 98.948 96.577 92.976 88.381
34 100.000 100.000 98.975 96.658 93.127 88.608
35 100.000 100.000 98.994 96.717 93.246 88.796
36 100.000 100.000 99.012 96.776 93.365 88.986
37 100.000 100.000 99.022 96.805 93.419 89.073
38 100.000 100.000 99.086 96.988 93.749 89.552
39 100.000 100.000 99.105 97.051 93.881 89.768
40 100.000 100.000 99.123 97.122 94.037 90.033
41 100.000 100.000 99.160 97.226 94.225 90.307
42 100.000 100.000 99.188 97.314 94.396 90.573
43 100.000 100.000 99.225 97.418 94.583 90.846
44 100.000 100.000 99.289 97.597 94.900 91.303
45 100.000 100.000 99.298 97.628 94.965 91.411
46 100.000 100.000 99.317 97.686 95.076 91.584
47 100.000 100.000 99.326 97.721 95.155 91.720
48 100.000 100.000 99.326 97.723 95.163 91.737
49 100.000 100.000 99.354 97.807 95.321 91.980
50 100.000 100.000 99.400 97.944 95.579 92.370
51 100.000 100.000 99.437 98.055 95.790 92.690
52 100.000 100.000 99.455 98.110 95.892 92.848
53 100.000 100.000 99.465 98.139 95.951 92.941
54 100.000 100.000 99.465 98.144 95.969 92.982
55 100.000 100.000 99.474 98.175 96.034 93.090
56 100.000 100.000 99.474 98.178 96.048 93.123
57 100.000 100.000 99.492 98.243 96.183 93.346
58 100.000 100.000 99.492 98.246 96.195 93.373
59 100.000 100.000 99.511 98.302 96.303 93.539
60 100.000 100.000 99.529 98.356 96.405 93.694
61 100.000 100.000 99.603 98.566 96.781 94.241
62 100.000 100.000 99.603 98.568 96.789 94.260
63 100.000 100.000 99.640 98.680 97.005 94.593
64 100.000 100.000 99.640 98.682 97.012 94.610
65 100.000 100.000 99.640 98.685 97.020 94.628
66 100.000 100.000 99.668 98.772 97.194 94.906
67 100.000 100.000 99.668 98.774 97.202 94.925
68 100.000 100.000 99.695 98.860 97.368 95.184
69 100.000 100.000 99.705 98.895 97.448 95.323
70 100.000 100.000 99.705 98.905 97.485 95.402
71 100.000 100.000 99.705 98.912 97.508 95.453
72 100.000 100.000 99.714 98.940 97.563 95.543
73 100.000 100.000 99.723 98.973 97.637 95.672
74 100.000 100.000 99.751 99.056 97.795 95.914
75 100.000 100.000 99.751 99.058 97.803 95.932
76 100.000 100.000 99.760 99.088 97.861 96.027
77 100.000 100.000 99.769 99.121 97.934 96.153
78 100.000 100.000 99.769 99.122 97.938 96.162
79 100.000 100.000 99.797 99.204 98.093 96.396
80 100.000 100.000 99.797 99.213 98.123 96.461
81 100.000 100.000 99.797 99.215 98.131 96.480
82 100.000 100.000 99.806 99.244 98.190 96.577
83 100.000 100.000 99.815 99.271 98.240 96.654
84 100.000 100.000 99.825 99.299 98.295 96.739
85 100.000 100.000 99.825 99.309 98.331 96.817
86 100.000 100.000 99.825 99.311 98.335 96.826
87 100.000 100.000 99.834 99.341 98.397 96.928
88 100.000 100.000 99.834 99.343 98.407 96.953
89 100.000 100.000 99.834 99.347 98.422 96.987
90 100.000 100.000 99.843 99.377 98.484 97.091
91 100.000 100.000 99.843 99.382 98.503 97.132
92 100.000 100.000 99.843 99.383 98.508 97.145
93 100.000 100.000 99.852 99.415 98.574 97.255
94 100.000 100.000 99.852 99.419 98.590 97.292
95 100.000 100.000 99.852 99.423 98.605 97.325
96 100.000 100.000 99.862 99.453 98.667 97.428
97 100.000 100.000 99.862 99.462 98.697 97.495
98 100.000 100.000 99.862 99.463 98.703 97.512
99 100.000 100.000 99.862 99.464 98.708 97.527
100 100.000 100.000 99.862 99.466 98.716 97.544
101 100.000 100.000 99.862 99.466 98.717 97.550
102 100.000 100.000 99.862 99.468 98.725 97.567
103 100.000 100.000 99.862 99.469 98.730 97.579
104 100.000 100.000 99.862 99.473 98.745 97.615
105 100.000 100.000 99.862 99.478 98.765 97.660
106 100.000 100.000 99.862 99.481 98.776 97.685
107 100.000 100.000 99.862 99.483 98.784 97.703
108 100.000 100.000 99.871 99.510 98.835 97.779
109 100.000 100.000 99.871 99.514 98.846 97.806
110 100.000 100.000 99.871 99.514 98.847 97.808
111 100.000 100.000 99.871 99.517 98.859 97.835
112 100.000 100.000 99.871 99.519 98.867 97.854
113 100.000 100.000 99.880 99.548 98.926 97.951
114 100.000 100.000 99.880 99.550 98.934 97.973
115 100.000 100.000 99.880 99.555 98.953 98.014
116 100.000 100.000 99.880 99.559 98.969 98.048
117 100.000 100.000 99.880 99.562 98.976 98.066
118 100.000 100.000 99.880 99.563 98.981 98.079
119 100.000 100.000 99.880 99.563 98.982 98.083
120 100.000 100.000 99.880 99.563 98.983 98.087
121 100.000 100.000 99.889 99.594 99.049 98.197
122 100.000 100.000 99.889 99.595 99.053 98.207
123 100.000 100.000 99.889 99.599 99.068 98.240
124 100.000 100.000 99.889 99.603 99.084 98.278
125 100.000 100.000 99.889 99.605 99.093 98.300
126 100.000 100.000 99.889 99.607 99.100 98.317
127 100.000 100.000 99.889 99.610 99.112 98.345
128 100.000 100.000 99.889 99.613 99.121 98.365
129 100.000 100.000 99.889 99.613 99.121 98.366
130 100.000 100.000 99.889 99.616 99.132 98.391
131 100.000 100.000 99.889 99.618 99.140 98.409
132 100.000 100.000 99.889 99.618 99.141 98.413
133 100.000 100.000 99.889 99.618 99.141 98.413
134 100.000 100.000 99.889 99.618 99.141 98.414
135 100.000 100.000 99.889 99.618 99.142 98.417
136 100.000 100.000 99.898 99.645 99.193 98.496
137 100.000 100.000 99.898 99.645 99.193 98.497
138 100.000 100.000 99.898 99.647 99.201 98.516
139 100.000 100.000 99.898 99.647 99.202 98.521
140 100.000 100.000 99.898 99.648 99.207 98.532
141 100.000 100.000 99.898 99.648 99.207 98.534
142 100.000 100.000 99.898 99.648 99.207 98.535
143 100.000 100.000 99.898 99.648 99.208 98.536
144 100.000 100.000 99.898 99.649 99.212 98.545
145 100.000 100.000 99.898 99.649 99.212 98.547
146 100.000 100.000 99.898 99.649 99.212 98.549
147 100.000 100.000 99.908 99.676 99.263 98.627
148 100.000 100.000 99.917 99.703 99.314 98.703
149 100.000 100.000 99.917 99.705 99.322 98.722
150 100.000 100.000 99.917 99.705 99.322 98.723
151 100.000 100.000 99.917 99.705 99.322 98.725
152 100.000 100.000 99.917 99.708 99.334 98.750
153 100.000 100.000 99.917 99.709 99.338 98.760
154 100.000 100.000 99.926 99.739 99.396 98.855
155 100.000 100.000 99.926 99.739 99.397 98.860
156 100.000 100.000 99.926 99.739 99.398 98.861
157 100.000 100.000 99.926 99.740 99.402 98.870
158 100.000 100.000 99.926 99.741 99.406 98.883
159 100.000 100.000 99.926 99.744 99.419 98.911
160 100.000 100.000 99.926 99.744 99.419 98.912
161 100.000 100.000 99.926 99.746 99.426 98.930
162 100.000 100.000 99.935 99.773 99.477 99.007
163 100.000 100.000 99.935 99.775 99.485 99.026
164 100.000 100.000 99.935 99.777 99.493 99.043
165 100.000 100.000 99.935 99.778 99.497 99.052
166 100.000 100.000 99.935 99.779 99.501 99.063
167 100.000 100.000 99.935 99.780 99.505 99.073
168 100.000 100.000 99.935 99.781 99.509 99.082
169 100.000 100.000 99.935 99.783 99.517 99.101
170 100.000 100.000 99.935 99.783 99.518 99.102
171 100.000 100.000 99.935 99.783 99.518 99.103
172 100.000 100.000 99.935 99.783 99.518 99.104
173 100.000 100.000 99.935 99.783 99.519 99.109
174 100.000 100.000 99.935 99.784 99.523 99.118
175 100.000 100.000 99.935 99.784 99.523 99.119
176 100.000 100.000 99.935 99.785 99.527 99.128
177 100.000 100.000 99.935 99.785 99.527 99.130
178 100.000 100.000 99.935 99.786 99.531 99.139
179 100.000 100.000 99.935 99.789 99.540 99.159
180 100.000 100.000 99.935 99.789 99.540 99.159
181 100.000 100.000 99.935 99.789 99.540 99.162
182 100.000 100.000 99.935 99.790 99.544 99.172
183 100.000 100.000 99.935 99.790 99.545 99.174
184 100.000 100.000 99.935 99.790 99.545 99.176
185 100.000 100.000 99.935 99.790 99.546 99.178
186 100.000 100.000 99.935 99.791 99.550 99.188
187 100.000 100.000 99.935 99.791 99.550 99.188
188 100.000 100.000 99.935 99.792 99.554 99.197
189 100.000 100.000 99.935 99.792 99.555 99.201
190 100.000 100.000 99.935 99.794 99.563 99.219
191 100.000 100.000 99.935 99.794 99.563 99.221
192 100.000 100.000 99.935 99.796 99.571 99.239
193 100.000 100.000 99.935 99.796 99.571 99.240
194 100.000 100.000 99.935 99.796 99.571 99.240
195 100.000 100.000 99.935 99.796 99.572 99.244
196 100.000 100.000 99.935 99.796 99.572 99.246
197 100.000 100.000 99.935 99.796 99.572 99.246
198 100.000 100.000 99.935 99.797 99.577 99.256
199 100.000 100.000 99.935 99.797 99.577 99.257
200 100.000 100.000 99.935 99.797 99.577 99.259
201 100.000 100.000 99.935 99.797 99.577 99.260
202 100.000 100.000 99.935 99.797 99.577 99.260
203 100.000 100.000 99.935 99.797 99.578 99.261
204 100.000 100.000 99.935 99.798 99.582 99.271
205 100.000 100.000 99.935 99.798 99.582 99.274
206 100.000 100.000 99.935 99.798 99.583 99.275
207 100.000 100.000 99.935 99.799 99.586 99.284
208 100.000 100.000 99.935 99.799 99.586 99.284
209 100.000 100.000 99.935 99.799 99.587 99.285
210 100.000 100.000 99.935 99.799 99.587 99.285
211 100.000 100.000 99.935 99.799 99.587 99.288
212 100.000 100.000 99.935 99.799 99.588 99.289
213 100.000 100.000 99.935 99.799 99.588 99.290
214 100.000 100.000 99.935 99.800 99.592 99.300
215 100.000 100.000 99.935 99.800 99.592 99.301
216 100.000 100.000 99.935 99.800 99.592 99.301
217 100.000 100.000 99.935 99.800 99.592 99.301
218 100.000 100.000 99.935 99.800 99.592 99.301
219 100.000 100.000 99.945 99.827 99.643 99.378
220 100.000 100.000 99.954 99.854 99.693 99.455
221 100.000 100.000 99.954 99.855 99.697 99.464
222 100.000 100.000 99.954 99.855 99.698 99.469
223 100.000 100.000 99.954 99.855 99.699 99.472
224 100.000 100.000 99.954 99.855 99.699 99.473
225 100.000 100.000 99.954 99.855 99.700 99.474
226 100.000 100.000 99.954 99.855 99.700 99.474
227 100.000 100.000 99.954 99.855 99.700 99.476
228 100.000 100.000 99.954 99.855 99.700 99.478
229 100.000 100.000 99.954 99.856 99.705 99.489
230 100.000 100.000 99.954 99.856 99.705 99.490
231 100.000 100.000 99.954 99.856 99.705 99.490
232 100.000 100.000 99.954 99.856 99.705 99.491
233 100.000 100.000 99.954 99.856 99.705 99.492
234 100.000 100.000 99.954 99.856 99.705 99.492
235 100.000 100.000 99.954 99.856 99.705 99.492
236 100.000 100.000 99.954 99.857 99.709 99.501
237 100.000 100.000 99.954 99.857 99.710 99.503
238 100.000 100.000 99.954 99.857 99.710 99.504
239 100.000 100.000 99.954 99.857 99.710 99.504
240 100.000 100.000 99.954 99.857 99.710 99.506
241 100.000 100.000 99.954 99.857 99.710 99.507
242 100.000 100.000 99.954 99.857 99.710 99.507
243 100.000 100.000 99.954 99.859 99.718 99.525
244 100.000 100.000 99.954 99.859 99.719 99.526
245 100.000 100.000 99.954 99.859 99.719 99.527
246 100.000 100.000 99.963 99.886 99.769 99.604
247 100.000 100.000 99.963 99.886 99.769 99.604
248 100.000 100.000 99.963 99.886 99.769 99.604
249 100.000 100.000 99.963 99.886 99.769 99.605
250 100.000 100.000 99.963 99.886 99.769 99.605
251 100.000 100.000 99.963 99.886 99.770 99.605
252 100.000 100.000 99.963 99.886 99.770 99.605
253 100.000 100.000 99.963 99.886 99.770 99.605
254 100.000 100.000 99.963 99.886 99.770 99.606
255 100.000 100.000 99.963 99.886 99.770 99.607
256 100.000 100.000 99.963 99.886 99.770 99.608
257 100.000 100.000 99.963 99.886 99.770 99.608
258 100.000 100.000 99.963 99.886 99.770 99.609
259 100.000 100.000 99.963 99.886 99.770 99.609
260 100.000 100.000 99.963 99.888 99.774 99.618
261 100.000 100.000 99.963 99.888 99.774 99.618
262 100.000 100.000 99.963 99.888 99.774 99.618
263 100.000 100.000 99.963 99.888 99.774 99.619
264 100.000 100.000 99.963 99.889 99.778 99.629
265 100.000 100.000 99.972 99.916 99.829 99.706
266 100.000 100.000 99.972 99.916 99.829 99.706
267 100.000 100.000 99.972 99.916 99.829 99.706
268 100.000 100.000 99.972 99.916 99.830 99.708
269 100.000 100.000 99.972 99.916 99.830 99.709
270 100.000 100.000 99.972 99.916 99.830 99.709
271 100.000 100.000 99.972 99.916 99.830 99.709
272 100.000 100.000 99.972 99.916 99.830 99.710
273 100.000 100.000 99.972 99.916 99.830 99.710
274 100.000 100.000 99.972 99.916 99.830 99.712
275 100.000 100.000 99.972 99.916 99.830 99.712
276 100.000 100.000 99.972 99.916 99.830 99.713
277 100.000 100.000 99.982 99.943 99.881 99.789
278 100.000 100.000 99.982 99.943 99.881 99.791
279 100.000 100.000 99.982 99.943 99.882 99.792
280 100.000 100.000 99.982 99.943 99.882 99.792
281 100.000 100.000 99.982 99.943 99.882 99.794
282 100.000 100.000 99.982 99.943 99.882 99.794
283 100.000 100.000 99.982 99.943 99.882 99.795
284 100.000 100.000 99.982 99.944 99.886 99.805
285 100.000 100.000 99.982 99.944 99.886 99.805
286 100.000 100.000 99.982 99.945 99.890 99.814
287 100.000 100.000 99.982 99.945 99.890 99.814
288 100.000 100.000 99.982 99.945 99.890 99.814
289 100.000 100.000 99.982 99.945 99.890 99.814
290 100.000 100.000 99.982 99.945 99.891 99.815
291 100.000 100.000 99.982 99.945 99.891 99.815
292 100.000 100.000 99.982 99.946 99.895 99.825
293 100.000 100.000 99.982 99.946 99.895 99.825
294 100.000 100.000 99.982 99.946 99.895 99.825
295 100.000 100.000 99.982 99.946 99.895 99.826
296 100.000 100.000 99.982 99.946 99.895 99.827
297 100.000 100.000 99.982 99.946 99.895 99.827
298 100.000 100.000 99.982 99.946 99.895 99.827
299 100.000 100.000 99.982 99.946 99.895 99.827
300 100.000 100.000 99.991 99.973 99.946 99.904
301 100.000 100.000 99.991 99.973 99.946 99.904
302 100.000 100.000 99.991 99.973 99.946 99.904
303 100.000 100.000 99.991 99.973 99.946 99.906
304 100.000 100.000 99.991 99.973 99.946 99.908
305 100.000 100.000 99.991 99.973 99.947 99.909
306 100.000 100.000 99.991 99.973 99.947 99.910
307 100.000 100.000 99.991 99.973 99.947 99.910
308 100.000 100.000 99.991 99.973 99.947 99.910
309 100.000 100.000 99.991 99.973 99.947 99.910
310 100.000 100.000 99.991 99.973 99.947 99.911
311 100.000 100.000 99.991 99.973 99.947 99.911
312 100.000 100.000 99.991 99.973 99.947 99.911
313 100.000 100.000 99.991 99.973 99.947 99.912
314 100.000 100.000 99.991 99.973 99.947 99.912
315 100.000 100.000 99.991 99.973 99.948 99.913
316 100.000 100.000 99.991 99.973 99.948 99.913
317 100.000 100.000 99.991 99.973 99.948 99.915
318 100.000 100.000 99.991 99.973 99.948 99.915
319 100.000 100.000 99.991 99.973 99.948 99.915
320 100.000 100.000 99.991 99.973 99.948 99.916
321 100.000 100.000 99.991 99.973 99.948 99.916
322 100.000 100.000 99.991 99.973 99.948 99.916
323 100.000 100.000 99.991 99.973 99.948 99.916
324 100.000 100.000 99.991 99.973 99.948 99.917
325 100.000 100.000 99.991 99.973 99.948 99.917
326 100.000 100.000 99.991 99.973 99.948 99.917
327 100.000 100.000 99.991 99.973 99.948 99.917
328 100.000 100.000 100.000 100.000 99.999 99.994
329 100.000 100.000 100.000 100.000 99.999 99.994
330 100.000 100.000 100.000 100.000 99.999 99.994
331 100.000 100.000 100.000 100.000 99.999 99.994
332 100.000 100.000 100.000 100.000 99.999 99.996
333 100.000 100.000 100.000 100.000 99.999 99.996
334 100.000 100.000 100.000 100.000 99.999 99.996
335 100.000 100.000 100.000 100.000 99.999 99.996
336 100.000 100.000 100.000 100.000 99.999 99.996
337 100.000 100.000 100.000 100.000 100.000 99.997
338 100.000 100.000 100.000 100.000 100.000 99.997
339 100.000 100.000 100.000 100.000 100.000 99.997
340 100.000 100.000 100.000 100.000 100.000 99.998
341 100.000 100.000 100.000 100.000 100.000 99.998
342 100.000 100.000 100.000 100.000 100.000 99.998
343 100.000 100.000 100.000 100.000 100.000 99.998
344 100.000 100.000 100.000 100.000 100.000 99.998
345 100.000 100.000 100.000 100.000 100.000 99.998
346 100.000 100.000 100.000 100.000 100.000 99.998
347 100.000 100.000 100.000 100.000 100.000 99.998
348 100.000 100.000 100.000 100.000 100.000 99.998
349 100.000 100.000 100.000 100.000 100.000 99.998
350 100.000 100.000 100.000 100.000 100.000 99.999
351 100.000 100.000 100.000 100.000 100.000 99.999
352 100.000 100.000 100.000 100.000 100.000 99.999
353 100.000 100.000 100.000 100.000 100.000 99.999
354 100.000 100.000 100.000 100.000 100.000 99.999
355 100.000 100.000 100.000 100.000 100.000 99.999
356 100.000 100.000 100.000 100.000 100.000 99.999
357 100.000 100.000 100.000 100.000 100.000 99.999
358 100.000 100.000 100.000 100.000 100.000 100.000
359 100.000 100.000 100.000 100.000 100.000 100.000
360 100.000 100.000 100.000 100.000 100.000 100.000
361 100.000 100.000 100.000 100.000 100.000 100.000
362 100.000 100.000 100.000 100.000 100.000 100.000
363 100.000 100.000 100.000 100.000 100.000 100.000
364 100.000 100.000 100.000 100.000 100.000 100.000
365 100.000 100.000 100.000 100.000 100.000 100.000
366 100.000 100.000 100.000 100.000 100.000 100.000

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 6, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 83.721 67.323 53.245 41.856 32.878 25.880
3 88.372 74.739 61.885 50.692 41.300 33.569
4 93.023 82.039 70.341 59.331 49.545 41.115
5 95.349 86.095 75.436 64.899 55.176 46.539
6 95.349 87.370 77.993 68.381 59.189 50.751
7 97.674 91.078 82.470 73.207 64.071 55.493
8 97.674 92.121 84.603 76.165 67.538 59.194
9 100.000 95.597 88.627 80.389 71.745 63.244
10 100.000 96.524 90.529 83.039 74.868 66.598
11 100.000 96.871 91.415 84.477 76.767 68.822
12 100.000 97.567 92.827 86.440 79.090 71.339
13 100.000 97.914 93.649 87.727 80.763 73.293
14 100.000 98.146 94.240 88.703 82.084 74.887
15 100.000 98.262 94.646 89.486 83.244 76.369
16 100.000 98.494 95.218 90.411 84.477 77.838
17 100.000 98.610 95.597 91.120 85.512 79.148
18 100.000 98.841 96.151 91.998 86.669 80.518
19 100.000 98.957 96.511 92.654 87.609 81.694
20 100.000 99.073 96.843 93.239 88.435 82.720
21 100.000 99.189 97.166 93.822 89.288 83.823
22 100.000 99.189 97.268 94.099 89.767 84.494
23 100.000 99.189 97.369 94.384 90.274 85.223
24 100.000 99.305 97.646 94.838 90.904 86.014
25 100.000 99.305 97.738 95.092 91.347 86.640
26 100.000 99.421 97.997 95.487 91.860 87.247
27 100.000 99.537 98.255 95.888 92.392 87.893
28 100.000 99.652 98.505 96.261 92.872 88.461
29 100.000 99.652 98.569 96.446 93.209 88.956
30 100.000 99.768 98.818 96.815 93.675 89.499
31 100.000 99.884 99.068 97.188 94.155 90.067
32 100.000 99.884 99.132 97.382 94.520 90.618
33 100.000 100.000 99.382 97.749 94.982 91.153
34 100.000 100.000 99.428 97.883 95.226 91.514
35 100.000 100.000 99.474 98.023 95.493 91.918
36 100.000 100.000 99.520 98.155 95.734 92.273
37 100.000 100.000 99.566 98.295 95.999 92.678
38 100.000 100.000 99.603 98.405 96.208 92.997
39 100.000 100.000 99.640 98.520 96.429 93.337
40 100.000 100.000 99.677 98.628 96.630 93.640
41 100.000 100.000 99.714 98.740 96.842 93.966
42 100.000 100.000 99.751 98.852 97.058 94.300
43 100.000 100.000 99.788 98.958 97.254 94.592
44 100.000 100.000 99.806 99.022 97.386 94.808
45 100.000 100.000 99.825 99.088 97.526 95.040
46 100.000 100.000 99.843 99.152 97.661 95.263
47 100.000 100.000 99.852 99.191 97.753 95.429
48 100.000 100.000 99.862 99.218 97.804 95.507
49 100.000 100.000 99.871 99.253 97.884 95.648
50 100.000 100.000 99.880 99.286 97.953 95.766
51 100.000 100.000 99.889 99.317 98.019 95.876
52 100.000 100.000 99.898 99.348 98.085 95.988
53 100.000 100.000 99.908 99.381 98.159 96.116
54 100.000 100.000 99.917 99.409 98.213 96.201
55 100.000 100.000 99.926 99.437 98.264 96.278
56 100.000 100.000 99.935 99.471 98.341 96.413
57 100.000 100.000 99.945 99.498 98.393 96.494
58 100.000 100.000 99.954 99.525 98.443 96.572
59 100.000 100.000 99.963 99.554 98.502 96.667
60 100.000 100.000 99.972 99.582 98.557 96.755
61 100.000 100.000 99.982 99.609 98.608 96.833
62 100.000 100.000 99.991 99.637 98.659 96.911
63 100.000 100.000 100.000 99.664 98.709 96.988
64 100.000 100.000 100.000 99.674 98.746 97.068
65 100.000 100.000 100.000 99.680 98.769 97.116
66 100.000 100.000 100.000 99.691 98.806 97.199
67 100.000 100.000 100.000 99.692 98.812 97.217
68 100.000 100.000 100.000 99.699 98.838 97.273
69 100.000 100.000 100.000 99.704 98.858 97.317
70 100.000 100.000 100.000 99.713 98.887 97.379
71 100.000 100.000 100.000 99.722 98.921 97.452
72 100.000 100.000 100.000 99.728 98.943 97.500
73 100.000 100.000 100.000 99.731 98.955 97.527
74 100.000 100.000 100.000 99.731 98.955 97.528
75 100.000 100.000 100.000 99.736 98.974 97.570
76 100.000 100.000 100.000 99.744 99.000 97.628
77 100.000 100.000 100.000 99.747 99.012 97.656
78 100.000 100.000 100.000 99.758 99.052 97.742
79 100.000 100.000 100.000 99.768 99.087 97.819
80 100.000 100.000 100.000 99.773 99.106 97.862
81 100.000 100.000 100.000 99.782 99.140 97.936
82 100.000 100.000 100.000 99.789 99.163 97.988
83 100.000 100.000 100.000 99.797 99.193 98.052
84 100.000 100.000 100.000 99.802 99.212 98.096
85 100.000 100.000 100.000 99.806 99.228 98.132
86 100.000 100.000 100.000 99.809 99.240 98.159
87 100.000 100.000 100.000 99.815 99.259 98.201
88 100.000 100.000 100.000 99.816 99.263 98.212
89 100.000 100.000 100.000 99.820 99.279 98.248
90 100.000 100.000 100.000 99.826 99.301 98.297
91 100.000 100.000 100.000 99.826 99.301 98.298
92 100.000 100.000 100.000 99.830 99.317 98.334
93 100.000 100.000 100.000 99.831 99.321 98.345
94 100.000 100.000 100.000 99.831 99.322 98.347
95 100.000 100.000 100.000 99.838 99.345 98.397
96 100.000 100.000 100.000 99.843 99.364 98.440
97 100.000 100.000 100.000 99.843 99.365 98.445
98 100.000 100.000 100.000 99.848 99.384 98.487
99 100.000 100.000 100.000 99.853 99.403 98.530
100 100.000 100.000 100.000 99.853 99.404 98.535
101 100.000 100.000 100.000 99.860 99.431 98.594
102 100.000 100.000 100.000 99.863 99.439 98.613
103 100.000 100.000 100.000 99.871 99.469 98.677
104 100.000 100.000 100.000 99.877 99.491 98.728
105 100.000 100.000 100.000 99.878 99.496 98.739
106 100.000 100.000 100.000 99.880 99.504 98.760
107 100.000 100.000 100.000 99.883 99.516 98.785
108 100.000 100.000 100.000 99.885 99.524 98.804
109 100.000 100.000 100.000 99.886 99.528 98.813
110 100.000 100.000 100.000 99.890 99.540 98.842
111 100.000 100.000 100.000 99.891 99.544 98.851
112 100.000 100.000 100.000 99.894 99.556 98.881
113 100.000 100.000 100.000 99.899 99.575 98.923
114 100.000 100.000 100.000 99.902 99.587 98.949
115 100.000 100.000 100.000 99.902 99.587 98.951
116 100.000 100.000 100.000 99.907 99.607 98.996
117 100.000 100.000 100.000 99.907 99.608 99.001
118 100.000 100.000 100.000 99.908 99.613 99.014
119 100.000 100.000 100.000 99.909 99.618 99.027
120 100.000 100.000 100.000 99.909 99.618 99.028
121 100.000 100.000 100.000 99.913 99.630 99.054
122 100.000 100.000 100.000 99.915 99.637 99.071
123 100.000 100.000 100.000 99.915 99.638 99.076
124 100.000 100.000 100.000 99.916 99.643 99.086
125 100.000 100.000 100.000 99.917 99.647 99.097
126 100.000 100.000 100.000 99.921 99.662 99.133
127 100.000 100.000 100.000 99.924 99.675 99.162
128 100.000 100.000 100.000 99.925 99.679 99.174
129 100.000 100.000 100.000 99.925 99.680 99.178
130 100.000 100.000 100.000 99.926 99.685 99.189
131 100.000 100.000 100.000 99.927 99.689 99.198
132 100.000 100.000 100.000 99.927 99.689 99.200
133 100.000 100.000 100.000 99.927 99.690 99.203
134 100.000 100.000 100.000 99.931 99.705 99.236
135 100.000 100.000 100.000 99.931 99.705 99.237
136 100.000 100.000 100.000 99.931 99.705 99.238
137 100.000 100.000 100.000 99.933 99.713 99.258
138 100.000 100.000 100.000 99.934 99.718 99.269
139 100.000 100.000 100.000 99.936 99.726 99.289
140 100.000 100.000 100.000 99.939 99.734 99.307
141 100.000 100.000 100.000 99.941 99.742 99.325
142 100.000 100.000 100.000 99.941 99.742 99.327
143 100.000 100.000 100.000 99.942 99.746 99.337
144 100.000 100.000 100.000 99.947 99.765 99.378
145 100.000 100.000 100.000 99.949 99.773 99.396
146 100.000 100.000 100.000 99.951 99.781 99.413
147 100.000 100.000 100.000 99.952 99.785 99.424
148 100.000 100.000 100.000 99.952 99.786 99.429
149 100.000 100.000 100.000 99.952 99.787 99.435
150 100.000 100.000 100.000 99.953 99.792 99.445
151 100.000 100.000 100.000 99.953 99.792 99.446
152 100.000 100.000 100.000 99.953 99.792 99.446
153 100.000 100.000 100.000 99.953 99.792 99.449
154 100.000 100.000 100.000 99.955 99.800 99.467
155 100.000 100.000 100.000 99.956 99.804 99.476
156 100.000 100.000 100.000 99.958 99.812 99.494
157 100.000 100.000 100.000 99.961 99.824 99.520
158 100.000 100.000 100.000 99.963 99.828 99.530
159 100.000 100.000 100.000 99.965 99.836 99.549
160 100.000 100.000 100.000 99.966 99.840 99.559
161 100.000 100.000 100.000 99.967 99.844 99.570
162 100.000 100.000 100.000 99.967 99.844 99.571
163 100.000 100.000 100.000 99.967 99.845 99.574
164 100.000 100.000 100.000 99.967 99.846 99.576
165 100.000 100.000 100.000 99.968 99.850 99.585
166 100.000 100.000 100.000 99.968 99.850 99.587
167 100.000 100.000 100.000 99.968 99.850 99.588
168 100.000 100.000 100.000 99.968 99.851 99.590
169 100.000 100.000 100.000 99.970 99.859 99.609
170 100.000 100.000 100.000 99.970 99.859 99.612
171 100.000 100.000 100.000 99.970 99.860 99.614
172 100.000 100.000 100.000 99.973 99.871 99.640
173 100.000 100.000 100.000 99.975 99.880 99.661
174 100.000 100.000 100.000 99.975 99.881 99.664
175 100.000 100.000 100.000 99.977 99.889 99.686
176 100.000 100.000 100.000 99.978 99.894 99.696
177 100.000 100.000 100.000 99.978 99.894 99.698
178 100.000 100.000 100.000 99.979 99.898 99.709
179 100.000 100.000 100.000 99.979 99.899 99.711
180 100.000 100.000 100.000 99.981 99.906 99.728
181 100.000 100.000 100.000 99.982 99.910 99.737
182 100.000 100.000 100.000 99.983 99.914 99.747
183 100.000 100.000 100.000 99.983 99.915 99.748
184 100.000 100.000 100.000 99.983 99.915 99.750
185 100.000 100.000 100.000 99.983 99.915 99.750
186 100.000 100.000 100.000 99.983 99.916 99.753
187 100.000 100.000 100.000 99.984 99.920 99.763
188 100.000 100.000 100.000 99.984 99.920 99.766
189 100.000 100.000 100.000 99.985 99.924 99.775
190 100.000 100.000 100.000 99.985 99.924 99.775
191 100.000 100.000 100.000 99.986 99.929 99.785
192 100.000 100.000 100.000 99.986 99.929 99.787
193 100.000 100.000 100.000 99.986 99.929 99.788
194 100.000 100.000 100.000 99.986 99.929 99.790
195 100.000 100.000 100.000 99.986 99.930 99.792
196 100.000 100.000 100.000 99.988 99.934 99.801
197 100.000 100.000 100.000 99.988 99.934 99.804
198 100.000 100.000 100.000 99.989 99.939 99.813
199 100.000 100.000 100.000 99.989 99.939 99.814
200 100.000 100.000 100.000 99.989 99.939 99.814
201 100.000 100.000 100.000 99.989 99.939 99.815
202 100.000 100.000 100.000 99.989 99.939 99.817
203 100.000 100.000 100.000 99.989 99.939 99.817
204 100.000 100.000 100.000 99.990 99.943 99.826
205 100.000 100.000 100.000 99.991 99.948 99.837
206 100.000 100.000 100.000 99.991 99.948 99.838
207 100.000 100.000 100.000 99.991 99.948 99.841
208 100.000 100.000 100.000 99.991 99.949 99.842
209 100.000 100.000 100.000 99.991 99.949 99.842
210 100.000 100.000 100.000 99.992 99.953 99.853
211 100.000 100.000 100.000 99.992 99.953 99.855
212 100.000 100.000 100.000 99.992 99.954 99.856
213 100.000 100.000 100.000 99.992 99.954 99.856
214 100.000 100.000 100.000 99.992 99.954 99.859
215 100.000 100.000 100.000 99.992 99.955 99.860
216 100.000 100.000 100.000 99.992 99.955 99.861
217 100.000 100.000 100.000 99.994 99.962 99.878
218 100.000 100.000 100.000 99.994 99.963 99.879
219 100.000 100.000 100.000 99.994 99.963 99.880
220 100.000 100.000 100.000 99.994 99.963 99.882
221 100.000 100.000 100.000 99.994 99.964 99.884
222 100.000 100.000 100.000 99.994 99.964 99.885
223 100.000 100.000 100.000 99.994 99.964 99.887
224 100.000 100.000 100.000 99.994 99.964 99.887
225 100.000 100.000 100.000 99.994 99.965 99.890
226 100.000 100.000 100.000 99.994 99.965 99.891
227 100.000 100.000 100.000 99.994 99.965 99.891
228 100.000 100.000 100.000 99.994 99.966 99.894
229 100.000 100.000 100.000 99.994 99.966 99.895
230 100.000 100.000 100.000 99.994 99.967 99.897
231 100.000 100.000 100.000 99.995 99.970 99.906
232 100.000 100.000 100.000 99.996 99.974 99.915
233 100.000 100.000 100.000 99.996 99.975 99.916
234 100.000 100.000 100.000 99.996 99.975 99.918
235 100.000 100.000 100.000 99.996 99.975 99.919
236 100.000 100.000 100.000 99.996 99.975 99.920
237 100.000 100.000 100.000 99.996 99.976 99.921
238 100.000 100.000 100.000 99.997 99.979 99.930
239 100.000 100.000 100.000 99.997 99.979 99.930
240 100.000 100.000 100.000 99.997 99.980 99.931
241 100.000 100.000 100.000 99.997 99.980 99.933
242 100.000 100.000 100.000 99.997 99.980 99.933
243 100.000 100.000 100.000 99.997 99.980 99.933
244 100.000 100.000 100.000 99.997 99.981 99.936
245 100.000 100.000 100.000 99.997 99.981 99.936
246 100.000 100.000 100.000 99.997 99.981 99.936
247 100.000 100.000 100.000 99.997 99.981 99.936
248 100.000 100.000 100.000 99.997 99.981 99.937
249 100.000 100.000 100.000 99.997 99.982 99.941
250 100.000 100.000 100.000 100.000 99.993 99.966
251 100.000 100.000 100.000 100.000 99.994 99.969
252 100.000 100.000 100.000 100.000 99.994 99.969
253 100.000 100.000 100.000 100.000 99.994 99.971
254 100.000 100.000 100.000 100.000 99.994 99.972
255 100.000 100.000 100.000 100.000 99.995 99.974
256 100.000 100.000 100.000 100.000 99.995 99.974
257 100.000 100.000 100.000 100.000 99.995 99.974
258 100.000 100.000 100.000 100.000 99.995 99.975
259 100.000 100.000 100.000 100.000 99.995 99.976
260 100.000 100.000 100.000 100.000 99.995 99.976
261 100.000 100.000 100.000 100.000 99.995 99.977
262 100.000 100.000 100.000 100.000 99.995 99.977
263 100.000 100.000 100.000 100.000 99.996 99.978
264 100.000 100.000 100.000 100.000 99.996 99.978
265 100.000 100.000 100.000 100.000 99.996 99.980
266 100.000 100.000 100.000 100.000 99.996 99.981
267 100.000 100.000 100.000 100.000 99.997 99.983
268 100.000 100.000 100.000 100.000 99.997 99.983
269 100.000 100.000 100.000 100.000 99.997 99.983
270 100.000 100.000 100.000 100.000 99.997 99.984
271 100.000 100.000 100.000 100.000 99.997 99.985
272 100.000 100.000 100.000 100.000 99.997 99.986
273 100.000 100.000 100.000 100.000 99.997 99.986
274 100.000 100.000 100.000 100.000 99.998 99.987
275 100.000 100.000 100.000 100.000 99.998 99.987
276 100.000 100.000 100.000 100.000 99.998 99.989
277 100.000 100.000 100.000 100.000 99.998 99.989
278 100.000 100.000 100.000 100.000 99.998 99.990
279 100.000 100.000 100.000 100.000 99.998 99.990
280 100.000 100.000 100.000 100.000 99.998 99.990
281 100.000 100.000 100.000 100.000 99.999 99.992
282 100.000 100.000 100.000 100.000 99.999 99.993
283 100.000 100.000 100.000 100.000 99.999 99.993
284 100.000 100.000 100.000 100.000 99.999 99.994
285 100.000 100.000 100.000 100.000 99.999 99.994
286 100.000 100.000 100.000 100.000 99.999 99.995
287 100.000 100.000 100.000 100.000 99.999 99.995
288 100.000 100.000 100.000 100.000 99.999 99.995
289 100.000 100.000 100.000 100.000 99.999 99.996
290 100.000 100.000 100.000 100.000 99.999 99.996
291 100.000 100.000 100.000 100.000 99.999 99.996
292 100.000 100.000 100.000 100.000 99.999 99.996
293 100.000 100.000 100.000 100.000 99.999 99.997
294 100.000 100.000 100.000 100.000 99.999 99.997
295 100.000 100.000 100.000 100.000 99.999 99.997
296 100.000 100.000 100.000 100.000 100.000 99.997
297 100.000 100.000 100.000 100.000 100.000 99.998
298 100.000 100.000 100.000 100.000 100.000 99.998
299 100.000 100.000 100.000 100.000 100.000 99.998
300 100.000 100.000 100.000 100.000 100.000 99.999
301 100.000 100.000 100.000 100.000 100.000 100.000
302 100.000 100.000 100.000 100.000 100.000 100.000
303 100.000 100.000 100.000 100.000 100.000 100.000
304 100.000 100.000 100.000 100.000 100.000 100.000
305 100.000 100.000 100.000 100.000 100.000 100.000
306 100.000 100.000 100.000 100.000 100.000 100.000
307 100.000 100.000 100.000 100.000 100.000 100.000
308 100.000 100.000 100.000 100.000 100.000 100.000
309 100.000 100.000 100.000 100.000 100.000 100.000
310 100.000 100.000 100.000 100.000 100.000 100.000

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 86.047 69.757 55.183 43.241 33.810 26.482
3 90.698 77.520 64.405 52.735 42.856 34.707
4 95.349 84.820 72.833 61.307 51.003 42.134
5 97.674 88.992 78.169 67.202 57.001 47.927
6 97.674 90.382 80.966 71.012 61.382 52.508
7 100.000 94.206 85.581 75.916 66.245 57.134
8 100.000 95.133 87.520 78.650 69.490 60.627
9 100.000 96.060 89.440 81.341 72.672 64.048
10 100.000 96.871 91.101 83.662 75.421 67.021
11 100.000 97.335 92.172 85.305 77.519 69.430
12 100.000 97.567 92.837 86.461 79.119 71.370
13 100.000 97.914 93.677 87.797 80.876 73.438
14 100.000 98.146 94.295 88.837 82.289 75.138
15 100.000 98.378 94.886 89.813 83.611 76.733
16 100.000 98.610 95.431 90.664 84.718 78.032
17 100.000 98.725 95.791 91.331 85.696 79.281
18 100.000 98.841 96.141 91.975 86.634 80.476
19 100.000 99.073 96.668 92.783 87.679 81.702
20 100.000 99.189 97.000 93.368 88.501 82.722
21 100.000 99.305 97.305 93.887 89.229 83.630
22 100.000 99.305 97.415 94.193 89.767 84.395
23 100.000 99.305 97.508 94.456 90.239 85.081
24 100.000 99.305 97.600 94.716 90.703 85.751
25 100.000 99.305 97.683 94.953 91.129 86.368
26 100.000 99.421 97.941 95.352 91.655 87.003
27 100.000 99.537 98.200 95.753 92.187 87.649
28 100.000 99.537 98.274 95.966 92.577 88.225
29 100.000 99.652 98.532 96.362 93.093 88.839
30 100.000 99.768 98.782 96.729 93.553 89.368
31 100.000 99.884 99.031 97.102 94.036 89.943
32 100.000 100.000 99.280 97.468 94.491 90.461
33 100.000 100.000 99.345 97.651 94.820 90.938
34 100.000 100.000 99.391 97.794 95.095 91.363
35 100.000 100.000 99.437 97.928 95.343 91.734
36 100.000 100.000 99.483 98.066 95.601 92.120
37 100.000 100.000 99.520 98.176 95.808 92.433
38 100.000 100.000 99.557 98.285 96.009 92.735
39 100.000 100.000 99.594 98.396 96.221 93.061
40 100.000 100.000 99.631 98.514 96.453 93.423
41 100.000 100.000 99.668 98.628 96.674 93.762
42 100.000 100.000 99.705 98.742 96.893 94.101
43 100.000 100.000 99.742 98.852 97.102 94.421
44 100.000 100.000 99.778 98.962 97.307 94.733
45 100.000 100.000 99.806 99.049 97.480 95.006
46 100.000 100.000 99.834 99.137 97.653 95.280
47 100.000 100.000 99.852 99.201 97.792 95.515
48 100.000 100.000 99.862 99.230 97.850 95.608
49 100.000 100.000 99.871 99.265 97.927 95.744
50 100.000 100.000 99.880 99.303 98.019 95.911
51 100.000 100.000 99.889 99.340 98.102 96.055
52 100.000 100.000 99.898 99.380 98.198 96.227
53 100.000 100.000 99.908 99.408 98.253 96.312
54 100.000 100.000 99.917 99.437 98.307 96.397
55 100.000 100.000 99.926 99.472 98.388 96.541
56 100.000 100.000 99.935 99.499 98.439 96.618
57 100.000 100.000 99.945 99.526 98.489 96.696
58 100.000 100.000 99.954 99.553 98.540 96.774
59 100.000 100.000 99.963 99.581 98.595 96.860
60 100.000 100.000 99.972 99.608 98.646 96.938
61 100.000 100.000 99.982 99.635 98.697 97.016
62 100.000 100.000 99.991 99.663 98.748 97.095
63 100.000 100.000 100.000 99.690 98.799 97.176
64 100.000 100.000 100.000 99.692 98.809 97.200
65 100.000 100.000 100.000 99.697 98.827 97.240
66 100.000 100.000 100.000 99.708 98.868 97.327
67 100.000 100.000 100.000 99.713 98.883 97.363
68 100.000 100.000 100.000 99.723 98.922 97.449
69 100.000 100.000 100.000 99.727 98.937 97.484
70 100.000 100.000 100.000 99.733 98.959 97.531
71 100.000 100.000 100.000 99.735 98.967 97.549
72 100.000 100.000 100.000 99.740 98.983 97.586
73 100.000 100.000 100.000 99.749 99.018 97.663
74 100.000 100.000 100.000 99.750 99.022 97.673
75 100.000 100.000 100.000 99.756 99.045 97.726
76 100.000 100.000 100.000 99.761 99.065 97.771
77 100.000 100.000 100.000 99.767 99.084 97.815
78 100.000 100.000 100.000 99.771 99.100 97.849
79 100.000 100.000 100.000 99.780 99.133 97.920
80 100.000 100.000 100.000 99.792 99.173 98.004
81 100.000 100.000 100.000 99.797 99.191 98.044
82 100.000 100.000 100.000 99.801 99.207 98.082
83 100.000 100.000 100.000 99.806 99.227 98.127
84 100.000 100.000 100.000 99.811 99.246 98.171
85 100.000 100.000 100.000 99.818 99.270 98.223
86 100.000 100.000 100.000 99.823 99.289 98.268
87 100.000 100.000 100.000 99.825 99.297 98.287
88 100.000 100.000 100.000 99.829 99.312 98.321
89 100.000 100.000 100.000 99.834 99.331 98.361
90 100.000 100.000 100.000 99.838 99.343 98.389
91 100.000 100.000 100.000 99.840 99.351 98.406
92 100.000 100.000 100.000 99.840 99.351 98.409
93 100.000 100.000 100.000 99.842 99.359 98.427
94 100.000 100.000 100.000 99.847 99.378 98.467
95 100.000 100.000 100.000 99.851 99.393 98.503
96 100.000 100.000 100.000 99.852 99.397 98.512
97 100.000 100.000 100.000 99.857 99.417 98.556
98 100.000 100.000 100.000 99.858 99.422 98.570
99 100.000 100.000 100.000 99.863 99.437 98.605
100 100.000 100.000 100.000 99.863 99.437 98.606
101 100.000 100.000 100.000 99.864 99.442 98.619
102 100.000 100.000 100.000 99.869 99.462 98.662
103 100.000 100.000 100.000 99.872 99.473 98.687
104 100.000 100.000 100.000 99.879 99.499 98.744
105 100.000 100.000 100.000 99.888 99.528 98.807
106 100.000 100.000 100.000 99.889 99.533 98.820
107 100.000 100.000 100.000 99.890 99.538 98.835
108 100.000 100.000 100.000 99.893 99.550 98.859
109 100.000 100.000 100.000 99.895 99.558 98.880
110 100.000 100.000 100.000 99.896 99.562 98.888
111 100.000 100.000 100.000 99.897 99.566 98.900
112 100.000 100.000 100.000 99.897 99.567 98.902
113 100.000 100.000 100.000 99.900 99.579 98.931
114 100.000 100.000 100.000 99.900 99.580 98.936
115 100.000 100.000 100.000 99.900 99.580 98.936
116 100.000 100.000 100.000 99.900 99.581 98.939
117 100.000 100.000 100.000 99.900 99.581 98.941
118 100.000 100.000 100.000 99.900 99.581 98.942
119 100.000 100.000 100.000 99.900 99.582 98.944
120 100.000 100.000 100.000 99.900 99.582 98.945
121 100.000 100.000 100.000 99.902 99.590 98.963
122 100.000 100.000 100.000 99.905 99.602 98.991
123 100.000 100.000 100.000 99.905 99.602 98.991
124 100.000 100.000 100.000 99.907 99.610 99.009
125 100.000 100.000 100.000 99.907 99.611 99.013
126 100.000 100.000 100.000 99.909 99.619 99.033
127 100.000 100.000 100.000 99.917 99.645 99.087
128 100.000 100.000 100.000 99.920 99.657 99.118
129 100.000 100.000 100.000 99.925 99.676 99.158
130 100.000 100.000 100.000 99.928 99.687 99.183
131 100.000 100.000 100.000 99.930 99.695 99.200
132 100.000 100.000 100.000 99.930 99.695 99.201
133 100.000 100.000 100.000 99.931 99.700 99.213
134 100.000 100.000 100.000 99.931 99.700 99.216
135 100.000 100.000 100.000 99.931 99.701 99.218
136 100.000 100.000 100.000 99.932 99.705 99.227
137 100.000 100.000 100.000 99.933 99.709 99.239
138 100.000 100.000 100.000 99.935 99.717 99.256
139 100.000 100.000 100.000 99.936 99.721 99.265
140 100.000 100.000 100.000 99.936 99.721 99.266
141 100.000 100.000 100.000 99.941 99.736 99.299
142 100.000 100.000 100.000 99.942 99.740 99.309
143 100.000 100.000 100.000 99.944 99.748 99.329
144 100.000 100.000 100.000 99.947 99.761 99.358
145 100.000 100.000 100.000 99.947 99.761 99.361
146 100.000 100.000 100.000 99.949 99.769 99.378
147 100.000 100.000 100.000 99.949 99.769 99.379
148 100.000 100.000 100.000 99.950 99.773 99.390
149 100.000 100.000 100.000 99.952 99.782 99.411
150 100.000 100.000 100.000 99.953 99.786 99.421
151 100.000 100.000 100.000 99.955 99.794 99.438
152 100.000 100.000 100.000 99.956 99.798 99.447
153 100.000 100.000 100.000 99.958 99.806 99.469
154 100.000 100.000 100.000 99.960 99.814 99.488
155 100.000 100.000 100.000 99.960 99.814 99.488
156 100.000 100.000 100.000 99.960 99.815 99.490
157 100.000 100.000 100.000 99.960 99.816 99.493
158 100.000 100.000 100.000 99.960 99.816 99.493
159 100.000 100.000 100.000 99.960 99.816 99.496
160 100.000 100.000 100.000 99.960 99.816 99.497
161 100.000 100.000 100.000 99.961 99.821 99.509
162 100.000 100.000 100.000 99.961 99.822 99.513
163 100.000 100.000 100.000 99.964 99.830 99.532
164 100.000 100.000 100.000 99.966 99.839 99.552
165 100.000 100.000 100.000 99.967 99.843 99.562
166 100.000 100.000 100.000 99.968 99.847 99.573
167 100.000 100.000 100.000 99.969 99.851 99.582
168 100.000 100.000 100.000 99.972 99.862 99.607
169 100.000 100.000 100.000 99.972 99.863 99.608
170 100.000 100.000 100.000 99.972 99.863 99.611
171 100.000 100.000 100.000 99.973 99.867 99.621
172 100.000 100.000 100.000 99.976 99.879 99.647
173 100.000 100.000 100.000 99.978 99.887 99.664
174 100.000 100.000 100.000 99.979 99.891 99.675
175 100.000 100.000 100.000 99.979 99.891 99.676
176 100.000 100.000 100.000 99.979 99.891 99.677
177 100.000 100.000 100.000 99.979 99.891 99.678
178 100.000 100.000 100.000 99.980 99.896 99.689
179 100.000 100.000 100.000 99.982 99.904 99.707
180 100.000 100.000 100.000 99.982 99.904 99.708
181 100.000 100.000 100.000 99.983 99.908 99.718
182 100.000 100.000 100.000 99.983 99.908 99.720
183 100.000 100.000 100.000 99.983 99.909 99.722
184 100.000 100.000 100.000 99.984 99.913 99.731
185 100.000 100.000 100.000 99.985 99.917 99.740
186 100.000 100.000 100.000 99.985 99.917 99.740
187 100.000 100.000 100.000 99.985 99.917 99.741
188 100.000 100.000 100.000 99.985 99.917 99.742
189 100.000 100.000 100.000 99.985 99.917 99.744
190 100.000 100.000 100.000 99.985 99.918 99.748
191 100.000 100.000 100.000 99.986 99.922 99.757
192 100.000 100.000 100.000 99.986 99.922 99.758
193 100.000 100.000 100.000 99.989 99.930 99.775
194 100.000 100.000 100.000 99.989 99.931 99.779
195 100.000 100.000 100.000 99.989 99.931 99.779
196 100.000 100.000 100.000 99.989 99.931 99.780
197 100.000 100.000 100.000 99.989 99.932 99.784
198 100.000 100.000 100.000 99.989 99.933 99.786
199 100.000 100.000 100.000 99.992 99.944 99.812
200 100.000 100.000 100.000 99.992 99.945 99.817
201 100.000 100.000 100.000 99.992 99.946 99.818
202 100.000 100.000 100.000 99.992 99.946 99.820
203 100.000 100.000 100.000 99.992 99.946 99.821
204 100.000 100.000 100.000 99.992 99.946 99.821
205 100.000 100.000 100.000 99.993 99.950 99.831
206 100.000 100.000 100.000 99.993 99.951 99.835
207 100.000 100.000 100.000 99.993 99.951 99.835
208 100.000 100.000 100.000 99.993 99.952 99.837
209 100.000 100.000 100.000 99.993 99.952 99.837
210 100.000 100.000 100.000 99.993 99.952 99.838
211 100.000 100.000 100.000 99.993 99.952 99.839
212 100.000 100.000 100.000 99.993 99.952 99.840
213 100.000 100.000 100.000 99.993 99.953 99.844
214 100.000 100.000 100.000 99.993 99.953 99.845
215 100.000 100.000 100.000 99.993 99.953 99.845
216 100.000 100.000 100.000 99.993 99.954 99.846
217 100.000 100.000 100.000 99.995 99.961 99.864
218 100.000 100.000 100.000 99.995 99.961 99.864
219 100.000 100.000 100.000 99.995 99.962 99.865
220 100.000 100.000 100.000 99.995 99.962 99.867
221 100.000 100.000 100.000 99.995 99.963 99.871
222 100.000 100.000 100.000 99.995 99.963 99.871
223 100.000 100.000 100.000 99.995 99.963 99.872
224 100.000 100.000 100.000 99.995 99.963 99.872
225 100.000 100.000 100.000 99.995 99.963 99.872
226 100.000 100.000 100.000 99.995 99.964 99.874
227 100.000 100.000 100.000 99.995 99.964 99.876
228 100.000 100.000 100.000 99.995 99.965 99.879
229 100.000 100.000 100.000 99.995 99.965 99.880
230 100.000 100.000 100.000 99.995 99.965 99.881
231 100.000 100.000 100.000 99.995 99.965 99.881
232 100.000 100.000 100.000 99.995 99.965 99.882
233 100.000 100.000 100.000 99.995 99.966 99.884
234 100.000 100.000 100.000 99.995 99.966 99.886
235 100.000 100.000 100.000 99.995 99.966 99.886
236 100.000 100.000 100.000 99.995 99.967 99.888
237 100.000 100.000 100.000 99.995 99.967 99.890
238 100.000 100.000 100.000 99.995 99.967 99.891
239 100.000 100.000 100.000 99.995 99.968 99.892
240 100.000 100.000 100.000 99.995 99.968 99.893
241 100.000 100.000 100.000 99.995 99.969 99.896
242 100.000 100.000 100.000 99.995 99.969 99.898
243 100.000 100.000 100.000 99.995 99.969 99.898
244 100.000 100.000 100.000 99.996 99.973 99.909
245 100.000 100.000 100.000 99.996 99.973 99.909
246 100.000 100.000 100.000 99.997 99.977 99.918
247 100.000 100.000 100.000 99.997 99.977 99.919
248 100.000 100.000 100.000 99.997 99.978 99.920
249 100.000 100.000 100.000 99.997 99.978 99.922
250 100.000 100.000 100.000 99.997 99.978 99.922
251 100.000 100.000 100.000 99.997 99.978 99.922
252 100.000 100.000 100.000 99.997 99.978 99.923
253 100.000 100.000 100.000 99.997 99.978 99.923
254 100.000 100.000 100.000 99.997 99.978 99.923
255 100.000 100.000 100.000 99.997 99.978 99.924
256 100.000 100.000 100.000 99.997 99.979 99.924
257 100.000 100.000 100.000 99.997 99.979 99.925
258 100.000 100.000 100.000 99.997 99.979 99.926
259 100.000 100.000 100.000 99.997 99.979 99.927
260 100.000 100.000 100.000 99.997 99.979 99.927
261 100.000 100.000 100.000 99.997 99.979 99.927
262 100.000 100.000 100.000 99.998 99.983 99.937
263 100.000 100.000 100.000 99.998 99.983 99.937
264 100.000 100.000 100.000 99.998 99.983 99.937
265 100.000 100.000 100.000 99.998 99.984 99.939
266 100.000 100.000 100.000 99.998 99.984 99.939
267 100.000 100.000 100.000 99.999 99.988 99.948
268 100.000 100.000 100.000 99.999 99.988 99.950
269 100.000 100.000 100.000 99.999 99.988 99.950
270 100.000 100.000 100.000 99.999 99.988 99.952
271 100.000 100.000 100.000 99.999 99.988 99.953
272 100.000 100.000 100.000 99.999 99.989 99.953
273 100.000 100.000 100.000 99.999 99.989 99.954
274 100.000 100.000 100.000 99.999 99.989 99.955
275 100.000 100.000 100.000 99.999 99.989 99.955
276 100.000 100.000 100.000 99.999 99.989 99.956
277 100.000 100.000 100.000 99.999 99.989 99.956
278 100.000 100.000 100.000 99.999 99.989 99.957
279 100.000 100.000 100.000 99.999 99.990 99.958
280 100.000 100.000 100.000 99.999 99.990 99.959
281 100.000 100.000 100.000 99.999 99.990 99.960
282 100.000 100.000 100.000 99.999 99.990 99.961
283 100.000 100.000 100.000 99.999 99.990 99.961
284 100.000 100.000 100.000 99.999 99.990 99.961
285 100.000 100.000 100.000 99.999 99.991 99.962
286 100.000 100.000 100.000 99.999 99.991 99.963
287 100.000 100.000 100.000 99.999 99.991 99.964
288 100.000 100.000 100.000 99.999 99.991 99.964
289 100.000 100.000 100.000 99.999 99.991 99.964
290 100.000 100.000 100.000 99.999 99.991 99.964
291 100.000 100.000 100.000 99.999 99.991 99.964
292 100.000 100.000 100.000 100.000 99.995 99.974
293 100.000 100.000 100.000 100.000 99.995 99.974
294 100.000 100.000 100.000 100.000 99.995 99.976
295 100.000 100.000 100.000 100.000 99.996 99.977
296 100.000 100.000 100.000 100.000 99.996 99.978
297 100.000 100.000 100.000 100.000 99.996 99.979
298 100.000 100.000 100.000 100.000 99.996 99.981
299 100.000 100.000 100.000 100.000 99.997 99.981
300 100.000 100.000 100.000 100.000 99.997 99.982
301 100.000 100.000 100.000 100.000 99.997 99.982
302 100.000 100.000 100.000 100.000 99.997 99.982
303 100.000 100.000 100.000 100.000 99.997 99.983
304 100.000 100.000 100.000 100.000 99.997 99.983
305 100.000 100.000 100.000 100.000 99.997 99.984
306 100.000 100.000 100.000 100.000 99.997 99.984
307 100.000 100.000 100.000 100.000 99.997 99.985
308 100.000 100.000 100.000 100.000 99.997 99.985
309 100.000 100.000 100.000 100.000 99.997 99.985
310 100.000 100.000 100.000 100.000 99.998 99.986
311 100.000 100.000 100.000 100.000 99.998 99.986
312 100.000 100.000 100.000 100.000 99.998 99.987
313 100.000 100.000 100.000 100.000 99.998 99.987
314 100.000 100.000 100.000 100.000 99.998 99.988
315 100.000 100.000 100.000 100.000 99.998 99.989
316 100.000 100.000 100.000 100.000 99.998 99.990
317 100.000 100.000 100.000 100.000 99.998 99.990
318 100.000 100.000 100.000 100.000 99.998 99.990
319 100.000 100.000 100.000 100.000 99.998 99.991
320 100.000 100.000 100.000 100.000 99.998 99.991
321 100.000 100.000 100.000 100.000 99.998 99.991
322 100.000 100.000 100.000 100.000 99.999 99.991
323 100.000 100.000 100.000 100.000 99.999 99.991
324 100.000 100.000 100.000 100.000 99.999 99.991
325 100.000 100.000 100.000 100.000 99.999 99.992
326 100.000 100.000 100.000 100.000 99.999 99.992
327 100.000 100.000 100.000 100.000 99.999 99.992
328 100.000 100.000 100.000 100.000 99.999 99.992
329 100.000 100.000 100.000 100.000 99.999 99.993
330 100.000 100.000 100.000 100.000 99.999 99.993
331 100.000 100.000 100.000 100.000 99.999 99.993
332 100.000 100.000 100.000 100.000 99.999 99.993
333 100.000 100.000 100.000 100.000 99.999 99.993
334 100.000 100.000 100.000 100.000 99.999 99.994
335 100.000 100.000 100.000 100.000 99.999 99.994
336 100.000 100.000 100.000 100.000 99.999 99.994
337 100.000 100.000 100.000 100.000 99.999 99.994
338 100.000 100.000 100.000 100.000 99.999 99.994
339 100.000 100.000 100.000 100.000 99.999 99.995
340 100.000 100.000 100.000 100.000 99.999 99.996
341 100.000 100.000 100.000 100.000 99.999 99.996
342 100.000 100.000 100.000 100.000 99.999 99.996
343 100.000 100.000 100.000 100.000 99.999 99.996
344 100.000 100.000 100.000 100.000 99.999 99.996
345 100.000 100.000 100.000 100.000 100.000 99.997
346 100.000 100.000 100.000 100.000 100.000 99.998
347 100.000 100.000 100.000 100.000 100.000 99.998
348 100.000 100.000 100.000 100.000 100.000 99.999
349 100.000 100.000 100.000 100.000 100.000 99.999
350 100.000 100.000 100.000 100.000 100.000 99.999
351 100.000 100.000 100.000 100.000 100.000 99.999
352 100.000 100.000 100.000 100.000 100.000 99.999
353 100.000 100.000 100.000 100.000 100.000 99.999
354 100.000 100.000 100.000 100.000 100.000 99.999
355 100.000 100.000 100.000 100.000 100.000 99.999
356 100.000 100.000 100.000 100.000 100.000 100.000
357 100.000 100.000 100.000 100.000 100.000 100.000
358 100.000 100.000 100.000 100.000 100.000 100.000
359 100.000 100.000 100.000 100.000 100.000 100.000
360 100.000 100.000 100.000 100.000 100.000 100.000
361 100.000 100.000 100.000 100.000 100.000 100.000
362 100.000 100.000 100.000 100.000 100.000 100.000
363 100.000 100.000 100.000 100.000 100.000 100.000
364 100.000 100.000 100.000 100.000 100.000 100.000
365 100.000 100.000 100.000 100.000 100.000 100.000
366 100.000 100.000 100.000 100.000 100.000 100.000

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 6, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 74.855 62.088 50.933 41.540 33.785
4 93.023 82.271 70.747 59.812 50.024 41.548
5 97.674 89.108 78.418 67.553 57.408 48.346
6 97.674 90.267 80.809 70.883 61.312 52.501
7 97.674 91.194 82.756 73.646 64.612 56.076
8 97.674 91.889 84.252 75.820 67.271 59.027
9 100.000 95.597 88.646 80.443 71.843 63.383
10 100.000 96.292 90.123 82.566 74.414 66.213
11 100.000 96.987 91.572 84.624 76.891 68.930
12 100.000 97.335 92.403 85.931 78.593 70.917
13 100.000 97.798 93.427 87.456 80.499 73.075
14 100.000 98.146 94.268 88.782 82.224 75.085
15 100.000 98.262 94.618 89.430 83.175 76.305
16 100.000 98.378 94.988 90.118 84.182 77.588
17 100.000 98.494 95.348 90.779 85.139 78.797
18 100.000 98.725 95.892 91.631 86.251 80.105
19 100.000 98.725 96.049 92.045 86.945 81.049
20 100.000 98.725 96.151 92.332 87.459 81.793
21 100.000 98.841 96.483 92.921 88.294 82.835
22 100.000 98.841 96.603 93.249 88.864 83.638
23 100.000 98.957 96.917 93.795 89.637 84.608
24 100.000 99.189 97.415 94.537 90.584 85.717
25 100.000 99.305 97.692 94.988 91.204 86.488
26 100.000 99.305 97.748 95.157 91.525 86.978
27 100.000 99.305 97.812 95.346 91.875 87.499
28 100.000 99.305 97.877 95.534 92.218 88.005
29 100.000 99.305 97.932 95.697 92.523 88.462
30 100.000 99.305 97.969 95.818 92.764 88.841
31 100.000 99.305 98.025 95.981 93.064 89.286
32 100.000 99.421 98.283 96.380 93.590 89.920
33 100.000 99.421 98.320 96.495 93.816 90.276
34 100.000 99.537 98.569 96.870 94.304 90.861
35 100.000 99.537 98.606 96.985 94.526 91.206
36 100.000 99.537 98.643 97.098 94.745 91.548
37 100.000 99.537 98.680 97.210 94.957 91.872
38 100.000 99.537 98.698 97.280 95.112 92.135
39 100.000 99.537 98.735 97.390 95.320 92.452
40 100.000 99.652 98.985 97.759 95.787 92.997
41 100.000 99.768 99.234 98.127 96.254 93.543
42 100.000 99.768 99.252 98.193 96.395 93.779
43 100.000 99.884 99.502 98.559 96.851 94.300
44 100.000 100.000 99.751 98.924 97.308 94.824
45 100.000 100.000 99.769 98.988 97.441 95.045
46 100.000 100.000 99.788 99.051 97.575 95.267
47 100.000 100.000 99.825 99.156 97.767 95.553
48 100.000 100.000 99.834 99.195 97.857 95.715
49 100.000 100.000 99.843 99.233 97.949 95.880
50 100.000 100.000 99.852 99.270 98.033 96.028
51 100.000 100.000 99.862 99.306 98.116 96.176
52 100.000 100.000 99.871 99.342 98.196 96.314
53 100.000 100.000 99.880 99.376 98.273 96.449
54 100.000 100.000 99.889 99.410 98.350 96.583
55 100.000 100.000 99.889 99.423 98.395 96.681
56 100.000 100.000 99.898 99.455 98.464 96.799
57 100.000 100.000 99.898 99.466 98.502 96.881
58 100.000 100.000 99.898 99.476 98.539 96.964
59 100.000 100.000 99.908 99.507 98.605 97.075
60 100.000 100.000 99.908 99.517 98.640 97.151
61 100.000 100.000 99.908 99.526 98.674 97.229
62 100.000 100.000 99.908 99.534 98.705 97.297
63 100.000 100.000 99.908 99.542 98.731 97.355
64 100.000 100.000 99.908 99.549 98.758 97.414
65 100.000 100.000 99.908 99.556 98.785 97.474
66 100.000 100.000 99.908 99.564 98.812 97.535
67 100.000 100.000 99.908 99.570 98.835 97.585
68 100.000 100.000 99.908 99.576 98.858 97.638
69 100.000 100.000 99.908 99.582 98.880 97.686
70 100.000 100.000 99.917 99.609 98.932 97.766
71 100.000 100.000 99.926 99.637 98.982 97.843
72 100.000 100.000 99.935 99.664 99.033 97.920
73 100.000 100.000 99.945 99.691 99.084 97.997
74 100.000 100.000 99.954 99.718 99.134 98.074
75 100.000 100.000 99.963 99.745 99.185 98.151
76 100.000 100.000 99.963 99.751 99.208 98.202
77 100.000 100.000 99.972 99.778 99.259 98.280
78 100.000 100.000 99.982 99.805 99.310 98.359
79 100.000 100.000 99.991 99.832 99.360 98.437
80 100.000 100.000 100.000 99.859 99.411 98.515
81 100.000 100.000 100.000 99.865 99.431 98.558
82 100.000 100.000 100.000 99.870 99.450 98.602
83 100.000 100.000 100.000 99.875 99.469 98.645
84 100.000 100.000 100.000 99.880 99.489 98.688
85 100.000 100.000 100.000 99.885 99.508 98.732
86 100.000 100.000 100.000 99.890 99.523 98.765
87 100.000 100.000 100.000 99.894 99.538 98.798
88 100.000 100.000 100.000 99.898 99.553 98.831
89 100.000 100.000 100.000 99.902 99.569 98.867
90 100.000 100.000 100.000 99.906 99.584 98.902
91 100.000 100.000 100.000 99.910 99.600 98.940
92 100.000 100.000 100.000 99.915 99.616 98.975
93 100.000 100.000 100.000 99.919 99.632 99.011
94 100.000 100.000 100.000 99.923 99.647 99.046
95 100.000 100.000 100.000 99.927 99.663 99.083
96 100.000 100.000 100.000 99.931 99.679 99.121
97 100.000 100.000 100.000 99.935 99.694 99.154
98 100.000 100.000 100.000 99.940 99.710 99.189
99 100.000 100.000 100.000 99.943 99.722 99.219
100 100.000 100.000 100.000 99.946 99.734 99.247
101 100.000 100.000 100.000 99.949 99.746 99.274
102 100.000 100.000 100.000 99.952 99.759 99.305
103 100.000 100.000 100.000 99.954 99.768 99.326
104 100.000 100.000 100.000 99.956 99.776 99.348
105 100.000 100.000 100.000 99.958 99.785 99.370
106 100.000 100.000 100.000 99.960 99.793 99.390
107 100.000 100.000 100.000 99.963 99.801 99.409
108 100.000 100.000 100.000 99.965 99.809 99.427
109 100.000 100.000 100.000 99.967 99.818 99.448
110 100.000 100.000 100.000 99.969 99.826 99.469
111 100.000 100.000 100.000 99.971 99.834 99.488
112 100.000 100.000 100.000 99.973 99.843 99.507
113 100.000 100.000 100.000 99.975 99.850 99.525
114 100.000 100.000 100.000 99.977 99.858 99.543
115 100.000 100.000 100.000 99.979 99.866 99.561
116 100.000 100.000 100.000 99.981 99.874 99.579
117 100.000 100.000 100.000 99.983 99.882 99.599
118 100.000 100.000 100.000 99.985 99.891 99.620
119 100.000 100.000 100.000 99.988 99.898 99.638
120 100.000 100.000 100.000 99.990 99.907 99.658
121 100.000 100.000 100.000 99.991 99.911 99.668
122 100.000 100.000 100.000 99.992 99.915 99.678
123 100.000 100.000 100.000 99.993 99.919 99.687
124 100.000 100.000 100.000 99.994 99.923 99.697
125 100.000 100.000 100.000 99.995 99.928 99.710
126 100.000 100.000 100.000 99.996 99.933 99.722
127 100.000 100.000 100.000 99.997 99.937 99.732
128 100.000 100.000 100.000 99.998 99.941 99.743
129 100.000 100.000 100.000 99.999 99.946 99.754
130 100.000 100.000 100.000 100.000 99.950 99.764
131 100.000 100.000 100.000 100.000 99.950 99.767
132 100.000 100.000 100.000 100.000 99.951 99.772
133 100.000 100.000 100.000 100.000 99.952 99.776
134 100.000 100.000 100.000 100.000 99.953 99.779
135 100.000 100.000 100.000 100.000 99.953 99.781
136 100.000 100.000 100.000 100.000 99.954 99.785
137 100.000 100.000 100.000 100.000 99.954 99.786
138 100.000 100.000 100.000 100.000 99.955 99.791
139 100.000 100.000 100.000 100.000 99.956 99.791
140 100.000 100.000 100.000 100.000 99.957 99.796
141 100.000 100.000 100.000 100.000 99.957 99.796
142 100.000 100.000 100.000 100.000 99.957 99.799
143 100.000 100.000 100.000 100.000 99.958 99.802
144 100.000 100.000 100.000 100.000 99.959 99.805
145 100.000 100.000 100.000 100.000 99.959 99.808
146 100.000 100.000 100.000 100.000 99.960 99.813
147 100.000 100.000 100.000 100.000 99.961 99.817
148 100.000 100.000 100.000 100.000 99.962 99.819
149 100.000 100.000 100.000 100.000 99.962 99.819
150 100.000 100.000 100.000 100.000 99.962 99.821
151 100.000 100.000 100.000 100.000 99.963 99.824
152 100.000 100.000 100.000 100.000 99.963 99.826
153 100.000 100.000 100.000 100.000 99.963 99.826
154 100.000 100.000 100.000 100.000 99.964 99.828
155 100.000 100.000 100.000 100.000 99.965 99.831
156 100.000 100.000 100.000 100.000 99.965 99.835
157 100.000 100.000 100.000 100.000 99.966 99.839
158 100.000 100.000 100.000 100.000 99.967 99.841
159 100.000 100.000 100.000 100.000 99.967 99.841
160 100.000 100.000 100.000 100.000 99.967 99.843
161 100.000 100.000 100.000 100.000 99.968 99.847
162 100.000 100.000 100.000 100.000 99.969 99.851
163 100.000 100.000 100.000 100.000 99.969 99.854
164 100.000 100.000 100.000 100.000 99.969 99.854
165 100.000 100.000 100.000 100.000 99.970 99.855
166 100.000 100.000 100.000 100.000 99.970 99.856
167 100.000 100.000 100.000 100.000 99.971 99.859
168 100.000 100.000 100.000 100.000 99.971 99.861
169 100.000 100.000 100.000 100.000 99.972 99.865
170 100.000 100.000 100.000 100.000 99.972 99.868
171 100.000 100.000 100.000 100.000 99.973 99.870
172 100.000 100.000 100.000 100.000 99.973 99.872
173 100.000 100.000 100.000 100.000 99.974 99.872
174 100.000 100.000 100.000 100.000 99.974 99.875
175 100.000 100.000 100.000 100.000 99.974 99.877
176 100.000 100.000 100.000 100.000 99.975 99.878
177 100.000 100.000 100.000 100.000 99.975 99.880
178 100.000 100.000 100.000 100.000 99.976 99.882
179 100.000 100.000 100.000 100.000 99.976 99.883
180 100.000 100.000 100.000 100.000 99.976 99.885
181 100.000 100.000 100.000 100.000 99.977 99.888
182 100.000 100.000 100.000 100.000 99.977 99.889
183 100.000 100.000 100.000 100.000 99.978 99.891
184 100.000 100.000 100.000 100.000 99.978 99.892
185 100.000 100.000 100.000 100.000 99.978 99.893
186 100.000 100.000 100.000 100.000 99.979 99.898
187 100.000 100.000 100.000 100.000 99.980 99.901
188 100.000 100.000 100.000 100.000 99.980 99.904
189 100.000 100.000 100.000 100.000 99.980 99.904
190 100.000 100.000 100.000 100.000 99.980 99.904
191 100.000 100.000 100.000 100.000 99.980 99.904
192 100.000 100.000 100.000 100.000 99.980 99.904
193 100.000 100.000 100.000 100.000 99.981 99.906
194 100.000 100.000 100.000 100.000 99.981 99.907
195 100.000 100.000 100.000 100.000 99.982 99.911
196 100.000 100.000 100.000 100.000 99.982 99.912
197 100.000 100.000 100.000 100.000 99.982 99.912
198 100.000 100.000 100.000 100.000 99.983 99.914
199 100.000 100.000 100.000 100.000 99.983 99.915
200 100.000 100.000 100.000 100.000 99.983 99.917
201 100.000 100.000 100.000 100.000 99.983 99.918
202 100.000 100.000 100.000 100.000 99.983 99.918
203 100.000 100.000 100.000 100.000 99.983 99.919
204 100.000 100.000 100.000 100.000 99.983 99.919
205 100.000 100.000 100.000 100.000 99.984 99.920
206 100.000 100.000 100.000 100.000 99.984 99.922
207 100.000 100.000 100.000 100.000 99.984 99.922
208 100.000 100.000 100.000 100.000 99.984 99.923
209 100.000 100.000 100.000 100.000 99.985 99.924
210 100.000 100.000 100.000 100.000 99.985 99.926
211 100.000 100.000 100.000 100.000 99.985 99.926
212 100.000 100.000 100.000 100.000 99.985 99.927
213 100.000 100.000 100.000 100.000 99.985 99.927
214 100.000 100.000 100.000 100.000 99.986 99.928
215 100.000 100.000 100.000 100.000 99.986 99.928
216 100.000 100.000 100.000 100.000 99.986 99.929
217 100.000 100.000 100.000 100.000 99.986 99.931
218 100.000 100.000 100.000 100.000 99.986 99.932
219 100.000 100.000 100.000 100.000 99.987 99.934
220 100.000 100.000 100.000 100.000 99.987 99.934
221 100.000 100.000 100.000 100.000 99.987 99.934
222 100.000 100.000 100.000 100.000 99.987 99.935
223 100.000 100.000 100.000 100.000 99.987 99.936
224 100.000 100.000 100.000 100.000 99.987 99.936
225 100.000 100.000 100.000 100.000 99.987 99.938
226 100.000 100.000 100.000 100.000 99.988 99.939
227 100.000 100.000 100.000 100.000 99.988 99.939
228 100.000 100.000 100.000 100.000 99.988 99.941
229 100.000 100.000 100.000 100.000 99.988 99.942
230 100.000 100.000 100.000 100.000 99.988 99.943
231 100.000 100.000 100.000 100.000 99.988 99.943
232 100.000 100.000 100.000 100.000 99.989 99.946
233 100.000 100.000 100.000 100.000 99.989 99.947
234 100.000 100.000 100.000 100.000 99.989 99.947
235 100.000 100.000 100.000 100.000 99.990 99.948
236 100.000 100.000 100.000 100.000 99.990 99.949
237 100.000 100.000 100.000 100.000 99.990 99.949
238 100.000 100.000 100.000 100.000 99.990 99.950
239 100.000 100.000 100.000 100.000 99.991 99.953
240 100.000 100.000 100.000 100.000 99.991 99.953
241 100.000 100.000 100.000 100.000 99.991 99.953
242 100.000 100.000 100.000 100.000 99.991 99.955
243 100.000 100.000 100.000 100.000 99.991 99.955
244 100.000 100.000 100.000 100.000 99.991 99.957
245 100.000 100.000 100.000 100.000 99.991 99.957
246 100.000 100.000 100.000 100.000 99.992 99.958
247 100.000 100.000 100.000 100.000 99.992 99.959
248 100.000 100.000 100.000 100.000 99.992 99.959
249 100.000 100.000 100.000 100.000 99.993 99.962
250 100.000 100.000 100.000 100.000 99.993 99.965
251 100.000 100.000 100.000 100.000 99.993 99.966
252 100.000 100.000 100.000 100.000 99.993 99.967
253 100.000 100.000 100.000 100.000 99.994 99.969
254 100.000 100.000 100.000 100.000 99.994 99.970
255 100.000 100.000 100.000 100.000 99.994 99.970
256 100.000 100.000 100.000 100.000 99.994 99.970
257 100.000 100.000 100.000 100.000 99.994 99.970
258 100.000 100.000 100.000 100.000 99.994 99.971
259 100.000 100.000 100.000 100.000 99.995 99.972
260 100.000 100.000 100.000 100.000 99.995 99.972
261 100.000 100.000 100.000 100.000 99.995 99.973
262 100.000 100.000 100.000 100.000 99.995 99.974
263 100.000 100.000 100.000 100.000 99.995 99.976
264 100.000 100.000 100.000 100.000 99.996 99.978
265 100.000 100.000 100.000 100.000 99.996 99.978
266 100.000 100.000 100.000 100.000 99.996 99.981
267 100.000 100.000 100.000 100.000 99.996 99.981
268 100.000 100.000 100.000 100.000 99.997 99.982
269 100.000 100.000 100.000 100.000 99.997 99.982
270 100.000 100.000 100.000 100.000 99.997 99.983
271 100.000 100.000 100.000 100.000 99.997 99.984
272 100.000 100.000 100.000 100.000 99.998 99.986
273 100.000 100.000 100.000 100.000 99.998 99.986
274 100.000 100.000 100.000 100.000 99.998 99.987
275 100.000 100.000 100.000 100.000 99.998 99.989
276 100.000 100.000 100.000 100.000 99.998 99.989
277 100.000 100.000 100.000 100.000 99.998 99.990
278 100.000 100.000 100.000 100.000 99.999 99.992
279 100.000 100.000 100.000 100.000 99.999 99.992
280 100.000 100.000 100.000 100.000 99.999 99.993
281 100.000 100.000 100.000 100.000 99.999 99.993
282 100.000 100.000 100.000 100.000 99.999 99.993
283 100.000 100.000 100.000 100.000 99.999 99.994
284 100.000 100.000 100.000 100.000 99.999 99.995
285 100.000 100.000 100.000 100.000 99.999 99.995
286 100.000 100.000 100.000 100.000 99.999 99.995
287 100.000 100.000 100.000 100.000 99.999 99.995
288 100.000 100.000 100.000 100.000 99.999 99.995
289 100.000 100.000 100.000 100.000 99.999 99.996
290 100.000 100.000 100.000 100.000 99.999 99.996
291 100.000 100.000 100.000 100.000 99.999 99.996
292 100.000 100.000 100.000 100.000 100.000 99.997
293 100.000 100.000 100.000 100.000 100.000 99.997
294 100.000 100.000 100.000 100.000 100.000 99.997
295 100.000 100.000 100.000 100.000 100.000 99.998
296 100.000 100.000 100.000 100.000 100.000 99.998
297 100.000 100.000 100.000 100.000 100.000 99.998
298 100.000 100.000 100.000 100.000 100.000 99.998
299 100.000 100.000 100.000 100.000 100.000 99.999
300 100.000 100.000 100.000 100.000 100.000 99.999
301 100.000 100.000 100.000 100.000 100.000 99.999
302 100.000 100.000 100.000 100.000 100.000 99.999
303 100.000 100.000 100.000 100.000 100.000 99.999
304 100.000 100.000 100.000 100.000 100.000 100.000
305 100.000 100.000 100.000 100.000 100.000 100.000
306 100.000 100.000 100.000 100.000 100.000 100.000
307 100.000 100.000 100.000 100.000 100.000 100.000
308 100.000 100.000 100.000 100.000 100.000 100.000
309 100.000 100.000 100.000 100.000 100.000 100.000
310 100.000 100.000 100.000 100.000 100.000 100.000

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 75.203 62.670 51.590 42.164 34.322
4 93.023 82.619 71.310 60.426 50.586 42.011
5 93.023 84.241 74.522 64.740 55.482 47.070
6 95.349 87.949 79.073 69.737 60.617 52.110
7 97.674 91.657 83.550 74.561 65.493 56.839
8 97.674 92.468 85.258 76.996 68.422 60.040
9 100.000 96.292 89.874 81.906 73.305 64.704
10 100.000 96.987 91.351 84.031 75.886 67.551
11 100.000 97.219 91.997 85.148 77.433 69.437
12 100.000 97.335 92.403 85.936 78.610 70.949
13 100.000 97.683 93.234 87.241 80.305 72.924
14 100.000 97.798 93.612 87.955 81.354 74.261
15 100.000 98.030 94.194 88.907 82.636 75.804
16 100.000 98.146 94.563 89.593 83.637 77.074
17 100.000 98.262 94.941 90.299 84.663 78.369
18 100.000 98.494 95.504 91.199 85.854 79.783
19 100.000 98.610 95.846 91.812 86.728 80.879
20 100.000 98.841 96.381 92.643 87.810 82.152
21 100.000 98.957 96.695 93.189 88.586 83.130
22 100.000 98.957 96.797 93.474 89.091 83.856
23 100.000 99.073 97.092 93.975 89.797 84.746
24 100.000 99.305 97.591 94.712 90.728 85.825
25 100.000 99.305 97.655 94.906 91.091 86.370
26 100.000 99.305 97.729 95.117 91.475 86.932
27 100.000 99.305 97.785 95.286 91.796 87.422
28 100.000 99.305 97.840 95.454 92.115 87.909
29 100.000 99.305 97.905 95.641 92.458 88.414
30 100.000 99.305 97.960 95.807 92.767 88.876
31 100.000 99.421 98.209 96.186 93.267 89.484
32 100.000 99.421 98.246 96.304 93.499 89.847
33 100.000 99.421 98.283 96.420 93.727 90.203
34 100.000 99.421 98.320 96.536 93.952 90.551
35 100.000 99.421 98.357 96.650 94.173 90.891
36 100.000 99.421 98.394 96.763 94.388 91.220
37 100.000 99.421 98.421 96.855 94.574 91.522
38 100.000 99.537 98.671 97.225 95.048 92.081
39 100.000 99.537 98.708 97.335 95.253 92.391
40 100.000 99.537 98.726 97.402 95.402 92.644
41 100.000 99.537 98.763 97.511 95.603 92.945
42 100.000 99.537 98.800 97.619 95.804 93.246
43 100.000 99.652 99.049 97.987 96.267 93.783
44 100.000 99.768 99.298 98.353 96.727 94.313
45 100.000 99.884 99.548 98.720 97.187 94.843
46 100.000 99.884 99.566 98.783 97.321 95.067
47 100.000 100.000 99.815 99.149 97.776 95.586
48 100.000 100.000 99.843 99.233 97.938 95.836
49 100.000 100.000 99.862 99.295 98.065 96.046
50 100.000 100.000 99.880 99.355 98.188 96.245
51 100.000 100.000 99.889 99.391 98.268 96.382
52 100.000 100.000 99.889 99.405 98.319 96.492
53 100.000 100.000 99.898 99.440 98.397 96.629
54 100.000 100.000 99.898 99.452 98.441 96.725
55 100.000 100.000 99.898 99.465 98.486 96.824
56 100.000 100.000 99.898 99.476 98.527 96.915
57 100.000 100.000 99.908 99.508 98.598 97.038
58 100.000 100.000 99.908 99.520 98.639 97.125
59 100.000 100.000 99.908 99.529 98.672 97.198
60 100.000 100.000 99.908 99.539 98.706 97.273
61 100.000 100.000 99.908 99.548 98.741 97.349
62 100.000 100.000 99.908 99.556 98.771 97.418
63 100.000 100.000 99.908 99.565 98.802 97.486
64 100.000 100.000 99.908 99.573 98.833 97.556
65 100.000 100.000 99.908 99.579 98.856 97.605
66 100.000 100.000 99.917 99.606 98.907 97.684
67 100.000 100.000 99.926 99.633 98.958 97.762
68 100.000 100.000 99.935 99.660 99.008 97.840
69 100.000 100.000 99.945 99.688 99.059 97.918
70 100.000 100.000 99.945 99.694 99.083 97.972
71 100.000 100.000 99.954 99.721 99.133 98.049
72 100.000 100.000 99.963 99.748 99.184 98.127
73 100.000 100.000 99.972 99.775 99.236 98.207
74 100.000 100.000 99.982 99.802 99.287 98.286
75 100.000 100.000 99.991 99.829 99.338 98.364
76 100.000 100.000 100.000 99.856 99.390 98.446
77 100.000 100.000 100.000 99.861 99.409 98.491
78 100.000 100.000 100.000 99.867 99.428 98.533
79 100.000 100.000 100.000 99.872 99.448 98.578
80 100.000 100.000 100.000 99.877 99.468 98.624
81 100.000 100.000 100.000 99.882 99.487 98.666
82 100.000 100.000 100.000 99.888 99.506 98.709
83 100.000 100.000 100.000 99.893 99.525 98.751
84 100.000 100.000 100.000 99.898 99.544 98.794
85 100.000 100.000 100.000 99.902 99.560 98.831
86 100.000 100.000 100.000 99.906 99.577 98.873
87 100.000 100.000 100.000 99.910 99.593 98.910
88 100.000 100.000 100.000 99.915 99.611 98.952
89 100.000 100.000 100.000 99.919 99.626 98.985
90 100.000 100.000 100.000 99.923 99.642 99.024
91 100.000 100.000 100.000 99.927 99.657 99.057
92 100.000 100.000 100.000 99.931 99.672 99.090
93 100.000 100.000 100.000 99.934 99.683 99.115
94 100.000 100.000 100.000 99.938 99.696 99.146
95 100.000 100.000 100.000 99.941 99.708 99.175
96 100.000 100.000 100.000 99.944 99.721 99.206
97 100.000 100.000 100.000 99.947 99.733 99.235
98 100.000 100.000 100.000 99.949 99.742 99.256
99 100.000 100.000 100.000 99.951 99.750 99.273
100 100.000 100.000 100.000 99.953 99.757 99.291
101 100.000 100.000 100.000 99.955 99.766 99.312
102 100.000 100.000 100.000 99.957 99.775 99.334
103 100.000 100.000 100.000 99.959 99.783 99.355
104 100.000 100.000 100.000 99.961 99.792 99.378
105 100.000 100.000 100.000 99.964 99.800 99.397
106 100.000 100.000 100.000 99.966 99.808 99.415
107 100.000 100.000 100.000 99.968 99.817 99.437
108 100.000 100.000 100.000 99.970 99.825 99.454
109 100.000 100.000 100.000 99.972 99.833 99.476
110 100.000 100.000 100.000 99.974 99.841 99.493
111 100.000 100.000 100.000 99.976 99.851 99.518
112 100.000 100.000 100.000 99.978 99.859 99.538
113 100.000 100.000 100.000 99.980 99.867 99.557
114 100.000 100.000 100.000 99.982 99.875 99.575
115 100.000 100.000 100.000 99.984 99.883 99.595
116 100.000 100.000 100.000 99.986 99.891 99.615
117 100.000 100.000 100.000 99.989 99.900 99.636
118 100.000 100.000 100.000 99.991 99.908 99.654
119 100.000 100.000 100.000 99.993 99.916 99.674
120 100.000 100.000 100.000 99.994 99.921 99.687
121 100.000 100.000 100.000 99.995 99.925 99.697
122 100.000 100.000 100.000 99.996 99.929 99.707
123 100.000 100.000 100.000 99.997 99.933 99.717
124 100.000 100.000 100.000 99.998 99.938 99.728
125 100.000 100.000 100.000 99.999 99.942 99.738
126 100.000 100.000 100.000 100.000 99.946 99.750
127 100.000 100.000 100.000 100.000 99.947 99.753
128 100.000 100.000 100.000 100.000 99.947 99.754
129 100.000 100.000 100.000 100.000 99.948 99.757
130 100.000 100.000 100.000 100.000 99.948 99.759
131 100.000 100.000 100.000 100.000 99.949 99.761
132 100.000 100.000 100.000 100.000 99.949 99.763
133 100.000 100.000 100.000 100.000 99.950 99.765
134 100.000 100.000 100.000 100.000 99.950 99.766
135 100.000 100.000 100.000 100.000 99.951 99.770
136 100.000 100.000 100.000 100.000 99.951 99.770
137 100.000 100.000 100.000 100.000 99.951 99.772
138 100.000 100.000 100.000 100.000 99.952 99.774
139 100.000 100.000 100.000 100.000 99.952 99.776
140 100.000 100.000 100.000 100.000 99.952 99.777
141 100.000 100.000 100.000 100.000 99.953 99.779
142 100.000 100.000 100.000 100.000 99.953 99.782
143 100.000 100.000 100.000 100.000 99.955 99.788
144 100.000 100.000 100.000 100.000 99.955 99.790
145 100.000 100.000 100.000 100.000 99.955 99.791
146 100.000 100.000 100.000 100.000 99.956 99.795
147 100.000 100.000 100.000 100.000 99.957 99.796
148 100.000 100.000 100.000 100.000 99.957 99.797
149 100.000 100.000 100.000 100.000 99.957 99.798
150 100.000 100.000 100.000 100.000 99.957 99.799
151 100.000 100.000 100.000 100.000 99.958 99.803
152 100.000 100.000 100.000 100.000 99.959 99.806
153 100.000 100.000 100.000 100.000 99.960 99.811
154 100.000 100.000 100.000 100.000 99.960 99.814
155 100.000 100.000 100.000 100.000 99.962 99.819
156 100.000 100.000 100.000 100.000 99.962 99.820
157 100.000 100.000 100.000 100.000 99.962 99.822
158 100.000 100.000 100.000 100.000 99.963 99.824
159 100.000 100.000 100.000 100.000 99.963 99.825
160 100.000 100.000 100.000 100.000 99.963 99.827
161 100.000 100.000 100.000 100.000 99.964 99.828
162 100.000 100.000 100.000 100.000 99.964 99.829
163 100.000 100.000 100.000 100.000 99.964 99.831
164 100.000 100.000 100.000 100.000 99.965 99.834
165 100.000 100.000 100.000 100.000 99.965 99.835
166 100.000 100.000 100.000 100.000 99.966 99.838
167 100.000 100.000 100.000 100.000 99.966 99.839
168 100.000 100.000 100.000 100.000 99.966 99.840
169 100.000 100.000 100.000 100.000 99.967 99.841
170 100.000 100.000 100.000 100.000 99.967 99.844
171 100.000 100.000 100.000 100.000 99.968 99.846
172 100.000 100.000 100.000 100.000 99.968 99.849
173 100.000 100.000 100.000 100.000 99.969 99.851
174 100.000 100.000 100.000 100.000 99.969 99.853
175 100.000 100.000 100.000 100.000 99.970 99.858
176 100.000 100.000 100.000 100.000 99.970 99.858
177 100.000 100.000 100.000 100.000 99.971 99.860
178 100.000 100.000 100.000 100.000 99.971 99.864
179 100.000 100.000 100.000 100.000 99.971 99.864
180 100.000 100.000 100.000 100.000 99.972 99.866
181 100.000 100.000 100.000 100.000 99.972 99.866
182 100.000 100.000 100.000 100.000 99.972 99.869
183 100.000 100.000 100.000 100.000 99.973 99.869
184 100.000 100.000 100.000 100.000 99.973 99.871
185 100.000 100.000 100.000 100.000 99.973 99.873
186 100.000 100.000 100.000 100.000 99.974 99.874
187 100.000 100.000 100.000 100.000 99.975 99.878
188 100.000 100.000 100.000 100.000 99.975 99.881
189 100.000 100.000 100.000 100.000 99.976 99.886
190 100.000 100.000 100.000 100.000 99.977 99.888
191 100.000 100.000 100.000 100.000 99.977 99.888
192 100.000 100.000 100.000 100.000 99.977 99.889
193 100.000 100.000 100.000 100.000 99.977 99.891
194 100.000 100.000 100.000 100.000 99.978 99.892
195 100.000 100.000 100.000 100.000 99.978 99.893
196 100.000 100.000 100.000 100.000 99.979 99.896
197 100.000 100.000 100.000 100.000 99.979 99.898
198 100.000 100.000 100.000 100.000 99.979 99.899
199 100.000 100.000 100.000 100.000 99.979 99.900
200 100.000 100.000 100.000 100.000 99.980 99.904
201 100.000 100.000 100.000 100.000 99.981 99.905
202 100.000 100.000 100.000 100.000 99.981 99.906
203 100.000 100.000 100.000 100.000 99.981 99.908
204 100.000 100.000 100.000 100.000 99.982 99.910
205 100.000 100.000 100.000 100.000 99.982 99.912
206 100.000 100.000 100.000 100.000 99.983 99.914
207 100.000 100.000 100.000 100.000 99.983 99.915
208 100.000 100.000 100.000 100.000 99.983 99.916
209 100.000 100.000 100.000 100.000 99.983 99.917
210 100.000 100.000 100.000 100.000 99.983 99.919
211 100.000 100.000 100.000 100.000 99.984 99.921
212 100.000 100.000 100.000 100.000 99.984 99.921
213 100.000 100.000 100.000 100.000 99.984 99.923
214 100.000 100.000 100.000 100.000 99.985 99.924
215 100.000 100.000 100.000 100.000 99.985 99.925
216 100.000 100.000 100.000 100.000 99.985 99.926
217 100.000 100.000 100.000 100.000 99.985 99.927
218 100.000 100.000 100.000 100.000 99.985 99.927
219 100.000 100.000 100.000 100.000 99.986 99.929
220 100.000 100.000 100.000 100.000 99.986 99.930
221 100.000 100.000 100.000 100.000 99.986 99.931
222 100.000 100.000 100.000 100.000 99.986 99.932
223 100.000 100.000 100.000 100.000 99.987 99.934
224 100.000 100.000 100.000 100.000 99.987 99.935
225 100.000 100.000 100.000 100.000 99.987 99.935
226 100.000 100.000 100.000 100.000 99.987 99.936
227 100.000 100.000 100.000 100.000 99.987 99.936
228 100.000 100.000 100.000 100.000 99.987 99.937
229 100.000 100.000 100.000 100.000 99.988 99.938
230 100.000 100.000 100.000 100.000 99.988 99.939
231 100.000 100.000 100.000 100.000 99.988 99.940
232 100.000 100.000 100.000 100.000 99.988 99.941
233 100.000 100.000 100.000 100.000 99.988 99.942
234 100.000 100.000 100.000 100.000 99.989 99.945
235 100.000 100.000 100.000 100.000 99.989 99.946
236 100.000 100.000 100.000 100.000 99.990 99.947
237 100.000 100.000 100.000 100.000 99.990 99.948
238 100.000 100.000 100.000 100.000 99.990 99.948
239 100.000 100.000 100.000 100.000 99.990 99.948
240 100.000 100.000 100.000 100.000 99.990 99.949
241 100.000 100.000 100.000 100.000 99.990 99.950
242 100.000 100.000 100.000 100.000 99.990 99.950
243 100.000 100.000 100.000 100.000 99.990 99.950
244 100.000 100.000 100.000 100.000 99.990 99.951
245 100.000 100.000 100.000 100.000 99.990 99.951
246 100.000 100.000 100.000 100.000 99.991 99.953
247 100.000 100.000 100.000 100.000 99.991 99.955
248 100.000 100.000 100.000 100.000 99.991 99.955
249 100.000 100.000 100.000 100.000 99.991 99.956
250 100.000 100.000 100.000 100.000 99.991 99.956
251 100.000 100.000 100.000 100.000 99.991 99.956
252 100.000 100.000 100.000 100.000 99.992 99.957
253 100.000 100.000 100.000 100.000 99.992 99.958
254 100.000 100.000 100.000 100.000 99.992 99.958
255 100.000 100.000 100.000 100.000 99.992 99.959
256 100.000 100.000 100.000 100.000 99.992 99.961
257 100.000 100.000 100.000 100.000 99.993 99.961
258 100.000 100.000 100.000 100.000 99.993 99.962
259 100.000 100.000 100.000 100.000 99.993 99.962
260 100.000 100.000 100.000 100.000 99.993 99.962
261 100.000 100.000 100.000 100.000 99.993 99.963
262 100.000 100.000 100.000 100.000 99.993 99.963
263 100.000 100.000 100.000 100.000 99.993 99.963
264 100.000 100.000 100.000 100.000 99.993 99.964
265 100.000 100.000 100.000 100.000 99.993 99.964
266 100.000 100.000 100.000 100.000 99.993 99.965
267 100.000 100.000 100.000 100.000 99.993 99.966
268 100.000 100.000 100.000 100.000 99.994 99.967
269 100.000 100.000 100.000 100.000 99.994 99.970
270 100.000 100.000 100.000 100.000 99.994 99.970
271 100.000 100.000 100.000 100.000 99.995 99.971
272 100.000 100.000 100.000 100.000 99.995 99.971
273 100.000 100.000 100.000 100.000 99.995 99.971
274 100.000 100.000 100.000 100.000 99.995 99.972
275 100.000 100.000 100.000 100.000 99.995 99.972
276 100.000 100.000 100.000 100.000 99.995 99.972
277 100.000 100.000 100.000 100.000 99.995 99.972
278 100.000 100.000 100.000 100.000 99.995 99.976
279 100.000 100.000 100.000 100.000 99.996 99.976
280 100.000 100.000 100.000 100.000 99.996 99.976
281 100.000 100.000 100.000 100.000 99.996 99.976
282 100.000 100.000 100.000 100.000 99.996 99.976
283 100.000 100.000 100.000 100.000 99.996 99.977
284 100.000 100.000 100.000 100.000 99.996 99.978
285 100.000 100.000 100.000 100.000 99.996 99.978
286 100.000 100.000 100.000 100.000 99.996 99.980
287 100.000 100.000 100.000 100.000 99.996 99.980
288 100.000 100.000 100.000 100.000 99.996 99.980
289 100.000 100.000 100.000 100.000 99.997 99.981
290 100.000 100.000 100.000 100.000 99.997 99.981
291 100.000 100.000 100.000 100.000 99.997 99.983
292 100.000 100.000 100.000 100.000 99.997 99.984
293 100.000 100.000 100.000 100.000 99.998 99.985
294 100.000 100.000 100.000 100.000 99.998 99.985
295 100.000 100.000 100.000 100.000 99.998 99.985
296 100.000 100.000 100.000 100.000 99.998 99.986
297 100.000 100.000 100.000 100.000 99.998 99.987
298 100.000 100.000 100.000 100.000 99.998 99.987
299 100.000 100.000 100.000 100.000 99.998 99.988
300 100.000 100.000 100.000 100.000 99.998 99.988
301 100.000 100.000 100.000 100.000 99.998 99.988
302 100.000 100.000 100.000 100.000 99.998 99.988
303 100.000 100.000 100.000 100.000 99.998 99.989
304 100.000 100.000 100.000 100.000 99.998 99.989
305 100.000 100.000 100.000 100.000 99.998 99.989
306 100.000 100.000 100.000 100.000 99.998 99.989
307 100.000 100.000 100.000 100.000 99.998 99.989
308 100.000 100.000 100.000 100.000 99.998 99.989
309 100.000 100.000 100.000 100.000 99.999 99.991
310 100.000 100.000 100.000 100.000 99.999 99.991
311 100.000 100.000 100.000 100.000 99.999 99.991
312 100.000 100.000 100.000 100.000 99.999 99.991
313 100.000 100.000 100.000 100.000 99.999 99.992
314 100.000 100.000 100.000 100.000 99.999 99.992
315 100.000 100.000 100.000 100.000 99.999 99.992
316 100.000 100.000 100.000 100.000 99.999 99.993
317 100.000 100.000 100.000 100.000 99.999 99.993
318 100.000 100.000 100.000 100.000 99.999 99.994
319 100.000 100.000 100.000 100.000 99.999 99.994
320 100.000 100.000 100.000 100.000 99.999 99.994
321 100.000 100.000 100.000 100.000 99.999 99.994
322 100.000 100.000 100.000 100.000 99.999 99.994
323 100.000 100.000 100.000 100.000 99.999 99.994
324 100.000 100.000 100.000 100.000 99.999 99.994
325 100.000 100.000 100.000 100.000 99.999 99.995
326 100.000 100.000 100.000 100.000 99.999 99.995
327 100.000 100.000 100.000 100.000 99.999 99.995
328 100.000 100.000 100.000 100.000 100.000 99.996
329 100.000 100.000 100.000 100.000 100.000 99.996
330 100.000 100.000 100.000 100.000 100.000 99.996
331 100.000 100.000 100.000 100.000 100.000 99.996
332 100.000 100.000 100.000 100.000 100.000 99.996
333 100.000 100.000 100.000 100.000 100.000 99.996
334 100.000 100.000 100.000 100.000 100.000 99.996
335 100.000 100.000 100.000 100.000 100.000 99.997
336 100.000 100.000 100.000 100.000 100.000 99.997
337 100.000 100.000 100.000 100.000 100.000 99.997
338 100.000 100.000 100.000 100.000 100.000 99.998
339 100.000 100.000 100.000 100.000 100.000 99.998
340 100.000 100.000 100.000 100.000 100.000 99.998
341 100.000 100.000 100.000 100.000 100.000 99.998
342 100.000 100.000 100.000 100.000 100.000 99.998
343 100.000 100.000 100.000 100.000 100.000 99.998
344 100.000 100.000 100.000 100.000 100.000 99.998
345 100.000 100.000 100.000 100.000 100.000 99.998
346 100.000 100.000 100.000 100.000 100.000 99.998
347 100.000 100.000 100.000 100.000 100.000 99.998
348 100.000 100.000 100.000 100.000 100.000 99.998
349 100.000 100.000 100.000 100.000 100.000 99.998
350 100.000 100.000 100.000 100.000 100.000 99.999
351 100.000 100.000 100.000 100.000 100.000 99.999
352 100.000 100.000 100.000 100.000 100.000 99.999
353 100.000 100.000 100.000 100.000 100.000 99.999
354 100.000 100.000 100.000 100.000 100.000 99.999
355 100.000 100.000 100.000 100.000 100.000 99.999
356 100.000 100.000 100.000 100.000 100.000 99.999
357 100.000 100.000 100.000 100.000 100.000 100.000
358 100.000 100.000 100.000 100.000 100.000 100.000
359 100.000 100.000 100.000 100.000 100.000 100.000
360 100.000 100.000 100.000 100.000 100.000 100.000
361 100.000 100.000 100.000 100.000 100.000 100.000
362 100.000 100.000 100.000 100.000 100.000 100.000
363 100.000 100.000 100.000 100.000 100.000 100.000
364 100.000 100.000 100.000 100.000 100.000 100.000
365 100.000 100.000 100.000 100.000 100.000 100.000
366 100.000 100.000 100.000 100.000 100.000 100.000

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 6, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m interactions to cover and n test cases, let $I_i$ be the index of the first test case that covers the interaction $I$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^{m}{I_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

CASA Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 88.372 74.855 62.088 50.933 41.540 33.785
4 93.023 82.271 70.747 59.812 50.024 41.548
5 95.349 86.443 76.064 65.664 55.961 47.267
6 97.674 90.267 80.790 70.838 61.243 52.416
7 97.674 91.425 83.153 74.102 65.049 56.451
8 97.674 92.468 85.286 77.061 68.520 60.159
9 97.674 93.163 86.790 79.256 71.209 63.143
10 97.674 93.395 87.437 80.376 72.766 65.048
11 97.674 93.859 88.452 81.885 74.658 67.203
12 100.000 97.335 92.431 86.001 78.705 71.063
13 100.000 97.567 93.040 87.025 80.105 72.761
14 100.000 97.798 93.612 87.955 81.355 74.265
15 100.000 97.914 93.972 88.619 82.322 75.494
16 100.000 98.146 94.508 89.456 83.422 76.804
17 100.000 98.262 94.868 90.118 84.383 78.021
18 100.000 98.494 95.412 90.973 85.505 79.352
19 100.000 98.610 95.698 91.453 86.182 80.211
20 100.000 98.610 95.828 91.805 86.791 81.068
21 100.000 98.610 95.929 92.092 87.302 81.803
22 100.000 98.725 96.243 92.635 88.069 82.765
23 100.000 98.957 96.760 93.422 89.082 83.952
24 100.000 98.957 96.852 93.677 89.532 84.595
25 100.000 98.957 96.917 93.869 89.889 85.131
26 100.000 99.073 97.185 94.299 90.479 85.869
27 100.000 99.073 97.249 94.486 90.823 86.378
28 100.000 99.305 97.738 95.203 91.724 87.421
29 100.000 99.305 97.812 95.410 92.094 87.956
30 100.000 99.305 97.868 95.572 92.395 88.406
31 100.000 99.305 97.914 95.713 92.663 88.815
32 100.000 99.305 97.978 95.895 92.989 89.289
33 100.000 99.305 98.015 96.011 93.214 89.637
34 100.000 99.305 98.043 96.096 93.381 89.901
35 100.000 99.305 98.071 96.186 93.561 90.187
36 100.000 99.305 98.098 96.276 93.740 90.472
37 100.000 99.305 98.145 96.412 93.994 90.852
38 100.000 99.305 98.181 96.523 94.206 91.177
39 100.000 99.305 98.218 96.636 94.420 91.506
40 100.000 99.305 98.237 96.696 94.544 91.709
41 100.000 99.421 98.486 97.067 95.018 92.268
42 100.000 99.537 98.735 97.438 95.492 92.826
43 100.000 99.537 98.745 97.475 95.580 92.983
44 100.000 99.537 98.763 97.537 95.706 93.188
45 100.000 99.537 98.791 97.619 95.861 93.427
46 100.000 99.537 98.818 97.701 96.017 93.665
47 100.000 99.537 98.828 97.737 96.097 93.807
48 100.000 99.652 99.077 98.105 96.564 94.349
49 100.000 99.652 99.086 98.142 96.647 94.496
50 100.000 99.652 99.095 98.177 96.727 94.636
51 100.000 99.768 99.345 98.543 97.184 95.162
52 100.000 99.768 99.354 98.580 97.270 95.313
53 100.000 99.768 99.363 98.615 97.347 95.446
54 100.000 99.768 99.372 98.648 97.421 95.573
55 100.000 99.884 99.622 99.014 97.877 96.095
56 100.000 99.884 99.631 99.047 97.950 96.221
57 100.000 100.000 99.880 99.414 98.409 96.748
58 100.000 100.000 99.889 99.445 98.475 96.862
59 100.000 100.000 99.889 99.455 98.512 96.941
60 100.000 100.000 99.898 99.484 98.572 97.043
61 100.000 100.000 99.898 99.492 98.600 97.105
62 100.000 100.000 99.898 99.499 98.627 97.166
63 100.000 100.000 99.898 99.506 98.654 97.226
64 100.000 100.000 99.908 99.537 98.717 97.331
65 100.000 100.000 99.908 99.544 98.743 97.390
66 100.000 100.000 99.908 99.551 98.770 97.449
67 100.000 100.000 99.908 99.558 98.796 97.507
68 100.000 100.000 99.908 99.565 98.820 97.559
69 100.000 100.000 99.908 99.570 98.840 97.606
70 100.000 100.000 99.908 99.576 98.863 97.658
71 100.000 100.000 99.908 99.583 98.889 97.714
72 100.000 100.000 99.908 99.589 98.908 97.759
73 100.000 100.000 99.908 99.596 98.934 97.814
74 100.000 100.000 99.908 99.602 98.957 97.864
75 100.000 100.000 99.908 99.608 98.979 97.912
76 100.000 100.000 99.908 99.613 98.995 97.949
77 100.000 100.000 99.908 99.617 99.011 97.986
78 100.000 100.000 99.908 99.621 99.027 98.024
79 100.000 100.000 99.908 99.625 99.043 98.060
80 100.000 100.000 99.908 99.629 99.059 98.097
81 100.000 100.000 99.908 99.632 99.072 98.128
82 100.000 100.000 99.908 99.637 99.088 98.164
83 100.000 100.000 99.908 99.641 99.103 98.200
84 100.000 100.000 99.908 99.646 99.122 98.242
85 100.000 100.000 99.908 99.650 99.138 98.278
86 100.000 100.000 99.908 99.654 99.153 98.313
87 100.000 100.000 99.908 99.658 99.169 98.349
88 100.000 100.000 99.908 99.663 99.184 98.384
89 100.000 100.000 99.908 99.668 99.203 98.424
90 100.000 100.000 99.908 99.672 99.218 98.459
91 100.000 100.000 99.908 99.675 99.231 98.487
92 100.000 100.000 99.908 99.677 99.239 98.510
93 100.000 100.000 99.917 99.704 99.290 98.588
94 100.000 100.000 99.926 99.731 99.341 98.666
95 100.000 100.000 99.926 99.733 99.350 98.687
96 100.000 100.000 99.926 99.738 99.365 98.720
97 100.000 100.000 99.926 99.741 99.377 98.747
98 100.000 100.000 99.935 99.768 99.427 98.825
99 100.000 100.000 99.945 99.795 99.478 98.902
100 100.000 100.000 99.954 99.822 99.529 98.979
101 100.000 100.000 99.954 99.824 99.537 99.000
102 100.000 100.000 99.963 99.851 99.588 99.077
103 100.000 100.000 99.963 99.853 99.596 99.097
104 100.000 100.000 99.972 99.880 99.647 99.174
105 100.000 100.000 99.972 99.882 99.655 99.196
106 100.000 100.000 99.982 99.909 99.706 99.272
107 100.000 100.000 99.991 99.936 99.757 99.349
108 100.000 100.000 100.000 99.964 99.807 99.426
109 100.000 100.000 100.000 99.966 99.816 99.447
110 100.000 100.000 100.000 99.968 99.824 99.467
111 100.000 100.000 100.000 99.971 99.836 99.493
112 100.000 100.000 100.000 99.973 99.844 99.514
113 100.000 100.000 100.000 99.976 99.856 99.539
114 100.000 100.000 100.000 99.977 99.861 99.553
115 100.000 100.000 100.000 99.980 99.872 99.579
116 100.000 100.000 100.000 99.982 99.880 99.598
117 100.000 100.000 100.000 99.982 99.882 99.606
118 100.000 100.000 100.000 99.984 99.890 99.625
119 100.000 100.000 100.000 99.986 99.898 99.644
120 100.000 100.000 100.000 99.989 99.906 99.662
121 100.000 100.000 100.000 99.990 99.911 99.675
122 100.000 100.000 100.000 99.992 99.919 99.694
123 100.000 100.000 100.000 99.994 99.927 99.712
124 100.000 100.000 100.000 99.996 99.934 99.730
125 100.000 100.000 100.000 99.996 99.936 99.736
126 100.000 100.000 100.000 99.997 99.941 99.748
127 100.000 100.000 100.000 99.997 99.942 99.754
128 100.000 100.000 100.000 99.998 99.946 99.765
129 100.000 100.000 100.000 99.998 99.947 99.770
130 100.000 100.000 100.000 99.998 99.948 99.775
131 100.000 100.000 100.000 99.998 99.950 99.780
132 100.000 100.000 100.000 99.998 99.951 99.785
133 100.000 100.000 100.000 99.999 99.955 99.795
134 100.000 100.000 100.000 99.999 99.956 99.800
135 100.000 100.000 100.000 99.999 99.957 99.805
136 100.000 100.000 100.000 99.999 99.958 99.809
137 100.000 100.000 100.000 99.999 99.959 99.813
138 100.000 100.000 100.000 99.999 99.960 99.817
139 100.000 100.000 100.000 99.999 99.961 99.821
140 100.000 100.000 100.000 99.999 99.962 99.826
141 100.000 100.000 100.000 99.999 99.962 99.829
142 100.000 100.000 100.000 99.999 99.963 99.833
143 100.000 100.000 100.000 99.999 99.964 99.836
144 100.000 100.000 100.000 100.000 99.968 99.845
145 100.000 100.000 100.000 100.000 99.969 99.849
146 100.000 100.000 100.000 100.000 99.969 99.852
147 100.000 100.000 100.000 100.000 99.970 99.855
148 100.000 100.000 100.000 100.000 99.971 99.859
149 100.000 100.000 100.000 100.000 99.972 99.862
150 100.000 100.000 100.000 100.000 99.972 99.865
151 100.000 100.000 100.000 100.000 99.973 99.868
152 100.000 100.000 100.000 100.000 99.974 99.871
153 100.000 100.000 100.000 100.000 99.974 99.874
154 100.000 100.000 100.000 100.000 99.975 99.876
155 100.000 100.000 100.000 100.000 99.975 99.879
156 100.000 100.000 100.000 100.000 99.976 99.882
157 100.000 100.000 100.000 100.000 99.977 99.885
158 100.000 100.000 100.000 100.000 99.977 99.888
159 100.000 100.000 100.000 100.000 99.978 99.891
160 100.000 100.000 100.000 100.000 99.979 99.894
161 100.000 100.000 100.000 100.000 99.979 99.896
162 100.000 100.000 100.000 100.000 99.980 99.899
163 100.000 100.000 100.000 100.000 99.980 99.901
164 100.000 100.000 100.000 100.000 99.981 99.904
165 100.000 100.000 100.000 100.000 99.981 99.906
166 100.000 100.000 100.000 100.000 99.982 99.908
167 100.000 100.000 100.000 100.000 99.982 99.910
168 100.000 100.000 100.000 100.000 99.983 99.912
169 100.000 100.000 100.000 100.000 99.983 99.914
170 100.000 100.000 100.000 100.000 99.983 99.917
171 100.000 100.000 100.000 100.000 99.984 99.919
172 100.000 100.000 100.000 100.000 99.984 99.921
173 100.000 100.000 100.000 100.000 99.985 99.923
174 100.000 100.000 100.000 100.000 99.985 99.925
175 100.000 100.000 100.000 100.000 99.986 99.928
176 100.000 100.000 100.000 100.000 99.986 99.930
177 100.000 100.000 100.000 100.000 99.987 99.932
178 100.000 100.000 100.000 100.000 99.987 99.934
179 100.000 100.000 100.000 100.000 99.988 99.936
180 100.000 100.000 100.000 100.000 99.988 99.938
181 100.000 100.000 100.000 100.000 99.989 99.940
182 100.000 100.000 100.000 100.000 99.989 99.943
183 100.000 100.000 100.000 100.000 99.990 99.945
184 100.000 100.000 100.000 100.000 99.990 99.947
185 100.000 100.000 100.000 100.000 99.991 99.949
186 100.000 100.000 100.000 100.000 99.991 99.951
187 100.000 100.000 100.000 100.000 99.991 99.953
188 100.000 100.000 100.000 100.000 99.992 99.955
189 100.000 100.000 100.000 100.000 99.992 99.958
190 100.000 100.000 100.000 100.000 99.993 99.960
191 100.000 100.000 100.000 100.000 99.993 99.962
192 100.000 100.000 100.000 100.000 99.994 99.963
193 100.000 100.000 100.000 100.000 99.994 99.965
194 100.000 100.000 100.000 100.000 99.994 99.966
195 100.000 100.000 100.000 100.000 99.995 99.968
196 100.000 100.000 100.000 100.000 99.995 99.969
197 100.000 100.000 100.000 100.000 99.995 99.971
198 100.000 100.000 100.000 100.000 99.995 99.972
199 100.000 100.000 100.000 100.000 99.996 99.974
200 100.000 100.000 100.000 100.000 99.996 99.975
201 100.000 100.000 100.000 100.000 99.996 99.977
202 100.000 100.000 100.000 100.000 99.997 99.978
203 100.000 100.000 100.000 100.000 99.997 99.980
204 100.000 100.000 100.000 100.000 99.997 99.981
205 100.000 100.000 100.000 100.000 99.998 99.983
206 100.000 100.000 100.000 100.000 99.998 99.984
207 100.000 100.000 100.000 100.000 99.998 99.986
208 100.000 100.000 100.000 100.000 99.998 99.986
209 100.000 100.000 100.000 100.000 99.999 99.987
210 100.000 100.000 100.000 100.000 99.999 99.988
211 100.000 100.000 100.000 100.000 99.999 99.989
212 100.000 100.000 100.000 100.000 99.999 99.990
213 100.000 100.000 100.000 100.000 99.999 99.991
214 100.000 100.000 100.000 100.000 99.999 99.991
215 100.000 100.000 100.000 100.000 100.000 99.992
216 100.000 100.000 100.000 100.000 100.000 99.993
217 100.000 100.000 100.000 100.000 100.000 99.994
218 100.000 100.000 100.000 100.000 100.000 99.995
219 100.000 100.000 100.000 100.000 100.000 99.995
220 100.000 100.000 100.000 100.000 100.000 99.995
221 100.000 100.000 100.000 100.000 100.000 99.995
222 100.000 100.000 100.000 100.000 100.000 99.995
223 100.000 100.000 100.000 100.000 100.000 99.995
224 100.000 100.000 100.000 100.000 100.000 99.995
225 100.000 100.000 100.000 100.000 100.000 99.995
226 100.000 100.000 100.000 100.000 100.000 99.995
227 100.000 100.000 100.000 100.000 100.000 99.995
228 100.000 100.000 100.000 100.000 100.000 99.995
229 100.000 100.000 100.000 100.000 100.000 99.995
230 100.000 100.000 100.000 100.000 100.000 99.995
231 100.000 100.000 100.000 100.000 100.000 99.995
232 100.000 100.000 100.000 100.000 100.000 99.995
233 100.000 100.000 100.000 100.000 100.000 99.996
234 100.000 100.000 100.000 100.000 100.000 99.996
235 100.000 100.000 100.000 100.000 100.000 99.996
236 100.000 100.000 100.000 100.000 100.000 99.996
237 100.000 100.000 100.000 100.000 100.000 99.996
238 100.000 100.000 100.000 100.000 100.000 99.996
239 100.000 100.000 100.000 100.000 100.000 99.996
240 100.000 100.000 100.000 100.000 100.000 99.996
241 100.000 100.000 100.000 100.000 100.000 99.996
242 100.000 100.000 100.000 100.000 100.000 99.996
243 100.000 100.000 100.000 100.000 100.000 99.996
244 100.000 100.000 100.000 100.000 100.000 99.996
245 100.000 100.000 100.000 100.000 100.000 99.996
246 100.000 100.000 100.000 100.000 100.000 99.996
247 100.000 100.000 100.000 100.000 100.000 99.996
248 100.000 100.000 100.000 100.000 100.000 99.996
249 100.000 100.000 100.000 100.000 100.000 99.996
250 100.000 100.000 100.000 100.000 100.000 99.996
251 100.000 100.000 100.000 100.000 100.000 99.996
252 100.000 100.000 100.000 100.000 100.000 99.997
253 100.000 100.000 100.000 100.000 100.000 99.997
254 100.000 100.000 100.000 100.000 100.000 99.997
255 100.000 100.000 100.000 100.000 100.000 99.997
256 100.000 100.000 100.000 100.000 100.000 99.997
257 100.000 100.000 100.000 100.000 100.000 99.997
258 100.000 100.000 100.000 100.000 100.000 99.997
259 100.000 100.000 100.000 100.000 100.000 99.997
260 100.000 100.000 100.000 100.000 100.000 99.997
261 100.000 100.000 100.000 100.000 100.000 99.997
262 100.000 100.000 100.000 100.000 100.000 99.997
263 100.000 100.000 100.000 100.000 100.000 99.997
264 100.000 100.000 100.000 100.000 100.000 99.997
265 100.000 100.000 100.000 100.000 100.000 99.997
266 100.000 100.000 100.000 100.000 100.000 99.997
267 100.000 100.000 100.000 100.000 100.000 99.997
268 100.000 100.000 100.000 100.000 100.000 99.997
269 100.000 100.000 100.000 100.000 100.000 99.997
270 100.000 100.000 100.000 100.000 100.000 99.997
271 100.000 100.000 100.000 100.000 100.000 99.998
272 100.000 100.000 100.000 100.000 100.000 99.998
273 100.000 100.000 100.000 100.000 100.000 99.998
274 100.000 100.000 100.000 100.000 100.000 99.998
275 100.000 100.000 100.000 100.000 100.000 99.998
276 100.000 100.000 100.000 100.000 100.000 99.998
277 100.000 100.000 100.000 100.000 100.000 99.998
278 100.000 100.000 100.000 100.000 100.000 99.998
279 100.000 100.000 100.000 100.000 100.000 99.998
280 100.000 100.000 100.000 100.000 100.000 99.998
281 100.000 100.000 100.000 100.000 100.000 99.998
282 100.000 100.000 100.000 100.000 100.000 99.998
283 100.000 100.000 100.000 100.000 100.000 99.998
284 100.000 100.000 100.000 100.000 100.000 99.998
285 100.000 100.000 100.000 100.000 100.000 99.998
286 100.000 100.000 100.000 100.000 100.000 99.998
287 100.000 100.000 100.000 100.000 100.000 99.998
288 100.000 100.000 100.000 100.000 100.000 99.998
289 100.000 100.000 100.000 100.000 100.000 99.999
290 100.000 100.000 100.000 100.000 100.000 99.999
291 100.000 100.000 100.000 100.000 100.000 99.999
292 100.000 100.000 100.000 100.000 100.000 99.999
293 100.000 100.000 100.000 100.000 100.000 99.999
294 100.000 100.000 100.000 100.000 100.000 99.999
295 100.000 100.000 100.000 100.000 100.000 99.999
296 100.000 100.000 100.000 100.000 100.000 99.999
297 100.000 100.000 100.000 100.000 100.000 99.999
298 100.000 100.000 100.000 100.000 100.000 99.999
299 100.000 100.000 100.000 100.000 100.000 99.999
300 100.000 100.000 100.000 100.000 100.000 99.999
301 100.000 100.000 100.000 100.000 100.000 100.000
302 100.000 100.000 100.000 100.000 100.000 100.000
303 100.000 100.000 100.000 100.000 100.000 100.000
304 100.000 100.000 100.000 100.000 100.000 100.000
305 100.000 100.000 100.000 100.000 100.000 100.000
306 100.000 100.000 100.000 100.000 100.000 100.000
307 100.000 100.000 100.000 100.000 100.000 100.000
308 100.000 100.000 100.000 100.000 100.000 100.000
309 100.000 100.000 100.000 100.000 100.000 100.000
310 100.000 100.000 100.000 100.000 100.000 100.000

ACTS Coverage

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5 Cov. Str. t=6
1 67.442 47.045 33.730 24.737 18.489 14.043
2 81.395 64.774 51.112 40.252 31.739 25.102
3 86.047 72.190 59.753 49.088 40.161 32.791
4 90.698 79.606 68.393 57.924 48.583 40.480
5 93.023 83.778 73.728 63.822 54.588 46.284
6 93.023 84.936 76.110 67.128 58.456 50.393
7 95.349 88.528 80.375 71.690 63.063 54.872
8 95.349 89.340 82.083 74.125 65.992 58.073
9 95.349 90.035 83.578 76.296 68.642 61.007
10 97.674 93.395 87.381 80.244 72.569 64.811
11 97.674 93.511 87.815 81.102 73.854 66.460
12 97.674 94.206 89.246 83.113 76.256 69.081
13 97.674 94.438 89.809 84.025 77.486 70.571
14 97.674 94.670 90.400 85.003 78.814 72.178
15 97.674 94.670 90.547 85.405 79.510 73.154
16 97.674 94.902 91.110 86.307 80.707 74.583
17 100.000 98.146 94.646 89.845 84.113 77.794
18 100.000 98.262 94.978 90.438 84.963 78.868
19 100.000 98.378 95.255 90.896 85.607 79.688
20 100.000 98.378 95.320 91.088 85.966 80.229
21 100.000 98.494 95.643 91.655 86.772 81.242
22 100.000 98.494 95.717 91.869 87.163 81.819
23 100.000 98.610 96.058 92.479 88.031 82.903
24 100.000 98.610 96.123 92.668 88.379 83.423
25 100.000 98.610 96.206 92.903 88.801 84.036
26 100.000 98.725 96.520 93.445 89.562 84.982
27 100.000 98.957 97.018 94.180 90.486 86.046
28 100.000 98.957 97.065 94.321 90.754 86.457
29 100.000 99.073 97.332 94.747 91.330 87.167
30 100.000 99.073 97.378 94.885 91.592 87.564
31 100.000 99.305 97.868 95.593 92.467 88.556
32 100.000 99.305 97.923 95.754 92.761 88.992
33 100.000 99.305 97.951 95.845 92.948 89.293
34 100.000 99.305 98.006 96.005 93.238 89.722
35 100.000 99.305 98.034 96.095 93.421 90.016
36 100.000 99.305 98.071 96.207 93.633 90.341
37 100.000 99.305 98.108 96.325 93.867 90.705
38 100.000 99.305 98.135 96.413 94.040 90.979
39 100.000 99.305 98.181 96.549 94.294 91.360
40 100.000 99.305 98.200 96.615 94.434 91.593
41 100.000 99.305 98.218 96.679 94.568 91.814
42 100.000 99.305 98.255 96.786 94.766 92.113
43 100.000 99.305 98.274 96.849 94.899 92.333
44 100.000 99.305 98.292 96.911 95.026 92.540
45 100.000 99.305 98.320 96.996 95.191 92.797
46 100.000 99.421 98.569 97.365 95.659 93.343
47 100.000 99.421 98.578 97.401 95.743 93.492
48 100.000 99.537 98.828 97.768 96.203 94.024
49 100.000 99.537 98.828 97.779 96.245 94.115
50 100.000 99.537 98.828 97.790 96.283 94.201
51 100.000 99.537 98.837 97.822 96.354 94.325
52 100.000 99.652 99.086 98.188 96.811 94.849
53 100.000 99.768 99.335 98.554 97.271 95.380
54 100.000 99.768 99.345 98.588 97.344 95.508
55 100.000 99.884 99.594 98.953 97.801 96.030
56 100.000 99.884 99.603 98.987 97.874 96.157
57 100.000 99.884 99.612 99.018 97.941 96.272
58 100.000 100.000 99.862 99.384 98.400 96.799
59 100.000 100.000 99.880 99.439 98.503 96.959
60 100.000 100.000 99.880 99.449 98.540 97.040
61 100.000 100.000 99.889 99.481 98.610 97.159
62 100.000 100.000 99.889 99.490 98.640 97.227
63 100.000 100.000 99.889 99.500 98.677 97.307
64 100.000 100.000 99.889 99.508 98.708 97.374
65 100.000 100.000 99.898 99.539 98.771 97.481
66 100.000 100.000 99.908 99.568 98.830 97.581
67 100.000 100.000 99.908 99.575 98.857 97.641
68 100.000 100.000 99.908 99.582 98.884 97.699
69 100.000 100.000 99.908 99.588 98.904 97.747
70 100.000 100.000 99.908 99.593 98.924 97.793
71 100.000 100.000 99.908 99.599 98.947 97.844
72 100.000 100.000 99.908 99.604 98.967 97.890
73 100.000 100.000 99.908 99.610 98.990 97.941
74 100.000 100.000 99.908 99.616 99.009 97.985
75 100.000 100.000 99.908 99.623 99.035 98.041
76 100.000 100.000 99.908 99.628 99.055 98.085
77 100.000 100.000 99.908 99.632 99.071 98.124
78 100.000 100.000 99.908 99.638 99.090 98.167
79 100.000 100.000 99.908 99.643 99.110 98.211
80 100.000 100.000 99.908 99.647 99.126 98.248
81 100.000 100.000 99.908 99.651 99.142 98.285
82 100.000 100.000 99.908 99.655 99.157 98.321
83 100.000 100.000 99.908 99.659 99.173 98.357
84 100.000 100.000 99.908 99.663 99.185 98.386
85 100.000 100.000 99.908 99.666 99.198 98.416
86 100.000 100.000 99.908 99.671 99.217 98.457
87 100.000 100.000 99.908 99.676 99.235 98.497
88 100.000 100.000 99.908 99.679 99.247 98.526
89 100.000 100.000 99.908 99.682 99.259 98.554
90 100.000 100.000 99.908 99.687 99.274 98.587
91 100.000 100.000 99.908 99.690 99.286 98.615
92 100.000 100.000 99.908 99.692 99.295 98.638
93 100.000 100.000 99.908 99.694 99.304 98.660
94 100.000 100.000 99.917 99.721 99.355 98.737
95 100.000 100.000 99.926 99.748 99.405 98.815
96 100.000 100.000 99.935 99.775 99.456 98.892
97 100.000 100.000 99.945 99.802 99.507 98.969
98 100.000 100.000 99.954 99.829 99.557 99.046
99 100.000 100.000 99.963 99.856 99.608 99.123
100 100.000 100.000 99.972 99.883 99.658 99.200
101 100.000 100.000 99.982 99.910 99.709 99.276
102 100.000 100.000 99.991 99.938 99.760 99.353
103 100.000 100.000 100.000 99.965 99.810 99.430
104 100.000 100.000 100.000 99.967 99.819 99.451
105 100.000 100.000 100.000 99.970 99.830 99.477
106 100.000 100.000 100.000 99.971 99.835 99.491
107 100.000 100.000 100.000 99.973 99.843 99.511
108 100.000 100.000 100.000 99.974 99.848 99.524
109 100.000 100.000 100.000 99.976 99.856 99.543
110 100.000 100.000 100.000 99.978 99.865 99.563
111 100.000 100.000 100.000 99.980 99.872 99.581
112 100.000 100.000 100.000 99.982 99.880 99.599
113 100.000 100.000 100.000 99.984 99.888 99.617
114 100.000 100.000 100.000 99.985 99.893 99.629
115 100.000 100.000 100.000 99.986 99.897 99.640
116 100.000 100.000 100.000 99.986 99.899 99.647
117 100.000 100.000 100.000 99.986 99.900 99.653
118 100.000 100.000 100.000 99.988 99.905 99.664
119 100.000 100.000 100.000 99.989 99.909 99.676
120 100.000 100.000 100.000 99.989 99.910 99.682
121 100.000 100.000 100.000 99.990 99.915 99.693
122 100.000 100.000 100.000 99.991 99.919 99.704
123 100.000 100.000 100.000 99.991 99.920 99.709
124 100.000 100.000 100.000 99.992 99.925 99.720
125 100.000 100.000 100.000 99.993 99.929 99.731
126 100.000 100.000 100.000 99.994 99.933 99.741
127 100.000 100.000 100.000 99.994 99.934 99.746
128 100.000 100.000 100.000 99.995 99.939 99.756
129 100.000 100.000 100.000 99.995 99.940 99.761
130 100.000 100.000 100.000 99.995 99.941 99.766
131 100.000 100.000 100.000 99.996 99.945 99.776
132 100.000 100.000 100.000 99.996 99.946 99.781
133 100.000 100.000 100.000 99.996 99.947 99.786
134 100.000 100.000 100.000 99.996 99.948 99.791
135 100.000 100.000 100.000 99.996 99.949 99.796
136 100.000 100.000 100.000 99.997 99.953 99.806
137 100.000 100.000 100.000 99.997 99.954 99.810
138 100.000 100.000 100.000 99.998 99.958 99.820
139 100.000 100.000 100.000 99.999 99.962 99.830
140 100.000 100.000 100.000 99.999 99.963 99.834
141 100.000 100.000 100.000 99.999 99.964 99.838
142 100.000 100.000 100.000 99.999 99.965 99.842
143 100.000 100.000 100.000 100.000 99.969 99.851
144 100.000 100.000 100.000 100.000 99.970 99.855
145 100.000 100.000 100.000 100.000 99.971 99.858
146 100.000 100.000 100.000 100.000 99.972 99.862
147 100.000 100.000 100.000 100.000 99.972 99.865
148 100.000 100.000 100.000 100.000 99.973 99.868
149 100.000 100.000 100.000 100.000 99.974 99.871
150 100.000 100.000 100.000 100.000 99.974 99.874
151 100.000 100.000 100.000 100.000 99.975 99.876
152 100.000 100.000 100.000 100.000 99.975 99.879
153 100.000 100.000 100.000 100.000 99.976 99.882
154 100.000 100.000 100.000 100.000 99.977 99.885
155 100.000 100.000 100.000 100.000 99.977 99.888
156 100.000 100.000 100.000 100.000 99.978 99.891
157 100.000 100.000 100.000 100.000 99.979 99.893
158 100.000 100.000 100.000 100.000 99.979 99.896
159 100.000 100.000 100.000 100.000 99.980 99.899
160 100.000 100.000 100.000 100.000 99.980 99.902
161 100.000 100.000 100.000 100.000 99.981 99.905
162 100.000 100.000 100.000 100.000 99.982 99.907
163 100.000 100.000 100.000 100.000 99.982 99.910
164 100.000 100.000 100.000 100.000 99.983 99.913
165 100.000 100.000 100.000 100.000 99.983 99.915
166 100.000 100.000 100.000 100.000 99.984 99.917
167 100.000 100.000 100.000 100.000 99.984 99.919
168 100.000 100.000 100.000 100.000 99.985 99.921
169 100.000 100.000 100.000 100.000 99.985 99.924
170 100.000 100.000 100.000 100.000 99.986 99.926
171 100.000 100.000 100.000 100.000 99.986 99.928
172 100.000 100.000 100.000 100.000 99.987 99.930
173 100.000 100.000 100.000 100.000 99.987 99.932
174 100.000 100.000 100.000 100.000 99.988 99.934
175 100.000 100.000 100.000 100.000 99.988 99.937
176 100.000 100.000 100.000 100.000 99.988 99.939
177 100.000 100.000 100.000 100.000 99.989 99.941
178 100.000 100.000 100.000 100.000 99.989 99.943
179 100.000 100.000 100.000 100.000 99.990 99.945
180 100.000 100.000 100.000 100.000 99.990 99.947
181 100.000 100.000 100.000 100.000 99.991 99.949
182 100.000 100.000 100.000 100.000 99.991 99.951
183 100.000 100.000 100.000 100.000 99.992 99.953
184 100.000 100.000 100.000 100.000 99.992 99.956
185 100.000 100.000 100.000 100.000 99.993 99.957
186 100.000 100.000 100.000 100.000 99.993 99.959
187 100.000 100.000 100.000 100.000 99.993 99.960
188 100.000 100.000 100.000 100.000 99.993 99.961
189 100.000 100.000 100.000 100.000 99.994 99.963
190 100.000 100.000 100.000 100.000 99.994 99.964
191 100.000 100.000 100.000 100.000 99.994 99.966
192 100.000 100.000 100.000 100.000 99.995 99.967
193 100.000 100.000 100.000 100.000 99.995 99.969
194 100.000 100.000 100.000 100.000 99.995 99.970
195 100.000 100.000 100.000 100.000 99.996 99.971
196 100.000 100.000 100.000 100.000 99.996 99.973
197 100.000 100.000 100.000 100.000 99.996 99.974
198 100.000 100.000 100.000 100.000 99.997 99.976
199 100.000 100.000 100.000 100.000 99.997 99.977
200 100.000 100.000 100.000 100.000 99.997 99.978
201 100.000 100.000 100.000 100.000 99.997 99.979
202 100.000 100.000 100.000 100.000 99.997 99.979
203 100.000 100.000 100.000 100.000 99.998 99.980
204 100.000 100.000 100.000 100.000 99.998 99.981
205 100.000 100.000 100.000 100.000 99.998 99.982
206 100.000 100.000 100.000 100.000 99.998 99.982
207 100.000 100.000 100.000 100.000 99.998 99.983
208 100.000 100.000 100.000 100.000 99.998 99.984
209 100.000 100.000 100.000 100.000 99.998 99.984
210 100.000 100.000 100.000 100.000 99.999 99.985
211 100.000 100.000 100.000 100.000 99.999 99.986
212 100.000 100.000 100.000 100.000 99.999 99.987
213 100.000 100.000 100.000 100.000 99.999 99.987
214 100.000 100.000 100.000 100.000 99.999 99.988
215 100.000 100.000 100.000 100.000 99.999 99.989
216 100.000 100.000 100.000 100.000 100.000 99.990
217 100.000 100.000 100.000 100.000 100.000 99.991
218 100.000 100.000 100.000 100.000 100.000 99.991
219 100.000 100.000 100.000 100.000 100.000 99.992
220 100.000 100.000 100.000 100.000 100.000 99.992
221 100.000 100.000 100.000 100.000 100.000 99.992
222 100.000 100.000 100.000 100.000 100.000 99.992
223 100.000 100.000 100.000 100.000 100.000 99.992
224 100.000 100.000 100.000 100.000 100.000 99.992
225 100.000 100.000 100.000 100.000 100.000 99.992
226 100.000 100.000 100.000 100.000 100.000 99.992
227 100.000 100.000 100.000 100.000 100.000 99.993
228 100.000 100.000 100.000 100.000 100.000 99.993
229 100.000 100.000 100.000 100.000 100.000 99.993
230 100.000 100.000 100.000 100.000 100.000 99.993
231 100.000 100.000 100.000 100.000 100.000 99.993
232 100.000 100.000 100.000 100.000 100.000 99.993
233 100.000 100.000 100.000 100.000 100.000 99.993
234 100.000 100.000 100.000 100.000 100.000 99.993
235 100.000 100.000 100.000 100.000 100.000 99.993
236 100.000 100.000 100.000 100.000 100.000 99.993
237 100.000 100.000 100.000 100.000 100.000 99.993
238 100.000 100.000 100.000 100.000 100.000 99.993
239 100.000 100.000 100.000 100.000 100.000 99.993
240 100.000 100.000 100.000 100.000 100.000 99.993
241 100.000 100.000 100.000 100.000 100.000 99.994
242 100.000 100.000 100.000 100.000 100.000 99.994
243 100.000 100.000 100.000 100.000 100.000 99.994
244 100.000 100.000 100.000 100.000 100.000 99.994
245 100.000 100.000 100.000 100.000 100.000 99.994
246 100.000 100.000 100.000 100.000 100.000 99.994
247 100.000 100.000 100.000 100.000 100.000 99.994
248 100.000 100.000 100.000 100.000 100.000 99.994
249 100.000 100.000 100.000 100.000 100.000 99.994
250 100.000 100.000 100.000 100.000 100.000 99.994
251 100.000 100.000 100.000 100.000 100.000 99.994
252 100.000 100.000 100.000 100.000 100.000 99.994
253 100.000 100.000 100.000 100.000 100.000 99.994
254 100.000 100.000 100.000 100.000 100.000 99.994
255 100.000 100.000 100.000 100.000 100.000 99.994
256 100.000 100.000 100.000 100.000 100.000 99.995
257 100.000 100.000 100.000 100.000 100.000 99.995
258 100.000 100.000 100.000 100.000 100.000 99.995
259 100.000 100.000 100.000 100.000 100.000 99.995
260 100.000 100.000 100.000 100.000 100.000 99.995
261 100.000 100.000 100.000 100.000 100.000 99.995
262 100.000 100.000 100.000 100.000 100.000 99.995
263 100.000 100.000 100.000 100.000 100.000 99.995
264 100.000 100.000 100.000 100.000 100.000 99.995
265 100.000 100.000 100.000 100.000 100.000 99.995
266 100.000 100.000 100.000 100.000 100.000 99.995
267 100.000 100.000 100.000 100.000 100.000 99.995
268 100.000 100.000 100.000 100.000 100.000 99.995
269 100.000 100.000 100.000 100.000 100.000 99.995
270 100.000 100.000 100.000 100.000 100.000 99.995
271 100.000 100.000 100.000 100.000 100.000 99.995
272 100.000 100.000 100.000 100.000 100.000 99.995
273 100.000 100.000 100.000 100.000 100.000 99.995
274 100.000 100.000 100.000 100.000 100.000 99.995
275 100.000 100.000 100.000 100.000 100.000 99.995
276 100.000 100.000 100.000 100.000 100.000 99.996
277 100.000 100.000 100.000 100.000 100.000 99.996
278 100.000 100.000 100.000 100.000 100.000 99.996
279 100.000 100.000 100.000 100.000 100.000 99.996
280 100.000 100.000 100.000 100.000 100.000 99.996
281 100.000 100.000 100.000 100.000 100.000 99.996
282 100.000 100.000 100.000 100.000 100.000 99.996
283 100.000 100.000 100.000 100.000 100.000 99.996
284 100.000 100.000 100.000 100.000 100.000 99.996
285 100.000 100.000 100.000 100.000 100.000 99.996
286 100.000 100.000 100.000 100.000 100.000 99.996
287 100.000 100.000 100.000 100.000 100.000 99.996
288 100.000 100.000 100.000 100.000 100.000 99.996
289 100.000 100.000 100.000 100.000 100.000 99.996
290 100.000 100.000 100.000 100.000 100.000 99.996
291 100.000 100.000 100.000 100.000 100.000 99.996
292 100.000 100.000 100.000 100.000 100.000 99.996
293 100.000 100.000 100.000 100.000 100.000 99.997
294 100.000 100.000 100.000 100.000 100.000 99.997
295 100.000 100.000 100.000 100.000 100.000 99.997
296 100.000 100.000 100.000 100.000 100.000 99.997
297 100.000 100.000 100.000 100.000 100.000 99.997
298 100.000 100.000 100.000 100.000 100.000 99.997
299 100.000 100.000 100.000 100.000 100.000 99.997
300 100.000 100.000 100.000 100.000 100.000 99.997
301 100.000 100.000 100.000 100.000 100.000 99.997
302 100.000 100.000 100.000 100.000 100.000 99.997
303 100.000 100.000 100.000 100.000 100.000 99.997
304 100.000 100.000 100.000 100.000 100.000 99.997
305 100.000 100.000 100.000 100.000 100.000 99.997
306 100.000 100.000 100.000 100.000 100.000 99.997
307 100.000 100.000 100.000 100.000 100.000 99.997
308 100.000 100.000 100.000 100.000 100.000 99.998
309 100.000 100.000 100.000 100.000 100.000 99.998
310 100.000 100.000 100.000 100.000 100.000 99.998
311 100.000 100.000 100.000 100.000 100.000 99.998
312 100.000 100.000 100.000 100.000 100.000 99.998
313 100.000 100.000 100.000 100.000 100.000 99.998
314 100.000 100.000 100.000 100.000 100.000 99.998
315 100.000 100.000 100.000 100.000 100.000 99.998
316 100.000 100.000 100.000 100.000 100.000 99.998
317 100.000 100.000 100.000 100.000 100.000 99.998
318 100.000 100.000 100.000 100.000 100.000 99.998
319 100.000 100.000 100.000 100.000 100.000 99.998
320 100.000 100.000 100.000 100.000 100.000 99.998
321 100.000 100.000 100.000 100.000 100.000 99.998
322 100.000 100.000 100.000 100.000 100.000 99.998
323 100.000 100.000 100.000 100.000 100.000 99.998
324 100.000 100.000 100.000 100.000 100.000 99.998
325 100.000 100.000 100.000 100.000 100.000 99.998
326 100.000 100.000 100.000 100.000 100.000 99.998
327 100.000 100.000 100.000 100.000 100.000 99.998
328 100.000 100.000 100.000 100.000 100.000 99.998
329 100.000 100.000 100.000 100.000 100.000 99.998
330 100.000 100.000 100.000 100.000 100.000 99.998
331 100.000 100.000 100.000 100.000 100.000 99.998
332 100.000 100.000 100.000 100.000 100.000 99.999
333 100.000 100.000 100.000 100.000 100.000 99.999
334 100.000 100.000 100.000 100.000 100.000 99.999
335 100.000 100.000 100.000 100.000 100.000 99.999
336 100.000 100.000 100.000 100.000 100.000 99.999
337 100.000 100.000 100.000 100.000 100.000 99.999
338 100.000 100.000 100.000 100.000 100.000 99.999
339 100.000 100.000 100.000 100.000 100.000 99.999
340 100.000 100.000 100.000 100.000 100.000 99.999
341 100.000 100.000 100.000 100.000 100.000 99.999
342 100.000 100.000 100.000 100.000 100.000 99.999
343 100.000 100.000 100.000 100.000 100.000 99.999
344 100.000 100.000 100.000 100.000 100.000 99.999
345 100.000 100.000 100.000 100.000 100.000 99.999
346 100.000 100.000 100.000 100.000 100.000 99.999
347 100.000 100.000 100.000 100.000 100.000 99.999
348 100.000 100.000 100.000 100.000 100.000 99.999
349 100.000 100.000 100.000 100.000 100.000 99.999
350 100.000 100.000 100.000 100.000 100.000 99.999
351 100.000 100.000 100.000 100.000 100.000 99.999
352 100.000 100.000 100.000 100.000 100.000 99.999
353 100.000 100.000 100.000 100.000 100.000 100.000
354 100.000 100.000 100.000 100.000 100.000 100.000
355 100.000 100.000 100.000 100.000 100.000 100.000
356 100.000 100.000 100.000 100.000 100.000 100.000
357 100.000 100.000 100.000 100.000 100.000 100.000
358 100.000 100.000 100.000 100.000 100.000 100.000
359 100.000 100.000 100.000 100.000 100.000 100.000
360 100.000 100.000 100.000 100.000 100.000 100.000
361 100.000 100.000 100.000 100.000 100.000 100.000
362 100.000 100.000 100.000 100.000 100.000 100.000
363 100.000 100.000 100.000 100.000 100.000 100.000
364 100.000 100.000 100.000 100.000 100.000 100.000
365 100.000 100.000 100.000 100.000 100.000 100.000
366 100.000 100.000 100.000 100.000 100.000 100.000

CASA Coverage

ACTS Coverage

CASA Fault Detection

ACTS Fault Detection

flex

Generation Strength: 6, Prioritisation Strength: 6

Results for te