New criteria for test case prioritization for software product lines. A replication and extension study

Abstract Testing software product lines represents a challenging task mainly because there are many derivable products. To facilitate this issue, multiple solutions were developed to reduce the number of products that are tested while maintaining a good percentage of coverage. However, the order of testing products has received little consideration. The purpose of this research is twofold: first, to replicate the results of a previous study (which uses two specific metrics for prioritization, namely, Variability Coverage & Cyclomatic Complexity - VC&CC, and Coefficient of Connectivity-Density - CoC), and second, to investigate two new metrics to be used as prioritization criteria (Ratio of Variability - RoV, and Flexibility of Configuration - FoC). The APFD (Average Percentage of Faults Detected) metric is used to evaluate the results obtained. In the investigation, a set of 9 feature models with various numbers of features, grouped in three intervals, was used. The results show that the original findings are confirmed for all feature models used. Regarding the new criteria used, FoC and RoV outperformed the CoC metric in 6 out of 9 cases, and also obtained the best results in 3 out of 9 cases. In the other 6 out of 9 cases the VC&CC criterion obtained the best results.