Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts

by   Kush Jain, et al.

An "adequate" test suite should effectively find all inconsistencies between a system's requirements/specifications and its implementation. Practitioners frequently use code coverage to approximate adequacy, while academics argue that mutation score may better approximate true (oracular) adequacy coverage. High code coverage is increasingly attainable even on large systems via automatic test generation, including fuzzing. In light of all of these options for measuring and improving testing effort, how should a QA engineer spend their time? We propose a new framework for reasoning about the extent, limits, and nature of a given testing effort based on an idea we call the oracle gap, or the difference between source code coverage and mutation score for a given software element. We conduct (1) a large-scale observational study of the oracle gap across popular Maven projects, (2) a study that varies testing and oracle quality across several of those projects and (3) a small-scale observational study of highly critical, well-tested code across comparable blockchain projects. We show that the oracle gap surfaces important information about the extent and quality of a test effort beyond either adequacy metric alone. In particular, it provides a way for practitioners to identify source files where it is likely a weak oracle tests important code.


Practical Mutation Testing at Scale

Mutation analysis assesses a test suite's adequacy by measuring its abil...

A Brief Survey on Oracle-based Test Adequacy Metrics

Even though code coverage is a widespread and popular test adequacy metr...

Measuring Coverage of Prolog Programs Using Mutation Testing

Testing is an important aspect in professional software development, bot...

Predictive Mutation Analysis via Natural Language Channel in Source Code

Mutation analysis can provide valuable insights into both System Under T...

TestSelector: Automatic Test Suite Selection for Student Projects – Extended Version

Computer Science course instructors routinely have to create comprehensi...

Homo in Machina: Improving Fuzz Testing Coverage via Compartment Analysis

Fuzz testing is often automated, but also frequently augmented by expert...

P4Testgen: An Extensible Test Oracle For P4

We present P4Testgen, a test oracle for the P4_16 language. P4Testgen su...

Please sign up or login with your details

Forgot password? Click here to reset