Transparently improving regression testing using symbolic execution

• Main Author: Marinescu, Paul Dan
• Other Authors: Cadar, Cristian
• Published: Imperial College London 2014
• Subjects: 004
• Online Access: http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.650721

Software testing is an expensive and time-consuming process, often involving the manual creation of comprehensive regression test suites. Current testing methodologies, however, do not take full advantage of these tests. In this thesis, we present two techniques for amplifying the effect of existing test suites using a lightweight symbolic execution mechanism. We approach the problem from two complementary perspectives: first, we aim to execute the code that was never executed by the regression tests by combining the existing tests, symbolic execution and a set of heuristics based on program analysis. Second, we thoroughly check all sensitive operations (e.g., pointer dereferences) executed by the test suite for errors, and explore additional paths around sensitive operations. We have implemented these approaches into two tools - katch and zesti - which we have used to test a large body of open-source code. We have applied katch to all the patches written in a combined period of approximately six years for nineteen mature programs from the popular GNU diffutils, GNU binutils and GNU findutils application suites, which are shipped with virtually all UNIX-based distributions. Our results show that katch can automatically synthesise inputs that significantly increase the patch coverage achieved by the existing manual test suites, and find bugs at the moment they are introduced. We have applied zesti to three open-source code bases - GNU Coreutils, libdwarf and readelf - where it found 52 previously unknown bugs, many of which are out of reach of standard symbolic execution. Our technique works transparently to the tester, requiring no additional human effort or changes to source code or tests. Furthermore, we have conducted a systematic empirical study to examine how code and tests co-evolve in six popular open-source systems and assess the applicability of katch and zesti to other systems.