A Comparative Study of Automated Test Explorers

• Main Author: Gustavsson, Johan
• Format: Others
• Language: English
• Published: Linköpings universitet, Programvara och system 2015
• Subjects: automated testing; graphwalker; ddmin; Minimizing Delta Debugging Algorithm; Lstar; L*; L-star; mbt; model based testing; De Bruijn
• Online Access: http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-122904

With modern computer systems becoming more and more complicated, theimportance of rigorous testing to ensure the quality of the product increases.This, however, means that the cost to perform tests also increases. In orderto address this problem, a lot of research has been conducted during thelast years to find a more automated way of testing software systems. Inthis thesis, different algorithms to automatically explore and test a systemhave been implemented and evaluated. In addition to this, a second setof algorithms have been implemented with the objective to isolate whichinteractions with the system were responsible for a failure. These algorithmswere also evaluated and compared against each other. In the first evaluationtwo explorers, which I called DeBruijn and LStarExplorer, were consideredsuperior to the other. The first used a DeBruijn sequence to brute forcea solution while the second used the L*-algorithm to build an FSM overthe system under test. This FSM could then be used to provide a moreaccurate description for when the failure occurred. The result from thesecond evaluation were two reducers which both tried to recreate a failureby first applying interactions performed just before the failure occurred. Ifthis was not successful, they tried interactions further and further away, untilthe failure was triggered. In addition to this, the thesis contains descriptionsabout the framework used to run the different strategies. === D ̊a v ̊ara moderna datasystem blir allt mer komplicerade,  ̈okar detta st ̈andigtbehovet av rigor ̈osa tester f ̈or att s ̈akerst ̈alla kvaliteten p ̊a den slutgiltiga pro-dukten. Det h ̈ar inneb ̈ar dock att kostnaden f ̈or att utf ̈ora testerna ocks ̊ao  ̈ kar. F ̈or att f ̈ors ̈oka hitta en l ̈osning p ̊a det h ̈ar problemet har forsknin-gen under senare tid arbetat med att ta fram automatiserade metoder atttesta mjukvarusystem. I den h ̈ar uppsatsen har olika algoritmer, f ̈or attutforska och testa ett system, implementerats och utv ̈arderats. D ̈arut ̈overhar ocks ̊a en grupp algoritmer implementerats som ska kunna isolera vilkainteraktioner med ett system som f ̊ar det att fallera.  ̈aven dessa algoritmerhar utv ̈arderats och testats mot varandra. Resultatet fr ̊an det f ̈orsta ex-perimentet var tv ̊a explorers, h ̈ar kallade DeBruijn och LStarExplorer, somvisade sig vara b ̈attre  ̈an de andra. Den f ̈orsta av dessa anv ̈ande en DeBruijn-sekvens f ̈or att hitta felen, medan den andra anv ̈ande en L*-algoritm f ̈or attbygga upp en FSM  ̈over systemet. Den h ̈ar FSM:en kunde sedan anv ̈andasf ̈or att mer precist beskriva n ̈ar felet uppstod. Resultatet fr ̊an det andraexperimentet var tv ̊a reducers, vilka b ̊ada f ̈ors ̈okte  ̊aterskapa fel genom attf ̈orst applicera interaktioner som ursprungligen utf ̈ordes percis innan feletuppstod. Om felet inte kunde  ̊aterskapas p ̊a detta s ̈att, fortsatte de medatt applicera interaktioner l ̈angre bort tills felet kunde  ̊aterskapas. Ut ̈overdetta inneh ̊aller uppsatsen ocks ̊a beskrivningar av ramverken som anv ̈andsf ̈or att k ̈ora de olika strategierna.