Distinguishing between human and computer generated questions

• Main Author: Emtell, Sebastian
• Format: Others
• Language: English
• Published: KTH, Skolan för elektroteknik och datavetenskap (EECS) 2020
• Subjects: Computer Sciences; Datavetenskap (datalogi)
• Online Access: http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-283041

This thesis aims to determine how accurate humans are at distinguishing between human written programming questions and programming questions generated by Heilman’s algorithm. Not a lot of previous work has been done comparing computer generated questions to human written questions. Heilman’s algorithm is relatively old but widely researched. A survey was shared with personal contacts of the author of this paper in order to gather responses. The survey aimed at making the choice harder than the simple format of Is this questions written by a human or generated by a computer? by using groups of three questions and asking the questions Which question is generated by a computer? and Which question is generated by a human? respectively, including the option None of the above. The results showed that people were able to distinguish between human written programming questions and programming questions generated by Heilman’s algorithm about 44.5 percent of the time. However, this number was slightly higher for the participants with programming experience (54.5 percent) and slightly lower for the participants without programming experience (38.5 percent). The results showed no significant di_erence whether the question was Which question is generated by a computer? or Which question is generated by a human?. === Denna avhandling syftar till att avgöra hur bra människor kan skilja mellan programmeringsfrågor skrivna av människor och programmeringsfrågor genererade med Heilman’s algoritm. Inte mycket tidigare arbete har genomförts inom området att jämföra datorgenererade frågor med frågor skrivna av människor. Heilman’s algoritm är relativt gammal men har undersökts mycket. En enkät delades ut till personliga kontakter av författaren till denna uppsats. Enkäten hade till syfte att göra valet svårare än det enkla formatet av Is this questions written by a human or generated by a computer? Genom att använda grupper om tre frågor och ställa frågorna Which question is generated by a computer? respektive Which question is generated by a human? och inkludera alternativet None of the above. Resultaten visade att människor kunde skilja mellan mänskligt skrivna programmeringsfrågor och programmeringsfrågor genererade med Heilman’s algorithm ungefär 44,5 procent av tiden. Detta antal var dock något högre för deltagarna med programmeringserfarenhet (54,5 procent) och något lägre för deltagarna utan programmeringserfarenhet (38,5 procent). Resultaten visade ingen signifikant skillnad oavsätt om frågan var Which question is generated by a computer? eller Which question is generated by a human?.