Spark-based Application for Abnormal Log Detection

• Main Author: Koutsoumpakis, Georgios
• Format: Others
• Language: English
• Published: Uppsala universitet, Institutionen för informationsteknologi 2014
• Online Access: http://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-233335

Ericsson is a world-leader in the rapidly-changing environment of communications technology and thus it is important to provide reliable and high quality networks. Automated test loops are executed frequently, trying to find problems  in Ericsson's products but, since test cases alone are not always adequate,  machine learning techniques  are sometimes  used to find abnormal system behaviour. The Awesome Automatic Log Analysis Application (AALAA) tries to find such behaviour by checking the log files produced during the testing, using machine learning techniques. Unfortunately,  its performance is not sufficient as it requires  a lot of time to process the logs and to train the model. This thesis manages to improve AALAAs performance by implementing a new version that uses Apache Spark, a general purpose  cluster  computing system, for both the processing of the data and for the training of the machine learning algorithm.