Potential threats mining methods based on correlation analysis of multi‐type logs

• Main Authors: Tao Qin, Yuli Gao, Lingyan Wei, Zhaoli Liu, Chenxu Wang
• Format: Article
• Language: English
• Published: Wiley 2018-09-01
• Series: IET Networks
• Subjects: correlation analysis; potential attacks; heterogeneous features; feature normalisation; anomaly detection platform; massive logs
• Online Access: https://doi.org/10.1049/iet-net.2017.0188

Log analysis is an efficiency way to detect threats by scrutinizing the events recorded by the operating systems and devices. However, it is more and more difficult to discover threats accurately due to the massive amount of logs and their various formats. Focusing on this problem, the authors propose a method for potential threats mining based on the correlation analysis of multi‐type logs. Firstly, they extract 12 features, including behavior‐related, attribute‐related and measurable features, from multi‐type logs based on the characteristics of known and potential attacks. They also propose normalization method to deal with these heterogeneous features. Secondly, focusing on solving the problem that analyzing a single type of log can only detect some specific attacks, they employ the logistic regression model to perform correlation analysis on multi‐type logs. Finally, they construct an anomaly detection platform integrated with parallel processing mechanism to process the massive records. The experimental results based on logs collected show that the proposed method has high detection accuracy and low computational complexity, which can be applied to mine potential threats and abnormal users from the massive logs in an actual network environment.