Distributed Classification of Asynchronous Partial Model for Non-regular Drifting Data

• Main Authors: Yu-Fen Chen, 陳玉芬
• Other Authors: 陳銘憲
• Format: Others
• Language: en_US
• Published: 2015
• Online Access: http://ndltd.ncl.edu.tw/handle/96478050764203518879

博士 === 國立臺灣大學 === 電機工程學研究所 === 103 === Big Data emphasizes on 5Vs (Volume, Velocity, Variety, Value and Veracity) relevant to variety of data (scientific and engineering, social network, sensor/IoT/IoE, and multimedia-audio, video, image, etc) that contribute to the Big Data challenges. This phenomenon introduces the urgent requirement for efficiently managing data to structured information. One predominate approach is distributed classification ensemble, which improve prediction efficiency by using ensemble of distributed model or integrated by combining distributed information via statistics, to allow multiple devices collect data concurrently. With the popularity of Big Data applications, wireless and mobile technology, the amount of data in various characteristics generated by distributed devices has been tremendously increasing. As a result, distributed classification in Big Data has new challenges. There are three main challenges in distributed big data systems: 1) the distributed classification models are asynchronous and incomplete from distributed devices. Traditional distributed classification algorithms, which rely on horizontal sub-databases or vertical sub-databases, cannot be applied in this scenario. 2) Due to various characteristics of Big Data, simply separating data to equal size for constructing models takes away the significant performance benefit of classification models. In particular, non-regular recurring data are especially vulnerable to models derived from equally separated windows because noise data interfere most of the models in fixed-size buckets. 3) In our distributed environment, arbitrarily transforming popular lazy models to rules will increase the diversity of local models and reduce additional transmission bandwidth consumption. This dissertation tries to solve the above problems. First, this dissertation focuses on distributed streaming environment scenario, and proposes a rule-based distributed classification for asynchronous partial data (DIP). Our proposed method DIP selects models based on the amount of local databases and the quality of local models such that the performance gain can be fully utilized. DIP saves the communication bandwidth by transferring organized information, instead of individual instances. In addition, our distributed classification method DIP enables local devices collect various amount of local data. Due to data diversity and change diversification, the performance of classification models built from fixed-size windows or chunks declines. We investigate the data characteristic of non-regular data and introduce sequential clustering which adaptively forms sequential clusters of data based on data distributions and time to reduce noise data of inter-cluster interference and enhance classification prediction of derived models. Finally, this dissertation proposes two model transformation methods, which transforms data distributions to rules, to facilitate popular lazy classifiers in our distributed classifier. In both theory analysis and tested experiments, the proposed distributed classification framework can achieve a significant performance gain and bigger scope, as compared to the traditional distributed classification ensemble and existing dynamically changing methods.