Deep Learning Based Integration and Optimization of Big Data Analytics Platforms

• Main Authors: LIAO, PO-HAO, 廖柏豪
• Other Authors: CHANG, BAO-RONG
• Format: Others
• Language: en_US
• Published: 2018
• Online Access: http://ndltd.ncl.edu.tw/handle/349jtp

碩士 === 國立高雄大學 === 資訊工程學系碩士班 === 106 === This study focused on big data analysis job scheduling mechanism, predicting the time for big data analysis based on deep learning DNN (Deep Neural Network), and shortening the average waiting time of overall work by intelligent scheduling optimization. The proposed mechanism is expected to enhance the execution efficiency of big data analysis platform greatly. A multi-platform big data processing system, characterized by high efficiency, high availability and high expandability, is integrated with Hadoop and Spark to make the platform support R command-based data analysis capability. The time complexity, priority and data size of working program can influence the efficiency of overall execution work and the average waiting time for fulfilling the work, especially in the environment of big data, the average waiting time for fulfilling the work is prolonged. This problems can be solved only by designing optimal scheduling to enhance system effectiveness. This study uses DNN to predict the execution time for R program, and implements intelligent scheduling according to Shortest Job First, the optimal program execution platform is selected, so as to shorten the average waiting time for fulfilling the work to optimize the multiple big data platforms.