An Efficient Design Flow for Hadoop on Embedded Heterogeneous Platforms

• Main Authors: Chen,Sheng-Yen, 陳聖諺
• Other Authors: 賴伯承
• Format: Others
• Language: en_US
• Published: 2016
• Online Access: http://ndltd.ncl.edu.tw/handle/nuu8t2

碩士 === 國立交通大學 === 電子工程學系 電子研究所 === 104 === Hadoop is a widely adopted distributed processing framework. The Hadoop framework achieves good portability by assuming each computing node a conventional CPU-based system with local memory. However, the current flow of this framework cannot effectively take full advantage of an embedded heterogeneous many-core platform. The main challenge stems from the mismatch of data collection and management paradigms between the Hadoop environment and embedded heterogeneous systems. To address the above design concerns, this paper proposes a workflow that enables Hadoop applications to efficiently leverage the distributed embedded heterogeneous many-core systems. By taking the same data layout of conventional Hadoop applications, the proposed flow introduces efficient manners to collect and manage the fine-grained data chunks. Using Principle Component Analysis (PCA) as an application driver, the proposed Hadoop design on a Tegra K1 cluster has achieved 6.4x performance enhancement when running the PCA analysis on an input matrix of 16K × 16K data. The proposed design also demonstrated much better energy efficiency when compared with the designs on conventional PC-based clusters.