Using Multi-Task Queues to Improve Data Locality in Hadoop

• Main Authors: Jhong-YiChen, 陳仲毅
• Other Authors: Sun-Yuan Hsieh
• Format: Others
• Language: en_US
• Published: 2013
• Online Access: http://ndltd.ncl.edu.tw/handle/60951468672167042635

碩士 === 國立成功大學 === 資訊工程學系碩博士班 === 101 === Distributed computing system as cloud computing becomes more and more popular, and Hadoop is one of the familiar systems. Because the default task scheduling in Hadoop is FCFS that is not efficient, Master probably selects tasks without data locality for slaves. It causes many unnecessary data transfer within slaves that directly increase jobs' execution times and make racks become the network bandwidth bottleneck. In this paper, we present a scheduling algorithm to globally consider task with data locality and load-balance. First, we create multiple queues for every slave and put each task to the queue with best data locality. Next, we compute the time balancing limit, and we start to shift tasks according the limit to get a better task assignment. In contrast with default scheduling, the proposed method could always keep less data transfer and improve the computing system performance.