Reducing Communication Overhead and Computation Costs in a Cloud Network by Early Combination of Partial Results

• Main Authors: Jun-neng Huang, 黃俊能
• Other Authors: Steve W.Haga
• Format: Others
• Language: en_US
• Published: 2011
• Online Access: http://ndltd.ncl.edu.tw/handle/09176936188923723605

碩士 === 國立中山大學 === 資訊工程學系研究所 === 99 === This thesis describes a method of reducing communication overheads within the MapReduce infrastructure of a cloud computing environment. MapReduce is an framework for parallelizing the processing on massive data systems stored across a distributed computer network. One of the benefits of MapReduce is that the computation is usually performed on a computer (node) that holds the data file. Not only does this approach achieve parallelism, but it also benefits from a characteristic common to many applications: that the answer derived from a computation is often smaller than the size of the input file. Our new method benefits also from this feature. We delay the transmission of individual answers out a given node, so as to allow these answers to be combined locally, first. This combination has two advantages. First, it allows for a further reduction in the amount of data to ultimately transmit. And second, it allows for additional computation across files (such as a merge-sort). There is a limit to the benefit of delaying transmission, however, because the reducer stage of MapReduce cannot begin its work until the nodes transmit their answers. We therefore consider a mechanism to allow the user to adjust the amount of delay before data transmission out of each node.