Improving Parallel Job Scheduling Performance in Multi-cluster through Look-Ahead Processor Allocation

• Main Authors: Shih, Po-Chi, 史伯其
• Other Authors: Chung, Yeh-Ching
• Format: Others
• Language: en_US
• Published: 2012
• Online Access: http://ndltd.ncl.edu.tw/handle/33033892034123787824

博士 === 國立清華大學 === 資訊工程學系 === 100 === Multi-cluster are an important and commonly used architecture in supercomputing, grid computing and the emerging cloud computing paradigm. Techniques for efficiently exploiting multi-cluster resources become increasingly significant. A critical aspect of exploiting these resources is the challenge of scheduling, in which how to allocate resources to each job is an important issue. Intelligent allocation algorithm can make good use of the information provided by the submitted jobs and current resource status to improve the system performance. In this dissertation we focus on the issues of processor allocation in heterogeneous multi-cluster (HMC) system. In a HMC system, processor allocation is responsible for choosing available processors among clusters for job execution. Traditionally, processor allocation in HMC considers only single performance factor, resource fragmentation or processor heterogeneity, which leads to heuristics such as Best-Fit (BF) and Fastest-First (FF). However, those heuristics only favor certain types of workloads and cannot be changed adaptively. In this dissertation we propose the look-ahead processor allocation technique, which make use of the information of the waiting jobs, such as amount, execution sequence, processor requirement, and runtime estimation, to guide the decision of processor allocation. Thus, the allocation decision is made dynamically according to current workload and resource configurations. Extensive simulations that consider different workload and resource configurations are conducted for performance evaluation. Simulation results show the effectiveness of look-ahead processor allocation methods, which can achieve better performance than traditional allocation methods for most cases. Results also indicate that the information of estimated job runtime can efficiently improve the system performance, even with high estimation error. With precise job runtime, the performance improvement can be made up to four times over traditional methods.