Estimating runtime of a job in Hadoop MapReduce

• Main Authors: Narges Peyravi, Ali Moeini
• Format: Article
• Language: English
• Published: SpringerOpen 2020-07-01
• Series: Journal of Big Data
• Subjects: Hadoop; MapReduce; Runtime of a job; Estimating the runtime
• Online Access: http://link.springer.com/article/10.1186/s40537-020-00319-4

Abstract Hadoop MapReduce is a framework to process vast amounts of data in the cluster of machines in a reliable and fault-tolerant manner. Since being aware of the runtime of a job is crucial to subsequent decisions of this platform and being better management, in this paper we propose a new method to estimate the runtime of a job. For this purpose, after analysis the anatomy of processing a job in Hadoop MapReduce precisely, we consider two cases: when a job runs for the first time or a job has run previously. In the first case, by considering essential and efficient parameters that higher impact on runtime we formulate each phase of the Hadoop execution pipeline and state them by mathematical expressions to calculate runtime of a job. In the second case, by referring to the profile or history of a job in the database and use a weighting system the runtime is estimated. The results show the average error rate is less than 12% in the estimation of runtime for the first run and less than 8.5% when the profile or history of the job has existed.