Performance optimization research based on Hive

• Main Authors: Wang Kang, Chen Haiguang, Li Dongjing
• Format: Article
• Language: English
• Published: Academic Journals Center of Shanghai Normal University 2017-08-01
• Series: Journal of Shanghai Normal University (Natural Sciences)
• Subjects: data warehouse; job optimization; performance optimization; compression; storage format
• Online Access: http://qktg.shnu.edu.cn/zrb/shsfqkszrb/ch/reader/view_abstract.aspx?file_no=20170411&flag=1

This paper research Hive performance optimization mainly from the two aspects of MapReduce scheduling and Hive performance tuning.MapReduce's programming model and its implementation process is analyzed,and parameters are tuned from the map side and reduce side.Then Hive's framework is researched from the aspects of the partition table,the external surface and common data file compression,the line storage and column type storage.The experimental results show that snappy compression and orcfile/parquet storage format can improve the efficiency of query for the column type queries, and has good compatibility.