Correlation Aware Technique for SQL to NoSQL Transformation

• Main Authors: Hsu, Jen-Chun, 徐仁淳
• Other Authors: Hsu, Ching-Hsien
• Format: Others
• Language: zh-TW
• Published: 2014
• Online Access: http://ndltd.ncl.edu.tw/handle/77732462514698260699

碩士 === 中華大學 === 資訊工程學系碩士班 === 102 === For better efficiency of parallel and distributed computing, Apache Hadoop distributes the imported data randomly on data nodes. This mechanism provides some advantages for general data analysis, when the data have the relationship between the data sets (example: Database), it’s a popular issue that stores the data with relevance. With the data sets increasing, a lot of data to be stored in a database, if we still use traditional database that has been unable to capable of providing an efficient service to real-time system. Most people wanted to use Hadoop to improve database performance. At this time, Apache provided a tool named Sqoop that can import all databases to Hadoop environment by command line interface. Have the same concept with Hadoop, Apache Sqoop separates each table into four parts and randomly distributes them on data nodes. However, there is still a database performance concern with this data placement mechanism. This paper proposes a Correlation Aware method on Sqoop (CA_Sqoop) to improve the data placement. By gathering related data as close as it could be to reduce the data transformation cost of the network and improve the performance in terms of database usage. The CA_Sqoop also considers the table correlation and size for better data locality and query efficiency. Simulation results show the data locality of CA_Sqoop is two times better than that of original Apache Sqoop.