Choosing Between Remote I/O versus Staging in Distributed Environment

• Main Author: Suslu, Ibrahim Hakki
• Other Authors: Kosar, Tevfik
• Format: Others
• Language: en
• Published: LSU 2010
• Subjects: Computer Science
• Online Access: http://etd.lsu.edu/docs/available/etd-06082010-092441/

Today, scientific applications and experiments have become increasingly complex and more demanding in terms of their computational and data requirements. The amount of data generated and used has grown at a very rapid rate. As tens or hundreds of terabytes of data for a single application is very common today; petabytes and even exabytes of data will be very common in a few years. One of the major challenges in distributed computing environments is how to access these large datasets remotely over the network. Data staging and remote I/O are the most widely used data access methods for distributed applications. Application developers generally chose one over the other intuitively without making any scientific comparison specific to their applications since there is no generic model available that they can use. In this thesis, we develop generic models and set guidelines for the application developers which would help them to choose the most appropriate data access method for their application. We define the parameters that potentially affect the end-to-end performance of the distributed applications which need to access remote data. To achieve our goal, we implement a series of synthetic benchmark applications to simulate different data access patterns. We run these benchmark applications on different distributed computing settings with different parameters, such as network bandwidth, server and client capabilities, and data access ratio. We also use different remote I/O protocols to show the importance of the protocol in making a decision. We use regression analysis to develop applicable generic models for comparing different data access methods, and test our models in a real life application. The main contribution of our thesis is generic models that can be applied to most data-intensive distributed applications to decide the best data access technique for those applications. Our models provide the scientists and application developers an opportunity to choose the best data access method before actually running the application.