A new adaptive data transfer library for model coupling

Data transfer means transferring data fields from a sender to a receiver. It is a fundamental and frequently used operation of a coupler. Most versions of state-of-the-art couplers currently use an implementation based on the point-to-point (P2P) communication of the message passing interface (MPI)...

Full description

Bibliographic Details
Main Authors: C. Zhang, L. Liu, G. Yang, R. Li, B. Wang
Format: Article
Language:English
Published: Copernicus Publications 2016-06-01
Series:Geoscientific Model Development
Online Access:http://www.geosci-model-dev.net/9/2099/2016/gmd-9-2099-2016.pdf
id doaj-20ad36425c88412a944f61038e1de966
record_format Article
spelling doaj-20ad36425c88412a944f61038e1de9662020-11-24T23:24:31ZengCopernicus PublicationsGeoscientific Model Development1991-959X1991-96032016-06-01962099211310.5194/gmd-9-2099-2016A new adaptive data transfer library for model couplingC. Zhang0L. Liu1G. Yang2R. Li3B. Wang4Ministry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaMinistry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaDepartment of Computer Science and Technology, Tsinghua University, Beijing, ChinaMinistry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaMinistry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaData transfer means transferring data fields from a sender to a receiver. It is a fundamental and frequently used operation of a coupler. Most versions of state-of-the-art couplers currently use an implementation based on the point-to-point (P2P) communication of the message passing interface (MPI) (referred to as “P2P implementation” hereafter). In this paper, we reveal the drawbacks of the P2P implementation when the parallel decompositions of the sender and the receiver are different, including low communication bandwidth due to small message size, variable and high number of MPI messages, as well as network contention. To overcome these drawbacks, we propose a butterfly implementation for data transfer. Although the butterfly implementation outperforms the P2P implementation in many cases, it degrades the performance when the sender and the receiver have similar parallel decompositions or when the number of processes used for running models is small. To ensure data transfer with optimal performance, we design and implement an adaptive data transfer library that combines the advantages of both butterfly implementation and P2P implementation. As the adaptive data transfer library automatically uses the best implementation for data transfer, it outperforms the P2P implementation in many cases while it does not decrease the performance in any cases. Now, the adaptive data transfer library is open to the public and has been imported into the C-Coupler1 coupler for performance improvement of data transfer. We believe that other couplers can also benefit from this.http://www.geosci-model-dev.net/9/2099/2016/gmd-9-2099-2016.pdf
collection DOAJ
language English
format Article
sources DOAJ
author C. Zhang
L. Liu
G. Yang
R. Li
B. Wang
spellingShingle C. Zhang
L. Liu
G. Yang
R. Li
B. Wang
A new adaptive data transfer library for model coupling
Geoscientific Model Development
author_facet C. Zhang
L. Liu
G. Yang
R. Li
B. Wang
author_sort C. Zhang
title A new adaptive data transfer library for model coupling
title_short A new adaptive data transfer library for model coupling
title_full A new adaptive data transfer library for model coupling
title_fullStr A new adaptive data transfer library for model coupling
title_full_unstemmed A new adaptive data transfer library for model coupling
title_sort new adaptive data transfer library for model coupling
publisher Copernicus Publications
series Geoscientific Model Development
issn 1991-959X
1991-9603
publishDate 2016-06-01
description Data transfer means transferring data fields from a sender to a receiver. It is a fundamental and frequently used operation of a coupler. Most versions of state-of-the-art couplers currently use an implementation based on the point-to-point (P2P) communication of the message passing interface (MPI) (referred to as “P2P implementation” hereafter). In this paper, we reveal the drawbacks of the P2P implementation when the parallel decompositions of the sender and the receiver are different, including low communication bandwidth due to small message size, variable and high number of MPI messages, as well as network contention. To overcome these drawbacks, we propose a butterfly implementation for data transfer. Although the butterfly implementation outperforms the P2P implementation in many cases, it degrades the performance when the sender and the receiver have similar parallel decompositions or when the number of processes used for running models is small. To ensure data transfer with optimal performance, we design and implement an adaptive data transfer library that combines the advantages of both butterfly implementation and P2P implementation. As the adaptive data transfer library automatically uses the best implementation for data transfer, it outperforms the P2P implementation in many cases while it does not decrease the performance in any cases. Now, the adaptive data transfer library is open to the public and has been imported into the C-Coupler1 coupler for performance improvement of data transfer. We believe that other couplers can also benefit from this.
url http://www.geosci-model-dev.net/9/2099/2016/gmd-9-2099-2016.pdf
work_keys_str_mv AT czhang anewadaptivedatatransferlibraryformodelcoupling
AT lliu anewadaptivedatatransferlibraryformodelcoupling
AT gyang anewadaptivedatatransferlibraryformodelcoupling
AT rli anewadaptivedatatransferlibraryformodelcoupling
AT bwang anewadaptivedatatransferlibraryformodelcoupling
AT czhang newadaptivedatatransferlibraryformodelcoupling
AT lliu newadaptivedatatransferlibraryformodelcoupling
AT gyang newadaptivedatatransferlibraryformodelcoupling
AT rli newadaptivedatatransferlibraryformodelcoupling
AT bwang newadaptivedatatransferlibraryformodelcoupling
_version_ 1725560318033133568