A new adaptive data transfer library for model coupling
Data transfer means transferring data fields from a sender to a receiver. It is a fundamental and frequently used operation of a coupler. Most versions of state-of-the-art couplers currently use an implementation based on the point-to-point (P2P) communication of the message passing interface (MPI)...
Main Authors: | , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Copernicus Publications
2016-06-01
|
Series: | Geoscientific Model Development |
Online Access: | http://www.geosci-model-dev.net/9/2099/2016/gmd-9-2099-2016.pdf |
id |
doaj-20ad36425c88412a944f61038e1de966 |
---|---|
record_format |
Article |
spelling |
doaj-20ad36425c88412a944f61038e1de9662020-11-24T23:24:31ZengCopernicus PublicationsGeoscientific Model Development1991-959X1991-96032016-06-01962099211310.5194/gmd-9-2099-2016A new adaptive data transfer library for model couplingC. Zhang0L. Liu1G. Yang2R. Li3B. Wang4Ministry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaMinistry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaDepartment of Computer Science and Technology, Tsinghua University, Beijing, ChinaMinistry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaMinistry of Education Key Laboratory for Earth System Modeling, Center for Earth System Science (CESS), Tsinghua University, Beijing, ChinaData transfer means transferring data fields from a sender to a receiver. It is a fundamental and frequently used operation of a coupler. Most versions of state-of-the-art couplers currently use an implementation based on the point-to-point (P2P) communication of the message passing interface (MPI) (referred to as “P2P implementation” hereafter). In this paper, we reveal the drawbacks of the P2P implementation when the parallel decompositions of the sender and the receiver are different, including low communication bandwidth due to small message size, variable and high number of MPI messages, as well as network contention. To overcome these drawbacks, we propose a butterfly implementation for data transfer. Although the butterfly implementation outperforms the P2P implementation in many cases, it degrades the performance when the sender and the receiver have similar parallel decompositions or when the number of processes used for running models is small. To ensure data transfer with optimal performance, we design and implement an adaptive data transfer library that combines the advantages of both butterfly implementation and P2P implementation. As the adaptive data transfer library automatically uses the best implementation for data transfer, it outperforms the P2P implementation in many cases while it does not decrease the performance in any cases. Now, the adaptive data transfer library is open to the public and has been imported into the C-Coupler1 coupler for performance improvement of data transfer. We believe that other couplers can also benefit from this.http://www.geosci-model-dev.net/9/2099/2016/gmd-9-2099-2016.pdf |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
C. Zhang L. Liu G. Yang R. Li B. Wang |
spellingShingle |
C. Zhang L. Liu G. Yang R. Li B. Wang A new adaptive data transfer library for model coupling Geoscientific Model Development |
author_facet |
C. Zhang L. Liu G. Yang R. Li B. Wang |
author_sort |
C. Zhang |
title |
A new adaptive data transfer library for model coupling |
title_short |
A new adaptive data transfer library for model coupling |
title_full |
A new adaptive data transfer library for model coupling |
title_fullStr |
A new adaptive data transfer library for model coupling |
title_full_unstemmed |
A new adaptive data transfer library for model coupling |
title_sort |
new adaptive data transfer library for model coupling |
publisher |
Copernicus Publications |
series |
Geoscientific Model Development |
issn |
1991-959X 1991-9603 |
publishDate |
2016-06-01 |
description |
Data transfer means transferring data fields from a sender to a receiver. It
is a fundamental and frequently used operation of a coupler. Most versions of
state-of-the-art couplers currently use an implementation based on the point-to-point (P2P) communication of the message passing interface (MPI)
(referred to as “P2P implementation” hereafter). In this paper, we reveal
the drawbacks of the P2P implementation when the parallel decompositions of
the sender and the receiver are different, including low communication
bandwidth due to small message size, variable and high number of MPI
messages, as well as network contention. To overcome these drawbacks, we
propose a butterfly implementation for data transfer. Although the butterfly
implementation outperforms the P2P implementation in many cases, it degrades
the performance when the sender and the receiver have similar parallel
decompositions or when the number of processes used for running models is
small. To ensure data transfer with optimal performance, we design and
implement an adaptive data transfer library that combines the advantages of
both butterfly implementation and P2P implementation. As the adaptive data
transfer library automatically uses the best implementation for data
transfer, it outperforms the P2P implementation in many cases while it does
not decrease the performance in any cases. Now, the adaptive data transfer
library is open to the public and has been imported into the C-Coupler1
coupler for performance improvement of data transfer. We believe that other
couplers can also benefit from this. |
url |
http://www.geosci-model-dev.net/9/2099/2016/gmd-9-2099-2016.pdf |
work_keys_str_mv |
AT czhang anewadaptivedatatransferlibraryformodelcoupling AT lliu anewadaptivedatatransferlibraryformodelcoupling AT gyang anewadaptivedatatransferlibraryformodelcoupling AT rli anewadaptivedatatransferlibraryformodelcoupling AT bwang anewadaptivedatatransferlibraryformodelcoupling AT czhang newadaptivedatatransferlibraryformodelcoupling AT lliu newadaptivedatatransferlibraryformodelcoupling AT gyang newadaptivedatatransferlibraryformodelcoupling AT rli newadaptivedatatransferlibraryformodelcoupling AT bwang newadaptivedatatransferlibraryformodelcoupling |
_version_ |
1725560318033133568 |