Efficient Video Frame Interpolation Using Generative Adversarial Networks
Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation method...
Main Authors: | , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2020-09-01
|
Series: | Applied Sciences |
Subjects: | |
Online Access: | https://www.mdpi.com/2076-3417/10/18/6245 |
id |
doaj-b6276ad62665428ba8b0aa0aad626865 |
---|---|
record_format |
Article |
spelling |
doaj-b6276ad62665428ba8b0aa0aad6268652020-11-25T03:31:03ZengMDPI AGApplied Sciences2076-34172020-09-01106245624510.3390/app10186245Efficient Video Frame Interpolation Using Generative Adversarial NetworksQuang Nhat Tran0Shih-Hsuan Yang1Department of Computer Science and Information Engineering, National Taipei University of Technology, Taipei 10608, TaiwanDepartment of Computer Science and Information Engineering, National Taipei University of Technology, Taipei 10608, TaiwanFrame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.https://www.mdpi.com/2076-3417/10/18/6245Deep learninggenerative adversarial network (GAN)frame interpolation |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Quang Nhat Tran Shih-Hsuan Yang |
spellingShingle |
Quang Nhat Tran Shih-Hsuan Yang Efficient Video Frame Interpolation Using Generative Adversarial Networks Applied Sciences Deep learning generative adversarial network (GAN) frame interpolation |
author_facet |
Quang Nhat Tran Shih-Hsuan Yang |
author_sort |
Quang Nhat Tran |
title |
Efficient Video Frame Interpolation Using Generative Adversarial Networks |
title_short |
Efficient Video Frame Interpolation Using Generative Adversarial Networks |
title_full |
Efficient Video Frame Interpolation Using Generative Adversarial Networks |
title_fullStr |
Efficient Video Frame Interpolation Using Generative Adversarial Networks |
title_full_unstemmed |
Efficient Video Frame Interpolation Using Generative Adversarial Networks |
title_sort |
efficient video frame interpolation using generative adversarial networks |
publisher |
MDPI AG |
series |
Applied Sciences |
issn |
2076-3417 |
publishDate |
2020-09-01 |
description |
Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation. |
topic |
Deep learning generative adversarial network (GAN) frame interpolation |
url |
https://www.mdpi.com/2076-3417/10/18/6245 |
work_keys_str_mv |
AT quangnhattran efficientvideoframeinterpolationusinggenerativeadversarialnetworks AT shihhsuanyang efficientvideoframeinterpolationusinggenerativeadversarialnetworks |
_version_ |
1724574047467995136 |