Efficient Video Frame Interpolation Using Generative Adversarial Networks

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation method...

Full description

Bibliographic Details
Main Authors: Quang Nhat Tran, Shih-Hsuan Yang
Format: Article
Language:English
Published: MDPI AG 2020-09-01
Series:Applied Sciences
Subjects:
Online Access:https://www.mdpi.com/2076-3417/10/18/6245
id doaj-b6276ad62665428ba8b0aa0aad626865
record_format Article
spelling doaj-b6276ad62665428ba8b0aa0aad6268652020-11-25T03:31:03ZengMDPI AGApplied Sciences2076-34172020-09-01106245624510.3390/app10186245Efficient Video Frame Interpolation Using Generative Adversarial NetworksQuang Nhat Tran0Shih-Hsuan Yang1Department of Computer Science and Information Engineering, National Taipei University of Technology, Taipei 10608, TaiwanDepartment of Computer Science and Information Engineering, National Taipei University of Technology, Taipei 10608, TaiwanFrame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.https://www.mdpi.com/2076-3417/10/18/6245Deep learninggenerative adversarial network (GAN)frame interpolation
collection DOAJ
language English
format Article
sources DOAJ
author Quang Nhat Tran
Shih-Hsuan Yang
spellingShingle Quang Nhat Tran
Shih-Hsuan Yang
Efficient Video Frame Interpolation Using Generative Adversarial Networks
Applied Sciences
Deep learning
generative adversarial network (GAN)
frame interpolation
author_facet Quang Nhat Tran
Shih-Hsuan Yang
author_sort Quang Nhat Tran
title Efficient Video Frame Interpolation Using Generative Adversarial Networks
title_short Efficient Video Frame Interpolation Using Generative Adversarial Networks
title_full Efficient Video Frame Interpolation Using Generative Adversarial Networks
title_fullStr Efficient Video Frame Interpolation Using Generative Adversarial Networks
title_full_unstemmed Efficient Video Frame Interpolation Using Generative Adversarial Networks
title_sort efficient video frame interpolation using generative adversarial networks
publisher MDPI AG
series Applied Sciences
issn 2076-3417
publishDate 2020-09-01
description Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.
topic Deep learning
generative adversarial network (GAN)
frame interpolation
url https://www.mdpi.com/2076-3417/10/18/6245
work_keys_str_mv AT quangnhattran efficientvideoframeinterpolationusinggenerativeadversarialnetworks
AT shihhsuanyang efficientvideoframeinterpolationusinggenerativeadversarialnetworks
_version_ 1724574047467995136