Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model

The Sentinel-2 (S2) constellation provides multispectral images at 10 m, 20 m, and 60 m resolution bands. Obtaining all bands at 10 m resolution would benefit many applications. Recently, many model-based and deep learning (DL)-based sharpening methods have been proposed. However, the downside of th...

Full description

Bibliographic Details
Main Authors: Han V. Nguyen, Magnus O. Ulfarsson, Johannes R. Sveinsson, Mauro Dalla Mura
Format: Article
Language:English
Published: IEEE 2021-01-01
Series:IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Subjects:
Online Access:https://ieeexplore.ieee.org/document/9464640/
id doaj-bc08cf61f0f944f9aa8ea764dea80bfa
record_format Article
spelling doaj-bc08cf61f0f944f9aa8ea764dea80bfa2021-07-21T23:00:07ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing2151-15352021-01-01146882689610.1109/JSTARS.2021.30922869464640Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation ModelHan V. Nguyen0Magnus O. Ulfarsson1https://orcid.org/0000-0002-0461-040XJohannes R. Sveinsson2https://orcid.org/0000-0001-6309-3126Mauro Dalla Mura3https://orcid.org/0000-0002-9656-9087Faculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, IcelandFaculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, IcelandFaculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, IcelandGIPSA-Lab, Grenoble Institute of Technology, Grenoble, FranceThe Sentinel-2 (S2) constellation provides multispectral images at 10 m, 20 m, and 60 m resolution bands. Obtaining all bands at 10 m resolution would benefit many applications. Recently, many model-based and deep learning (DL)-based sharpening methods have been proposed. However, the downside of those methods is that the DL-based methods need to be trained separately for the 20 m and the 60 m bands in a supervised manner at reduced resolution, while the model-based methods heavily depend on the hand-crafted image priors. To break the gap, this article proposes a novel unsupervised DL-based S2 sharpening method using a single convolutional neural network (CNN) to sharpen the 20 and 60 m bands at the same time at full resolution. The proposed method replaces the hand-crafted image prior by the deep image prior (DIP) provided by a CNN structure whose parameters are easily optimized using a DL optimizer. We also incorporate the modulation transfer function-based degradation model as a network layer, and add all bands to both network input and output. This setting improves the DIP and exploits the advantage of multitask learning since all S2 bands are highly correlated. Extensive experiments with real S2 data show that our proposed method outperforms competitive methods for reduced-resolution evaluation and yields very high quality sharpened image for full-resolution evaluation.https://ieeexplore.ieee.org/document/9464640/Convolutional neural networks (CNNs)image fusionMTF-based degradationSentinel-2 image sharpeningsuper-resolutionunsupervised deep learning
collection DOAJ
language English
format Article
sources DOAJ
author Han V. Nguyen
Magnus O. Ulfarsson
Johannes R. Sveinsson
Mauro Dalla Mura
spellingShingle Han V. Nguyen
Magnus O. Ulfarsson
Johannes R. Sveinsson
Mauro Dalla Mura
Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model
IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Convolutional neural networks (CNNs)
image fusion
MTF-based degradation
Sentinel-2 image sharpening
super-resolution
unsupervised deep learning
author_facet Han V. Nguyen
Magnus O. Ulfarsson
Johannes R. Sveinsson
Mauro Dalla Mura
author_sort Han V. Nguyen
title Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model
title_short Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model
title_full Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model
title_fullStr Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model
title_full_unstemmed Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model
title_sort sentinel-2 sharpening using a single unsupervised convolutional neural network with mtf-based degradation model
publisher IEEE
series IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
issn 2151-1535
publishDate 2021-01-01
description The Sentinel-2 (S2) constellation provides multispectral images at 10 m, 20 m, and 60 m resolution bands. Obtaining all bands at 10 m resolution would benefit many applications. Recently, many model-based and deep learning (DL)-based sharpening methods have been proposed. However, the downside of those methods is that the DL-based methods need to be trained separately for the 20 m and the 60 m bands in a supervised manner at reduced resolution, while the model-based methods heavily depend on the hand-crafted image priors. To break the gap, this article proposes a novel unsupervised DL-based S2 sharpening method using a single convolutional neural network (CNN) to sharpen the 20 and 60 m bands at the same time at full resolution. The proposed method replaces the hand-crafted image prior by the deep image prior (DIP) provided by a CNN structure whose parameters are easily optimized using a DL optimizer. We also incorporate the modulation transfer function-based degradation model as a network layer, and add all bands to both network input and output. This setting improves the DIP and exploits the advantage of multitask learning since all S2 bands are highly correlated. Extensive experiments with real S2 data show that our proposed method outperforms competitive methods for reduced-resolution evaluation and yields very high quality sharpened image for full-resolution evaluation.
topic Convolutional neural networks (CNNs)
image fusion
MTF-based degradation
Sentinel-2 image sharpening
super-resolution
unsupervised deep learning
url https://ieeexplore.ieee.org/document/9464640/
work_keys_str_mv AT hanvnguyen sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel
AT magnusoulfarsson sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel
AT johannesrsveinsson sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel
AT maurodallamura sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel
_version_ 1721292330218029056