Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model
The Sentinel-2 (S2) constellation provides multispectral images at 10 m, 20 m, and 60 m resolution bands. Obtaining all bands at 10 m resolution would benefit many applications. Recently, many model-based and deep learning (DL)-based sharpening methods have been proposed. However, the downside of th...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2021-01-01
|
Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/9464640/ |
id |
doaj-bc08cf61f0f944f9aa8ea764dea80bfa |
---|---|
record_format |
Article |
spelling |
doaj-bc08cf61f0f944f9aa8ea764dea80bfa2021-07-21T23:00:07ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing2151-15352021-01-01146882689610.1109/JSTARS.2021.30922869464640Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation ModelHan V. Nguyen0Magnus O. Ulfarsson1https://orcid.org/0000-0002-0461-040XJohannes R. Sveinsson2https://orcid.org/0000-0001-6309-3126Mauro Dalla Mura3https://orcid.org/0000-0002-9656-9087Faculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, IcelandFaculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, IcelandFaculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, IcelandGIPSA-Lab, Grenoble Institute of Technology, Grenoble, FranceThe Sentinel-2 (S2) constellation provides multispectral images at 10 m, 20 m, and 60 m resolution bands. Obtaining all bands at 10 m resolution would benefit many applications. Recently, many model-based and deep learning (DL)-based sharpening methods have been proposed. However, the downside of those methods is that the DL-based methods need to be trained separately for the 20 m and the 60 m bands in a supervised manner at reduced resolution, while the model-based methods heavily depend on the hand-crafted image priors. To break the gap, this article proposes a novel unsupervised DL-based S2 sharpening method using a single convolutional neural network (CNN) to sharpen the 20 and 60 m bands at the same time at full resolution. The proposed method replaces the hand-crafted image prior by the deep image prior (DIP) provided by a CNN structure whose parameters are easily optimized using a DL optimizer. We also incorporate the modulation transfer function-based degradation model as a network layer, and add all bands to both network input and output. This setting improves the DIP and exploits the advantage of multitask learning since all S2 bands are highly correlated. Extensive experiments with real S2 data show that our proposed method outperforms competitive methods for reduced-resolution evaluation and yields very high quality sharpened image for full-resolution evaluation.https://ieeexplore.ieee.org/document/9464640/Convolutional neural networks (CNNs)image fusionMTF-based degradationSentinel-2 image sharpeningsuper-resolutionunsupervised deep learning |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Han V. Nguyen Magnus O. Ulfarsson Johannes R. Sveinsson Mauro Dalla Mura |
spellingShingle |
Han V. Nguyen Magnus O. Ulfarsson Johannes R. Sveinsson Mauro Dalla Mura Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Convolutional neural networks (CNNs) image fusion MTF-based degradation Sentinel-2 image sharpening super-resolution unsupervised deep learning |
author_facet |
Han V. Nguyen Magnus O. Ulfarsson Johannes R. Sveinsson Mauro Dalla Mura |
author_sort |
Han V. Nguyen |
title |
Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model |
title_short |
Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model |
title_full |
Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model |
title_fullStr |
Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model |
title_full_unstemmed |
Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model |
title_sort |
sentinel-2 sharpening using a single unsupervised convolutional neural network with mtf-based degradation model |
publisher |
IEEE |
series |
IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
issn |
2151-1535 |
publishDate |
2021-01-01 |
description |
The Sentinel-2 (S2) constellation provides multispectral images at 10 m, 20 m, and 60 m resolution bands. Obtaining all bands at 10 m resolution would benefit many applications. Recently, many model-based and deep learning (DL)-based sharpening methods have been proposed. However, the downside of those methods is that the DL-based methods need to be trained separately for the 20 m and the 60 m bands in a supervised manner at reduced resolution, while the model-based methods heavily depend on the hand-crafted image priors. To break the gap, this article proposes a novel unsupervised DL-based S2 sharpening method using a single convolutional neural network (CNN) to sharpen the 20 and 60 m bands at the same time at full resolution. The proposed method replaces the hand-crafted image prior by the deep image prior (DIP) provided by a CNN structure whose parameters are easily optimized using a DL optimizer. We also incorporate the modulation transfer function-based degradation model as a network layer, and add all bands to both network input and output. This setting improves the DIP and exploits the advantage of multitask learning since all S2 bands are highly correlated. Extensive experiments with real S2 data show that our proposed method outperforms competitive methods for reduced-resolution evaluation and yields very high quality sharpened image for full-resolution evaluation. |
topic |
Convolutional neural networks (CNNs) image fusion MTF-based degradation Sentinel-2 image sharpening super-resolution unsupervised deep learning |
url |
https://ieeexplore.ieee.org/document/9464640/ |
work_keys_str_mv |
AT hanvnguyen sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel AT magnusoulfarsson sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel AT johannesrsveinsson sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel AT maurodallamura sentinel2sharpeningusingasingleunsupervisedconvolutionalneuralnetworkwithmtfbaseddegradationmodel |
_version_ |
1721292330218029056 |