Deep multi-level up-projection network for single image super-resolution

Most convolutional neural network-based single image super-resolution (SR) methods do not take full account of the hierarchical features of the original low-resolution (LR) images, including the intra-channel spatial feature information and the inter-channel feature information, which decreases the representational capacity of the network. A deep multi-level upprojection network (DMUN) is proposed to solve this problem. Local feature up-projection unit is adopted in DMUN to obtain high-resolution (HR) feature of different levels and then to reconstruct the SR image. Residual up-projection group in DMUN mines the hierarchical LR feature information and its corresponding HR residual information recursively. A residual recorrection mechanism is further introduced, which adopts HR residual information to re-correct HR features and enrich the details of the output image. Finally, the original residual block with spatial-and-channel attention mechanism is improved, which adaptively recalibrates features by considering the intra-channel spatial relationships and the inter-channel pixel-wise interdependencies simultaneously. Experiments on benchmark datasets show that DMUN achieves favourable performance against state-of-the-art methods.


INTRODUCTION
Single image super-resolution (SISR) is a classical problem in low-level computer vision, aiming to reconstruct a highresolution (HR) image from its degraded low-resolution (LR) measurement. Various methods, including interpolation-based methods, reconstruction-based methods and learning-based methods have been proposed for SR in the past few decades. With the great progress made by convolutional neural networks (CNNs) in high-level computer vision, such as image classification and object detection, CNN-based SR methods [1,2,4] have also achieved notable improvements over conventional methods.
Among them, SRCNN [1] is considered as the first attempt to introduce a three-layer CNN into SISR. Kim et al. [2] proposed very deep SR and achieved great improvements over SRCNN by increasing the network depth. The depth of network was proved to be important for improving network performance, while very deep network also brings the problem of training difficulty.
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2020 The Authors. IET Image Processing published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology Therefore, IRCNN [3] introduced dilated convolution to tackle low-level vision tasks, thus enlarging the receptive field of a network without the sacrifice of computational cost. Kim et al. further introduced recursive learning to ease the problem of overfitting and control the parameter number of very deep model. However, those networks mentioned above have a simple structure consisting of only a single path, which simply stacks convolution layer in a chain way and limits the improvement of SR performance. Subsequently, residual block, [4] memory block, [5] Laplacian Pyramid, [6] and dense connection [7] were introduced to enhance the performance of CNN in SISR. Residual channel attention block [8] was also introduced into SISR to adaptively rescale channel-wise features. However, most deep learning based methods neglect to take full account of the hierarchical features for reconstruction. Channel attention mechanism cannot model the intra-channel spatial relationships of the feature map, which is considered more important than the correlations between channels, especially for low-level computer vision tasks. Although most CNN-based methods can achieve sound performance, but they still have some drawbacks. Firstly, there are two kinds of SR network structures in the past. The first one adopts the bicubic interpolation [9] to upscale LR input to HR before entering the network, which results in the loss of the original structural information of the image. The second one uses LR image as input and adopts the feature information at last level to reconstruct the HR image, whose superiority have been demonstrated in [7,8,11]. However, these methods neglect the relationships between the feature information of different levels, which reduces the utilization of image features in the network. Secondly, most of these methods treat features of one channel and of different channels equally, therefore neglect the spatial-and-channel attention mechanism, which can be used to exploit the intra-channel spatial relationships and the inter-channel pixel-wise interdependencies to reduce the unnecessary computing burden and improve the flexibility and reconstruction quality of network in processing different types of information.
To address these drawbacks, we propose a novel deep multi-level up-projection network (DMUN), which consists of shallow feature extraction (SFE) unit, local feature upprojection (LFU) unit and feature reconstruction (FR) unit. In order to improve the ability of the network to obtain feature information at different levels, we do not choose to reconstruct the HR image at the end of the network, but propose a local feature up-projection (LFU) unit. Within each LFU unit, we use four residual up-projection groups (RUGs) to obtain the LR feature information at the current level and the corresponding HR residual information. We further introduce a residual re-correction mechanism, where the HR residual information is then used to re-correct the HR features obtained by the upscale module to enrich the details of the output image. Our proposed DMUN uses multiple stacked LFU units to obtain the HR features of different levels, which serves as the input to the FR unit to obtain the super-resolved image.
Inspired by squeeze-and-excitation (SE), [10] we propose a novel residual activation block (RAB) to introduce the spatialand-channel attention mechanism in low-level computer vision. RAB improves the original residual block [4] with spatial-andchannel attention mechanism, which adaptively recalibrates features through modelling the intra-channel spatial relationships and the inter-channel pixel-wise interdependencies. In this way, the spatial activation and channel activation of the feature map are implemented simultaneously. Such a spatial-and-channel attention mechanism encourages the network to focus on informative and important features, which further enhances the representational ability of the network. Experiments on benchmark datasets clearly show that our proposed DMUN achieves sound performance against state-of-the-art methods for SISR.
In summary, the main contributions of this work are as follows: (1) We propose a novel deep multi-level up-projection network (DMUN) for SISR, which achieves better SR reconstruction performance than previous state-of-art CNNbased methods. (2) We propose LFU unit to exploit the hierarchical HR feature information, which serves as the input of FR unit to achieve feature fusion at different levels and greatly boost the representational power of the network. (3) We introduce a residual re-correction mechanism, which adopts the computed HR residual information to re-correct HR features to enrich the details of the output image. (4) We propose RAB to introduce a spatial-and-channel attention into SISR, which adaptively recalibrates features through modelling the spatial relationships within channel and the pixel-wise interdependencies between channels simultaneously.
The remainder of this paper is organized as follows. Section 2 briefly reviews related work on single image super-resolution and attention mechanism. Section 3 describes the proposed DMUN model in detail. In Section 4, extensive experiments are presented and the results are reported. Finally, several concluding remarks are given in Section 5.

Single image super-resolution
Recently, deep learning-based methods have achieved superior performance against conventional methods in image super-resolution. Specifically, Dong et al. [1] firstly proposed a super-resolution convolutional neural network (SRCNN) for the SISR problem and proved the effectiveness of convolutional neural network in SISR. However, the performance of SRCNN with only three convolutional layers is limited due to the shallow architecture. Since then, Kim et al. [2] proposed very deep super-resolution (VDSR), which achieved remarkable performance through increasing the depth of the network. The increase of network depth inevitably lead to additional parameters in the model. To address the long-term dependency problem, Tai et al. [5] introduced a novel persistent memory network for image super-resolution, which consisted of a recursive unit and a gate unit. However, all of these methods mentioned above need to interpolate the LR image to desired size before applying them into models. This pre-processing step not only increases the computational cost and often blurs the original LR image. To fix this issue, ESPCN, [11] which introduced an efficient sub-pixel convolution layer to upscale the LR feature maps into the HR output, was proposed to extract the feature maps in the LR space. Lai et al. [6] proposed the laplacian pyramid super-resolution network (LapSRN) that also took LR images as input and progressively reconstructs the subband residuals of HR images. Very deep convolutional neural network can achieve more complex modelling mappings, but it also makes the network more difficult to train. Since the residual connection makes the training of very deep networks possible, some very deep models have been proposed to achieve more competitive results. Enhanced deep SR (EDSR) [4] achieved significant improvement by removing unnecessary modules in residual networks and expanding the model size. EDSR had a large number of filters (256 filters), leading to a very wide network with high number of parameters. Subsequently, Zhang et al. [7] introduced dense connections into a residual dense network (RDN) model. In [12], a deep back-projection network (DBPN) was proposed to address the mutual dependencies of LR images and HR images, which iteratively performed backprojections to learn the feedback error signal between LR and HR images. In [13], a multi-scale residual block (MSRB) was proposed to adaptive detect features and achieve feature fusion at different scales. However, these deep networks discussed above neglect the attention mechanism and take into account the consistent importance of all spatial locations and channels for super-resolution. Therefore, Zhang et al. [8] proposed the very deep residual channel attention networks (RCAN), which introduced channel attention mechanism to SISR. One shortcoming of the RCAN is its high computational complexity and a large number of parameters (23M). Li et al. [14] proposed an image super-resolution feedback network (SRFBN) with the feedback mechanism, which carried high-level information back to previous layers to refine low-level encoded information.
Making full use of the information in the LR images is the key to improve network performance. To investigate multi-level features and highlight the important features of the input, we introduce a novel spatial-and-channel attention mechanism, and propose local feature up-projection unit, which can be used to capture the feature information at different levels.

Attention mechanism
One does not try to process the whole scene at the same time, which is a striking feature of the human visual system. The human visual system selectively pays attention to salient parts of the whole image [15] for more detailed information and ignores other less useful information. Attention mechanism can be seen as a guidance to the rational allocation of available processing resources. [10,15] Recently, several attempts have been proposed to apply attention mechanism into high-level computer vision tasks in many different scenarios, [10,16] including image classification, [10,16] understanding in images [17] and sequence-based models. [18] Wang et al. [16] proposed residual attention network for image classification which introduced a powerful trunk-and-mask attention mechanism. Hu et al. [10] proposed squeeze-and-excitation (SE) block, which could be seamlessly integrated into the existing network structure, to exploit the inter-channel relationship by performing feature recalibration on the feature map. However, spatial information relationship in the feature map is not considered, and there are few works are presented to investigate the impact of attention mechanism for low-level vision tasks such as SISR. To handle this case, we propose a novel RAB to exploit the intra-channel spatial relationships and inter-channel pixel-wise interdependencies simultaneously to selectively emphasize those informative features and suppress less important ones. Our proposed model will be detailed in the next section.

Network structure
In this section, the proposed network will be described from the whole to the part. As shown in Figure 2, the proposed DMUN mainly consists of three parts: shallow feature extraction (SFE) unit, multiple stacked local feature up-projection (LFU) units and feature reconstruction (FR) unit. Let x and y be the input and output of the DMUN, respectively. Firstly, one convolutional layer is utilized to extract the shallow features from the original LR image, where F SFE denotes the function of the shallow feature extraction (SFE) unit and L 0 represents the shallow features and is then used as the input to next part. Then D local feature upprojection (LFU) units are stacked by using chained mode. So we can further have where F d LFU denotes the operation of the d-th LFU unit. L d indicates the LR feature information output from the d-th LFU unit and serves as input to the following LFU units. H d denotes the HR feature information corresponding to L d and is used to reconstruct the final HR image.
Unlike the previous SR models selecting the features output from the last LFU unit to reconstruct HR image, we adopt D LFU units to obtain the HR feature information at different levels, and use them as input to the FR unit to obtain the final HR image, be a training set, where x i is the corresponding corrupted patch of the ground truth patchx i , N is the number of training patches. Lim et al. [4] suggested that better experimental results could be achieved when training with mean absolute error (MAE). Therefore, MAE is adopted here as loss function for SISR: where Θ represents the parameter set of the whole network.

Local feature up-projection unit
To make full use of the hierarchical features from the LR image, we propose a local feature up-projection (LFU) unit, as shown in Figure 3. The proposed LFU unit mainly contains two components: four residual up-projection groups and a upscale module. Detailed description of the residual up-projection group (RUG) is given in Section 3.3.
In general, the LFU unit takes L d −1 = M 0 as input, and outputs the HR feature H d and the LR feature L d . Formally, this procedure is described as: where F k RUG denotes the operation of the k-th RUG. M k indicates the LR feature information output from the k-th RUG and serves as the input to following RUGs and the upscale module. R k denotes the HR residual information corresponding to M k and is used to re-correct the HR features obtained by the upscale module.
To take full advantage of the LR features, we adopt multi-level LR feature information as input to upscale module. Inspired by [4], we utilize ESPCN [11] in upscale module F UP . So the HR features M SR can be obtained by Furthermore, we introduce a residual re-correction mechanism, which adopts the HR residual information R k to recorrect the HR features M SR , so we have

Residual up-projection group
This section details about residual up-projection group (RUG) (Figure 4), which contains four residual activation blocks (RAB) and a upscale module. We propose RUG to exploit the hierarchical LR feature information M k and HR residual information R k , which can be divided into two stages. The RAB in stage 2 shares same parameters with its corresponding RAB in stage 1. So the stage 1 can be formulated as, where F i RAB denotes the operation of the i-th RAB. B i−1 ] indicate the output and input of the i-th RAB in stage 1.
To enable each RAB capture both the deep features and shallow features of the input image, we adopt the opposite sequence of stage 1 in stage 2. We have where [B i−1 ] and B (2) i indicate the input and output of the i-th RAB in stage 2. Furthermore, skip connection is introduced into RUG to stabilize the training of network.
where * represents the operation of convolution, W k is the weight of convolution layer with kernel size of 1×1 in the k-th RUG, where the bias term is omitted for simplicity. [B 1 , … , B i ] refers to the concatenation of the different levels residual information produced by the previous RABs and also acts as input to upscale module F UP to obtain HR residual information R k , which can be expressed as RUG uses a recursive method to mine LR feature information and its corresponding HR residual information, which enables each RAB combine shallow feature information and deep feature information, and thus greatly improves the representational capacity of the network.

Residual activation block
To enhance the discriminative learning ability of network and make the network focus on more informative features (e.g. edges and texture), we propose residual activation block (RAB) (Figure 1) to improve residual block [4] with spatial-and-channel attention.
Assume an input feature map X ∈ ℝ H ×W ×C ′ that passes through convolution layers to generate output feature map U ∈ ℝ H ×W ×C . As shown in Figure 5 as the learned set of filters, where v i refers to the i-th convolution filter.
We propose first generating the intra-channel spatial statistics by using a depthwise convolution layer, denoted as the H sa function, producing the output feature map U s = [u s,1 , u s,2 , … , u s,C ], where v i is a 2D spatial convolution filter with kernel size of 3×3. Depthwise convolution is adopted here to overcome the entanglement between spatial relationship and channel correlation introduced by the traditional convolution. Equation (13) indicates that the i-th channel of V only acts on the corresponding i-th channel of U, which completes the modelling of the spatial relationships within channels. Then, we adopt a set of pointwise convolution layers to generate the pixel-wise channel statistics.
where H ca represents the operation to model the interchannel pixel-wise correlations, denotes the ReLU function, ×C . The inter-channel pixel-wise interdependencies are evaluated on the features in the same position of the feature maps on channels "u s,1 " to "u s,C ." Pointwise convolution does not affect the surrounding feature values at the position (j, k) and thus avoids mixing the intra-channel spatial feature information when modelling the pixel-wise interdependencies between channels. Meanwhile, we achieve the compression mechanism by introducing a dimensionality reduction layer with parameters W pw1 , which also distills the inter-channel pixel-wise relationships. As discussed in [10], since we expect that the multiple channels and multiple spatial locations can be emphasized opposed to one-hot activation, the activation function must learn a non-mutually-exclusive relationship. So the output map U c is passed through a sigmoid layer (⋅) to obtain activation values, which are used to recalibrate or activate the feature map U, where H scale denotes element-wise multiplication. The spatialand-channel attention mechanism can be considered a guide to allocate high activation values to the features in important channels with abundant spatial information, which selectively allows salient features to be preserved and improves the ability of RAB to distinguish different types of features effectively. We empirically found that spatial-and-channel attention mechanism can also accelerate the convergence speed of DMUN because activation values can provide a supervision function to some extent when reconstructing the HR images.

Implementation details
In this section, we describe the implementation details of our proposed DMUN. RGB colour channels are used for both input and output image in the DMUN. We set the kernel size of all the convolutional layers to 3×3, except for the upscale module and pointwise convolution layers in RAB, whose kernel sizes are set to 1×1. For convolutional layer with kernel size 3×3, we pad zeros to each side of the input to keep size fixed. The number D of LFU unit is set to 6. In each LFU unit, we set the number of RUGs as K = 4. Convolution layers in SFE unit and LFU unit have 64 filters, except for that in the upscale module of RAB. At the end of the DMUN, we adopt 3 convolutional filters to generate the final HR images with three colour channels.

Settings
Following, [4] we use 800 images from DIV2K [19] dataset to construct the training set. For testing, we choose five widely used benchmark datasets: Set5, [20] Set14, [21] BSDS100, [22] Urban100, [23] and Manga109. [24] In order to fully demonstrate the effectiveness of our proposed DMUN, we conduct experiments with Bicubic (BI) and blur-downscale (BD) degradation models. We use BI model to simulate LR images by bicubic [9] downsampling the HR images with three specific scale factors (2, 3 and 4). We regard BD as a degradation model which applies Gaussian blur followed by downsampling to HR images. The size of the Gaussian kernel is 7×7 with standard deviation of 1.6. Scaling, flipping and rotating are used to augment the training data. The LR RGB images are cropped into patches with the size of 48×48 as inputs.
The Adam optimizer is used to minimize the L 1 loss function with parameters 1 = 0.9, 2 = 0.999, = 0.01 and = 1e-8. Due to the memory space, we use the batch size of 16 for DMUN. The parameter of epoch is set as 1000 for DMUN. The learning rate is initialized to 0.0001 and decreases half for every 200 epochs. The proposed network is implemented with the Pytorch framework running on a PC with Intel(R) Xeon(R) E5-2650V4 CPU 2.2GHz and an Nvidia GTX1080Ti GPU.

Spatial-and-channel attention
In [10], channel attention was firstly proposed to tackle the highlevel computer vision tasks (e.g. image classification). SE proposes a structural unit to model the interdependencies between channels and achieves adaptive recalibration of channel-wise features. Subsequently, RCAN [8] directly uses channel attention mechanism to construct residual channel attention block for image SR. However, the channel attention does not consider the intra-channel spatial relationships, which is considered more important and even better than the channel-wise relationships, especially for low-level computer vision tasks. Unlike the channel-wise relationships considered in SE, we chose to exploit the pixel-wise interdependencies between channels because pixel-level information is what we pay more attention to in SISR. We improve residual block with spatial-andchannel attention mechanism, which adaptively recalibrates features by simultaneously considering the spatial relationships within channels and the pixel-wise interdependencies between channels.
To demonstrate the effect of spatial-and-channel attention mechanism, we replace RAB in DMUN with residual block (RB) [4] and residual channel attention block (RCAB), [8] resulting in DMUN-RB and DMUN-RCAB, respectively. Table 1 shows the ablation investigation on the effects of different blocks. As shown in Table 1, DMUN-RCAB outperforms DMUN-RB by 0.06,0.05,0.06 and 0.05 dB on Set5, Set14, BSD100 and Urban100, respectively, which demonstrates the effectiveness of considering channel attention mechanism to reconstruct HR images. Furthermore, our proposed spatial-and-channel attention mechanism achieves the great improvement on all datasets. Therefore, it can be validated that our spatial-and-channel attention indeed performs better than channel attention for SISR and our proposed DMUN can greatly improve the representational power of the network.

Residual recorrection mechanism
In order to make full use of the feature information and improve the intrinsic expressive ability of the network, we introduce a residual re-correction mechanism into LFU unit, which adopts HR residual information to correct the HR features obtained by the upscale module. We believe that the residual network structure in the past has not paid enough attention to the residual information, which can be regarded as another form of image content in a sense. As shown in Figure 6, the HR residual information in the network contains many details of the image feature. So we input the LR residual information into upscale module to obtain HR residual feature information, and use it to re-correct the HR features to enrich the details of the output SR image. It can be seen that HR features obtained from multi-level LR features are relatively smooth, while the corrected HR features contain more details and texture information. Experimental results in Section 4.3 also demon-

Number of parameters
To prove our method has a better trade-off between the performance and model size, we show performance and number of network parameters from our network and the existing deep SR networks in Figure 7. The results are evaluated on Set5 dataset for ×4 enlargement. The DMUN can achieve the best SR results among all the networks except for RCAN. Compared with EDSR and RDN, our proposed DMUN and DMUN+ achieve higher PSNR value, while the parameters are reduced by 77% and 55% respectively. Furthermore, our DMUN+ can achieve competitive results compared to RCAN, while only needs the 63% parameters of RCAN. Meanwhile, in comparison with the networks with parameters fewer than 10M, such as D-DBPN, MDSR and MSRN, our approach can achieve better performance with great advantages. This demonstrates our proposed

Results with BI degradation model
For BI degradation model, the proposed DMUN is compared with several state-of-the-art image SR approaches, including VDSR, [2] LapSRN, [6] MemNet, [5] EDSR, [4] SRMD, [25] D-DBPN, [12] CARN, [26] MSRN, [13] RCAN, [8] SDNND, [27] SRFBN [14] and HRAN. [28] Following, [8] we introduce self-ensemble strategy to further improve our DMUN performance and denote the self-ensembled one as DMUN+. Table 2 shows the quantitative comparisons of the performances over the benchmark datasets. PSNR and SSIM are calculated on the luminance channel of the transformed YCbCr space. As one can see, the proposed DMUN performs the best on all test datasets in most scaling factors by a large margin compared with all previous methods except for RCAN. The reason why our DMUN has no similar advantage over RCAN is that RCAN has 16M parameters while our method only has 10M parameters for training. As we all know, a deeper network allows the model to capture more feature information by using larger receptive field. Moreover, our DMUN+ can achieve comparable or even better results than those by RCAN on some test datasets. This also indicates the better effectiveness of our residual activation block (RAB) over residual channel attention block in RCAN.
Furthermore, even with some networks with equivalent parameters, our proposed DMUN can still achieve a large lead on most datasets. Especially on the dataset Urban100, which contains detailed urban scene images such as buildings with regular structures, our DMUN achieves higher PSNR than D-DBPN with an increase of 0.28dB under scale factor ×2 and 0.28dB under scale factor ×4. The results demonstrate that the multiple stacked local feature up-projection (LFU) units allow our network to extract more informative features and improve the performance.
The qualitatively visual comparisons of different methods are given in Figure 8 and Figure 9. Figure 8 shows visual comparisons on scale ×2. In image "Img_060", we observe that most of the compared methods fail to recover the fringes and suffer from blurry artefacts. But our DMUN can better mitigate the blur artefacts. In Figure 9, we show visual comparisons on scale ×4. It can also be seen from the SR reconstruction results that the proposed DMUN can restore clearer textures and sharper edges and achieve higher performance than other methods.

Results with blur-downscale (BD) degradation model
Following, [8] we further show the SR results with BD degradation model. We compare ×3 SR results with eight state-of-theart methods: SPMSR, [29] SRCNN, [1] VDSR, [2] IRCNN, [3] SRMDNF, [25] RDN, [7] RCAN [8] and SRFBN. [14] Table 3 shows the results of our methods and the compared methods in terms of PSNR and SSIM. As shown in Table 3, the proposed DMUN can achieve comparable results with RCAN, and can also achieve significant improvement compared with the remaining methods. Furthermore, our DMUN+ performs the best on all the test datasets with all scale factors compared with all previous methods. Fig. 10 shows visual comparisons for ×3 SR under the blurdown degradation model. As we can see, some methods cannot even reconstruct the image details, while others suffer from  slight blurring artefacts. In contrast, our proposed DMUN can accurately and clearly recover the texture patterns and structures of the image. This demonstrates that the proposed model is more efficient and also has the ability to restore images from the blur-down degradation model.

Super-resolving real-world image
We also carry out comparative SR experiments on the representative real-world LR benchmark, namely the "Chip" (with 109×55 pixels) image, which contains many edges. Different from the above experiments on BI and BD degradation models, the real-world image super-resolution often lacks the corresponding ground-truth HR image, and the degradation model is unknown either. So we only provide visual comparison. As shown in Figure 11, we compare our DMUN with several representative CNN-based methods: VDSR, [2] CARN, [26] MSRN [13] and RCAN. [8] Both our proposed DMUN and RCAN [8] can restore richer edge and detail information than other state-of-the-art methods. When comparing the magnified letters in the red insets, it also shows that our DMUN can produce image of better quality with minimal artefacts. The results with real-world image further demonstrate the effectiveness and robustness of our DMUN model.

CONCLUSIONS
In this paper, we propose a novel deep multi-level up-projection network (DMUN) for highly accurate image SR. The core idea behind our network is to fully mine the hierarchical features from the original low-resolution images and improve the representational ability of the network. Firstly, Multiple local feature up-projection (LFU) units are stacked to obtain feature information at different levels. In each LFU unit, RUGs use a recursive method to exploit LR feature information, which enables each RAB capture both shallow feature information and deep feature information. We also propose a residual re-correction mechanism, which adopts the high-level residual information from RUG to re-correct HR features to enrich the details of the output image. Secondly, in order to enhance the discriminative ability for different types of features, we improve the original residual block with the spatial-and-channel attention mechanism, which adaptively recalibrates features by considering the spatial relationships within channels as well as the pixel-wise interdependencies between channels simultaneously. Finally, we adopt the same DMUN structure to handle two degradation models. Extensive experiments and model analysis have demonstrated the superiority of our proposed DMUN in comparison with the state-of-the-art methods.