CN112699844B - Image super-resolution method based on multi-scale residual hierarchy close-coupled network - Google Patents

Image super-resolution method based on multi-scale residual hierarchy close-coupled network Download PDF

Info

Publication number
CN112699844B
CN112699844B CN202110043244.8A CN202110043244A CN112699844B CN 112699844 B CN112699844 B CN 112699844B CN 202110043244 A CN202110043244 A CN 202110043244A CN 112699844 B CN112699844 B CN 112699844B
Authority
CN
China
Prior art keywords
network model
connection
block
hierarchy
scale residual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110043244.8A
Other languages
Chinese (zh)
Other versions
CN112699844A (en
Inventor
严伊彤
刘闯闯
金龙存
彭新一
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Publication of CN112699844A publication Critical patent/CN112699844A/en
Application granted granted Critical
Publication of CN112699844B publication Critical patent/CN112699844B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/14—Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/08—Feature extraction

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Mathematics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses an image super-resolution method based on a multi-scale residual hierarchy close-connected network. The method comprises the following steps: creating a training data set and a verification data set; establishing a multi-scale residual hierarchy close-connected network model; initializing a multi-scale residual hierarchy close-coupled network model, determining a loss function, selecting an optimizer, and setting training parameters; training a multi-scale residual hierarchy close-connected network model, wherein each epoch uses a verification set to test the performance of the model to obtain a trained multi-scale residual hierarchy close-connected network model; acquiring a test data set, inputting the test data set into a trained multi-scale residual hierarchy close-connected network model to execute a test, and generating a super-resolution image; and calculating peak signal-to-noise ratio and structural similarity between the generated super-resolution image and the real high-resolution image.

Description

Image super-resolution method based on multi-scale residual hierarchy close-coupled network
Technical Field
The invention relates to the field of computer vision, in particular to an image super-resolution method based on a multi-scale residual hierarchy close-connected network.
Background
With advances in technology and equipment updates, smart phones, digital cameras, and other devices are very common in life, and people often use these imaging devices to record and share life, and high-definition images and videos can record more and clearer figures and landscapes. In addition, in the fields of remote sensing satellite images, traffic monitoring, security, medical treatment, military reconnaissance and the like, images and videos are important record carriers, and high-quality images and videos play a vital role. Meanwhile, in the high-order task of computer vision, high-quality images and videos are helpful for improving the performance of the task.
An image is an important information carrier, and image resolution is one of the important criteria for measuring image quality, and refers to the amount of information stored in an image, which is the number of pixels contained in an image per inch. The higher resolution image contains more pixels, provides more detail and texture information, and can meet the requirements in various fields. However, in real life, the quality of the obtained image is often not high due to physical limitations of the imaging device, insufficient shooting conditions, limitation of network bandwidth, possible information loss in network transmission, and the like. The super-resolution of the image mainly means that the resolution of the image is improved through a software technology, and the quality of the image is improved. This technique is simpler, more economical and easier to implement than enhancing image resolution by hardware means. Therefore, the research of advanced image super-resolution technology has important significance and wide application.
Image super-resolution refers to recovering a high-quality high-resolution image from a given low-resolution image. Image interpolation increases the size of an image through an interpolation method, and the reconstructed image often cannot meet the requirement although the reconstruction speed is high. In recent years, deep learning with strong learning ability is widely used in the field of computer vision, and has made great progress. Dong Chao et al in 2014 proposed that the paper srcan (Dong C, et al, "Image Super-Resolution Using Deep Convolutional Networks") first applied a deep convolutional neural network to an Image Super-resolution task, and utilized the neural network to learn a mapping relationship between a low-resolution Image and a high-resolution Image end to end, so as to obtain a better reconstruction effect and a faster reconstruction speed than a traditional learning-based method, such as sparse representation-based Image Super-resolution (Yang et al, "Image Super-resolution via sparse representation"). Subsequently, a number of convolutional neural network-based image super-Resolution methods, such as LapSRN (La, et al, "Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution"), EDSR (Lim B, et al, "Enhanced Deep Residual Networks for Single Image Super-Resolution"), RDN (Zhang et al, "Residual Dense Network for Image Super-Resolution"), have been proposed, and the reconstruction quality has been improved.
The scale information in the image is also important for the super resolution of the image, and different objects need different scale information; on the other hand, most image super-resolution network structures are underutilized for multi-level neural network features, different level features having different receptive fields and containing different information, these networks neglecting interactions between the level features.
According to the method, an image super-resolution method based on multi-scale residual hierarchy close connection is designed according to the characteristics of image super-resolution reconstruction and the characteristics of images, a two-layer close connection structure is used for extracting multi-level features, gradient propagation is enhanced, information propagation and reuse in a network are optimized, a multi-scale module is used for extracting information of different scales, a plurality of residual connection are used for stabilizing training of the network, and the reconstruction performance of the network is improved.
Disclosure of Invention
The invention provides an image super-resolution method based on a multi-scale residual hierarchy close-connected network for solving the technical problems, and relates to improvement of an image super-resolution reconstruction method and realization and application of the multi-scale residual hierarchy close-connected network in the field of image super-resolution.
The object of the invention is achieved by at least one of the following technical solutions.
The image super-resolution method based on the multi-scale residual hierarchy close-connected network comprises a training process and a testing process, and specifically comprises the following steps:
s1, manufacturing a training data set and a verification data set;
s2, establishing a multi-scale residual hierarchy close-connected network model;
s3, initializing the multi-scale residual hierarchy close-connected network model established in the step S2, determining a loss function, selecting an optimizer, and setting parameters for training the multi-scale residual hierarchy close-connected network model;
s4, training a multi-scale residual hierarchy close-connected network model, wherein each epoch uses a verification set to test the performance of the model, and a trained multi-scale residual hierarchy close-connected network model is obtained;
s5, acquiring a test data set, inputting the test data set into a trained multi-scale residual hierarchy close-connected network model to execute a test, and generating a super-resolution image;
s6, calculating peak signal-to-noise ratio and structural similarity between the generated super-resolution image and the real high-resolution image.
Further, in step S1, several 2K images of DIV2K are used to make pairs of low-resolution-high-resolution images as training data sets; using the dataset DIV2K dataset disclosed in the NTIRE2017 image super-resolution game, which contains 1000 high quality 2K resolution images that contain rich scenes, 800 for training the network, 100 for verifying the performance of the network, the remaining 100 for testing the reconstruction ability of the network (not disclosed); performing downsampling processing of a scaling multiple on the 2K image by adopting a bicubic kernel interpolation method to obtain a low-resolution image (LR), and forming a training sample pair with a real image (HR) for training a multi-scale residual hierarchy close-connected network model; because the original image is too large in size, the direct input into the network model for training can cause excessive calculation of the network model and slow down the training speed, so that the low-resolution image is randomly cut, and is cut into image blocks with the size of A multiplied by D, the corresponding high-resolution image is cut into Ar multiplied by Dr, wherein A and D respectively represent the height and the width of the image blocks, and r is an amplification factor; to enhance the diversity of data and the amount of expanded data in the training dataset, the low resolution image and the high resolution image in the training dataset are flipped and rotated, including 90 °, 180 °, and 270 °;
The verification data Set adopts Set5 and consists of 5 images, and is used for evaluating the reconstruction performance of the generated network in the network training process, thereby being beneficial to observing the convergence degree of the generated network model.
Further, in step S2, the multi-scale residual hierarchy close-coupled network model includes a shallow feature extraction module, a depth feature extraction module, and a reconstruction module;
the shallow feature extraction module comprises a convolution layer, and is used for converting a 3-channel input image into a 64-channel shallow feature map, as follows:
H 0 ＝F SF (I LR )；
wherein ,H0 Is a shallow feature, and I LR Is an input low resolution image, F SF Representing a shallow feature extraction module;
the deep feature extraction module comprises M multi-scale residual hierarchy secret connection blocks, a feature fusion block and global jump connection; multi-scale residual hierarchy secret connection block slave shallow layer characteristic H 0 The multi-scale and multi-layer deep features are extracted, the feature fusion block fuses the extracted multi-scale and multi-layer deep features, the global jump connection transmits shallow features to the tail part of a deep feature extraction module in the multi-scale residual hierarchy close connection network model, so that the main part of the network model, namely the multi-scale residual hierarchy close connection block, is focused on extracting high-frequency information, and meanwhile, the convergence rate of the network model is accelerated, and the method comprises the following steps of:
H DF ＝F DF (H 0 )＝H 0 +F GF ([H 1 ，H 2 ，…，H m ，…，H M ])；
wherein ,HDF Is an extracted deep layer feature, F DF Represents a deep feature extraction module, H m Is the extracted feature of the m-th multi-scale residual hierarchy secret connection block, F GF Representing feature fusion blocks, []Represents a series connection;
the reconstruction module includes an upsampling block and a convolution layerFor amplifying the extracted deep features by corresponding times and reconstructing the final high-resolution image I SR The method comprises the steps of carrying out a first treatment on the surface of the The up-sampling block adopts sub-pixel convolution, and is specifically as follows:
I SR ＝F REC (H DF )＝F conv (F up (H DF ))；
wherein ,FREC Representing the reconstruction part, F conv and Fup Representing the upsampled block and the convolutional layer, respectively.
Further, the multi-scale residual error level secret connection block comprises a level secret connection module, a memory unit, a multi-scale block and a local jump connection;
the hierarchical secret connection module is used for extracting multi-level characteristics, and consists of K secret connection blocks connected in a secret connection mode, and the hierarchical secret connection module is as follows:
Figure BDA0002896115370000031
wherein ,
Figure BDA0002896115370000032
is the extracted feature of the hierarchical dense connection module in the m-th multi-scale residual error hierarchical dense connection block,
Figure BDA0002896115370000033
hierarchical secret connection module representing mth multi-scale residual hierarchy secret connection block, < ->
Figure BDA0002896115370000034
Represents K secondary secret connection blocks in the hierarchical secret connection module, S k Is the extracted feature of the kth secondary secret connection block, [ · ] ]Representing a tandem operation;
the memory unit is used for integrating the characteristics extracted by the hierarchical secret connection module, extracting unified information in a self-adaptive mode, reducing the number of channels of the characteristic diagram, and therefore reducing the calculated amount of the subsequent hierarchical secret connection blocks in the multi-scale residual error hierarchical secret connection blocks, and the method is as follows:
Figure BDA0002896115370000035
wherein ,
Figure BDA0002896115370000036
memory cell +.>
Figure BDA0002896115370000037
Extracting characteristics;
the multi-scale block comprises an expansion space pyramid pooling structure and a jump connection; the expansion space pyramid pooling structure comprises parallel 1×1 convolution, three 3×3 convolution layers with expansion rates of 1, 2 and 4 respectively, and pooling layers, wherein one convolution layer fusing the 1×1 convolution layers of the feature graphs extracted by the convolution layers is connected with a jump for extracting the fused features
Figure BDA0002896115370000038
Features under different receptive fields, which is helpful for improving the reconstruction performance of the whole network; the jump connection is used for connecting the memory unit +.>
Figure BDA0002896115370000039
Extracted features->
Figure BDA00028961153700000310
The output connection with the pyramid pooling structure of the expansion space is beneficial to improving the efficiency and stability of the network; the following is shown:
Figure BDA00028961153700000311
wherein ,
Figure BDA00028961153700000312
representing the expansion space pyramid pooling structure in the mth multi-scale residual hierarchy secret joint block,
Figure BDA0002896115370000041
Is the feature extracted by the multi-scale block of the m-th multi-scale residual error level secret connection block;
feature H extracted from m-th multi-scale residual hierarchy secret connection block m The following is shown:
Figure BDA0002896115370000042
further, the secondary secret connection block is used for extracting local multi-level characteristics and comprises a characteristic compression block, a local secret connection group, a fusion block, input jump connection and compression jump connection;
the characteristic compression block compresses the channel number of the input characteristics of the secondary secret connection block so as to reduce the calculated amount of the local secret connection group; the convolution layer number of the local dense connection group is determined by the channel number k multiplied by G of the input characteristic of the secondary dense connection block, and G is the growth rate; the following is shown:
Figure BDA0002896115370000043
wherein ,
Figure BDA0002896115370000044
is the partial secret connection group in the kth secret connection block +.>
Figure BDA0002896115370000045
Extracted features, S BLC Is the compressed characteristic of the characteristic compression block, S k-1,d Is the feature extracted by the d convolution layer of the local dense connection group in the k secondary dense connection block;
the fusion block fuses and compresses the features extracted by the local dense connection group, and the features S extracted by the kth secondary dense connection block k The following is shown:
S k ＝S BLC ten F FB (S k-1 Ten S LDG )；
wherein ,FFB Representing a fusion block, the input jump connection extracting the features S of the kth-1 next-nearest connection block k-1 Transferring to a fusion block, wherein the compression jump connection compresses the characteristics S of the characteristics compression block BLC A tail portion transferred to the secondary seal connection layer; the input jump connection and the compression jump connection help to stabilize the training of the multi-scale residual hierarchy close-connected network model and improve the performance of the network model.
Further, in step S3, a kaiming gaussian initialization method is adopted to initialize the weight of the convolution layer of the multi-scale residual hierarchy close-connected network model; parameters for training the multi-scale residual hierarchy close-coupled network model include: specifying paths of a training data set and a verification data set, specifying a magnification factor r, inputting a batch data quantity B of a network model, an initial learning rate Lr_initial, the iteration number e of training the network model, and inputting a high-resolution image block size patch_size of the network model;
in order to reduce the computational complexity of the multi-scale residual hierarchy close-coupled network model, L is selected 1 Optimizing a multi-scale residual hierarchy close-coupled network model as a loss function; in the iterative training process, the loss function may generate oscillation, which indicates that the current learning rate is too large, and prevents the convergence of the network model, namely, the convergence curve oscillates near the extreme point, so that after E epochs are trained on the network, the learning rate is halved, the convergence of the network model is accelerated, and the performance of the network model is improved;
And selecting an ADAM optimizer to perform inverse gradient propagation on the multi-scale residual hierarchy close-connected network model, and updating model parameters.
Further, in step S4, a training set is given
Figure BDA0002896115370000046
L 1 The function definition is as follows:
Figure BDA0002896115370000051
wherein W, H is the length of the low resolution imageWidth, C is the number of channels, r is the amplification factor, F θ Representing a multi-scale residual hierarchy close-coupled network model, wherein θ represents a network parameter set, and optimizing in the whole network training process;
each epoch training verifies the model and downsamples the original high resolution image (HR) using the bicubic interpolation method to obtain a corresponding low resolution image (LR).
Further, in step S6, the super-resolution image I reconstructed by the multi-scale residual hierarchy close-connected network model is reconstructed SR And original high resolution image I HR Converting to YCbCr color space, calculating peak signal-to-noise ratio and structural similarity on a Y channel, and measuring the reconstruction quality of a multi-scale residual hierarchy close-connected network model;
the peak signal-to-noise ratio (PSNR) measures the quality of image reconstruction on global information, and the calculation formula is as follows:
Figure BDA0002896115370000052
Figure BDA0002896115370000053
wherein H, W is the length and width of the low-resolution image, r is the magnification factor, and X is the real high-resolution image, so as to generate a super-resolution image; MSE is mean square error, n is the number of bits per pixel; the unit of PSNR is decibel (dB), and the larger the value is, the smaller the distortion is, and the better the reconstruction quality is;
The Structural Similarity (SSIM) measures the structural similarity of images, and the structural similarity of test images and reference images is measured under the global statistical characteristics by means of the mean value and the variance, and the calculation formula is as follows:
Figure BDA0002896115370000054
wherein ,μx ,μ y Mean of the images x, y, respectively；σ x ,σ y Variance of images x, y, respectively; wherein mu x ,μ y The average value of the images x and y respectively; sigma (sigma) x ,σ y Variance of images x, y, respectively; c 1 ＝k 1 R，c 2 ＝k 2 R，k 1 ＝0.01，k 2 =0.03, r is the dynamic range of the pixel value, and the pixel value of the gray image is [0,255]；c 1 ,c 2 To prevent denominator from being 0;
the SSIM measures the similarity of the image structure in terms of brightness, contrast and structure, the value range is [0,1], the larger the value is, the more similar the two images are, the more dissimilar the two images are, and when the two images are completely identical, the SSIM value is 1.
Compared with the prior art, the invention has the advantages that:
the invention fully utilizes multi-scale information of an image and multi-level characteristic information of a deep convolutional neural network, and provides an image super-resolution method based on a multi-scale residual hierarchy close-connected network. Meanwhile, the network adopts a plurality of jump connections to stabilize training, and the performance of the network is improved.
Drawings
FIG. 1 is a training flow chart and a testing flow chart in an embodiment of the present invention;
FIG. 2 is a schematic diagram of the overall network structure of a multi-scale residual hierarchy tight junction network in an embodiment of the present invention;
FIG. 3 is a schematic diagram of a feature fusion block in an embodiment of the present invention;
FIG. 4 is a schematic diagram of an upsampling block structure in an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a multi-scale residual hierarchy interconnect block in an embodiment of the present invention;
FIG. 6 is a schematic diagram of a Multi-Scale Block structure in an embodiment of the invention;
FIG. 7 is a schematic diagram of a secondary-level secret Block (Sub-Dense Block) according to an embodiment of the present invention;
FIG. 8 is a graph of partial reconstruction effects in test dataset Urban100 and Manga109 for the present invention and other image super resolution methods under bicubic interpolation downsampling conditions (BI) at magnification of 4, where the reconstruction effect of the present invention is the best;
FIG. 9 is a graph of partial reconstruction effects in test datasets Urban100, manga109 and BSD100 under double-cube blurred downsampling conditions (BD) at a magnification of 3 for the present invention and other image super-resolution methods, where the reconstruction effect of the present invention is the best;
FIG. 10 is a graph of the trade-off between the model parameters and the reconstruction performance (peak signal to noise ratio, PSNR) at a magnification of 4 and Set5 for the present invention versus other image super-resolution methods, and the present invention achieves better reconstruction performance when the parameters are comparable.
Detailed Description
The invention provides an image super-resolution method based on a multi-scale residual hierarchy close-connected network, which utilizes a two-layer close-connected structure and multi-scale blocks to extract multi-layer and multi-scale feature images and can reconstruct high-resolution images efficiently.
Specific embodiments of the present invention will be described in further detail below with reference to examples and drawings, but embodiments of the present invention are not limited thereto.
Examples:
an image super-resolution method based on a multi-scale residual hierarchy close-coupled network, as shown in fig. 1, comprises a training process and a testing process, and specifically comprises the following steps:
s1, manufacturing a training data set and a verification data set;
in this embodiment, 800 2K images of DIV2K are used to make pairs of low-resolution-high-resolution images as training data sets; using the dataset DIV2K dataset disclosed in the NTIRE2017 image super-resolution game, which contains 1000 high quality 2K resolution images that contain rich scenes, 800 for training the network, 100 for verifying the performance of the network, the remaining 100 for testing the reconstruction ability of the network (not disclosed); performing downsampling processing of a scaling multiple on the 2K image by adopting a bicubic kernel interpolation method to obtain a low-resolution image (LR), and forming a training sample pair with a real image (HR) for training a multi-scale residual hierarchy close-connected network model; because the original image is too large in size, the direct input to the network model for training can cause excessive calculation of the network model and slow down the training speed, so that the low-resolution image is randomly cut, cut into image blocks with the size of A multiplied by D, the corresponding high-resolution image is cut into Ar multiplied by Dr, wherein A and D respectively represent the height and the width of the image blocks, in the embodiment, the image blocks with the size of 48 multiplied by 48 are cut into image blocks with the size of 48 multiplied by 48, and the corresponding high-resolution image is cut into 48 multiplied by 48, and r is an amplification factor; in order to enhance the diversity of data in the training data set and expand the data volume, in this embodiment, the low resolution image and the high resolution image in the training data set are flipped and rotated, including 90 °, 180 °, and 270 °;
In this embodiment, the verification data Set is Set5 and is composed of 5 images, and is used for evaluating the reconstruction performance of the generated network in the network training process, so as to facilitate observation of the convergence degree of the generated network model.
S2, establishing a multi-scale residual hierarchy close-connected network model;
as shown in fig. 2, the multi-scale residual hierarchy close-coupled network model includes a shallow feature extraction module, a depth feature extraction module, and a reconstruction module;
in this embodiment, the shallow feature extraction module includes a 3×3 convolution layer, which is configured to convert a 3-channel input image into a 64-channel shallow feature map, as follows:
H 0 ＝F SF (I LR )；
wherein ,H0 Is a shallow feature, and I LR Is an input low resolution image, F SF Representing a shallow feature extraction module;
the deep feature extraction module comprises M multi-scale residual hierarchy secret connection blocks, a feature fusion block and global jump connection, wherein M is set to be 10 in the embodiment; multi-scale residual hierarchy tie-inJunction from shallow features H 0 In this embodiment, as shown in fig. 3, the feature fusion block includes a serial layer (concat), a 1×1 convolution layer (Conv) and a ReLu activation layer, the feature fusion block fuses the extracted multi-scale and multi-level deep features, and the global jump connection transfers the shallow features to the tail of the deep feature extraction module in the multi-scale residual hierarchy close-connected network model, so that the main part of the network model, namely the multi-scale residual hierarchy close-connected block, is focused on extracting high-frequency information, and simultaneously speeds up the convergence rate of the network model, and the method is as follows:
H DF ＝F DF (H 0 )＝H 0 +F GF ([H 1 ，H 2 ，…，H m ，…，H M ])；
wherein ,HDF Is an extracted deep layer feature, F DF Represents a deep feature extraction module, H m Is the extracted feature of the m-th multi-scale residual hierarchy secret connection block, F G Representing feature fusion blocks, []Represents a series connection;
in this embodiment, the reconstruction module includes an upsampling block and a 3×3 convolution layer for performing corresponding multiple amplification on the extracted deep features and reconstructing the final high resolution image I SR The method comprises the steps of carrying out a first treatment on the surface of the The up-sampling block is composed of a 3×3 convolution Layer (Conv), a ReLu activation Layer, an up-sampling Layer (Upsampling Layer), a 3×3 convolution Layer (Conv), and a ReLu activation Layer, as shown in fig. 4, the up-sampling Layer adopts a sub-pixel Layer (sub-pixel Layer) for amplifying the extracted deep features by corresponding multiples and reconstructing the final high-resolution image I SR The method is characterized by comprising the following steps:
I SR ＝F REC (H DF )＝F conv (F up (H DF ))；
wherein ,FREC Representing the reconstruction part, F conv and Fup Representing the upsampled block and the convolutional layer, respectively.
As shown in fig. 5, the multi-scale residual hierarchy secret connection block comprises a hierarchy secret connection module, a memory unit, a multi-scale block and a local jump connection;
the hierarchical secret connection module is used for extracting multi-level characteristics, and is formed by connecting K secret connection blocks in a secret connection mode, in the embodiment, K is set to be 5, and the method is as follows:
Figure BDA0002896115370000071
wherein ,
Figure BDA0002896115370000072
is the extracted feature of the hierarchical dense connection module in the m-th multi-scale residual error hierarchical dense connection block,
Figure BDA0002896115370000073
hierarchical secret connection module representing mth multi-scale residual hierarchy secret connection block, < ->
Figure BDA0002896115370000074
Represents K secondary secret connection blocks in the hierarchical secret connection module, S k Is the extracted feature of the kth secondary secret connection block, [ · ]]Representing a tandem operation;
the memory unit is used for integrating the characteristics extracted by the hierarchical secret connection module, extracting unified information in a self-adaptive mode, reducing the number of channels of the characteristic diagram, and therefore reducing the calculated amount of the subsequent hierarchical secret connection blocks in the multi-scale residual error hierarchical secret connection blocks, and the method is as follows:
Figure BDA0002896115370000081
wherein ,
Figure BDA0002896115370000082
memory cell +.>
Figure BDA0002896115370000083
Extracting characteristics;
in this embodiment, the memory unit is composed of a 1×1 convolution layer, the feature images extracted from the previous sub-dense connection blocks are first subjected to a series operation, and feature compression is performed by 1×1 convolution to obtain a feature image with a size of w×h×g, where W, H is the length and width of the low resolution image block, and G is the growth rate (growth rate), and in this embodiment, G is set to 64.
The multi-scale block comprises an expansion space pyramid pooling structure and a jump connection; in this embodiment, as shown in fig. 6, the expansion space pyramid pooling structure includes parallel 1×1 convolution, three 3×3 convolution layers with expansion rates of 1, 2, and 4, and pooling layers, and one 1×1 convolution layer fusing the feature graphs extracted by the convolution layers and jump connection for extracting fused features
Figure BDA0002896115370000084
Features under different receptive fields, which is helpful for improving the reconstruction performance of the whole network; the jump connection is used for connecting the memory unit +.>
Figure BDA0002896115370000085
Extracted features->
Figure BDA0002896115370000086
The output connection with the pyramid pooling structure of the expansion space is beneficial to improving the efficiency and stability of the network; the following is shown:
Figure BDA0002896115370000087
wherein ,
Figure BDA0002896115370000088
representing the expansion space pyramid pooling structure in the mth multi-scale residual hierarchy secret joint block,
Figure BDA0002896115370000089
is the feature extracted by the multi-scale block of the m-th multi-scale residual error level secret connection block;
mth multi-scale residual hierarchyCharacteristic H extracted from the close-coupled block m The following is shown:
Figure BDA00028961153700000810
as shown in fig. 7, the secondary secret connection block is used for extracting local multi-level features, including a feature compression block, a local secret connection group, a fusion block, an input jump connection and a compression jump connection;
in this embodiment, the feature compression block is composed of a 3×3 convolution layer and ReLu, and performs channel number compression on the input features of the secondary secret connection block, so as to reduce the calculation amount of the local secret connection group; the convolution layer number of the local close connection group is determined by the channel number k multiplied by G of the input characteristic of the secondary close connection block, G is the growth rate, and is set to be 64, and the local close connection group consists of k-1 convolution blocks, wherein the local close connection group comprises a series connection, a 3 multiplied by 3 convolution layer and a ReLu; the following is shown:
Figure BDA00028961153700000811
wherein ,
Figure BDA00028961153700000812
is the partial secret connection group in the kth secret connection block +.>
Figure BDA00028961153700000813
Extracted features, S BLC Is the compressed characteristic of the characteristic compression block, S k-1,d Is the feature extracted by the d convolution layer of the local dense connection group in the k secondary dense connection block;
in this embodiment, the fusion block is a 1×1 convolution layer, and fuses and compresses the features extracted from the local dense connection group, and the features S extracted from the kth dense connection block k The following is shown:
S k ＝S BLC ten F FB (S k-1 Ten S LDG )；
wherein ,FFB Representation fusionA block for inputting the features S extracted by the jump connection of the kth-1 sub-secret connection block k-1 Transferring to a fusion block, wherein the compression jump connection compresses the characteristics S of the characteristics compression block BLC A tail portion transferred to the secondary seal connection layer; the input jump connection and the compression jump connection help to stabilize the training of the multi-scale residual hierarchy close-connected network model and improve the performance of the network model.
S3, initializing the multi-scale residual hierarchy close-connected network model established in the step S2, determining a loss function, selecting an optimizer, and setting parameters for training the multi-scale residual hierarchy close-connected network model;
in the embodiment, a kaiming Gaussian initialization method is adopted to initialize the weight of a convolution layer of a multi-scale residual hierarchy close-connected network model; parameters for training the multi-scale residual hierarchy close-coupled network model include: specifying paths of a training data set and a verification data set, specifying a magnification factor r, inputting a batch data quantity B of a network model, an initial learning rate Lr_initial, the iteration number e of training the network model, and inputting a high-resolution image block size patch_size of the network model;
In the embodiment, 2 times, 3 times, 4 times and 8 times of multi-scale residual hierarchy close-connected network models are trained respectively, so that the amplification factors are 2, 3, 4 and 8 respectively; the lot data amount B input to the network model is set to 16; the initial learning rate Lr_initial is set to 1×10 -4 The method comprises the steps of carrying out a first treatment on the surface of the Setting the iteration times of the network training as 1000 epochs; the low resolution image block size input to the network model is 48×48, and the high resolution image block size is 48r×48r, so when the magnification factor is 4, its corresponding high resolution image block size patch_size is set to 192.
The loss function commonly used in the task of image super resolution is L 1 、L 2 In order to reduce the computation complexity of the multi-scale residual hierarchy close-connected network model, L is selected for perception loss, countermeasures loss and the like 1 Optimizing a multi-scale residual hierarchy close-coupled network model as a loss function; in the iterative training process, the loss function may oscillate, which indicates that the current learning rate is too large, and prevents the convergence, i.e. the recovery, of the network modelThe converging curve oscillates near the extreme point, so after training E epochs on the network, the learning rate is halved, the convergence of the network model is accelerated, and the performance of the network model is improved, in the embodiment, E is set to be 200;
Selecting an ADAM optimizer to perform inverse gradient propagation on a multi-scale residual hierarchy close-connected network model, and updating model parameters, wherein in the embodiment, the parameters of the ADAM are set as follows: beta 1 ＝0.9，β 2 =0.999 and ε=10 -8 。
S4, training a multi-scale residual hierarchy close-connected network model, wherein each epoch uses a verification set to test the performance of the model, and a trained multi-scale residual hierarchy close-connected network model is obtained;
in this embodiment, the multi-scale residual hierarchy close-coupled network model is trained for 1000 cycles.
Given training set
Figure BDA0002896115370000091
L 1 The function definition is as follows:
Figure BDA0002896115370000092
wherein W, H is the length and width of the low resolution image, C is the number of channels, r is the magnification factor, F θ Representing a multi-scale residual hierarchy close-coupled network model, wherein θ represents a network parameter set, and optimizing in the whole network training process;
each epoch training verifies the model and downsamples the original high resolution image (HR) using the bicubic interpolation method to obtain a corresponding low resolution image (LR).
S5, acquiring a test data set, inputting the test data set into a trained multi-scale residual hierarchy close-connected network model to execute a test, and generating a super-resolution image;
in this embodiment, five standard test data sets are used to verify the effect of the image super-resolution model. The five test sets were: set5, set14, BSD100, urban100, and Manga109.Set5, set14, BSD100 are a collection of some natural images; urban100 is a collection of 100 city images with high frequency information; manga109 is a collection of 109 Japanese comic images. These datasets are widely used in a variety of super-resolution model validations, with excellent representativeness and convincing. And firstly, carrying out downsampling operation on the high-resolution image of the data set to obtain a corresponding low-resolution image. Low resolution images that need to be magnified during production and life can also be acquired as input to the test.
A test operation is performed, generating a super resolution image (SR).
S6, calculating peak signal-to-noise ratio and structural similarity between the generated super-resolution image and the real high-resolution image;
super-resolution image I reconstructed by multi-scale residual hierarchy close-coupled network model SR And original high resolution image I HR Converting to YCbCr color space, calculating peak signal-to-noise ratio and structural similarity on a Y channel, and measuring the reconstruction quality of a multi-scale residual hierarchy close-connected network model;
the peak signal-to-noise ratio (PSNR) measures the quality of image reconstruction on global information, and the calculation formula is as follows:
Figure BDA0002896115370000101
Figure BDA0002896115370000102
wherein H, W is the length and width of the low-resolution image, r is the magnification factor, X is the real image, and is the generated super-resolution image; MSE is mean square error, n is the number of bits per pixel, e.g. 8, 16, because the pixel of the gray scale image has a value range of [0,255], so in this example n is 8; the unit of PSNR is decibel (dB), and the larger the value is, the smaller the distortion is, and the better the reconstruction quality is;
the Structural Similarity (SSIM) measures the structural similarity of images, and the structural similarity of test images and reference images is measured under the global statistical characteristics by means of the mean value and the variance, and the calculation formula is as follows:
Figure BDA0002896115370000103
wherein ,μx ,μ y The average value of the images x and y respectively; sigma (sigma) x ,σ y Variance of images x, y, respectively; wherein mu x ,μ y The average value of the images x and y respectively; sigma (sigma) x ,σ y Variance of images x, y, respectively; c 1 ＝k 1 R，c 2 ＝k 2 R，k 1 ＝0.01，k 2 =0.03, r is the dynamic range of the pixel value, and the pixel value of the gray image is [0,255]R in this example is 255; c 1 ,c 2 To prevent denominator from being 0;
the SSIM measures the similarity of the image structure in terms of brightness, contrast and structure, the value range is [0,1], the larger the value is, the more similar the two images are, the more dissimilar the two images are, and when the two images are completely identical, the SSIM value is 1.
It can be seen from tables 1 and 2 that the results of the present invention (MS-RHDN) and its self-integration (MS-rhdn+) are better than the previous results in both the double cube downsampling and the fuzzy downsampling cases, indicating the effectiveness of the super-resolution reconstruction of the image of the present invention. Fig. 8 and 9 are visual effects to be compared, and the method (MS-RHDN) according to the present invention has better recovery effect than other methods.
Table 1 comparison of quantitative experimental results under the condition of double cubic fuzzy downsampling (BD)
Figure BDA0002896115370000111
In table 1, the data shown in bold are the best results, and the underlined data are the second best results.
Table 2 comparison table of quantitative experimental results under double cubic downsampling conditions
Figure BDA0002896115370000112
Figure BDA0002896115370000121
/>
In table 2, the data shown in bold are the best results, and the data shown in underline are the second best results.
If low resolution images acquired from other sources are used as input for the test, only the generated super resolution images (SRs) are saved, and PSNR and SSIM are not calculated, because these low resolution images do not have corresponding high resolution images, and calculation of PNSR and SSIM requires reference images (high resolution images).
FIG. 10 is a tradeoff between parametric and model performance of the method of the present invention and other image super-resolution methods. From the figures, it can be seen that the method of the present invention achieves the best reconstruction performance with a small amount of parameters, demonstrating the effectiveness of the present invention.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.

Claims (1)

1. The image super-resolution method based on the multi-scale residual hierarchy close-connected network is characterized by comprising a training process and a testing process, and specifically comprises the following steps of:
S1, manufacturing a training data set and a verification data set;
using 800 2K images of DIV2K to make pairs of low-resolution-high-resolution images as training data sets; using the dataset DIV2K dataset disclosed in the NTIRE2017 image super-resolution game, which contains 1000 high quality 2K resolution images that contain rich scenes, 800 for training the network, 100 for verifying the performance of the network, the remaining 100 for testing the reconstruction ability of the network; performing downsampling processing of a scaling multiple on the 2K image by adopting a bicubic kernel interpolation method to obtain a low-resolution image (LR), and forming a training sample pair with a real image (HR) for training a multi-scale residual hierarchy close-connected network model; because the original image is too large in size, the direct input into the network model for training can cause excessive calculation of the network model and slow down the training speed, so that the low-resolution image is randomly cut, cut into image blocks with the size of A multiplied by D, the corresponding high-resolution image is cut into Ar multiplied by Dr, wherein A and D respectively represent the height and the width of the image blocks, cut into image blocks with the size of 48 multiplied by 48, the corresponding high-resolution image is cut into 48 multiplied by 48, and r is an amplification factor; to enhance the diversity of data and the amount of expanded data in the training dataset, the low resolution image and the high resolution image in the training dataset are flipped and rotated, including 90 °, 180 °, and 270 °;
The verification data Set adopts Set5 and consists of 5 images, and is used for evaluating the reconstruction performance of the generated network in the network training process, so that the convergence degree of the generated network model is observed;
s2, establishing a multi-scale residual hierarchy close-connected network model;
the multi-scale residual hierarchy close-coupled network model comprises a shallow layer feature extraction module, a deep layer feature extraction module and a reconstruction module;
the shallow feature extraction module comprises a 3×3 convolution layer, and is configured to convert a 3-channel input image into a 64-channel shallow feature map, as follows:
H 0 ＝F SF (I LR )；
wherein ,H0 Is a shallow feature, and I LR Is an input low resolution image, F SF Representing a shallow feature extraction module;
the deep feature extraction module comprises M multi-scale residual hierarchy secret connection blocks, a feature fusion block and global jump connection, wherein M is set to be 10; multi-scale residual hierarchy secret connection block slave shallow layer characteristic H 0 The feature fusion block is used for fusing the extracted multi-scale and multi-level deep features, and the overall jump connection is used for transmitting the shallow features to the tail of a deep feature extraction module in the multi-scale residual hierarchy close connection network model, so that the main part of the network model, namely the multi-scale residual hierarchy close connection block, is focused on extracting high-frequency information, and meanwhile, the convergence rate of the network model is accelerated, and the method comprises the following steps of:
H DF ＝F DF (H 0 )＝H 0 +F GF ([H 1 ，H 2 ，…，H m ，…，H M ])；
wherein ,HDF Is an extracted deep layer feature, F DF Represents a deep feature extraction module, H m Is the extracted feature of the m-th multi-scale residual hierarchy secret connection block, F GF Representing feature fusion blocks, []Represents a series connection;
the reconstruction module comprises an up-sampling block and a 3×3 convolution layer for amplifying the extracted deep features by corresponding times and reconstructing the final high-resolution image I SR The method comprises the steps of carrying out a first treatment on the surface of the The up-sampling block consists of a 3×3 convolution Layer (Conv), a ReLu activation Layer, an up-sampling Layer (Upsampling Layer), a 3×3 convolution Layer (Conv) and a ReLu activation Layer, wherein the up-sampling Layer adopts a sub-pixel Layer (sub-pixel Layer) for amplifying the extracted deep features by corresponding multiples and reconstructing a final high-resolution image I SR The method is characterized by comprising the following steps:
I SR ＝F REC (H DF )＝F conv (F up (H DF ))；
wherein ,FREC Representing the reconstruction part, F conv and Fup Representing the upsampled block and the convolutional layer, respectively;
the multi-scale residual error level secret connection block comprises a level secret connection module, a memory unit, a multi-scale block and local jump connection;
the hierarchical secret connection module is used for extracting multi-level characteristics, and is formed by connecting K secret connection blocks in a secret connection mode, wherein K is set to be 5, and the method is as follows:
Figure QLYQS_1
wherein ,
Figure QLYQS_2
is the extracted feature of the hierarchical dense connection module in the mth multi-scale residual hierarchy dense connection block,/- >
Figure QLYQS_3
Hierarchical secret connection module representing mth multi-scale residual hierarchy secret connection block, < ->
Figure QLYQS_4
Represents K secondary secret connection blocks in the hierarchical secret connection module, S k Is the extracted feature of the kth secondary secret connection block, [ · ]]Representing a tandem operation;
the memory unit is used for integrating the characteristics extracted by the hierarchical secret connection module, extracting unified information in a self-adaptive mode, reducing the number of channels of the characteristic diagram, and therefore reducing the calculated amount of the subsequent hierarchical secret connection blocks in the multi-scale residual error hierarchical secret connection blocks, and the method is as follows:
Figure QLYQS_5
wherein ,
Figure QLYQS_6
memory cell +.>
Figure QLYQS_7
Extracting characteristics;
the memory unit is composed of a 1X 1 convolution layer, firstly, the characteristic diagram extracted from the previous secondary-secret connection block is subjected to series operation, and characteristic compression is carried out through 1X 1 convolution to obtain a characteristic diagram with the size of W X H X G, wherein W, H is the length and the width of a low-resolution image block, G is the growth rate, and G is set to 64;
the multi-scale block comprises an expansion space pyramid pooling structure and a jump connection; the expansion space pyramid pooling structure comprises parallel 1×1 convolution, three 3×3 convolution layers with expansion rates of 1, 2 and 4 respectively, and pooling layers, wherein one convolution layer fusing the 1×1 convolution layers of the feature graphs extracted by the convolution layers is connected with a jump for extracting the fused features
Figure QLYQS_8
Features under different receptive fields, which is helpful for improving the reconstruction performance of the whole network; the jump connection is used for connecting the memory unit +.>
Figure QLYQS_9
Extracted features
Figure QLYQS_10
The output connection with the pyramid pooling structure of the expansion space is beneficial to improving the efficiency and stability of the network; the following is shown:
Figure QLYQS_11
wherein ,
Figure QLYQS_12
representing the expansion space pyramid pooling structure in the mth multi-scale residual hierarchy secret connection block,/->
Figure QLYQS_13
Is the feature extracted by the multi-scale block of the m-th multi-scale residual error level secret connection block;
feature H extracted from m-th multi-scale residual hierarchy secret connection block m The following is shown:
Figure QLYQS_14
the secondary secret connection block is used for extracting local multi-level characteristics and comprises a characteristic compression block, a local secret connection group, a fusion block, an input jump connection and a compression jump connection;
the characteristic compression block is composed of a 3X 3 convolution layer and ReLu, and the characteristic compression block compresses the channel number of the input characteristics of the secondary secret connection block so as to reduce the calculated amount of the local secret connection group; the convolution layer number of the local close connection group is determined by the channel number k multiplied by G of the input characteristic of the secondary close connection block, G is the growth rate, and is set to be 64, and the local close connection group consists of k-1 convolution blocks, wherein the local close connection group comprises a series connection, a 3 multiplied by 3 convolution layer and a ReLu; the following is shown:
Figure QLYQS_15
wherein ,
Figure QLYQS_16
is the partial secret connection group in the kth secret connection block +.>
Figure QLYQS_17
Extracted features, S BLC Is the compressed characteristic of the characteristic compression block, S k-1，d Is the feature extracted by the d convolution layer of the local dense connection group in the k secondary dense connection block;
the fusion block is a 1 multiplied by 1 convolution layer, and is used for fusing and compressing the characteristics extracted by the local dense connection group and the characteristics S extracted by the kth secondary dense connection block k The following is shown:
S k ＝S BLC +F FB (S k-1 +S LDG )；
wherein ,FFB Representing a fusion block, the input jump connection extracting the features S of the kth-1 next-nearest connection block k-1 Transferring to a fusion block, wherein the compression jump connection compresses the characteristics S of the characteristics compression block BLC A tail portion transferred to the secondary seal connection layer; input jump connection and compression jump connection stable multi-scale residual hierarchy close connection networkTraining a network model and improving the performance of the network model;
s3, initializing the multi-scale residual hierarchy close-connected network model established in the step S2, determining a loss function, selecting an optimizer, and setting parameters for training the multi-scale residual hierarchy close-connected network model;
initializing a convolution layer weight of a multi-scale residual hierarchy close-connected network model by adopting a kaiming Gaussian initialization method; parameters for training the multi-scale residual hierarchy close-coupled network model include: specifying paths of a training data set and a verification data set, specifying a magnification factor r, inputting a batch data quantity B of a network model, an initial learning rate Lr_initial, the iteration number e of training the network model, and inputting a high-resolution image block size patch_size of the network model;
2 times, 3 times, 4 times and 8 times of multi-scale residual hierarchy close-connected network models are trained respectively, so that the amplification factors are 2, 3, 4 and 8 respectively; the lot data amount B input to the network model is set to 16; the initial learning rate Lr_initial is set to 1×10 -4 The method comprises the steps of carrying out a first treatment on the surface of the Setting the iteration times of the network training as 1000 epochs; the low resolution image block size input to the network model is 48×48, and the high resolution image block size is 48r×48r, so when the magnification factor is 4, its corresponding high resolution image block size patch_size is set to 192;
the loss function commonly used in the task of image super resolution is L 1 、L 2 Perception loss and countermeasures loss, L is selected for reducing the computation complexity of the multi-scale residual hierarchy close-connected network model 1 Optimizing a multi-scale residual hierarchy close-coupled network model as a loss function; in the iterative training process, the loss function may generate oscillation, which indicates that the current learning rate is too large, so as to prevent the convergence of the network model, namely, the convergence curve oscillates near the extreme point, after E epochs are trained on the network, the learning rate is halved, the convergence of the network model is accelerated, the performance of the network model is improved, and E is set to be 200;
selecting an ADAM optimizer to carry out inverse gradient propagation on a multi-scale residual hierarchy close-connected network model, updating model parameters, wherein the ADAM parameters are set as follows: beta 1 ＝0.9，β 2 =0.999 and ε=10 -8 ；
S4, training a multi-scale residual hierarchy close-connected network model, wherein each epoch uses a verification set to test the performance of the model, and a trained multi-scale residual hierarchy close-connected network model is obtained;
training the multi-scale residual hierarchy close-connected network model for 1000 cycles;
given training set
Figure QLYQS_18
L 1 The function definition is as follows:
Figure QLYQS_19
wherein W, H is the length and width of the low resolution image, C is the number of channels, r is the magnification factor, F θ Representing a multi-scale residual hierarchy close-coupled network model, wherein θ represents a network parameter set, and optimizing in the whole network training process;
each epoch training verifies the model, and uses a bicubic interpolation method to downsample the original high-resolution image (HR) to obtain a corresponding low-resolution image (LR);
s5, acquiring a test data set, inputting the test data set into a trained multi-scale residual hierarchy close-connected network model to execute a test, and generating a super-resolution image;
five standard test data sets are adopted to verify the effect of the image super-resolution model, and the five test sets are as follows: set5, set14, BSD100, urban100 and Manga109, set5, set14, BSD100 are a collection of some natural images; urban100 is a collection of 100 city images with high frequency information; manga109 is a collection of 109 Japanese comic images; these datasets are widely used in a variety of super-resolution model validations by first performing a downsampling operation on a high-resolution image of the dataset to obtain a corresponding low-resolution image; the low-resolution images which need to be amplified in production and life can be obtained as the input of the test;
Performing a test operation to generate a super resolution image (SR);
s6, calculating peak signal-to-noise ratio and structural similarity between the generated super-resolution image and the real high-resolution image;
super-resolution image I reconstructed by multi-scale residual hierarchy close-coupled network model SR And original high resolution image I HR Converting to YCbCr color space, calculating peak signal-to-noise ratio and structural similarity on a Y channel, and measuring the reconstruction quality of a multi-scale residual hierarchy close-connected network model;
the peak signal-to-noise ratio (PSNR) measures the quality of image reconstruction on global information, and the calculation formula is as follows:
Figure QLYQS_20
Figure QLYQS_21
wherein H, W is the length and width of the low-resolution image, r is the magnification factor, X is the real image, and is the generated super-resolution image; MSE is mean square error, n is bit number of each pixel, n is 8; the unit of PSNR is decibel (dB), and the larger the value is, the smaller the distortion is, and the better the reconstruction quality is;
the Structural Similarity (SSIM) measures the structural similarity of images, and the structural similarity of test images and reference images is measured under the global statistical characteristics by means of the mean value and the variance, and the calculation formula is as follows:
Figure QLYQS_22
wherein ,μx ，μ y The average value of the images x and y respectively; sigma (sigma) x ，σ y Variance of images x, y, respectively; wherein mu x ，μ y The average value of the images x and y respectively; sigma (sigma) x ，σ y Variance of images x, y, respectively; c 1 ＝k 1 R，c 2 ＝k 2 R，k 1 ＝0.01，k 2 =0.03, r is the dynamic range of the pixel value, and the pixel value of the gray image ranges from [0, 255]R in this example is 255; c 1 ，c 2 To prevent denominator from being 0;
the sSIM measures the similarity of the image structure in terms of brightness, contrast and structure, the value range is [0,1], the larger the value is, the more similar the two images are, the more dissimilar the two images are, and when the two images are completely identical, the SSIM value is 1.
CN202110043244.8A 2020-04-23 2021-01-13 Image super-resolution method based on multi-scale residual hierarchy close-coupled network Active CN112699844B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010328982 2020-04-23
CN2020103289822 2020-04-23

Publications (2)

Publication Number Publication Date
CN112699844A CN112699844A (en) 2021-04-23
CN112699844B true CN112699844B (en) 2023-06-20

Family

ID=75514393

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110043244.8A Active CN112699844B (en) 2020-04-23 2021-01-13 Image super-resolution method based on multi-scale residual hierarchy close-coupled network

Country Status (1)

Country Link
CN (1) CN112699844B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113066013B (en) * 2021-05-18 2023-02-10 广东奥普特科技股份有限公司 Method, system, device and storage medium for generating visual image enhancement
CN113139907B (en) * 2021-05-18 2023-02-14 广东奥普特科技股份有限公司 Generation method, system, device and storage medium for visual resolution enhancement
CN113344997B (en) * 2021-06-11 2022-07-26 方天圣华(北京)数字科技有限公司 Method and system for rapidly acquiring high-definition foreground image only containing target object
CN113421188B (en) * 2021-06-18 2024-01-05 广东奥普特科技股份有限公司 Method, system, device and storage medium for image equalization enhancement
CN114066727A (en) * 2021-07-28 2022-02-18 华侨大学 Multi-stage progressive image super-resolution method

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110675321A (en) * 2019-09-26 2020-01-10 兰州理工大学 Super-resolution image reconstruction method based on progressive depth residual error network

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018053340A1 (en) * 2016-09-15 2018-03-22 Twitter, Inc. Super resolution using a generative adversarial network

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110675321A (en) * 2019-09-26 2020-01-10 兰州理工大学 Super-resolution image reconstruction method based on progressive depth residual error network

Also Published As

Publication number Publication date
CN112699844A (en) 2021-04-23

Similar Documents

Publication Publication Date Title
CN112699844B (en) Image super-resolution method based on multi-scale residual hierarchy close-coupled network
CN111583109B (en) Image super-resolution method based on generation of countermeasure network
CN113362223B (en) Image super-resolution reconstruction method based on attention mechanism and two-channel network
CN110599401A (en) Remote sensing image super-resolution reconstruction method, processing device and readable storage medium
CN111353940B (en) Image super-resolution reconstruction method based on deep learning iterative up-down sampling
CN111192200A (en) Image super-resolution reconstruction method based on fusion attention mechanism residual error network
CN111598778B (en) Super-resolution reconstruction method for insulator image
CN111462013B (en) Single-image rain removing method based on structured residual learning
CN112150521B (en) Image stereo matching method based on PSMNet optimization
CN111932461B (en) Self-learning image super-resolution reconstruction method and system based on convolutional neural network
CN112507997A (en) Face super-resolution system based on multi-scale convolution and receptive field feature fusion
CN110675321A (en) Super-resolution image reconstruction method based on progressive depth residual error network
CN108921783B (en) Satellite image super-resolution reconstruction method based on mixed loss function constraint
CN111861884B (en) Satellite cloud image super-resolution reconstruction method based on deep learning
CN110363068B (en) High-resolution pedestrian image generation method based on multiscale circulation generation type countermeasure network
CN110517195B (en) Unsupervised SAR image denoising method
CN110288524B (en) Deep learning super-resolution method based on enhanced upsampling and discrimination fusion mechanism
CN113538234A (en) Remote sensing image super-resolution reconstruction method based on lightweight generation model
CN113034361B (en) Remote sensing image super-resolution reconstruction method based on improved ESRGAN
CN111127325A (en) Satellite video super-resolution reconstruction method and system based on cyclic neural network
CN115393186A (en) Face image super-resolution reconstruction method, system, device and medium
CN113538246A (en) Remote sensing image super-resolution reconstruction method based on unsupervised multi-stage fusion network
CN114494047B (en) Biological image denoising method based on dual-enhancement residual error network
CN115526779A (en) Infrared image super-resolution reconstruction method based on dynamic attention mechanism
CN109272450A (en) A kind of image oversubscription method based on convolutional neural networks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant