CN111932461B - Self-learning image super-resolution reconstruction method and system based on convolutional neural network - Google Patents
Self-learning image super-resolution reconstruction method and system based on convolutional neural network Download PDFInfo
- Publication number
- CN111932461B CN111932461B CN202010802461.6A CN202010802461A CN111932461B CN 111932461 B CN111932461 B CN 111932461B CN 202010802461 A CN202010802461 A CN 202010802461A CN 111932461 B CN111932461 B CN 111932461B
- Authority
- CN
- China
- Prior art keywords
- image
- neural network
- convolutional neural
- reconstructed
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4046—Scaling the whole image or part thereof using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a self-learning image super-resolution reconstruction method and a self-learning image super-resolution reconstruction system based on a convolutional neural network, wherein the method comprises the following steps: step 1, obtaining a training sample of an image to be reconstructed; step 2, constructing a convolutional neural network; the convolutional neural network includes: the device comprises a feature extraction unit, a feature enhancement unit, a residual error unit and a reconstruction unit; step 3, training the convolutional neural network constructed in the step 2 based on the training sample obtained in the step 1 to obtain a trained reconstructed convolutional neural network; and 4, performing super-resolution reconstruction on the image to be reconstructed based on the reconstructed convolutional neural network trained in the step 3. The method can not only effectively solve the problem of insufficient training samples of the self-learning algorithm, but also avoid the phenomenon of over-fitting of the network; meanwhile, a high-resolution image with higher peak signal-to-noise ratio and better visual effect can be obtained.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a self-learning image super-resolution reconstruction method and system based on a convolutional neural network.
Background
With the rapid development of social intelligence and informatization, images have become an important way for human beings to acquire information, and have very important application values in the fields of monitoring equipment, satellite image remote sensing, video restoration, medical images and the like. Because of the high pixel density, the high resolution image can provide more important detail information for digital image processing, however, due to the limitations of imaging equipment, illumination and other conditions, the resolution of the acquired image is often lower. How to effectively improve the quality of an imaged image becomes a critical and important task for image processing. Image super-resolution reconstruction technology is one of the main means for improving the image resolution at present.
From the viewpoint of the image super-resolution algorithm model, the existing algorithms are divided into three types: interpolation-based, reconstruction-based, and learning-based. Interpolation-based algorithms are most widely used, and reconstruction-based and learning-based algorithms tend to combine interpolation-based algorithms. The basic idea of the image super-resolution algorithm based on reconstruction is to reconstruct a high-resolution image by using the inverse process of a degradation model, and total variation regularization is a popular algorithm in the model based on reconstruction, however, a large amount of artifacts are formed at the edge by the traditional total variation regularization algorithm, so that the visual quality of the high-resolution image is seriously affected. Learning-based algorithms can be divided into two categories: external learning and self-learning. The external learning algorithm contains two non-negligible disadvantages: first, such algorithms take a significant amount of time to model train; second, parameters derived from one magnification training are often only applicable to this magnification, requiring training of multiple sets of different parameters for other magnifications. The self-learning algorithm does not need to depend on an external database, and can complete learning and training processes by utilizing the information of the image, so that the defects of the external learning algorithm are effectively avoided.
At present, most of better-effect image super-resolution uses a convolutional neural network to perform feature extraction and image reconstruction. Shcher et al successfully combined the convolutional neural network with the self-learning algorithm, and realized super-resolution reconstruction of the image by using a single image, and obtained a better image reconstruction effect. Because the image blocks in a single image have structural self-similarity, the characteristic can provide a certain number of samples for the self-learning algorithm, but the self-learning algorithm also has a certain defect, firstly, the training samples are insufficient, secondly, the problem of image overfitting can be caused by the samples generated by utilizing the multi-scale self-similarity of the images, and therefore, how to construct a convolution network suitable for the self-learning algorithm is the problem which is mainly solved by the invention.
In summary, the existing self-learning image super-resolution invention capable of obtaining high-resolution images generally has the problems of insufficient training samples and easy network overfitting, and a new self-learning image super-resolution reconstruction method based on a convolutional neural network is needed.
Disclosure of Invention
The invention aims to provide a self-learning image super-resolution reconstruction method and system based on a convolutional neural network, so as to solve one or more technical problems. The method can not only effectively solve the problem of insufficient training samples of the self-learning algorithm, but also avoid the phenomenon of over-fitting of the network; meanwhile, a high-resolution image with higher peak signal-to-noise ratio and better visual effect can be obtained.
In order to achieve the above purpose, the invention adopts the following technical scheme:
the invention discloses a self-learning image super-resolution reconstruction method based on a convolutional neural network, which comprises the following steps of:
step 1, obtaining a training sample of an image to be reconstructed; the training sample comprises: a high-low resolution training sample pair;
step 2, constructing a convolutional neural network; the convolutional neural network includes:
a feature extraction unit comprising: a convolution layer; the input of the convolution layer is an image to be reconstructed, and the output is a feature map;
a feature enhancement unit comprising: a plurality of convolution layers; the input of the first layer in the plurality of convolution layers is the feature diagram extracted by the feature extraction unit, and the input of the remaining layers is the output of the previous layer; the outputs of the convolution layers are feature graphs;
a residual unit comprising: a plurality of convolution layers; the residual unit is provided with:
the long jump connection is used for connecting the image to be reconstructed with a result obtained through the convolutional neural network;
the short jump connection is used for transmitting the output of each convolution layer of the characteristic enhancement unit to each convolution layer of the residual unit respectively;
step 3, training the convolutional neural network constructed in the step 2 based on the training sample obtained in the step 1 to obtain a trained reconstructed neural network;
and 4, performing super-resolution reconstruction on the image to be reconstructed based on the reconstructed convolutional neural network trained in the step 3.
The invention further improves that in the step 1, the step of obtaining the training sample of the image to be reconstructed specifically comprises the following steps:
step 1.1, downsampling the image I to be reconstructed with different multiplying powers to obtain an image and downsampled versions I with different multiplying powers n ，n∈Z + Obtaining an initial sample;
step 1.2, expanding the initial sample obtained in the step 1.1 to obtain a training sample of an image to be reconstructed;
wherein, the expansion mode is:
I e ＝f(I n ,A,M)；
wherein: i e Is an expanded image sample, f is a sample set I n And performing enhancement operation, wherein A is rotation of the image at different angles, and M is mirror-image inversion of the image.
In the step 2, the convolution kernel size is 3×3 in the feature extraction unit of the constructed convolution neural network; and selecting the filter of [3, 64 ].
The invention further improves that in the step 2, the extracted features are linearly stacked in a feature enhancement extraction unit of the constructed convolutional neural network, and the stacking process expression is as follows:
F n+1 ＝a*F n +(1-a)*F n-1 ；
wherein: f (F) n Representing the image features extracted by the current layer, F n-1 Representing the output characteristics of the previous layer, F n+1 Representing the input of the next layer of the current layer, wherein n represents the number of layers of the hidden layer, and a is a product factor obtained by a plurality of experiments;
adding an active layer for enhancement after the linear stacking operation is completed, wherein the expression is as follows:
wherein: c represents a linear stacking operation, and R represents an activating operation.
The invention is further improved in that the value of n is 2, 3, 4 and 5; a is taken to be 0.6.
A further improvement of the invention is that, in step 2, in the residual units of the convolutional neural network constructed,
the expression for a long jump connection is:
I output＝ I input +F final ；
wherein I is input And I output Respectively represent the input and the output of the convolutional neural network, F final Representing the last layer learned by the network training.
A further improvement of the invention is that, in step 2, in the residual units of the convolutional neural network constructed,
the expression for a short jump connection is:
F p+1 ＝F p +F q-p ；
wherein F represents an operation of extracting a feature, F p+1 Refers to the input of the p+1 layer, F p And F q-p For the output of each hidden layer, p, q each represent the number of layers of the network.
A further improvement of the present invention is that it further comprises:
and 5, taking the super-resolution reconstructed image obtained in the step 4 as an image to be reconstructed, and repeating the steps 1 to 4.
The invention is further improved in that the step 5 specifically comprises the following steps:
feeding back the super-resolution reconstructed image obtained in the step 4 to an input end through a back propagation algorithm, and repeating the steps 1 to 4; and in the repetition process, using the mean square error as a loss function, adjusting the parameter according to the loss function, and repeatedly iterating to obtain the target image with the best final super-resolution effect.
The invention discloses a self-learning image super-resolution reconstruction system based on a convolutional neural network, which comprises the following steps:
the sample acquisition module is used for acquiring training samples of the image to be reconstructed; the training sample comprises: a high-low resolution training sample pair;
a convolutional neural network, comprising:
a feature extraction unit comprising: a convolution layer; the input of the convolution layer is an image to be reconstructed, and the output is a feature map;
a feature enhancement unit comprising: a plurality of convolution layers; the input of the first layer in the plurality of convolution layers is the feature diagram extracted by the feature extraction unit, and the input of the remaining layers is the output of the previous layer; the outputs of the convolution layers are feature graphs;
a residual unit comprising: a plurality of convolution layers; the residual unit is provided with:
the long jump connection is used for connecting the image to be reconstructed with a result obtained through the convolutional neural network;
the short jump connection is used for transmitting the output of each convolution layer of the characteristic enhancement unit to each convolution layer of the residual unit respectively;
the training reconstruction module is used for training the convolutional neural network according to the obtained training sample to obtain a trained reconstruction neural network; and performing super-resolution reconstruction on the image to be reconstructed based on the trained reconstruction convolutional neural network.
Compared with the prior art, the invention has the following beneficial effects:
according to the self-learning image super-resolution reconstruction method based on the convolutional neural network, self super-resolution reconstruction is achieved by utilizing a single image. The invention discloses a lightweight convolutional neural network, which comprises a feature extraction unit, a feature enhancement unit, a residual error unit and a reconstruction unit. In the training process, an input image is downsampled to form a high-low resolution image pair, and a training network is trained by the image pair, so that richer high-low resolution details can be obtained, more high-frequency details are recovered, and the loss of image details is avoided; the invention aims to obtain more high-resolution details, so that the residual error unit is added, the loss of image information in the training process is avoided, and a result with better reconstruction effect can be obtained. Through the combination of the operations, the high-resolution image with higher peak signal-to-noise ratio and better visual effect can be obtained.
In the invention, the sample enhancement is carried out on a single image, so that the network under fitting can be avoided.
In the invention, a convolutional neural network is built, and the image is reconstructed by learning the mapping relation between the high-resolution training sample pair and the low-resolution training sample pair through the network, because the training data are obtained from the image itself, the image has self-similarity, the data distribution is centralized, and the convolutional neural network is different from the network which relies on external training and needs a plurality of high-resolution sample pairs and low-resolution sample pairs with certain differences, the network can be converged quickly, the network structure is relatively simple, and the complex unit is not required to be built to learn the mapping relation between the high-resolution sample pair and the low-resolution sample pair. A large number of previous experiments show that a single image contains abundant inherent information, and the invention utilizes the reproducibility of the information to build a relatively light and simple network, can adapt to different settings of each image, and can obtain a better superscore result by extracting characteristic information. The invention constructs the residual error unit, and the information loss is unavoidable when the image is transferred between the convolution layers, so that the residual error unit can be added to compensate the information loss of the image.
A large number of facts indicate that the low-resolution image contains a lot of abundant low-frequency details, but the invention only performs shallow feature extraction on the image in the feature extraction stage, if the shallow features are transmitted to a later hidden layer to continue feature extraction as in the prior deep network structure, a lot of important detail information is obviously lost, and the inherent deep information of the image and the strength of a deep learning network are not fully utilized. The present invention seeks to introduce a feature enhancement unit to extract deep features of an image. The invention stacks the image features extracted from the previous layer with the features extracted from the current layer, and because of a lot of repeated information between the two layers of feature images, if the redundancy removing operation is not performed on the two layers of feature images, a lot of time is spent for learning some repeated information. The invention mainly extracts the shallow features extracted from the front layer of the network in a deep layer without introducing redundant parameters, and experiments show that the invention not only improves the convergence rate of the network, but also has better superdivision result.
The invention can recover the high-resolution image with better visual effect, and the high-resolution image has wide application in work and life. For example: the method has important application value in the fields of monitoring equipment, satellite image remote sensing, digital high definition, microscopic imaging, video coding communication, video restoration, medical imaging and the like. The high-resolution image can provide more important detail information for digital image processing due to high pixel density, and lays a good foundation for image post-processing. In summary, the invention has wider application range and great significance.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description of the embodiments or the drawings used in the description of the prior art will make a brief description; it will be apparent to those of ordinary skill in the art that the drawings in the following description are of some embodiments of the invention and that other drawings may be derived from them without undue effort.
FIG. 1 is a schematic flow chart of a self-learning image super-resolution reconstruction method based on a convolutional neural network according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a long hop connection used in a network residual unit in an embodiment of the present invention;
FIG. 3 is a schematic diagram of a short hop connection used in a network residual unit in an embodiment of the present invention;
FIG. 4 is a schematic diagram of a network architecture according to an embodiment of the present invention;
FIG. 5 is a graph showing the comparison of the super-division results of face images of the embodiment by the method and various methods according to the embodiment of the invention;
FIG. 6 is a graph showing the comparison of the superdivision results of the building image 1 of the embodiment according to the method of the present invention and various methods;
FIG. 7 is a graph showing the comparison of the superdivision results of the method of the present invention and various methods for the building image 2 of the example.
Detailed Description
In order to make the purposes, technical effects and technical solutions of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention are clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention; it will be apparent that the described embodiments are some of the embodiments of the present invention. Other embodiments, which may be made by those of ordinary skill in the art based on the disclosed embodiments without undue burden, are within the scope of the present invention.
Referring to fig. 1 to fig. 4, a self-learning image super-resolution reconstruction method based on a convolutional neural network according to an embodiment of the present invention includes the following specific steps:
step 1, selecting a proper sample enhancement mode, and enhancing and expanding a training sample;
and 2, constructing a convolutional neural network, and reconstructing an image by learning a mapping relation between high-resolution and low-resolution training sample pairs through the network, wherein training data are obtained from the image, the image has self-similarity, data distribution is centralized, and the convolutional neural network is different from a network which relies on external training and needs a large number of high-resolution and low-resolution sample pairs with certain difference, so that the network can be converged quickly, and the network structure is relatively simple, and does not need to construct a complex unit to learn the mapping relation between the high-resolution and low-resolution sample pairs. A large number of previous experiments show that a single image contains abundant inherent information, and the invention utilizes the reproducibility of the information to build a relatively light and simple network, can adapt to different settings of each image, and can obtain a better superscore result by extracting characteristic information.
In the embodiment of the invention, the specific steps of the step 1 comprise:
step 1.1, inputting a low-resolution image I;
step 1.2, because of reconstructing a single image, there is no external sample, and there are limited training samples. The invention firstly performs downsampling of different multiplying factors on an input low-resolution image I to obtain the image itself and a plurality of downsampled versions I of different multiplying factors n (n∈Z + ) These images are used as input data for training the network;
step 1.3, although the number of samples is expanded in the step 1.2, the training samples are far insufficient and are very liable to cause the network to be under fitted, so that the invention also carries out some enhancement operations such as rotation, mirror image and the like on the training samples to expand the samples; the specific expansion mode in the embodiment of the invention is shown as the following formula:
I e ＝f(I n ,A,M)；
wherein: i e Is an expanded image sample, f is a sample set I n And performing enhancement operation, wherein A is rotation of the image at different angles, and M is mirror-image inversion of the image.
In the embodiment of the invention, the specific steps of the step 2 include:
step 2.1, constructing a feature extraction unit, and performing shallow layer extraction on the features of the image by using a convolution layer;
step 2.2, constructing a feature enhancement unit, and performing depth extraction again on the image features obtained in the step 2.1;
step 2.3, constructing a residual error unit, wherein the information loss is unavoidable when the image is transferred between the convolution layers, so that the residual error unit can be added to compensate the information loss of the image;
and 2.4, inputting the test image into a trained network by the image reconstruction unit, and performing super-resolution reconstruction.
In the embodiment of the invention, the specific steps of step 2.1 include:
(1) Inputting a low resolution image I;
(2) The feature extraction layer is designed, because the feature extraction is directly performed on the downsampled low-resolution image, in order to avoid information loss and control the calculated amount, the convolution kernel size selected by the invention is 3×3, and the input image is a three-channel RGB image, so that the filters of [3, 64] are selected for extracting the shallow features of the image.
In the embodiment of the present invention, the specific steps in step 2.2 include:
(1) And (3) removing redundancy of the features extracted in the step 2.1. A large number of facts indicate that the low-resolution image contains a lot of abundant low-frequency details, but the invention only performs shallow feature extraction on the image in the feature extraction stage, if the shallow features are transmitted to a later hidden layer to continue feature extraction as in the prior deep network structure, a lot of important detail information is obviously lost, and the inherent deep information of the image and the strength of a deep learning network are not fully utilized. The present invention seeks to introduce an enhancement unit to extract deep features of the image. Thus, the present invention stacks the image features extracted from the previous layer with the features extracted from the current layer, and since there is a lot of repeated information between the two layers of feature maps, it takes a lot of time to learn some of the repeated information if the redundancy removing operation is not performed thereon.
In the embodiment of the invention, the linear stacking is performed, and the stacking process is shown as the following formula:
F n+1 ＝a*F n +(1-a)*F n-1 ；
wherein: f (F) n Representing the image features extracted by the current layer, F n-1 Representing the output characteristics of the previous layer, F n+1 Representing the input of the next layer of the current layer, n represents the layer number (n=2..5) of the hidden layer, in order to control the depth of the network, only 4 layers are selected for enhancement, a is a product factor obtained by a large number of experiments, and when a is 0.6, the network performance is the best, and the image reconstruction effect is the best.
(2) After the linear stacking operation is completed, an activating layer is added to better fit the relation between the image features extracted by the two hidden layers.
In the embodiment of the invention, the whole enhancement process is shown as the following formula:
wherein: c represents a linear stacking operation, R represents an activating operation; the invention mainly extracts the shallow features extracted from the front layer of the network in a deep layer without introducing redundant parameters, and experiments show that the invention not only improves the convergence rate of the network, but also has better superdivision result.
In the embodiment of the invention, the specific steps of the step 2.3 include:
(1) In order to fully utilize the intrinsic information of the image, the invention also introduces a residual unit. In this unit the invention uses long-skip-connection and short-skip-connection. Because the low-resolution image contains abundant low-frequency information, the invention can be directly used for super-division of the image, and therefore, the invention uses a long jump connection to connect the input low-frequency image with details obtained by a convolutional neural network, and the following formula is shown:
I output＝ I input +F final ；
because the invention builds an end-to-end network structure, I input And I output Respectively represent the input and output of the network, F final Representing the last layer of training learning from the network, which contains many valuable high frequency details;
(2) Unlike the previous residual learning, which only transfers the current layer recursively to the next layer, the present invention uses several short jump connections to transfer the outputs of the first layers of the network to the next layers, and this recursion mode can be represented by the following formula:
F p+1 ＝F p +F q-p ；
wherein: f represents the operation of extracting the features, and it is noted that F p+1 Refers to the input of the p+1 layer, and F p And F q-p For the output of each hidden layer, p and q represent the number of layers of the network, except q is the total number of layers of the network, and the value range of p is set to (1 … q/2-1) in order not to increase the complexity of the network.
In the embodiment of the present invention, the specific steps in step 2.4 include:
(1) Inputting a low resolution image I;
(2) Downsampling the low resolution image I input in (1) to obtain a downsampled image I ↓S Inputting the downsampled image into a network to extract features and learning parameters to obtain I ↓S Corresponding super resolution reconstruction map HI ↓S ；
(3) Reconstructing the HI image obtained in step (2) ↓S The information of (a) is fed back to an input end through a back propagation algorithm (BP), all the steps are repeated again, in the process, the method uses a mean square error (Mean Squared Error, MSE) as a loss function, the parameter size is adjusted according to the loss function, and the target image I with the best super-resolution effect is finally obtained through repeated iteration ↑S 。
In summary, the method of the embodiment of the invention improves the resolution of a single image by a self-learning method, namely, a low-resolution image is used as a training and testing sample, and a convolutional neural network is trained to realize the reconstruction of the low-resolution image. The method aims at searching and utilizing the inherent information of a single image, namely when reconstructing a low resolution image I with super resolution, firstly downsampling the image to obtain a downsampled image I ↓S (s is a sampling factor), then a mapping relation between the two is learned by utilizing a network convolution layer, and finally the trained network is used for super-resolution reconstruction of the low-resolution image to obtain a reconstructed image I ↑S I.e. a high resolution image.
The embodiment of the invention discloses a self-learning image super-resolution reconstruction system based on a convolutional neural network, which comprises the following steps:
the sample acquisition module is used for acquiring training samples of the image to be reconstructed; the training sample comprises: a high-low resolution training sample pair;
a convolutional neural network, comprising:
a feature extraction unit comprising: a convolution layer; the input of the convolution layer is an image to be reconstructed, and the output is a feature map;
a feature enhancement unit comprising: a plurality of convolution layers; the input of the residual layer is the output of the previous layer; the outputs of the convolution layers are feature graphs;
a residual unit comprising: a plurality of convolution layers; the residual unit is provided with:
the long jump connection is used for connecting the image to be reconstructed with a result obtained through the convolutional neural network;
the short jump connection is used for transmitting the output of each convolution layer of the characteristic enhancement unit to each convolution layer of the residual unit respectively;
the training reconstruction module is used for training the convolutional neural network according to the obtained training sample to obtain a trained reconstructed convolutional neural network; and performing super-resolution reconstruction on the image to be reconstructed based on the trained reconstruction convolutional neural network.
The working principle of the method disclosed by the embodiment of the invention comprises the following steps: the invention is based on a convolutional neural network, improves the resolution of a single image by a self-learning method, namely, a low-resolution image is used as a training and testing sample, and a convolutional neural network is untrained so as to realize the reconstruction of the image. The invention aims to find the inherent information of a single image, namely when super-dividing a low-resolution image I, firstly downsampling the image to obtain a downsampled image I ↓S (s is a sampling factor), designing a convolutional neural network to learn a mapping relation between the two, and finally using the trained network for super-resolution reconstruction of the low-resolution image to obtain an image I ↑S I.e. a high resolution image.
Experimental comparative analysis of the examples of the present invention: the image super-resolution reconstruction effect is measured by comparing and calculating peak signal to noise ratio (PSNR).
The Mean Square Error (MSE) may reflect the difference between the reconstructed image and the original image as follows:
wherein:ζ is the row, column number, X of the image data i,j Is the pixel value of the ith row and jth column of the original image, Y i,j Is the pixel value of the ith row and jth column of the reconstructed image;
the peak signal-to-noise ratio (PSNR) reflects the fidelity of the reconstructed image and is calculated as follows:
wherein: l represents the dynamic range of the image pixel.
The network built by the invention comprises an enhancement unit, a residual error unit and a linear superposition unit, and in order to verify the necessity and effectiveness of adding the units, the invention designs four network structures which respectively carry out a comparison test on a data set Urban100 by using a sampling factor of 2X. The four network structures are respectively: structure 1: enhancement unit + residual unit; structure 2: an enhancement unit+a linear superposition unit; structure 3: residual error unit+linear superposition unit; structure 4: enhancement unit + residual unit + linear superposition unit. The peak signal-to-noise ratios for the four network structures are shown in table 1 below:
table 1: peak signal-to-noise ratio comparison results for four different network structures
As can be seen from analysis of table 1, the PSNR value of structure 4 is highest, i.e. the super-resolution reconstruction effect on the image is better when the network incorporates the enhancement unit, the residual unit and the linear superposition unit.
Referring to fig. 5 to 7, the comparison of data results is shown in table 2:
table 2: comparison of peak signal to noise ratios for different algorithms
As can be seen from the analysis results of table 2, when the images are reconstructed on the data sets Set5, set14 and Urban100 with the sampling factors of 2, 3 and 4, compared with the external learning method EDSR for deep learning and the self-learning method ZSSR for deep learning of the machine learning method a+, the method of the invention can obtain high-resolution images with higher peak signal-to-noise ratio and better visual effect, and the application range of the method of the invention is wider. Compared with the machine learning method A+, the external learning method EDSR of the deep learning and the self-learning method ZSSSR of the deep learning, the peak signal-to-noise ratio can be improved by 0.5-1dB.
In summary, the invention discloses a self-learning image super-resolution reconstruction method based on a convolutional neural network, which comprises the following steps: selecting a proper sample enhancement mode, and enhancing and expanding a training sample; building a convolutional neural network, and reconstructing an image by learning a mapping relation between a high-resolution training sample pair and a low-resolution training sample pair; constructing a feature extraction unit, and performing shallow layer extraction on the features of the image by using a convolution layer; constructing a feature enhancement unit, and deeply extracting the image features again; constructing a residual error unit, wherein the information loss of the image is unavoidable when the image is transferred between the convolution layers, so that the residual error unit can be added to compensate the information loss of the image; and (3) image reconstruction, namely inputting the test image into a trained network to perform super-resolution reconstruction. The single image contains rich inherent information, the image has self-similarity and concentrated data distribution, and a complex unit is not required to be constructed to learn the mapping relation between the high-resolution sample pair and the low-resolution sample pair, so that the network structure is relatively simple, and convergence can be quickly obtained. The self-learning image super-resolution reconstruction method based on the convolutional neural network can not only effectively solve the problem of insufficient training samples of the self-learning algorithm, but also avoid the phenomenon of over-fitting of the network, and can obtain a high-resolution image with higher peak signal-to-noise ratio and better visual effect.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the above embodiments, one skilled in the art may make modifications and equivalents to the specific embodiments of the present invention, and any modifications and equivalents not departing from the spirit and scope of the present invention are within the scope of the claims of the present invention.
Claims (10)
1. The self-learning image super-resolution reconstruction method based on the convolutional neural network is characterized by comprising the following steps of:
step 1, obtaining a training sample of an image to be reconstructed; the training sample comprises: a high-low resolution training sample pair;
step 2, constructing a convolutional neural network; the convolutional neural network includes:
a feature extraction unit comprising: a convolution layer; the input of the convolution layer is an image to be reconstructed, and the output is a feature map;
a feature enhancement unit comprising: a plurality of convolution layers; the input of the first layer in the plurality of convolution layers is the feature diagram extracted by the feature extraction unit, and the input of the remaining layers is the output of the previous layer; the outputs of the convolution layers are feature graphs;
a residual unit comprising: a plurality of convolution layers; the residual unit is provided with:
the long jump connection is used for connecting the image to be reconstructed with a result obtained through the convolutional neural network;
the short jump connection is used for transmitting the output of each convolution layer of the characteristic enhancement unit to each convolution layer of the residual unit respectively;
step 3, training the convolutional neural network constructed in the step 2 based on the training sample obtained in the step 1 to obtain a trained reconstructed convolutional neural network;
and 4, performing super-resolution reconstruction on the image to be reconstructed based on the reconstructed convolutional neural network trained in the step 3.
2. The self-learning image super-resolution reconstruction method based on a convolutional neural network according to claim 1, wherein in step 1, the step of obtaining training samples of the image to be reconstructed specifically comprises:
step 1.1, downsampling the image I to be reconstructed with different multiplying powers to obtain an image and downsampled versions I with different multiplying powers n ，n∈Z + Obtaining an initial sample;
step 1.2, expanding the initial sample obtained in the step 1.1 to obtain a training sample of an image to be reconstructed;
wherein, the expansion mode is:
I e ＝f(I n ,A,M)；
wherein: i e Is an expanded image sample, f is a sample set I n And performing enhancement operation, wherein A is rotation of the image at different angles, and M is mirror-image inversion of the image.
3. The self-learning image super-resolution reconstruction method based on a convolutional neural network as claimed in claim 1, wherein in the step 2, the size of a convolution kernel in a feature extraction unit of the constructed convolutional neural network is 3×3; and selecting the filter of [3, 64 ].
4. The self-learning image super-resolution reconstruction method based on a convolutional neural network according to claim 1, wherein in the step 2, the extracted features are linearly stacked in a feature enhancement unit of the convolutional neural network, and a stacking process expression is as follows:
F n+1 ＝a*F n +(1-a)*F n-1 ；
wherein: f (F) n Representing the image features extracted by the current layer, F n-1 Representing the output characteristics of the previous layer, F n+1 Representing the input of the next layer of the current layer, n represents the layer number of the hidden layer, and a is a product factor obtained through a large number of experiments;
adding an active layer for enhancement after the linear stacking operation is completed, wherein the expression is as follows:
wherein: c represents a linear stacking operation, and R represents an activating operation.
5. The self-learning image super-resolution reconstruction method based on the convolutional neural network as set forth in claim 4, wherein n has values of 2, 3, 4 and 5; a is taken to be 0.6.
6. The self-learning image super-resolution reconstruction method based on a convolutional neural network as claimed in claim 1, wherein in the step 2, in the constructed residual units of the convolutional neural network,
the expression for a long jump connection is:
I output ＝I input +F final ；
wherein I is input And I output Respectively represent the input and the output of the convolutional neural network, F final Representing the last layer learned by the network training.
7. The self-learning image super-resolution reconstruction method based on a convolutional neural network as claimed in claim 1, wherein in the step 2, in the constructed residual units of the convolutional neural network,
the expression for a short jump connection is:
F p+1 ＝F p +F q-p ；
wherein F represents an operation of extracting a feature, F p+1 Refers to the input of the p+1 layer, F p And F q-p For the output of each hidden layer, p, q each represent the number of layers of the network.
8. The self-learning image super-resolution reconstruction method based on a convolutional neural network of claim 1, further comprising:
and 5, taking the super-resolution reconstructed image obtained in the step 4 as an image to be reconstructed, and repeating the steps 1 to 4.
9. The self-learning image super-resolution reconstruction method based on convolutional neural network of claim 8, wherein step 5 specifically comprises:
feeding back the super-resolution reconstructed image obtained in the step 4 to an input end through a back propagation algorithm, and repeating the steps 1 to 4; and in the repetition process, using the mean square error as a loss function, adjusting the size of the network parameters according to the loss function, and repeatedly iterating to obtain a final target image.
10. The self-learning image super-resolution reconstruction system based on the convolutional neural network is characterized by comprising the following components:
the sample acquisition module is used for acquiring training samples of the image to be reconstructed; the training sample comprises: a high-low resolution training sample pair;
a convolutional neural network, comprising:
a feature extraction unit comprising: a convolution layer; the input of the convolution layer is an image to be reconstructed, and the output is a feature map;
a feature enhancement unit comprising: a plurality of convolution layers; the input of the first layer in the plurality of convolution layers is the feature diagram extracted by the feature extraction unit, and the input of the remaining layers is the output of the previous layer; the outputs of the convolution layers are feature graphs;
a residual unit comprising: a plurality of convolution layers; the residual unit is provided with:
the long jump connection is used for connecting the image to be reconstructed with a result obtained through the convolutional neural network;
the short jump connection is used for transmitting the output of each convolution layer of the characteristic enhancement unit to each convolution layer of the residual unit respectively;
the training reconstruction module is used for training the convolutional neural network according to the obtained training sample to obtain a trained reconstructed convolutional neural network; and performing super-resolution reconstruction on the image to be reconstructed based on the trained reconstruction convolutional neural network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010802461.6A CN111932461B (en) | 2020-08-11 | 2020-08-11 | Self-learning image super-resolution reconstruction method and system based on convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010802461.6A CN111932461B (en) | 2020-08-11 | 2020-08-11 | Self-learning image super-resolution reconstruction method and system based on convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111932461A CN111932461A (en) | 2020-11-13 |
| CN111932461B true CN111932461B (en) | 2023-07-25 |
Family
ID=73310662
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010802461.6A Active CN111932461B (en) | 2020-08-11 | 2020-08-11 | Self-learning image super-resolution reconstruction method and system based on convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111932461B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112308781A (en) * | 2020-11-23 | 2021-02-02 | 中国科学院深圳先进技术研究院 | Single image three-dimensional super-resolution reconstruction method based on deep learning |
| CN112580381A (en) * | 2020-12-23 | 2021-03-30 | 成都数之联科技有限公司 | Two-dimensional code super-resolution reconstruction enhancing method and system based on deep learning |
| CN112617850B (en) * | 2021-01-04 | 2022-08-30 | 苏州大学 | Premature beat and heart beat detection system for electrocardiosignals |
| CN112907446B (en) * | 2021-02-07 | 2022-06-07 | 电子科技大学 | Image super-resolution reconstruction method based on packet connection network |
| CN112734622B (en) * | 2021-03-30 | 2021-07-20 | 深圳大学 | Image steganalysis method and terminal based on Tucker decomposition |
| CN115880158B (en) * | 2023-01-30 | 2023-10-27 | 西安邮电大学 | Blind image super-resolution reconstruction method and system based on variation self-coding |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106228512A (en) * | 2016-07-19 | 2016-12-14 | 北京工业大学 | Based on learning rate adaptive convolutional neural networks image super-resolution rebuilding method |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109903226A (en) * | 2019-01-30 | 2019-06-18 | 天津城建大学 | Image super-resolution rebuilding method based on symmetrical residual error convolutional neural networks |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9213076B2 (en) * | 2012-02-27 | 2015-12-15 | Medimagemetric LLC | System, process and computer-accessible medium for providing quantitative susceptibility mapping |
-
2020
- 2020-08-11 CN CN202010802461.6A patent/CN111932461B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106228512A (en) * | 2016-07-19 | 2016-12-14 | 北京工业大学 | Based on learning rate adaptive convolutional neural networks image super-resolution rebuilding method |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109903226A (en) * | 2019-01-30 | 2019-06-18 | 天津城建大学 | Image super-resolution rebuilding method based on symmetrical residual error convolutional neural networks |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111932461A (en) | 2020-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111932461B (en) | Self-learning image super-resolution reconstruction method and system based on convolutional neural network | |
| CN109903228B (en) | Image super-resolution reconstruction method based on convolutional neural network | |
| CN111047515A (en) | Cavity convolution neural network image super-resolution reconstruction method based on attention mechanism | |
| CN109727195B (en) | Image super-resolution reconstruction method | |
| CN111861886B (en) | Image super-resolution reconstruction method based on multi-scale feedback network | |
| CN110889895A (en) | Face video super-resolution reconstruction method fusing single-frame reconstruction network | |
| CN112699844A (en) | Image super-resolution method based on multi-scale residual error level dense connection network | |
| CN111161146A (en) | Coarse-to-fine single-image super-resolution reconstruction method | |
| CN112950480A (en) | Super-resolution reconstruction method integrating multiple receptive fields and dense residual attention | |
| CN113781308A (en) | Image super-resolution reconstruction method and device, storage medium and electronic equipment | |
| CN114998458A (en) | Undersampled magnetic resonance image reconstruction method based on reference image and data correction | |
| CN115526779A (en) | Infrared image super-resolution reconstruction method based on dynamic attention mechanism | |
| Wu et al. | Feedback pyramid attention networks for single image super-resolution | |
| CN111311488A (en) | Efficient super-resolution reconstruction method based on deep learning | |
| CN112150356A (en) | Single compressed image super-resolution reconstruction method based on cascade framework | |
| CN109272450A (en) | A kind of image oversubscription method based on convolutional neural networks | |
| CN112435162B (en) | Terahertz image super-resolution reconstruction method based on complex domain neural network | |
| CN112967185A (en) | Image super-resolution algorithm based on frequency domain loss function | |
| CN116563110A (en) | Blind image super-resolution reconstruction method based on Bicubic downsampling image space alignment | |
| CN116797541A (en) | Transformer-based lung CT image super-resolution reconstruction method | |
| CN108510464B (en) | Compressed sensing network based on block observation and full-image reconstruction method | |
| CN114913262B (en) | Nuclear magnetic resonance imaging method and system with combined optimization of sampling mode and reconstruction algorithm | |
| CN115880158A (en) | Blind image super-resolution reconstruction method and system based on variational self-coding | |
| Wu et al. | Pyramidal dense attention networks for lightweight image super-resolution | |
| CN109087247A (en) | The method that a kind of pair of stereo-picture carries out oversubscription |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |