CN106683048B - Image super-resolution method and device - Google Patents
Image super-resolution method and device Download PDFInfo
- Publication number
- CN106683048B CN106683048B CN201611086392.3A CN201611086392A CN106683048B CN 106683048 B CN106683048 B CN 106683048B CN 201611086392 A CN201611086392 A CN 201611086392A CN 106683048 B CN106683048 B CN 106683048B
- Authority
- CN
- China
- Prior art keywords
- network
- super
- resolution image
- discrimination
- loss function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000012545 processing Methods 0.000 claims abstract description 36
- 238000012800 visualization Methods 0.000 claims abstract description 9
- 238000010606 normalization Methods 0.000 claims description 15
- 238000013528 artificial neural network Methods 0.000 claims description 12
- 230000000694 effects Effects 0.000 abstract description 6
- 230000006870 function Effects 0.000 description 108
- 238000010586 diagram Methods 0.000 description 14
- 238000012549 training Methods 0.000 description 14
- 230000004913 activation Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 6
- 238000005070 sampling Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000003042 antagnostic effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000013529 biological neural network Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000002620 method output Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The application discloses a super-resolution image generation method, after a generation network and a discrimination network are preset, a real image sample is input into the generation network to output a super-resolution image sample, discrimination probabilities output by the discrimination network after the real image sample and the super-resolution image sample are input respectively are obtained, a generation network loss function and a discrimination network loss function are determined according to the real image sample, the super-resolution image sample and the discrimination probabilities, and configuration parameters of the generation network and the discrimination network are adjusted according to the generation network loss function and the discrimination network loss function. After the adjustment is completed, the processed low-resolution image is received, a super-resolution image of the low-resolution image can be generated according to the generation network, and the super-resolution image is subjected to visualization processing. Thereby obviously improving the super-resolution effect of the image and the authenticity of the super-resolution image.
Description
Technical Field
The invention relates to the technical field of communication, in particular to an image super-resolution method. The invention also relates to an image super-resolution device.
Background
Image super-resolution reconstruction techniques utilize a set of low-quality, low-resolution images (or motion sequences) to produce a single high-quality, high-resolution image. The image super-resolution reconstruction application field is wide, and the method has important application prospects in the aspects of military affairs, medicine, public safety, computer vision and the like. In the field of computer vision, image super-resolution reconstruction techniques make it possible to transform an image from a detected level to a recognized level, or further to a fine resolution level.
Based on an image super-resolution reconstruction technology capable of improving the identification capability and the identification precision of an image, the prior art provides a single-frame image super-resolution method for generating a high-resolution image by using a single-frame low-resolution and under-sampled image. The single-frame image super-resolution reconstruction technology can realize the concentration analysis of the target object, so that the image with higher spatial resolution of the region of interest can be acquired without directly adopting the configuration of the high spatial resolution image with huge data volume.
With the continuous development of the field of artificial neural networks, technicians can realize single-frame image super-resolution based on a deep convolutional neural network at present, so that the single-frame image super-resolution technology has great progress. Convolutional neural networks are one type of artificial neural networks, and have become a hot research point in the field of current speech analysis and image recognition. The weight sharing network structure of the system is more similar to a biological neural network, the complexity of a network model is reduced, and the number of weights is reduced. The advantage is more obvious when the input of the network is a multi-dimensional image, so that the image can be directly used as the input of the network, and the complex characteristic extraction and data reconstruction process in the traditional recognition algorithm is avoided. Convolutional networks are a multi-layered perceptron specifically designed to recognize two-dimensional shapes, the structure of which is highly invariant to translation, scaling, tilting, or other forms of deformation.
At present, when the super-resolution of a single-frame image is based on a deep convolutional neural network, the method outputs a frame of high-resolution image by performing a series of convolution or deconvolution operations on an input low-resolution image. And calculating the Mean Square Error (MSE) of the output high-resolution image and the real high-resolution image to serve as a supervision signal for deep convolutional neural network training.
However, in the process of implementing the present application, the inventor finds that when the prior art performs large sampling factor processing on a picture (for example, 4x upsampling, i.e., the width and the height of an image are respectively enlarged to 4 times), the recovery of texture details still has a problem, so that the details are not clear enough. Even if the single-frame image super-resolution of the deep convolutional neural network is utilized, the result is usually excessively smooth when the sampling factor is large, and high-frequency detail information is lacked, so that the requirement cannot be met.
Disclosure of Invention
The invention provides a super-resolution image generation method which is used for better retaining detail information of an image to be processed and solving the problem that texture details of an output high-resolution image are not clear when a sampling factor is large in the prior art. The method presets a generating network and a judging network, wherein the generating network and the judging network are both deep neural networks, and the method also comprises the following steps:
inputting real image samples into the generation network to output super-resolution image samples after super-resolution processing;
acquiring discrimination probabilities output by the discrimination network after the real image sample and the super-resolution image sample are input respectively, wherein the discrimination probabilities are probabilities that an input image of the discrimination network is a real image;
determining a generation network loss function and a discrimination network loss function according to the real image sample, the super-resolution image sample and the discrimination probability, and adjusting configuration parameters of the generation network and the discrimination network according to the generation network loss function and the discrimination network loss function;
after the adjustment of the configuration parameters is completed, receiving a low-resolution image to be processed, generating a super-resolution image of the low-resolution image according to the generation network, and performing visualization processing on the super-resolution image;
when the loss function of the generation network is smaller, the degree of reality of the super-resolution image output by the generation network is higher;
and when the loss function of the discrimination network is smaller, the accuracy of the discrimination probability output by the discrimination network is higher.
Preferably, the loss function of the generating network is generated according to a competing loss function, a regular constraint loss function, and a pixel-level mean square error loss function, wherein:
the countermeasure loss function is generated according to the discrimination probability output by the discrimination network after the super-resolution image sample is input;
the rule constraint function is determined according to the spatial consistency of the super-resolution image samples;
the pixel-level mean square error loss function is determined from the super-resolution image samples and the true image samples.
Preferably, after inputting the real image samples into the generation network to output super-resolution processed super-resolution image samples, the method further includes:
respectively setting labels for the real image sample and the super-resolution image sample;
wherein the label of the real image sample is set to be 1, and the label of the super-resolution image sample is set to be 0.
Preferably, the loss function of the discriminant network is generated according to the labels of all the image samples and the discriminant probability of each image sample output through the discriminant network.
Preferably, before the real image sample is input into the generating network, scaling and normalization processing are performed on the real image sample; and carrying out normalization processing on the low-resolution image before the low-resolution image is input into the generating network.
Correspondingly, the application also discloses a super-resolution image generation device, which comprises:
the device comprises a presetting module, a data processing module and a data processing module, wherein the presetting module is used for presetting a generating network and a judging network, and the generating network and the judging network are both deep neural networks;
an input module, which inputs the real image sample into the generation network to output the super-resolution image sample after the super-resolution processing;
the acquisition module is used for acquiring the discrimination probabilities output by the discrimination network after the real image sample and the super-resolution image sample are input respectively, wherein the discrimination probabilities are the probabilities that the input image of the discrimination network is a real image;
the determining module is used for determining a generated network loss function and a judgment network loss function according to the real image sample, the super-resolution image sample and the judgment probability, and adjusting configuration parameters of the generated network and the judgment network according to the generated network loss function and the judgment network loss function;
the generation module is used for receiving a low-resolution image to be processed after the adjustment of the configuration parameters is finished, generating a super-resolution image of the low-resolution image according to the generation network and carrying out visualization processing on the super-resolution image;
when the loss function of the generation network is smaller, the degree of reality of the super-resolution image output by the generation network is higher;
and when the loss function of the discrimination network is smaller, the accuracy of the discrimination probability output by the discrimination network is higher.
Preferably, the loss function of the generating network is generated according to a competing loss function, a regular constraint loss function, and a pixel-level mean square error loss function, wherein:
the countermeasure loss function is generated according to the discrimination probability output by the discrimination network after the super-resolution image sample is input;
the rule constraint function is determined according to the spatial consistency of the super-resolution image samples;
the pixel-level mean square error loss function is determined from the super-resolution image samples and the true image samples.
Preferably, the method further comprises the following steps:
the label module is used for respectively setting labels for the real image sample and the super-resolution image sample;
wherein the label of the real image sample is set to be 1, and the label of the super-resolution image sample is set to be 0.
Preferably, the loss function of the discriminant network is generated according to the labels of all the image samples and the discriminant probability of each image sample output through the discriminant network.
Preferably, before the real image sample is input into the generating network, scaling and normalization processing are performed on the real image sample; and carrying out normalization processing on the low-resolution image before the low-resolution image is input into the generating network.
Therefore, by applying the technical scheme of the application, after the generation network and the discrimination network are preset, the real image sample is input into the generation network to output the super-resolution image sample after the super-resolution processing, the discrimination probability output by the discrimination network after the real image sample and the super-resolution image sample are respectively input is obtained, finally, the generation network loss function and the discrimination network loss function are determined according to the real image sample, the super-resolution image sample and the discrimination probability, and the configuration parameters of the generation network and the discrimination network are adjusted according to the generation network loss function and the discrimination network loss function. Thus, when the low-resolution image to be processed is received after the adjustment is completed, the super-resolution image of the low-resolution image can be generated according to the generation network and can be visualized. Thereby obviously improving the super-resolution effect of the image and the authenticity of the super-resolution image.
Drawings
Fig. 1 is a schematic structural diagram of a generation network disclosed in an embodiment of the present application;
fig. 2 is a schematic structural diagram of a discrimination network disclosed in the embodiment of the present application;
fig. 3 is a schematic flow chart of a super-resolution image generation method proposed in the present application;
FIG. 4 is a schematic diagram illustrating a work flow of a training process according to an embodiment of the present application;
FIG. 5 is a schematic diagram illustrating a super-resolution image calculation process in this embodiment of the present application;
fig. 6 is a schematic structural diagram of a super-resolution image generating apparatus according to the present application.
Detailed Description
As described in the background art, the single-frame image super-resolution based on the deep convolutional neural network makes great progress in the single-frame image super-resolution technology. But there is still a problem with the recovery of texture details when the upsampling factor is large (e.g. 4 times upsampling), resulting in less sharp details.
In view of this, the present application provides a super-resolution image generation method, which provides a confrontation network framework for single-frame image super-resolution, wherein the confrontation network framework includes two deep neural networks, one is a generation network and the other is a discrimination network, and the two networks are trained simultaneously and compete with each other. The purpose of the generation network is to generate a high-resolution image from a low-resolution image, and the generated high-resolution image is difficult to distinguish from a real image; the purpose of the discrimination network is to distinguish between real images and super-resolution generated images. By means of the antagonistic network training, the generated network outputs a high-resolution image as real as possible. For this reason, before elaborating the detailed steps of the technical solution of the present application, the present application sets a generation network and a discrimination network in advance, it should be noted that the generation network and the discrimination network are both types of deep neural networks, and the following description is made with reference to the two networks in combination with specific embodiments:
as shown in fig. 1, a schematic diagram of a structure of a generated network in the embodiment of the present application is shown, where specific meanings of each layer in the structure diagram of the generated network are as follows:
layer 1 InputLR represents an input low resolution image;
layers 2 and 3 represent a convolution layer and a ReLU (Rectified linear unit, which is one of deep learning activation functions) activation function layer, where the step size of the convolution operation is 1, the convolution kernel size is 3 × 3, and the number of convolution kernels is 64;
layers 4 to 9 are residual network function blocks, two groups of convolution layers are used and are followed by a batch normalization layer, ReLU is taken as an activation function, and finally an element-level addition layer is used, wherein the step size of convolution operation is 1, the size of a convolution kernel is 3 x 3, and the number of the convolution kernels is 64;
layers 10 to 33 are 4 residual network function blocks, each of which is the same as above;
layers 34 to 37 are two sets of deconvolution units for image upsampling. The step size of the deconvolution layer operation is 0.5, the convolution kernel size is 3 x 3, and the number of convolution kernels is 64; the scaling factor used in this example is 4, so two sets of deconvolution units are used; if the overscaling factor is 2, then a set of deconvolution units may be employed;
layer 38 is a convolutional layer, the convolution operation step size is 1, the convolution kernel size is 3 × 3, and the number of convolution kernels is 3, in order to generate 3 channels of RGB data.
The last layer of the generation network outputs a super-resolution image.
As shown in fig. 2, a schematic structural diagram of a discrimination network in the specific embodiment of the present application is shown, and specific meanings of each layer in the structure diagram of the discrimination network are as follows:
the 1 st layer Input HR/SR represents an Input high-resolution real image or super-resolution image; the real image is marked as 1, and the super-resolution image is marked as 0;
layer 2 and layer 3 represent a convolution layer and an activation function layer, convolution; the convolution layer step size is 1, the convolution kernel size is 3 x 3, and the number of convolution kernels is 64;
layers 4 to 6 represent a convolution layer, an activation function layer and a batch regularization layer; wherein the convolution layer step size is 2, the convolution kernel size is 3 x 3, and the number of convolution kernels is 64;
layers 7 to 9 represent a convolution layer, an activation function layer, and a batch regularization layer; wherein the convolution layer step size is 1, the convolution kernel size is 3 x 3, and the number of convolution kernels is 128;
layers 10 to 12 represent a convolution layer, an activation function layer, and a batch regularization layer; wherein the convolution layer step size is 2, the convolution kernel size is 3 x 3, and the number of convolution kernels is 128;
layers 13 through 18 are similar to layers 7 through 12, the only difference being that the number of convolution kernels is 256;
layers 19 through 24 are similar to layers 7 through 12, the only difference being that the number of convolution kernels is 512;
layers 25 and 26 are a fully connected layer and a ReLU activation function layer;
the 27 th layer and the 28 th layer are a full connection layer and a Sigmoid (a Sigmoid function is used as an activation function and is one of deep learning activation functions) activation function layer, wherein the number of nodes of the full connection layer is 1;
the last layer of the discrimination network outputs a probability value representing the probability that the input image is a real image.
The generation network and the discrimination network of the present application are described above with reference to a specific network structure and function distribution, however, it should be noted that the above structure is only a preferred embodiment scheme provided in the specific embodiment of the present application, and other modifications made by those skilled in the art on the basis of performing super-resolution processing on an input image into a super-resolution image based on a deep neural network and performing judgment on a real image/super-resolution image with respect to the input image all belong to the protection scope of the present application.
Based on the pre-created generation network and the discrimination network, a flow diagram of the super-resolution image generation method provided by the application is shown in fig. 3, and the method comprises the following steps:
s301, inputting the real image sample into the generation network to output the super-resolution image sample after the super-resolution processing.
The method mainly comprises the following steps that the deep neural networks with different functions are improved to optimize the output effect of the generation network, so that the method is mainly divided into two flow path parts, and the first part of the generation network outputs a high-resolution image according to an input low-resolution image; the second part judges whether the network judges the input image. The output image of the network is generated as the input image of the discrimination network, and the output of the discrimination network is used as the feedback signal of the generation network.
Based on the above description, this step first inputs the existing picture as a real image sample into the generation network, and before inputting into the generation network, the real image sample needs to be subjected to some preprocessing work, such as scaling processing, normalization processing, and the like, according to the convention. The specific scaling parameters and normalization parameters can be set by a technician according to actual conditions, and the adjustment belongs to the protection scope of the application.
Since the present application needs to check the verification accuracy of the discrimination network, and a real image sample has a certain similarity with a subsequently generated super-resolution image, in a preferred embodiment of the present application, labels are respectively set for the real image sample and the super-resolution image sample for discrimination, where the label of the real image sample is set to 1, and the label of the super-resolution image sample is set to 0.
S302, obtaining the discrimination probability output by the discrimination network after the real image sample and the super-resolution image sample are input respectively, wherein the discrimination probability is the probability that the input image of the discrimination network is a real image.
Based on the super-resolution image sample generated in S301 and the original real image sample, the present application inputs the super-resolution image sample and the original real image sample to the discrimination network at the same time, so as to detect the verification accuracy of the discrimination network. No matter the input real image sample or the super-resolution image sample, the discrimination network outputs a discrimination probability, which is used to describe the probability that the input image is a real image after the detection of the discrimination network. In a specific application scenario, the probability may correspond to the label value of each sample image, for example, the closer the probability value is to 1, the higher the probability representing the real image is; the closer the probability value is to 0, the higher the probability of representing a super-resolution image.
S303, determining a generated network loss function and a discriminant network loss function according to the real image sample, the super-resolution image sample and the discriminant probability, and adjusting configuration parameters of the generated network and the discriminant network according to the generated network loss function and the discriminant network loss function.
Because the two deep networks are used in the training stage to compete with each other based on the single-frame image super-resolution training mode, based on the super-resolution image in S301 and the discrimination probability in S302, the loss functions of the two deep networks are determined for the generation network and the discrimination network respectively.
As mentioned above, the generation network and the discrimination network are of the type of deep neural network, and each of the generation network and the discrimination network is composed of a convolution unit and a plurality of functions as functional modules, so that in a specific application scenario of the present application, the weight values of each network node in the generation network are adjusted based on the generation network loss function, and the weight values of each network node in the discrimination network are adjusted based on the discrimination network loss function.
It should be noted that, the above-mentioned true degree of the super-resolution image is that the super-resolution image sample is relative to the true image sample, and the accuracy is that the discrimination probability is relative to the true image/super-resolution image in the true situation, and the specific representation form can be set by the skilled person according to the actual situation. In addition, the parameter configuration for the generation network/the discrimination network can be adjusted by the technician according to the loss function, and all of them belong to the protection scope of the present application.
In a preferred embodiment of the present application, the loss function of the generating network is generated according to a countermeasure loss function, a rule constraint loss function, and a pixel-level mean square error loss function, and specifically, each function is generated as follows:
(1) and the countermeasure loss function is generated according to the discrimination probability output by the discrimination network after the super-resolution image sample is input.
In a specific application scenario, the countermeasure loss function is determined as follows:
wherein D (I)SR) The larger the value is, the closer the super-resolution output image is to the real image, and the smaller the resistance loss function value is.
(2) The rule constraint function is determined according to the spatial consistency of the super-resolution image samples.
In a specific application scenario of the present application, the rule constraint loss function is denoted as LGREGThe main purpose is to maintain the spatial consistency of the super-resolution image, and the determination mode is as follows:
(3) the pixel-level mean square error loss function is determined from the super-resolution image samples and the true image samples.
In the specific application scenario of the present application, the pixel-level mean square error loss function is denoted as LGMSEThe calculation formula is as follows:
wherein, IHRRepresenting a high-resolution real image, ISRRepresenting the super-resolved processed image.
Based on the above functions, the preferred embodiment of the present application may assign different weights to the accuracy of the super-resolution image according to different functions when generating the loss function of the generation network, and the specific weight value may be set by a technician according to the actual application, which all belong to the protection scope of the present application.
In another preferred embodiment of the present application, the loss function of the discrimination network is generated based on labels previously allocated to the real image samples and the super-resolution image samples and discrimination probabilities obtained by the discrimination network after inputting the images respectively. In a specific embodiment of the present application, assuming that the loss function of the discriminant network is LD, the corresponding determination is as follows:
wherein, aiRepresenting an input label, 1 representing a real image, and 0 representing a super-resolution image; y isiRepresenting the discriminating network output class probability.
The present application further utilizes these loss functions to adjust the configuration parameters of the generation network and the discrimination network based on the loss functions of the generation network and the discrimination network. It should be noted that the adjustment process of S301 to S303 in the present application is not limited to one time, but is repeated as many times as necessary according to the optimization degree of the network and the requirement of the technician on the accuracy of the super-resolution image generation, which all belong to the protection scope of the present application.
S304, after the adjustment of the configuration parameters is completed, receiving a low-resolution image to be processed, generating a super-resolution image of the low-resolution image according to the generation network, and performing visualization processing on the super-resolution image.
After the parameters of the network to be generated and the judgment network are adjusted through the steps of S301-S303, the generation network can output a high-resolution image as real as possible through the training of the countermeasure network. When a technician needs to generate a super-resolution image for the low-resolution image, the low-resolution image can be used as an input image and input to a generation network, and the super-resolution image is generated according to the input image. Compared with the existing pixel level error statistics, the method can better retain image detail information and solve the problem that the texture details of the output high-resolution image are not clear when the sampling factor is large.
To further illustrate the technical idea of the present invention, the technical solution of the present invention will now be described with reference to the training process shown in fig. 4 and the super-resolution image determination flow shown in fig. 5.
In the present embodiment, the training samples in each training process are N pictures with Width × Height randomly cropped from the training picture set, for example, N is 32, and Width × Height is 128. These training samples serve as both inputs to the generation network and the discrimination network. As shown in fig. 4, the training process workflow proposed for the embodiment of the present application includes the following steps:
step one, scaling and normalization processing. Carrying out bicubic interpolation downsampling on the training sample, wherein the sampling factor is 4; then normalizing the down-sampled image, namely dividing each channel pixel value by 255; as input data for generating a network;
and step two, generating network forward propagation. Traversing all layers in a forward direction according to the generated network structure diagram;
outputting a super-resolution image, and meanwhile, judging the input of the network and generating loss calculation of the network;
and step four, sample pretreatment. The sample is divided into two sources, one is a training sample, and each channel pixel value is divided by 255 to be used as a real image and marked as a category 1; the output of the step three is used as a super-resolution image, is generated by a program, is different from a real image, and is marked as a category 0; n samples of each type, and 2N samples in total;
and fifthly, judging the forward propagation of the network. Traversing all layers in a forward direction according to the judged network structure diagram;
outputting category probability, wherein the probability value is closer to 1, and the probability of representing a real image is higher; the closer the probability value is to 0, the higher the probability of representing the super-resolution image;
and seventhly, generating a network loss function for calculation. The generated network loss function is denoted as LG and consists of three parts:
1) mean square error loss function at pixel level, denoted LGMSEThe determination is as follows:
wherein, x and y represent the coordinate position of the pixel point in the image, IHRRepresenting a high-resolution real image, ISRRepresenting the super-resolved processed image.
2) The aim of the loss function is to count the authenticity of the super-resolution image, denoted as LGADV. The determination is as follows:
where N is the number of samples in step four, I represents the number of the current sample, and D (I)SR) The larger the value is, the closer the super-resolution output image is to the true image, and the smaller the countermeasure loss function value is.
3) Rule constraint loss function, denoted LGREGThe main purpose is to maintain spatial consistency of the super-resolution image. The determination is as follows:
wherein, ISR x，yRepresenting a certain in a super-resolution processed imageAnd (5) each pixel point.
The generated network loss function can be generated by weighted summation of the three parts, and the specific way is as follows:
LG＝γ0LGMSE+γ1LGADV+γ2LGREG
wherein gamma is0Can take the value of 0.9, gamma1Can take the values of 0.002, gamma2Can take the value 2 x 10-8These parameters may be specifically adjusted.
And step eight, generating network back propagation. According to the generated network loss function calculated in the step seven, the whole generated network is propagated reversely, namely, each network node in the generated network is adjusted by using the network loss function, and in a specific application scene, technicians can adjust the weight value of the generated network node according to the loss function;
and ninthly, judging the calculation of the network loss function. The discrimination network loss function is denoted as LD, and the calculation formula is as follows:
wherein a isiRepresenting an input label, 1 representing a real image, and 0 representing a super-resolution image; y isiRepresenting a discrimination probability of discriminating the network output;
step ten, judging the network back propagation. According to the discrimination network loss function calculated in the step nine, the whole discrimination network is propagated reversely, and the weight value of the discrimination network node is adjusted;
in an actual application process, technicians may simultaneously execute the steps seven, eight, nine and ten as required, and the specific sequence of executing the steps does not affect the protection scope of the present application.
After the configuration parameters are adjusted for the loss functions of the generation network and the discrimination network, the super-resolution image calculation flow in this embodiment is as shown in fig. 5, and includes the following steps:
step one, inputting a low-resolution image;
step two, carrying out normalization processing on the input image, namely dividing the pixel value of each channel of the image by 255;
and step three, generating network forward propagation. Traversing all layers in a forward direction according to the generated network structure diagram;
step four, outputting a super-resolution image;
and step five, image visualization, namely multiplying the pixel value of each channel of the super-resolution image output by the generated network by 255, and performing fixed-point processing.
By applying the technical scheme of the specific embodiment, the super-resolution effect of the single-frame image is improved, especially for the condition of large scaling factor, the reality of the super-resolution image of the single-frame image is also improved, and the super-resolution effect is not limited to the signal-to-noise ratio.
To achieve the above technical object, the present invention also provides a super-resolution image generating apparatus as shown in fig. 6, including:
the system comprises a presetting module 610, a generating network and a judging network are preset, wherein the generating network and the judging network are both deep neural networks;
an input module 620, which inputs the real image sample into the generation network to output the super-resolution image sample after the super-resolution processing;
an obtaining module 630, configured to obtain a discrimination probability that the discrimination network outputs after the real image sample and the super-resolution image sample are input, where the discrimination probability is a probability that an input image of the discrimination network is a real image;
a determining module 640, configured to determine a generated network loss function and a discriminant network loss function according to the real image sample, the super-resolution image sample and the discriminant probability, and adjust configuration parameters of the generated network and the discriminant network according to the generated network loss function and the discriminant network loss function;
the generating module 650 receives a low-resolution image to be processed after the adjustment of the configuration parameters is completed, generates a super-resolution image of the low-resolution image according to the generating network, and performs visualization processing on the super-resolution image;
when the loss function of the generation network is smaller, the degree of reality of the super-resolution image output by the generation network is higher;
and when the loss function of the discrimination network is smaller, the accuracy of the discrimination probability output by the discrimination network is higher.
In a specific application scenario, the loss function of the generating network is generated according to a confrontation loss function, a rule constraint loss function, and a pixel-level mean square error loss function, where:
the countermeasure loss function is generated according to the discrimination probability output by the discrimination network after the super-resolution image sample is input;
the rule constraint function is determined according to the spatial consistency of the super-resolution image samples;
the pixel-level mean square error loss function is determined from the super-resolution image samples and the true image samples.
In a specific application scenario, the method further includes:
the label module is used for respectively setting labels for the real image sample and the super-resolution image sample;
wherein the label of the real image sample is set to be 1, and the label of the super-resolution image sample is set to be 0.
In a specific application scenario, the loss function of the discrimination network is generated according to the labels of all the image samples and the discrimination probability of each image sample output by the discrimination network.
In a specific application scene, before the real image sample is input into the generation network, scaling and normalization processing are carried out on the real image sample; and carrying out normalization processing on the low-resolution image before the low-resolution image is input into the generating network.
Therefore, by applying the technical scheme of the application, after the generation network and the discrimination network are preset, the real image sample is input into the generation network to output the super-resolution image sample, the discrimination probabilities output by the discrimination network after the real image sample and the super-resolution image sample are respectively input are obtained, the generation network loss function and the discrimination network loss function are determined according to the real image sample, the super-resolution image sample and the discrimination probabilities, and the configuration parameters of the generation network and the discrimination network are adjusted according to the generation network loss function and the discrimination network loss function. After the adjustment of the configuration parameters is completed, the low-resolution image to be processed is received, a super-resolution image of the low-resolution image can be generated according to the generation network, and the super-resolution image is subjected to visualization processing. Thereby obviously improving the super-resolution effect of the image and the authenticity of the super-resolution image.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present invention may be implemented by hardware, or by software plus a necessary general hardware platform. Based on such understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.), and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method according to the implementation scenarios of the present invention.
Those skilled in the art will appreciate that the figures are merely schematic representations of one preferred implementation scenario and that the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those skilled in the art will appreciate that the modules in the devices in the implementation scenario may be distributed in the devices in the implementation scenario according to the description of the implementation scenario, or may be located in one or more devices different from the present implementation scenario with corresponding changes. The modules of the implementation scenario may be combined into one module, or may be further split into a plurality of sub-modules.
The above-mentioned invention numbers are merely for description and do not represent the merits of the implementation scenarios.
The above disclosure is only a few specific implementation scenarios of the present invention, however, the present invention is not limited thereto, and any variations that can be made by those skilled in the art are intended to fall within the scope of the present invention.
Claims (8)
1. A super-resolution image generation method is characterized in that a generation network and a discrimination network are preset, the generation network and the discrimination network are both of a deep neural network type, and the method further comprises the following steps:
inputting real image samples into the generation network to output super-resolution image samples after super-resolution processing;
respectively setting labels for the real image sample and the super-resolution image sample;
acquiring discrimination probabilities output by the discrimination network after the real image sample and the super-resolution image sample are input respectively, wherein the discrimination probabilities are probabilities that an input image of the discrimination network is a real image;
determining a generation network loss function and a discrimination network loss function according to the real image sample, the super-resolution image sample and the discrimination probability, and adjusting configuration parameters of the generation network and the discrimination network according to the generation network loss function and the discrimination network loss function;
after the adjustment of the configuration parameters is completed, receiving a low-resolution image to be processed, generating a super-resolution image of the low-resolution image according to the generation network, and performing visualization processing on the super-resolution image;
when the loss function of the generation network is smaller, the truth degree of the super-resolution image output by the generation network is higher;
when the loss function of the discrimination network is smaller, the accuracy of the discrimination probability output by the discrimination network is higher;
the loss function of the generating network is generated according to a countermeasure loss function, a rule constraint loss function and a pixel-level mean square error loss function;
and the loss function of the discrimination network is generated according to the labels of all the image samples and the discrimination probability of each image sample output by the discrimination network.
2. The method of claim 1, wherein the countermeasure loss function is generated according to a discriminant probability that the discriminant network outputs after inputting the super-resolution image samples;
the rule constraint function is determined according to the spatial consistency of the super-resolution image samples;
the pixel-level mean square error loss function is determined from the super-resolution image samples and the true image samples.
3. The method of claim 1 or 2, wherein the label of the real image sample is set to 1 and the label of the super-resolution image sample is set to 0.
4. The method of claim 1,
before the real image sample is input into the generating network, scaling and normalization processing are carried out on the real image sample;
and carrying out normalization processing on the low-resolution image before the low-resolution image is input into the generating network.
5. A super-resolution image generation device characterized by comprising:
the device comprises a presetting module, a data processing module and a data processing module, wherein the presetting module is used for presetting a generating network and a judging network, and the generating network and the judging network are both deep neural networks;
an input module, which inputs the real image sample into the generation network to output the super-resolution image sample after the super-resolution processing;
the label module is used for respectively setting labels for the real image sample and the super-resolution image sample;
the acquisition module is used for acquiring the discrimination probabilities output by the discrimination network after the real image sample and the super-resolution image sample are input respectively, wherein the discrimination probabilities are the probabilities that the input image of the discrimination network is a real image;
the determining module is used for determining a generated network loss function and a judged network loss function according to the real image sample, the super-resolution image sample and the judgment probability, and adjusting configuration parameters of the generated network and the judged network according to the generated network loss function and the judged network loss function;
the generation module is used for receiving a low-resolution image to be processed after the adjustment of the configuration parameters is finished, generating a super-resolution image of the low-resolution image according to the generation network and carrying out visualization processing on the super-resolution image;
when the loss function of the generation network is smaller, the degree of reality of the super-resolution image output by the generation network is higher;
when the loss function of the discrimination network is smaller, the accuracy of the discrimination probability output by the discrimination network is higher;
the loss function of the generating network is generated according to a confrontation loss function, a rule constraint loss function and a pixel-level mean square error loss function.
6. The apparatus of claim 5, wherein the countermeasure loss function is generated from a discrimination probability that the discrimination network outputs after inputting the super-resolution image samples;
the rule constraint function is determined according to the spatial consistency of the super-resolution image samples;
the pixel-level mean square error loss function is determined from the super-resolution image samples and the true image samples.
7. The apparatus of claim 5 or 6, wherein the label of the real image sample is set to 1 and the label of the super-resolution image sample is set to 0.
8. The apparatus of claim 5,
before the real image sample is input into the generating network, scaling and normalization processing are carried out on the real image sample; and carrying out normalization processing on the low-resolution image before the low-resolution image is input into the generating network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611086392.3A CN106683048B (en) | 2016-11-30 | 2016-11-30 | Image super-resolution method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611086392.3A CN106683048B (en) | 2016-11-30 | 2016-11-30 | Image super-resolution method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106683048A CN106683048A (en) | 2017-05-17 |
| CN106683048B true CN106683048B (en) | 2020-09-01 |
Family
ID=58867637
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611086392.3A Active CN106683048B (en) | 2016-11-30 | 2016-11-30 | Image super-resolution method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106683048B (en) |
Families Citing this family (36)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107133934B (en) * | 2017-05-18 | 2020-03-17 | 北京小米移动软件有限公司 | Image completion method and device |
| CN107274358A (en) * | 2017-05-23 | 2017-10-20 | 广东工业大学 | Image Super-resolution recovery technology based on cGAN algorithms |
| CN108960014B (en) * | 2017-05-23 | 2021-05-11 | 北京旷视科技有限公司 | Image processing method, device and system and storage medium |
| US10262243B2 (en) * | 2017-05-24 | 2019-04-16 | General Electric Company | Neural network point cloud generation system |
| CN107423700B (en) * | 2017-07-17 | 2020-10-20 | 广州广电卓识智能科技有限公司 | Method and device for verifying testimony of a witness |
| CN107578377A (en) * | 2017-08-31 | 2018-01-12 | 北京飞搜科技有限公司 | A kind of super-resolution image reconstruction method and system based on deep learning |
| CN107633218B (en) | 2017-09-08 | 2021-06-08 | 百度在线网络技术（北京）有限公司 | Method and apparatus for generating image |
| CN107590774A (en) * | 2017-09-18 | 2018-01-16 | 北京邮电大学 | A kind of car plate clarification method and device based on generation confrontation network |
| CN107862785A (en) * | 2017-10-16 | 2018-03-30 | 深圳市中钞信达金融科技有限公司 | Bill authentication method and device |
| CN107945204B (en) * | 2017-10-27 | 2021-06-25 | 西安电子科技大学 | Pixel-level image matting method based on generation countermeasure network |
| CN107977511A (en) * | 2017-11-30 | 2018-05-01 | 浙江传媒学院 | A kind of industrial design material high-fidelity real-time emulation algorithm based on deep learning |
| CN111328448B (en) * | 2017-12-01 | 2021-08-03 | 华为技术有限公司 | Method and apparatus for image processing |
| CN108090508B (en) | 2017-12-12 | 2020-01-31 | 腾讯科技（深圳）有限公司 | classification training method, device and storage medium |
| CN108122209B (en) * | 2017-12-14 | 2020-05-15 | 浙江捷尚视觉科技股份有限公司 | License plate deblurring method based on countermeasure generation network |
| CN108122249A (en) * | 2017-12-20 | 2018-06-05 | 长沙全度影像科技有限公司 | A kind of light stream method of estimation based on GAN network depth learning models |
| CN109993694B (en) * | 2017-12-29 | 2021-10-08 | Tcl科技集团股份有限公司 | Method and device for generating super-resolution image |
| CN108182669A (en) * | 2018-01-02 | 2018-06-19 | 华南理工大学 | A kind of Super-Resolution method of the generation confrontation network based on multiple dimension of pictures |
| CN108235058B (en) * | 2018-01-12 | 2021-09-17 | 广州方硅信息技术有限公司 | Video quality processing method, storage medium and terminal |
| CN110472457A (en) * | 2018-05-10 | 2019-11-19 | 成都视观天下科技有限公司 | Low-resolution face image identification, restoring method, equipment and storage medium |
| CN108961161B (en) * | 2018-05-24 | 2023-09-22 | 上海商汤智能科技有限公司 | Image data processing method, device and computer storage medium |
| CN108805188B (en) * | 2018-05-29 | 2020-08-21 | 徐州工程学院 | Image classification method for generating countermeasure network based on feature recalibration |
| CN109118470B (en) * | 2018-06-26 | 2020-12-15 | 腾讯科技（深圳）有限公司 | Image quality evaluation method and device, terminal and server |
| CN109377459B (en) * | 2018-09-30 | 2022-03-18 | 国网智能科技股份有限公司 | Super-resolution deblurring method of generative confrontation network |
| CN109525859B (en) * | 2018-10-10 | 2021-01-15 | 腾讯科技（深圳）有限公司 | Model training method, image sending method, image processing method and related device equipment |
| CN109636721B (en) * | 2018-11-29 | 2023-06-23 | 武汉大学 | Video super-resolution method based on countermeasure learning and attention mechanism |
| CN109801215B (en) * | 2018-12-12 | 2020-04-28 | 天津津航技术物理研究所 | Infrared super-resolution imaging method based on countermeasure generation network |
| CN109859113B (en) * | 2018-12-25 | 2021-08-20 | 北京奇艺世纪科技有限公司 | Model generation method, image enhancement method, device and computer-readable storage medium |
| CN109697470B (en) * | 2018-12-27 | 2021-02-09 | 百度在线网络技术（北京）有限公司 | Recognition model training method and device, electronic equipment and storage medium |
| CN109949219B (en) * | 2019-01-12 | 2021-03-26 | 深圳先进技术研究院 | Reconstruction method, device and equipment of super-resolution image |
| CN109934102B (en) * | 2019-01-28 | 2020-11-27 | 浙江理工大学 | Finger vein identification method based on image super-resolution |
| CN109816764B (en) * | 2019-02-02 | 2021-06-25 | 深圳市商汤科技有限公司 | Image generation method and device, electronic equipment and storage medium |
| CN110580682A (en) * | 2019-09-16 | 2019-12-17 | 电子科技大学 | Countermeasure network seismic data super-resolution reconstruction method based on optimization generation |
| CN111402133A (en) * | 2020-03-13 | 2020-07-10 | 北京字节跳动网络技术有限公司 | Image processing method, image processing device, electronic equipment and computer readable medium |
| CN113628121B (en) * | 2020-05-06 | 2023-11-14 | 阿里巴巴集团控股有限公司 | Method and device for processing and training multimedia data |
| WO2022108031A1 (en) | 2020-11-19 | 2022-05-27 | Samsung Electronics Co., Ltd. | Image generators with conditionally-independent pixel synthesis |
| KR102611121B1 (en) * | 2021-03-22 | 2023-12-07 | 재단법인대구경북과학기술원 | Method and apparatus for generating imaga classification model |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102693419A (en) * | 2012-05-24 | 2012-09-26 | 武汉大学 | Super-resolution face recognition method based on multi-manifold discrimination and analysis |
| CN103985107A (en) * | 2014-05-27 | 2014-08-13 | 武汉科技大学 | Self-adaptive image super-resolution reconstruction method based on visual perception |
| CN104778659A (en) * | 2015-04-15 | 2015-07-15 | 杭州电子科技大学 | Single-frame image super-resolution reconstruction method on basis of deep learning |
| CN104992410A (en) * | 2015-02-10 | 2015-10-21 | 国网重庆市电力公司电力科学研究院 | Monocular visual pattern processing method |
| CN106127684A (en) * | 2016-06-22 | 2016-11-16 | 中国科学院自动化研究所 | Image super-resolution Enhancement Method based on forward-backward recutrnce convolutional neural networks |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9652829B2 (en) * | 2015-01-22 | 2017-05-16 | Samsung Electronics Co., Ltd. | Video super-resolution by fast video segmentation for boundary accuracy control |
-
2016
- 2016-11-30 CN CN201611086392.3A patent/CN106683048B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102693419A (en) * | 2012-05-24 | 2012-09-26 | 武汉大学 | Super-resolution face recognition method based on multi-manifold discrimination and analysis |
| CN103985107A (en) * | 2014-05-27 | 2014-08-13 | 武汉科技大学 | Self-adaptive image super-resolution reconstruction method based on visual perception |
| CN104992410A (en) * | 2015-02-10 | 2015-10-21 | 国网重庆市电力公司电力科学研究院 | Monocular visual pattern processing method |
| CN104778659A (en) * | 2015-04-15 | 2015-07-15 | 杭州电子科技大学 | Single-frame image super-resolution reconstruction method on basis of deep learning |
| CN106127684A (en) * | 2016-06-22 | 2016-11-16 | 中国科学院自动化研究所 | Image super-resolution Enhancement Method based on forward-backward recutrnce convolutional neural networks |
Non-Patent Citations (1)
| Title |
|---|
| "图像超分辨率重建算法研究";杨宇翔;《中国博士学位论文全文数据库 信息科技辑》;20131015(第10期);第I138-38页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106683048A (en) | 2017-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106683048B (en) | Image super-resolution method and device | |
| CN109584248B (en) | Infrared target instance segmentation method based on feature fusion and dense connection network | |
| US11232286B2 (en) | Method and apparatus for generating face rotation image | |
| Wang et al. | Detect globally, refine locally: A novel approach to saliency detection | |
| CN108764085B (en) | Crowd counting method based on generation of confrontation network | |
| Rahaman et al. | An efficient multilevel thresholding based satellite image segmentation approach using a new adaptive cuckoo search algorithm | |
| CN107563433B (en) | Infrared small target detection method based on convolutional neural network | |
| CN110059728B (en) | RGB-D image visual saliency detection method based on attention model | |
| CN105139004A (en) | Face expression identification method based on video sequences | |
| CN111784624B (en) | Target detection method, device, equipment and computer readable storage medium | |
| CN110930378B (en) | Emphysema image processing method and system based on low data demand | |
| CN111079739A (en) | Multi-scale attention feature detection method | |
| CN110532959B (en) | Real-time violent behavior detection system based on two-channel three-dimensional convolutional neural network | |
| CN112818969A (en) | Knowledge distillation-based face pose estimation method and system | |
| CN109753996B (en) | Hyperspectral image classification method based on three-dimensional lightweight depth network | |
| CN110879982A (en) | Crowd counting system and method | |
| CN110246171B (en) | Real-time monocular video depth estimation method | |
| CN112257741B (en) | Method for detecting generative anti-false picture based on complex neural network | |
| CN117597703A (en) | Multi-scale converter for image analysis | |
| CN113344110B (en) | Fuzzy image classification method based on super-resolution reconstruction | |
| CN111666852A (en) | Micro-expression double-flow network identification method based on convolutional neural network | |
| CN117079098A (en) | Space small target detection method based on position coding | |
| CN115311550B (en) | Remote sensing image semantic change detection method and device, electronic equipment and storage medium | |
| CN115861756A (en) | Earth background small target identification method based on cascade combination network | |
| CN115546638A (en) | Change detection method based on Siamese cascade differential neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221116 Address after: 15 / F, Zhongchuang Plaza, 385 Hangtian Middle Road, national civil aerospace industrial base, Xi'an City, Shaanxi Province 710100 Patentee after: Xi'an Yu vision Mdt InfoTech Ltd. Address before: 2 / F, South Block, building 10, wanlun Science Park, 88 Jiangling Road, Binjiang District, Hangzhou City, Zhejiang Province, 310051 Patentee before: ZHEJIANG UNIVIEW TECHNOLOGIES Co.,Ltd. |
|
| TR01 | Transfer of patent right |