CN113269722A - Training method for generating countermeasure network and high-resolution image reconstruction method - Google Patents
Training method for generating countermeasure network and high-resolution image reconstruction method Download PDFInfo
- Publication number
- CN113269722A CN113269722A CN202110435203.3A CN202110435203A CN113269722A CN 113269722 A CN113269722 A CN 113269722A CN 202110435203 A CN202110435203 A CN 202110435203A CN 113269722 A CN113269722 A CN 113269722A
- Authority
- CN
- China
- Prior art keywords
- resolution image
- image set
- model
- countermeasure network
- edge detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/32—Determination of transform parameters for the alignment of images, i.e. image registration using correlation-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a training method for generating a countermeasure network for high-resolution picture reconstruction, and belongs to the field of deep learning. The training method for generating the countermeasure network comprises the following steps: the generator model acquires a low-resolution image set generated from an original high-resolution image set, and trains by using the low-resolution image set to obtain a generated high-resolution image set; the edge detection model respectively carries out edge detection on the original high-resolution image set and the generated high-resolution image set, and carries out iterative update on the generator model according to the result of the edge detection; and continuously comparing and identifying the original high-resolution image set and the generated high-resolution image set by the identifier model, and determining the training effect of the generator model according to the comparison and identification result. The embodiment of the invention introduces the image edge detection module in the generation of the countermeasure network, so that the generator model of the countermeasure network can be better trained and generated.
Description
Technical Field
The invention relates to the technical field of deep learning, in particular to a training method for generating a countermeasure network and a high-resolution image reconstruction method.
Background
The image resolution is an important index of the image detail presenting capability, and describes the number of pixel points contained in an image, which is a measure of the amount of image information. In practical applications, due to the limitation of hardware devices (such as imaging sensors) in the imaging system, people cannot acquire high-resolution images. In addition, in internet applications, due to network bandwidth and other reasons, users can only transmit images or videos with lower resolution, and actually users want images or videos with high resolution. Although the quality of the acquired image can be improved by upgrading the hardware device, it obviously increases the cost of acquiring the image, and the image super-resolution reconstruction technology is generated accordingly. How to effectively improve the resolution of the image and the quality of the image is a research focus of the image super-resolution reconstruction technology and an important subject in the field of visual image processing.
The image super-resolution reconstruction technology has wide application range and research significance in a plurality of fields. For example, the field of image compression: in occasions with higher real-time requirements such as video conferences and the like, the images can be compressed in advance before transmission, after the transmission is finished, the original image sequence is restored by a super-resolution reconstruction technology after the images are decoded by a receiving end, and the space required by storage and the bandwidth required by transmission are greatly reduced; the field of medical imaging: the super-resolution reconstruction is carried out on the medical image, the requirement on the imaging environment can be reduced on the basis of not increasing the cost of a high-resolution imaging technology, the accurate detection of lesion cells is realized through the restored clear medical image, and a doctor can make better diagnosis on the illness state of a patient; the field of remote sensing imaging: the development of the high-resolution remote sensing satellite has the characteristics of long time consumption, high price, complex flow and the like, so that a researcher introduces an image super-resolution reconstruction technology into the field and tries to solve the challenge that the high-resolution remote sensing imaging is difficult to obtain, so that the resolution of an observed image can be improved on the premise of not changing a detection system; the field of public security: videos collected by monitoring equipment in public places are often influenced by factors such as weather and distance, and the problems of image blurring, low resolution and the like exist. By performing super-resolution reconstruction on the acquired video, important information such as license plate numbers, clear faces and the like can be recovered for case handling personnel, and necessary clues are provided for case detection; the field of video perception: through the image super-resolution reconstruction technology, the effects of enhancing the image quality of the video, improving the quality of the video and improving the visual experience of a user can be achieved.
At present, the research on super-resolution of images is realized based on deep learning, and the reconstruction of super-resolution images based on a learning method is also a popular direction of the research at present. The existing various image super-resolution reconstruction algorithms are image super-resolution restoration performed on the basis of an existing data set, in the process of performing result testing, pictures with characteristics similar to those of a training set are adopted, and when actually acquired pictures are subjected to image restoration, the restoration effect is unsatisfactory due to different characteristics of the pictures.
Disclosure of Invention
The embodiment of the invention aims to provide a training method for generating a countermeasure network for high-resolution picture reconstruction, which can reconstruct images with higher quality.
In order to achieve the above object, an embodiment of the present invention provides a training method for generating a countermeasure network for high resolution picture reconstruction, where the countermeasure network includes a generator model, a discriminator model, and an edge detection model, and the training method for generating a countermeasure network for high resolution picture reconstruction includes: the generator model acquires a low-resolution image set generated from an original high-resolution image set, and trains by using the low-resolution image set to obtain a generated high-resolution image set; the edge detection model respectively carries out edge detection on the original high-resolution image set and the generated high-resolution image set, and carries out iterative update on the generator model according to the result of the edge detection; and continuously comparing and identifying the original high-resolution image set and the generated high-resolution image set by the identifier model, and determining the training effect of the generator model according to the comparison and identification result.
Optionally, the generator model for generating the countermeasure network includes 5 convolutional layers, 12 residual blocks, and 2 double-amplification upsampling layers, where the residual block performs a splicing operation on outputs of two adjacent convolutional layers through a ReLU activation function, and uses a tanh activation function in a last deconvolution layer.
Optionally, the training of the generator model by using the low-resolution image set to obtain the generated high-resolution image set includes: the generator model takes the low-resolution image set as input and is trained through a loss function of the generator model, wherein the loss function of the generator model is a weighted sum of content loss, countermeasure loss and edge loss.
Optionally, the content loss is calculated by using a mean square error of MSE, and the content loss is represented by the following formula:
where W, H denotes the width and height of the low resolution image,
representing the original high-resolution image or images,
representing the generated high resolution image and r represents an upsampling factor.
Optionally, the challenge loss is based on the result of the discrimination
In the generated high-resolution image set
Is determined and the challenge loss is represented by:
where N represents the number of samples of the generated high resolution image set.
Optionally, the edge loss is represented by:
where W, H denotes the width and height of the original high resolution image, E denotes the edge detection model algorithm, E (I)SR) Edge detection image representing an image generated by a generator, E (I)HR) An edge detection image representing the original high resolution image.
Optionally, the performing, by the edge detection model, edge detection on the original high-resolution image set and the generated high-resolution image set respectively, and performing iterative update on the generator model according to the result of the edge detection includes: respectively carrying out edge extraction on the original high-resolution image set and the generated high-resolution image set through a Canny operator of the edge detection model; determining an image edge threshold value through the maximum inter-class variance, respectively carrying out edge detection on the original high-resolution image set and the generated high-resolution image set according to the edge threshold value, and carrying out iterative updating on the generator model according to the edge detection result.
Optionally, the discriminator model includes 8 convolutional layers and a Sigmoid layer, where an input generated high-resolution image set passes through the 8 convolutional layers, an output result is sent to the Sigmoid layer to obtain a classification output probability, and each layer adopts batch normalization processing and a leak ReLU activation function.
Optionally, before the identifying the original high-resolution image set and the generated high-resolution image set respectively by the identifier model, the training method for generating an anti-adversarial network for high-resolution image reconstruction further includes: the discriminator model is trained by the loss function of the discriminator model.
Optionally, the identifying the original high-resolution image set and the generated high-resolution image set by the identifier model respectively, and determining a training effect of the generator model according to a result of the identifying includes: and continuously and respectively identifying the original high-resolution image set and the generated high-resolution image set through the trained identifier model, wherein the generated high-resolution image set can replace the original high-resolution image set to a satisfactory effect.
The embodiment of the invention also provides a high-resolution image reconstruction method based on the generation countermeasure network, wherein the generation countermeasure network comprises a generator model, a discriminator model and an edge detection model, and the high-resolution image reconstruction method based on the generation countermeasure network comprises the following steps: training the generation countermeasure network according to the training method for the high-resolution picture reconstruction generation countermeasure network; and reconstructing the high-resolution image through the trained generator model for generating the countermeasure network.
An embodiment of the present invention further provides a control device, where the control device includes: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the training method for generating an antagonistic network for high resolution picture reconstruction according to any one of the above and the high resolution image reconstruction method based on the generated antagonistic network.
The embodiment of the invention also provides a machine-readable storage medium, wherein the machine-readable storage medium stores instructions for enabling a machine to execute the training method for generating the countermeasure network for reconstructing the high-resolution picture and the high-resolution image reconstruction method based on the countermeasure network.
Through the technical scheme, the training method for generating the countermeasure network for high-resolution picture reconstruction, provided by the embodiment of the invention, introduces the image edge detection module into the generation of the countermeasure network, so that the high-frequency details of the generated high-resolution image and the original high-resolution image are better detected, and a generator model of the countermeasure network is better trained and generated.
Additional features and advantages of embodiments of the invention will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the embodiments of the invention without limiting the embodiments of the invention. In the drawings:
FIG. 1 is a schematic diagram of a prior art architecture for generating a countermeasure network;
FIG. 2 is a flow chart of a training method for generating a countermeasure network according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an architecture for generating a countermeasure network according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a convolution structure of a generator model;
FIG. 5 is a schematic diagram of the structure of an edge detection model;
FIG. 6 is a flow diagram illustrating the operation of the edge detection model;
fig. 7 is a schematic diagram of the convolution structure of the discriminator model.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating embodiments of the invention, are given by way of illustration and explanation only, not limitation.
Before the content of the embodiment of the invention is introduced, the existing reconstruction method of the high-resolution image and the defects thereof are analyzed:
at present, research on ultra-high resolution images is realized based on deep learning, and the reconstruction of super-resolution images based on a learning method is also a popular direction of the research at present. The method comprises the steps of extracting features of a high-resolution image and a low-resolution image, creating a sample library, learning to obtain a mapping relation between the high-resolution image and the low-resolution image, and guiding the reconstruction of the image by using the mapping relation. During reconstruction, detailed information possessed by the sample is mined, namely abundant high-frequency information is utilized to generate a high-resolution image.
At first, when the neural network is applied to super-resolution image reconstruction, namely SRCNN, there are only three convolutional layers, which are respectively the feature extraction of an image, the nonlinear mapping of features, and the reconstruction of an image, a low-resolution image needs to be subjected to upsampling and interpolation to obtain the same size as a high-resolution image, and then the low-resolution image is input into the network. The embarrassment that the SRCNN carries out feature extraction on a high-resolution image is solved.
Then, image super resolution techniques SRGAN, CinCGAN, and the like based on the generation countermeasure network are generated, and a high resolution image is generated using a low resolution image as an input of the generation network.
Fig. 1 shows the overall architecture of generating a countermeasure network. The generation of the countermeasure network originates from a game theory algorithm, and mainly adopts a competition mechanism to realize the updating of the network. Referring to fig. 1, the generation of the countermeasure network includes two competing neural network structures: the generator model G aims to learn the distribution of real data samples and generate a generated sample G (X) with similarity approximate to the real samples, and the discriminator model D aims to judge whether the training samples come from the real samples or the generated samples. The generation of a series of related technologies combining the countermeasure network technology and image processing realizes super-resolution reconstruction of images, and has a very good effect in practical application. According to the definition of Goodfellow et al on the generation network, when the generation countermeasure network is applied in the image super-resolution field, the model is trained in an alternate optimization mode to solve the problem of minimum-maximum antagonism, as shown in the following formula:
wherein, thetaDAnd thetaGParameters of the discriminator model D and the generator model G respectively,
representation of generating a real image I of a countering network inputHRIs determined by the probability density function of (a),
representing low resolution images ILRGenerated high resolution image G (I) obtained after passing through the generatorLR) By a probability density function ofDAnd thetaGThe network game problem is solved by the alternate optimization.
Equation (1) shows that a generator model G is trained, the goal being to fool a discriminator model D, which is trained to distinguish between super-resolved images and differences from real images. Accordingly, the generator model can learn to create an image that is highly similar to the real image.
However, various existing ultrahigh resolution image reconstruction algorithms perform image restoration on the basis of an existing data set, in the process of performing result testing, pictures with characteristics similar to those of a training set are also adopted, and when the actually acquired pictures perform image restoration, the restoration effect is unsatisfactory due to different characteristics of the pictures. Meanwhile, when a picture with a relatively obvious edge feature like a human face is taken, the edge of the picture is not detected well and restored.
Fig. 2 is a schematic flowchart of a training method for generating a countermeasure network according to an embodiment of the present invention, fig. 3 is a schematic structural diagram of the countermeasure network according to an embodiment of the present invention, and referring to fig. 3, the countermeasure network may include a generator model, a discriminator model, and an edge detection model. With reference to fig. 2, the training method for generating a countermeasure network is used for high resolution picture reconstruction, and the training method for generating a countermeasure network may include the following steps:
step S110: the generator model obtains a low-resolution image set generated from an original high-resolution image set, and trains by using the low-resolution image set to obtain a generated high-resolution image set.
As described above, the generator model G aims to learn the distribution of real data samples, and generates a generated sample with similarity approximating to a real sample through the processes of feature extraction of an image, nonlinear mapping of features, reconstruction generation of an image, and the like. Referring to fig. 3, a generator model G for generating a countermeasure network according to an embodiment of the present invention may utilize a low resolution image set X generated from an original high resolution image set Y to perform neural network training on the low resolution image set X, so as to obtain a generated high resolution image set G (X).
Preferably, the generator model for generating the countermeasure network may include 5 convolutional layers, 12 residual blocks, and 2 double-amplification upsampling layers, wherein the residual blocks perform a splicing operation on outputs of two adjacent convolutional layers through a ReLU activation function, and a tanh activation function is used in the last convolutional layer.
Referring to FIG. 4, by way of example, images in image set X pass through the convolution layer and the ReLU activation function layer; then entering iterative training, wherein the iterative training adopts a residual error network structure, which mainly comprises a convolution layer, an activation function layer, a convolution layer and input of splicing iterative training, and after 12 times of iterative training, the residual error network structure passes through the convolution layer and is spliced with the input before the initial iterative training; next, two upsampling layers are passed, wherein the upsampling layers comprise a convolution layer and an activation function layer besides a double-amplification upsampling layer, namely an x2 upsampling factor; finally, the model output result of x4 magnification can be obtained through the convolution layer.
For example, an image with a resolution of 32 × 32 is generated into an image of 128 × 128. The residual block can solve the dependency between the input and the output to a certain extent, the process carries out splicing operation on the output of each previous convolution layer and the output of the corresponding convolution layer through a ReLU activation function, and the tanh activation function is used in the last deconvolution layer. Accordingly, the low resolution image set X may be passed through the generator model G to obtain a generated high resolution image set G (X).
Preferably, the training of the generator model by using the low resolution image set to obtain the generated high resolution image set may include: the generator model takes the low-resolution image set as input and is trained through a loss function of the generator model, wherein the loss function of the generator model is a weighted sum of content loss, countermeasure loss and edge loss.
And calculating through a loss function in the generator model by taking the low-resolution image set as input, and iteratively updating parameters in the generator model so as to train the generator model.
By way of example, in the image super-resolution technique SRGAN based on generation of the countermeasure network, for training of generation of the countermeasure network, calculation is performed by loss functions of a generator model and a discriminator model respectively, so as to realize iterative update of parameters in the generation of the countermeasure network model.
Preferably, the content loss is calculated by using a mean square error of MSE, and the content loss can be represented by the following formula:
where W, H denotes the width and height of the low resolution image,
representing the original high-resolution image or images,
representing the generated high resolution image and r represents an upsampling factor.
The design of the loss function needs to take into account the competing losses from the generator model and the discriminator model in addition to the content loss.
Preferably, the challenge loss is based on the result of the identification
In the generated high-resolution image set
Is determined and the challenge loss is represented by:
where N represents the number of samples of the generated high resolution image set.
Further, the embodiment of the invention takes the high resolution image edge detail problem into consideration for the design of a generator model loss function for generating a countermeasure network, and introduces edge loss.
Preferably, the edge loss is represented by the following formula:
where W, H denotes the width and height of the original high resolution image, E denotes the edge detection model algorithm, E (I)SR) Edge detection image representing an image generated by a generator, E (I)HR) An edge detection image representing the original high resolution image.
And (4) calculating the mean square error of pixels between the pictures in the generated high-resolution image set X and the corresponding pictures in the original high-resolution image set Y by using the formula (4), so as to obtain the edge loss.
Loss function of the generator model (which may also be referred to as perceptual loss l)g_loss) The design is crucial to the performance of the generator model, and the embodiment of the invention can improve the perception loss based on the image super-resolution technology SRGAN for generating the countermeasure network, and increase the edge loss, namely the perception loss lg_lossMay be a content loss lmseTo combat the loss lgenEdge loss ledgeThe weighted sum of (a) and (b) can also be represented by the following formula:
lg_loss＝lmse+ledge+λlgen (5)
wherein λ represents an adjustable parameter, and l is adjusted according to the parametergenSize of influence
Step S120: and the edge detection model respectively carries out edge detection on the original high-resolution image set and the generated high-resolution image set, and carries out iterative update on the generator model according to the result of the edge detection.
The results of edge detection may be understood as identifying accurately points in the digital image where changes in brightness are significant, for example, identifying edges of people in the picture. The extraction of the image edge through the edge detection can be applied to high-resolution picture reconstruction, so that the picture reconstruction is more accurate.
Preferably, step S120 may include: respectively carrying out edge extraction on the original high-resolution image set and the generated high-resolution image set through a Canny operator of the edge detection model; determining an image edge threshold value through the maximum inter-class variance, respectively carrying out edge detection on the original high-resolution image set and the generated high-resolution image set according to the edge threshold value, and carrying out iterative updating on the generator model according to the edge detection result.
By way of example, fig. 5 is a schematic structural diagram of an edge detection model, please refer to fig. 3 and 5, the input of the edge detection model is a generated high-resolution image set X and an original high-resolution image set Y, the edge detection model detects the edge of the image by the maximum inter-class variance method, and outputs an edge image e (X) of the generated high-resolution image set and an edge image e (Y) of the original high-resolution image set. The edge detection model returns the edge detection result to the generator model G, so that the parameters of the generator model G are iteratively updated, that is, the parameters of the generator model are iteratively updated through the calculation of the formula (4), and the generator model G can obtain the edge information of the best high-resolution image, so as to accurately reconstruct the high-resolution image.
With further reference to fig. 6, fig. 6 is a schematic diagram illustrating an operation flow of an edge detection model, when performing image edge detection on the generated high-resolution image set X and the original high-resolution image set Y, extracting an edge of an image by using a Canny operator, and extracting a threshold of the image by using a maximum inter-class variance, where the process may include:
1) carrying out graying processing on the picture;
2) performing Gaussian filtering processing on the grayed picture;
3) calculating the gradient value and the direction of a vector corresponding to the processed picture;
4) selecting edge threshold selection through non-maximum value inhibition, wherein double threshold selection can be adopted;
5) and carrying out edge detection on the picture.
Accordingly, the generated high-resolution image set includes the edge information of the image through steps S110 to S120, and the generated high-resolution image set is further identified to detect the training effect of the generator model.
Step S130: and the discriminator model continuously compares and discriminates the original high-resolution image set and the generated high-resolution image set, and determines the training effect of the generator model according to the comparison and discrimination result.
Preferably, the discriminator model includes 8 convolutional layers and a Sigmoid layer, wherein the input generated high-resolution image set is sent to the Sigmoid layer to obtain a classification output probability after passing through the 8 convolutional layers, and each layer adopts batch normalization processing and a Leaky ReLU activation function.
By way of example, an input image of the discriminator model comprises two parts, one part is a generated image of a countermeasure network through a generator, the other part is a real image, the input image passes through a convolution layer and an activation function layer, an activation function adopts Leaky ReLU, feature extraction is realized through a dense network, each part of the dense network mainly comprises a Conv convolution layer and a Leaky ReLU activation function layer, after 7 dense networks with similar structures are passed, output results are sent to a full connection layer and the Leaky ReLU layer for feature classification, and finally, the output results are sent to a Sigmoid layer to obtain classification output probability.
Fig. 7 shows a neural convolution structure of the discriminator model, please refer to fig. 3 and fig. 7, which illustrate that the input data of the discriminator model D is the generated high-resolution image set g (x), and after 8 convolution layers, the output result is sent to a Sigmoid layer, so as to obtain the classification output probability, i.e. the probability that the generated high-resolution image set g (x) is in the original high-resolution image set. Each layer of the neural convolution structure of the discriminator model applies batch normalization processing and a Leaky ReLU activation function, and adopts a full connection layer to combine image characteristics.
Preferably, before performing step S130, the training method for generating a countermeasure network for high resolution picture reconstruction may further include: the discriminator model is trained by the loss function of the discriminator model.
Further preferably, step S130 may include: and continuously and respectively identifying the original high-resolution image set and the generated high-resolution image set through the trained identifier model, wherein the generated high-resolution image set can replace the original high-resolution image set to a satisfactory effect.
The loss function of the discriminator model can be designed based on the goal function of GAN, and can be represented by the following formula:
further, the parameters of the generator model are iteratively updated by the calculation of equation (5) to enable the generated high resolution image set to replace the original high resolution image set.
By way of example, the generator model is continuously trained, and when the discriminator model can be deceived by the generated high-resolution image, it can be understood that when the discriminator model cannot judge whether the current image is the original high-resolution image or the generated high-resolution image, the training of the generator model can be considered to achieve a satisfactory effect, that is, the high-resolution image can be generated by the generator model. It should be noted that the satisfactory effect is a relative effect, and when the satisfactory effect is achieved, the training of the generator model can be continued, so that the generator model can be better applied.
Therefore, the training method for generating the countermeasure network for reconstructing the high-resolution picture, provided by the embodiment of the invention, introduces the image edge detection module into the generation countermeasure network, so that the high-frequency details of the generated high-resolution image and the original high-resolution image are better detected, and a generator model of the countermeasure network is better trained and generated; during network training, different thresholds are adopted according to different images, so that edge detail parts of the images can be better extracted, the extraction effect is improved, and a network model is better trained; adopting an improved residual block, combining a loss function introduced by an edge detection model, combining to generate a loss function of a countermeasure network, and training the network integrally; the edge detection module is combined to generate a countermeasure network, so that the generation quality of the image is improved more effectively; and adjusting network structure parameters and training different network structure models aiming at different application scenes, wherein in the actual application process, the optimal network structure model can be selected according to different scenes to obtain the optimal image generation quality. Namely, the embodiment of the invention can better recover the texture details and the high-frequency information part of the high-resolution picture, and has a very good recovery effect on the image recovery in a specific scene.
The embodiment of the invention also provides a high-resolution image reconstruction method based on the generation countermeasure network, wherein the generation countermeasure network comprises a generator model, a discriminator model and an edge detection model, and the high-resolution image reconstruction method based on the generation countermeasure network comprises the following steps: training the generation countermeasure network according to the training method for the high-resolution picture reconstruction generation countermeasure network of the steps S110 to S130; and reconstructing the high resolution image by the trained generator model of the countermeasure network.
By way of example, the generator model after generating the training of the countermeasure network through steps S110-S130 may be used to reconstruct a high-resolution image, and according to the foregoing, after the generator model acquires a low-resolution image corresponding to an original high-resolution image, the generator model reconstructs the image through processes of feature extraction, nonlinear mapping of features, reconstruction of the image, residual processing, and the like of the image through the convolutional neural network as shown in fig. 4, so that the resolution of the generated high-resolution image is infinitely close to that of the original high-resolution image.
The high-resolution image reconstruction method based on the generation countermeasure network can better reconstruct the pixel abrupt change part of the image when reconstructing high-frequency details of the image.
It should be noted that the content and effect of the high-resolution image reconstruction method based on the generation countermeasure network provided by the embodiment of the present invention are similar to those of steps S110 to S130, and are not described herein again.
An embodiment of the present invention further provides a control device, where the control device includes: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the training method for generating an antagonistic network for high resolution picture reconstruction according to steps S110-S130 and the high resolution image reconstruction method based on the generating antagonistic network described above.
The embodiment of the invention also provides a machine-readable storage medium, wherein the machine-readable storage medium stores instructions for enabling a machine to execute the training method for generating the countermeasure network for reconstructing the high-resolution picture according to the steps S110 to S130 and the high-resolution image reconstruction method based on the countermeasure network.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (13)
1. A training method for generating a countermeasure network for high resolution picture reconstruction, the method comprising a generator model, a discriminator model, and an edge detection model, and the training method for generating a countermeasure network for high resolution picture reconstruction comprising:
the generator model acquires a low-resolution image set generated from an original high-resolution image set, and trains by using the low-resolution image set to obtain a generated high-resolution image set;
the edge detection model respectively carries out edge detection on the original high-resolution image set and the generated high-resolution image set, and carries out iterative update on the generator model according to the result of the edge detection; and
and the discriminator model continuously compares and discriminates the original high-resolution image set and the generated high-resolution image set, and determines the training effect of the generator model according to the comparison and discrimination result.
2. The training method for generation of countermeasure network for high resolution picture reconstruction as claimed in claim 1, wherein the generator model for generation of countermeasure network comprises 5 convolutional layers, 12 residual blocks, 2 double-magnification upsampling layers,
and the residual block carries out splicing operation on the outputs of two adjacent convolutional layers through a ReLU activation function, and the tanh activation function is used in the last convolutional layer.
3. The training method for generating an antagonistic network for the reconstruction of high resolution pictures according to claim 1 or 2, wherein the generator model is trained with the set of low resolution images, and the obtaining of the generated set of high resolution images comprises:
the generator model takes the low-resolution image set as input and is trained through a loss function of the generator model, wherein the loss function of the generator model is a weighted sum of content loss, countermeasure loss and edge loss.
4. The training method for generation of countermeasure network for high resolution picture reconstruction as claimed in claim 3, wherein the content loss is calculated using MSE mean square error and is represented by the following formula:
where W, H denotes the width and height of the low resolution image,
representing the original high-resolution image or images,
representing the generated high resolution image and r represents an upsampling factor.
5. The training method for generating countermeasure network for high resolution picture reconstruction as claimed in claim 3, wherein the countermeasure loss is based on the discrimination result
In the generated high-resolution image set
Is determined and the challenge loss is represented by:
where N represents the number of samples of the generated high resolution image set.
6. The training method for generation of countermeasure network for high resolution picture reconstruction of claim 3, wherein the edge loss is represented by the following formula:
where W, H denotes the width and height of the original high resolution image, E denotes the edge detection model algorithm, E (I)SR) An edge detection image representing an image generated by a generator, (I)HR) An edge detection image representing the original high resolution image.
7. The training method for generating an antagonistic network for the reconstruction of high resolution pictures according to claim 1, wherein the edge detection model respectively performs edge detection on the original high resolution image set and the generated high resolution image set, and the iterative updating of the generator model according to the result of the edge detection comprises:
respectively carrying out edge extraction on the original high-resolution image set and the generated high-resolution image set through a Canny operator of the edge detection model;
determining an image edge threshold value through the maximum inter-class variance, respectively carrying out edge detection on the original high-resolution image set and the generated high-resolution image set according to the edge threshold value, and carrying out iterative updating on the generator model according to the edge detection result.
8. The training method for generation of countermeasure networks for high resolution picture reconstruction as claimed in claim 1, wherein the discriminator model comprises 8 convolutional layers, Sigmoid layers,
after the input generated high-resolution image set passes through the 8 convolutional layers, the output result is sent to a Sigmoid layer to obtain a classification output probability, and each layer adopts batch normalization processing and a Leaky ReLU activation function.
9. The training method for generating an antagonistic network for high resolution image reconstruction according to claim 1 or 8, wherein before said identifying the original high resolution image set and the generated high resolution image set respectively by the identifier model, the training method for generating an antagonistic network for high resolution image reconstruction further comprises:
the discriminator model is trained by the loss function of the discriminator model.
10. The training method for generating an antagonistic network for the reconstruction of high resolution pictures according to claim 9, wherein the identifying the original high resolution image set and the generated high resolution image set by the identifier model respectively, and determining the training effect of the generator model according to the identifying result comprises:
continuously and respectively identifying the original high-resolution image set and the generated high-resolution image set through the trained identifier model,
it is satisfactory that the generated high-resolution image set can replace the original high-resolution image set.
11. A high-resolution image reconstruction method based on a generative countermeasure network, the generative countermeasure network comprising a generator model, a discriminator model, and an edge detection model, the high-resolution image reconstruction method based on the generative countermeasure network comprising: the training method for generating the countermeasure network for high resolution picture reconstruction according to any one of claims 1 to 10, wherein the generation countermeasure network is trained; and
and reconstructing the high-resolution image through the trained generator model for generating the countermeasure network.
12. A control device, characterized in that the control device comprises: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the training method for generating an antagonistic network for high resolution picture reconstruction according to any one of claims 1 to 10 and the high resolution image reconstruction method based on generating an antagonistic network according to claim 11.
13. A machine-readable storage medium having stored thereon instructions for causing a machine to execute the training method for generating a countermeasure network for high resolution picture reconstruction according to any one of claims 1 to 10 and the high resolution image reconstruction method based on generating a countermeasure network according to claim 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110435203.3A CN113269722A (en) | 2021-04-22 | 2021-04-22 | Training method for generating countermeasure network and high-resolution image reconstruction method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110435203.3A CN113269722A (en) | 2021-04-22 | 2021-04-22 | Training method for generating countermeasure network and high-resolution image reconstruction method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113269722A true CN113269722A (en) | 2021-08-17 |
Family
ID=77229135
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110435203.3A Pending CN113269722A (en) | 2021-04-22 | 2021-04-22 | Training method for generating countermeasure network and high-resolution image reconstruction method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113269722A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113808021A (en) * | 2021-09-17 | 2021-12-17 | 北京金山云网络技术有限公司 | Image processing method and device, image processing model training method and device, and electronic equipment |
| CN113840147A (en) * | 2021-11-26 | 2021-12-24 | 浙江智慧视频安防创新中心有限公司 | Video processing method and device based on intelligent digital retina |
| CN114331903A (en) * | 2021-12-31 | 2022-04-12 | 电子科技大学 | Image restoration method and storage medium |
| CN114881864A (en) * | 2021-10-12 | 2022-08-09 | 北京九章云极科技有限公司 | Training method and device for seal restoration network model |
| CN115953394A (en) * | 2023-03-10 | 2023-04-11 | 中国石油大学(华东) | Target segmentation-based detection method and system for mesoscale ocean vortexes |
| CN116681790A (en) * | 2023-07-18 | 2023-09-01 | 脉得智能科技(无锡)有限公司 | Training method of ultrasound contrast image generation model and image generation method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109509152A (en) * | 2018-12-29 | 2019-03-22 | 大连海事大学 | A kind of image super-resolution rebuilding method of the generation confrontation network based on Fusion Features |
| CN110689562A (en) * | 2019-09-26 | 2020-01-14 | 深圳市唯特视科技有限公司 | Trajectory loop detection optimization method based on generation of countermeasure network |
| CN110766609A (en) * | 2019-08-29 | 2020-02-07 | 王少熙 | Depth-of-field map super-resolution reconstruction method for ToF camera |
| CN110992262A (en) * | 2019-11-26 | 2020-04-10 | 南阳理工学院 | Remote sensing image super-resolution reconstruction method based on generation countermeasure network |
| CN111062872A (en) * | 2019-12-17 | 2020-04-24 | 暨南大学 | Image super-resolution reconstruction method and system based on edge detection |
| CN112001847A (en) * | 2020-08-28 | 2020-11-27 | 徐州工程学院 | Method for generating high-quality image by relatively generating antagonistic super-resolution reconstruction model |
-
2021
- 2021-04-22 CN CN202110435203.3A patent/CN113269722A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109509152A (en) * | 2018-12-29 | 2019-03-22 | 大连海事大学 | A kind of image super-resolution rebuilding method of the generation confrontation network based on Fusion Features |
| CN110766609A (en) * | 2019-08-29 | 2020-02-07 | 王少熙 | Depth-of-field map super-resolution reconstruction method for ToF camera |
| CN110689562A (en) * | 2019-09-26 | 2020-01-14 | 深圳市唯特视科技有限公司 | Trajectory loop detection optimization method based on generation of countermeasure network |
| CN110992262A (en) * | 2019-11-26 | 2020-04-10 | 南阳理工学院 | Remote sensing image super-resolution reconstruction method based on generation countermeasure network |
| CN111062872A (en) * | 2019-12-17 | 2020-04-24 | 暨南大学 | Image super-resolution reconstruction method and system based on edge detection |
| CN112001847A (en) * | 2020-08-28 | 2020-11-27 | 徐州工程学院 | Method for generating high-quality image by relatively generating antagonistic super-resolution reconstruction model |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113808021A (en) * | 2021-09-17 | 2021-12-17 | 北京金山云网络技术有限公司 | Image processing method and device, image processing model training method and device, and electronic equipment |
| CN114881864A (en) * | 2021-10-12 | 2022-08-09 | 北京九章云极科技有限公司 | Training method and device for seal restoration network model |
| CN113840147A (en) * | 2021-11-26 | 2021-12-24 | 浙江智慧视频安防创新中心有限公司 | Video processing method and device based on intelligent digital retina |
| WO2023093339A1 (en) * | 2021-11-26 | 2023-06-01 | 浙江智慧视频安防创新中心有限公司 | Video processing method and apparatus based on intelligent digital retina |
| CN114331903A (en) * | 2021-12-31 | 2022-04-12 | 电子科技大学 | Image restoration method and storage medium |
| CN115953394A (en) * | 2023-03-10 | 2023-04-11 | 中国石油大学(华东) | Target segmentation-based detection method and system for mesoscale ocean vortexes |
| CN115953394B (en) * | 2023-03-10 | 2023-06-23 | 中国石油大学(华东) | Ocean mesoscale vortex detection method and system based on target segmentation |
| CN116681790A (en) * | 2023-07-18 | 2023-09-01 | 脉得智能科技(无锡)有限公司 | Training method of ultrasound contrast image generation model and image generation method |
| CN116681790B (en) * | 2023-07-18 | 2024-03-22 | 脉得智能科技(无锡)有限公司 | Training method of ultrasound contrast image generation model and image generation method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111047516B (en) | Image processing method, image processing device, computer equipment and storage medium | |
| Alhashim et al. | High quality monocular depth estimation via transfer learning | |
| CN113269722A (en) | Training method for generating countermeasure network and high-resolution image reconstruction method | |
| CN110853033B (en) | Video detection method and device based on inter-frame similarity | |
| CN110599421B (en) | Model training method, video fuzzy frame conversion method, device and storage medium | |
| WO2023087636A1 (en) | Anomaly detection method and apparatus, and electronic device, storage medium and computer program product | |
| CN112598579A (en) | Image super-resolution method and device for monitoring scene and storage medium | |
| CN113538480A (en) | Image segmentation processing method and device, computer equipment and storage medium | |
| CN111784624B (en) | Target detection method, device, equipment and computer readable storage medium | |
| CN114419102B (en) | Multi-target tracking detection method based on frame difference time sequence motion information | |
| CN108229281B (en) | Neural network generation method, face detection device and electronic equipment | |
| CN110942097A (en) | Imaging-free classification method and system based on single-pixel detector | |
| Hovhannisyan et al. | AED-Net: A single image dehazing | |
| CN116824641B (en) | Gesture classification method, device, equipment and computer storage medium | |
| Wang et al. | Deep intensity guidance based compression artifacts reduction for depth map | |
| Liu et al. | Arbitrary-scale super-resolution via deep learning: A comprehensive survey | |
| Kim et al. | Long-term video generation with evolving residual video frames | |
| CN116129417A (en) | Digital instrument reading detection method based on low-quality image | |
| CN113807354B (en) | Image semantic segmentation method, device, equipment and storage medium | |
| CN112052863B (en) | Image detection method and device, computer storage medium and electronic equipment | |
| CN116415019A (en) | Virtual reality VR image recognition method and device, electronic equipment and storage medium | |
| CN114841870A (en) | Image processing method, related device and system | |
| Falahatnejad et al. | PTSRGAN: Power transmission lines single image super-resolution using a generative adversarial network | |
| CN113902647A (en) | Image deblurring method based on double closed-loop network | |
| CN113658231A (en) | Optical flow prediction method, optical flow prediction device, electronic device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |