CN111915491A - Weak supervision super-resolution reconstruction model and method based on distant and close scenes - Google Patents
Weak supervision super-resolution reconstruction model and method based on distant and close scenes Download PDFInfo
- Publication number
- CN111915491A CN111915491A CN202010820223.8A CN202010820223A CN111915491A CN 111915491 A CN111915491 A CN 111915491A CN 202010820223 A CN202010820223 A CN 202010820223A CN 111915491 A CN111915491 A CN 111915491A
- Authority
- CN
- China
- Prior art keywords
- range
- long
- image
- resolution
- discriminator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012549 training Methods 0.000 claims abstract description 106
- 238000010276 construction Methods 0.000 claims abstract description 5
- 238000012795 verification Methods 0.000 claims description 26
- 238000005457 optimization Methods 0.000 claims description 13
- 238000011176 pooling Methods 0.000 claims description 13
- 230000003247 decreasing effect Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 230000008485 antagonism Effects 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 238000000638 solvent extraction Methods 0.000 claims description 2
- 230000006870 function Effects 0.000 description 13
- 238000012360 testing method Methods 0.000 description 13
- 230000000694 effects Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000000452 restraining effect Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a remote and close scene based weakly supervised super resolution reconstruction model and a method thereof, wherein the model comprises the following components: the data set construction module is used for constructing a reconstruction data set of the far and near scene resolution for training; the training sample loading module loads a training sample; the first generator network is used for reconstructing the long-range view image of the loaded training sample to obtain a long-range view high-resolution image, inputting the obtained long-range view high-resolution image to the discriminator and inputting the obtained long-range view high-resolution image to the second generator network; the second generator network is used for reconstructing the input long-range high-resolution image and outputting a long-range low-resolution image, and the input long-range image, the first generator network output long-range high-resolution image and the second generator network output long-range low-resolution image form a closed loop; and the discriminator is used for discriminating the long-range high-resolution image reconstructed from the first generator network and the short-range image of the training sample and outputting a discrimination result.
Description
Technical Field
The invention relates to the technical field of computer image processing, in particular to a far and near view-based weak supervision super-resolution reconstruction method and device for learning far and near view images of traffic road conditions by using weak supervision learning and realizing super-resolution reconstruction of the far view images by using rich detail information of the near view images.
Background
In the prior art, under the condition that a vehicle shoots a distant view traffic road condition image in automatic driving and intelligent monitoring, super-resolution reconstruction is often required to be carried out on the low-resolution image containing the license plate number and the road sign information, so that the detail information is enriched.
At present, super-resolution reconstruction technology of a single natural image is developed and mature, various technologies can be realized, such as supervised super-resolution reconstruction methods based on deep learning, such as SRCNN, FSRCNN, RDN, EDSR, SRGAN and the like, but the methods cannot perform super-resolution reconstruction by using unpaired low-resolution images and high-resolution images, and therefore the effect of low-resolution image super-resolution reconstruction is not ideal when a true value of the low-resolution images cannot be obtained in a real scene.
Although the existing methods for performing super-resolution reconstruction based on weak supervised learning, such as CinCGAN and the like, can effectively solve the problem of super-resolution reconstruction of an unpaired single image, the resolution of a weak supervised image domain input into a discriminator is also required to be high, for example, if the data set required by the existing weak supervised super-resolution reconstruction method is an unpaired low-resolution image and a high-resolution image, in such a case, the weak supervised super-resolution reconstruction of an unpaired near-far image with the same resolution cannot be realized, and the discriminator of the existing weak supervised super-resolution reconstruction method can only input the generated high-resolution image and the image of the weak supervised high-resolution image domain with the same resolution.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a remote and near view-based weak supervision super-resolution reconstruction model and method, so as to achieve the purpose of super-resolution reconstruction of a remote view image by using rich detail information of the near view image.
In order to achieve the above object, the present invention provides a weakly supervised super resolution reconstruction model based on a close-range view, which includes:
the data set construction module is used for constructing a far and near view resolution reconstruction data set for training, and the data in the data set takes a far view image and a near view image which is randomly extracted as a training sample;
the training sample loading module is used for loading a training sample;
the first generator network is used for reconstructing a long-range image Fi of the loaded training sample to obtain a long-range high-resolution image Si, inputting the obtained long-range high-resolution image Si into the discriminator and inputting the obtained long-range high-resolution image Si into the second generator network;
the second generator network is used for reconstructing the input long-range high-resolution image Si and outputting a long-range low-resolution image Fi, inputting the long-range image Fi, outputting the long-range high-resolution image Si by the first generator network G and outputting the long-range low-resolution image Fi by the second generator network to form a closed loop;
and the discriminator is used for discriminating the long-range high-resolution image Si reconstructed from the first generator network and the short-range image Ni of the training sample and outputting a discrimination result.
Preferably, the training sample loading module loads a plurality of samples in batch, each training sample comprises a long-range view image and a randomly-extracted short-range view image, the size of each frame of picture is adjusted during loading, data enhancement is performed by rotation and inversion, and normalization processing is performed on pixel values of each picture.
Preferably, the discriminator uses a feature coding network of a CinCGAN discriminator to respectively extract and input corresponding feature maps, then uses a global pooling layer to pool two feature maps with different sizes to the same size, and finally uses a full connection layer to output a discrimination result of 0 or 1.
Preferably, the model is trained in a full process using a way to generate an confrontation.
Preferably, Adam is used as an optimizer for each generator and discriminator, and the reduced lronplateau mode is used as the learning rate attenuation mode.
In order to achieve the above object, the present invention further provides a method for reconstructing a weakly supervised super resolution based on a close and far view, comprising the following steps:
step S1, loading a training sample, and preprocessing the loaded training sample during loading, wherein the loaded training sample comprises a long-range view image and a short-range view image;
step S2, inputting a long-range image Fi with a training sample loaded in an iterative mode into a first generator network to be reconstructed to obtain a long-range high-resolution image Si, inputting the long-range high-resolution image Si obtained through the first generator network into a second generator network, and outputting a long-range low-resolution image Fi obtained by reconstructing the long-range high-resolution image Si;
step S3, inputting the long-range high-resolution image Si reconstructed in the first generator network and the short-range image Ni of the training sample into a discriminator, and outputting a discrimination result;
step S4, calculating the discriminator loss, and updating the discriminator parameter by back propagation according to the loss;
step S5, calculating loss of each generator, and updating generator parameters by back propagation according to the obtained loss;
step S6, after each training, using the sample in the verification set to perform one round of verification, and reducing the learning rate of each generator and each discriminator according to the average loss of each generator and each discriminator in the verification process;
and step S7, repeating the steps S1-S6, and carrying out iterative optimization until the training is finished.
Preferably, in step S1, a plurality of training samples are loaded in batch, and the size of each frame of picture is adjusted during loading, and data enhancement is performed by using rotation and inversion, and the pixel values of each picture are normalized.
Preferably, the input long-range image Fi, the first generator network outputs the long-range high-resolution image Si, and the second generator network finally outputs the long-range low-resolution image Fi to form a closed loop.
Preferably, in step S3, the feature coding network using the CinCGAN discriminator respectively extracts and inputs corresponding feature maps, then uses the global pooling layer to pool two feature maps of different sizes to the same size, and finally uses the full connection layer to output the discrimination result 0 or 1.
Preferably, in step S6, it is determined whether the average loss of each generator is always decreasing, and if the number of consecutive partitionings is kept constant, the generator learning rate is decreased by a set factor; if the average loss continuity probability of the discriminator is not decreased, the learning rate of the discriminator is decreased by a set factor.
Compared with the prior art, the remote and near view-based weak supervision super-resolution reconstruction model and method perform super-resolution reconstruction by using weak supervision learning, train the complete process by using a countermeasure generation mode, input a remote view image into a generator network to obtain a reconstructed high-resolution remote view image and a low-resolution remote view image recovered from the high-resolution remote view image, input a randomly extracted near view domain image and a reconstructed remote view high-resolution result into a discriminator to output a discrimination result, and achieve the purpose of realizing super-resolution reconstruction on the remote view image by using rich detail information of the near view image.
Drawings
FIG. 1 is a system architecture diagram of a close-range view-based weakly supervised super resolution reconstruction model according to the present invention;
FIG. 2 is a schematic diagram of a generator signature encoding network according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a discriminator feature coding network in an embodiment of the invention;
FIG. 4 is a flowchart illustrating steps of a super-resolution reconstruction method based on a close-and-distance view according to the present invention;
fig. 5 is a flowchart of weakly supervised super resolution reconstruction based on a far and near view image of traffic conditions in an embodiment of the present invention.
Detailed Description
Other advantages and capabilities of the present invention will be readily apparent to those skilled in the art from the present disclosure by describing the embodiments of the present invention with specific embodiments thereof in conjunction with the accompanying drawings. The invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention.
FIG. 1 is a system architecture diagram of a close-range view-based weakly supervised super resolution reconstruction model according to the present invention. As shown in fig. 1, the present invention provides a remote-view-based weakly supervised super-resolution reconstruction model, which includes:
and the data set construction module 101 is used for constructing a reconstruction data set for training with the far and near scene resolution.
In the embodiment of the present invention, the data set construction module 101 constructs a far-and-near-view traffic condition reconstruction data set for training by acquiring a far-and-near-view traffic condition image captured by a vehicle in automatic driving and intelligent monitoring, generally, it is required to ensure that the acquired image satisfies the discrimination of the far-and-near view without significant blur and noise interference, the image satisfying the above two points can be directly added into the far-and-near-view resolution reconstruction data set and the near-view resolution reconstruction data set, the data in the data set takes the far-view image and the randomly extracted near-view image as a training sample, the data set includes three parts of a training set, a verification set and a test set, and in the embodiment of the present invention, the acquired image is randomly selected according to a ratio of 1: 1: and 3, respectively dividing the training set into a test set, a verification set and a training set, wherein the training set is used during training, the verification set is input into the model to respectively calculate a generator loss function and a discriminator loss function after each epoch (current training round number) training, the current model training effect is judged through the loss functions, and the test set is used for finally evaluating the far and near scene super-resolution reconstruction effect of the model.
The training sample loading module 102 is configured to load a training sample, and perform preprocessing on the loaded training sample during loading, where the loaded training sample includes a distant view image and a near view image.
In the embodiment of the invention, batch loading is adopted for loading training samples, a plurality of samples are loaded each time, each training sample comprises a long-range view image and a randomly-extracted short-range view image, the size of each frame of picture is adjusted to 256 × 256 during loading, data enhancement is carried out by rotation and inversion, and normalization processing is carried out on the pixel value of each picture.
The first generator network 103 is configured to reconstruct a long-range image Fi of the loaded training sample to obtain a long-range high-resolution image Si, input the obtained long-range high-resolution image Si to the discriminator, and input the obtained long-range high-resolution image Si to the second generator network 104.
And the second generator network 104 is used for reconstructing the input high-resolution long-range image Si and outputting a low-resolution long-range image fi. The long-range image Fi input to the first generator network 103, the long-range high-resolution image Si output from the first generator network 103, and the long-range low-resolution image Fi output from the second generator network 104 are input to the first generator network 103 as the long-range image Fi to form a closed loop.
In an embodiment of the present invention, the first generator network 103 includes a feature coding network and an up-sampling network, the second generator network 104 includes a down-sampling network and a feature coding network, and the feature coding network portions of the first generator network 103 and the second generator network 104 both adopt network architectures used by a CinCGAN (unused Image Super-Resolution using Cycle-in-Cycle generated adaptive network) generator portion, as shown in fig. 2, in the figure, numbers after k, n, and s respectively represent a kernel size, a filter number, and a step size, for example, k3n64s1 refers to a convolutional layer including 64 filters, where a space size is 3 and a convolutional layer is 1, and for G generators G steps 1 and G2, 3 convolutional layers are used at the head and tail, and 6 residual blocks are used in the middle. Since the generator network adopted by the present invention is an existing generator network structure, it is not described herein.
And a discriminator 105 for discriminating the long-range high-resolution image Si reconstructed in the first generator network 103 from the short-range image Ni of the training sample, and outputting a discrimination result.
In the embodiment of the present invention, the discriminator used by CinCGAN is selected by the discriminator, except that a global pooling layer is added in front of the fully-connected layer of the discriminator used by CinCGAN to ensure that the randomly extracted low-resolution near-field image Ni and the reconstructed far-field high-resolution image Si obtain feature maps (feature maps) of the same size in front of the fully-connected layer of the discriminator, and the discriminator outputs the discrimination result, specifically, the feature maps corresponding to the feature maps are respectively extracted and input by using a feature coding network (as shown in fig. 3) of the CinCGAN discriminator, then the two feature maps of different sizes are pooled to the same size by using the global pooling layer, and finally the discrimination result 0 or 1 is output by using the fully-connected layer, fig. 3 is a structural diagram of the feature coding network of the CinCGAN discriminator in the embodiment of the present invention, and 70 × 70PatchGAN is used for the discriminator. The real and fake images refer to a near view image Ni of a training sample and a far view high resolution image Si obtained through reconstruction, because the discriminator of the invention adopts a PatchGAN structure, the final output of the input real and fake images is a feature map of 70x70, the final discriminator output result is obtained through averaging the output feature map, and because the global pooling of the discriminator is a very simple operation, the description is omitted here.
In the invention, on the basis of a training strategy, a mode of generating antagonism is used for training a model complete process, namely, a distant view image Fi is input into a first generator network to obtain a reconstructed high-resolution distant view image Si and a low-resolution distant view image Fi recovered from Si, a randomly extracted near view domain image Ni and a reconstructed distant view high-resolution result Si are input into a discriminator, corresponding feature maps are respectively extracted and input by using a feature coding network of a CinCGAN discriminator, then two feature maps with different sizes are pooled to the same size by using a global pooling layer (global pooling), and finally a discrimination result 0 or 1 is output by using a full connection layer, so that the antagonism training is carried out alternately under the same epoch.
In the invention, training data are loaded from a training set during training, and training in a model complete process is carried out in a mode of generating an confrontation, after each training round, the samples in the verification set are used for verification, so that a training result is judged, and after the complete training is successful, the samples in the test set are used for testing and evaluating the model, and the processes of verification and testing are similar to those of training, so that the details are not repeated.
Specifically, the invention uses Adam as an optimizer of a generator and a discriminator, and the learning rate attenuation mode uses a reduce LROnPlateau mode. In the loss function part, the loss function adopted by the generator comprises three parts: the method comprises the steps of confrontation loss, cycle consistency loss and identity loss, wherein the confrontation loss is responsible for improving the effect of generating a confrontation training mode, the cycle consistency loss ensures the cycle consistency of the whole generator, the weak supervision learning effect is improved, and the identity loss is beneficial to restraining the color and the brightness of the generated high-resolution image. And after the loss is solved, the network parameters are updated by back propagation, and iterative optimization is repeatedly completed until the training is completed. In the invention, the attenuation of the learning rate refers to that when the loss on the verification set does not decrease or even rises, the learning rate is selected to be reduced, so that the network can learn a local optimal point which is beneficial to reducing the loss on the verification set more easily. According to the invention, whether the learning rate is reduced or not is judged through the verification set, so that on one hand, network overfitting is effectively avoided, on the other hand, the network generalization performance is improved, and the training process is accelerated.
Fig. 4 is a flowchart of steps of a weakly supervised super resolution reconstruction method based on a distant view. As shown in fig. 4, the weakly supervised super resolution reconstruction method based on the close and far view of the present invention includes the following steps:
step S1, loading a training sample, and preprocessing the loaded training sample during loading, where the loaded training sample includes a distant view image and a near view image.
In the specific embodiment of the invention, firstly, a training far-and-near view resolution reconstruction data set needs to be constructed, the data in the data set takes a far-view image and a randomly-extracted near-view image as a training sample, and the data set comprises a training set, a verification set and a test set.
The loading of the training samples adopts batch loading, a plurality of samples are loaded each time, each training sample comprises a long-range view image and a randomly-extracted short-range view image, the size of each frame of picture is adjusted to be 256 x 256 during loading, data enhancement is carried out by rotation and overturning, and normalization processing is carried out on the pixel value of each picture.
Step S2, inputting the distant view image Fi loaded with the training sample in an iterative mode into a first generator network to reconstruct to obtain a distant view high-resolution image Si, inputting the distant view high-resolution image Si obtained by the first generator network into a second generator network, outputting a distant view low-resolution image Fi obtained by reconstructing the distant view high-resolution image Si, outputting the distant view high-resolution image Si by inputting the distant view image Fi, outputting the distant view high-resolution image Si by the first generator network, and finally outputting the distant view low-resolution image Fi by the second generator network to form a closed loop.
And step S3, inputting the long-range high-resolution image Si reconstructed from the first generator network and the short-range image Ni of the training sample into a discriminator, and outputting a discrimination result.
In the embodiment of the invention, the discriminator used by CinCGAN is selected by the discriminator, and the difference is that a global pooling layer is added in front of the full connection layer of the discriminator used by CinCGAN to ensure that the low-resolution near-field image Ni and the reconstructed far-field high-resolution image Si are randomly extracted to obtain feature maps (feature maps) with the same size in front of the full connection layer of the discriminator, and finally the discrimination result is output by the discriminator, specifically, the feature maps corresponding to the extraction and input of the feature maps are respectively extracted by using a feature coding network of the CinCGAN discriminator, then the two feature maps with different sizes are pooled to the same size by using the global pooling layer, and finally the discrimination result is output by using the full connection layer to be 0 or 1.
In the aspect of a training strategy, because the super-resolution reconstruction is carried out by using weak supervised learning, the method directly uses a countermeasure generation mode to carry out the training of the complete process, namely, a long-range image Fi is input into a first generator network to obtain a reconstructed high-resolution long-range image Si and a low-resolution long-range image Fi obtained by restoring Si through a second generator network. Inputting a randomly extracted near-field image Ni and a reconstructed far-field high-resolution result Si into a discriminator, respectively extracting and inputting corresponding feature maps by using a feature coding network of a CinCGAN discriminator, pooling the two feature maps with different sizes to the same size by using global posing, and outputting a discrimination result 0 or 1 by using a full-link layer, thereby alternately resisting training under the same epoch.
In step S4, the discriminator loss is calculated, and the discriminator parameter is updated by back propagation based on the calculated discriminator loss.
In the present invention, the arbiter penalty is the typical arbiter penalty of the generation countermeasure network, and its penalty function can be as follows:
since the loss function of the discriminator loss is consistent with the discriminator loss function of the generation countermeasure network typical in the prior art, it is not described herein.
Like other weak supervision generation countermeasure networks, when the discriminator is expected to be optimized, a high-resolution long-range image obtained by reconstruction is judged to be 0, and a short-range low-resolution image is judged to be 1; when the generator is optimized, the opposite is true, namely, the discrimination result of the discriminator is that the high-resolution long-range image is judged to be 1, and the low-resolution image of the short-range area is judged to be 0. The optimization of the generator and the arbiter is performed iteratively, the generator optimization is performed first, and then the arbiter optimization is performed, although the arbiter optimization may be performed first and then the generator optimization is performed, which have no influence on the two, and the present invention is not limited thereto.
In step S5, the loss of each generator is calculated, and the generator parameters are updated by back propagation based on the calculated loss. And calculating the loss of the two generators simultaneously, calculating the total loss, wherein the total loss function comprises the loss parts of the two generators respectively, and performing back propagation optimization after the total loss is obtained.
In a specific embodiment of the present invention, the loss function employed by the generator comprises three parts: the method comprises the steps of resisting loss, cycle consistency loss and identity loss, wherein the resisting loss is responsible for improving the effect of generating a resisting training mode, the cycle consistency loss ensures the cycle consistency of the whole generator and improves the weak supervision learning effect, the identity loss is beneficial to restraining the color and the brightness of a generated high-resolution image, the network parameters are updated by back propagation after the loss is solved, and iterative optimization is repeatedly completed until the training is finished. Since the generator network adopted in the present invention is a typical generator network structure in the prior art, the loss function is also disclosed in the prior art, and is not described herein again.
Step S6, after each round of training, using the sample in the verification set to perform a round of verification, and judging whether the average loss of each generator is reduced all the time in the verification process, if the continuous probability is kept not to be reduced, reducing the learning rate of each generator by a set multiple, wherein the average loss of the generator refers to the average loss of each epoch (current training round number) generator, because each iteration generates loss, and the average loss is the sum of all iteration losses under one epoch and is averaged; if the continuous probability of the average loss of the discriminator is not decreased, the learning rate of the discriminator is decreased by a set factor, where the average loss of the discriminator is the sum of the discriminator losses per iteration/total batch size.
In the present invention, the verification process is the same as the training process, but the verification process loads verification data in the verification set, which is not described herein again.
In step S7, steps S1-S6 are repeated, and iterative optimization is performed until training is completed, for example, when the learning rate (determiner or generator) is lower than a preset threshold, the training is completed, or the loss (determiner or generator) is not reduced.
Preferably, after the training and verification are successful, the test data of the test set may be used to perform test evaluation on the model after step S7, and the test process of the test data is the same as the training and verification process, which is not described herein again.
Examples
As shown in fig. 5, in this embodiment, a weakly supervised super resolution reconstruction process based on far and near scene images of traffic road conditions is as follows:
s1, firstly, a far-and-near-view super-resolution reconstruction data set for training is manufactured, the data set comprises a training set, a verification set and a test set, and the data takes a far-view image and a near-view image which is randomly extracted as a training sample.
And S2, loading data in batches, loading a plurality of samples each time, adjusting the size of each frame of picture to 256 × 256 during loading, performing data enhancement by using rotation and inversion, and then normalizing the pixel values.
S3, inputting the distant view image Fi into a generator network G, the generator G including two parts: feature coding networks and up-sampling networks. After the generator G outputs the generated high-resolution long-range image Si, the generator G inputs Si to a generator F, which also includes two parts: a downsampling network and a feature coding network. The generator F outputs a distant view low resolution image fi reconstructed from the output distant view high resolution image Si. The input Fi is used for outputting Si by the generator, and the final output Fi by the generator F is used for forming a closed loop. The feature encoding network part of generators G and F employs the network architecture used by the CinCGAN generator part.
S4, selecting a discriminator used by CinCGAN in the discriminator part, wherein the difference is that a global posing layer is added in front of a full connection layer of the discriminator to ensure that a low-resolution near-field image and a reconstructed far-field high-resolution image are randomly extracted to obtain a feature map with the same size in front of the full connection layer, and finally outputting a discrimination result by the discriminator. As with other weak supervision generation countermeasure networks, when the discriminator is expected to be optimized, the high-resolution long-range image obtained by reconstruction is judged to be 0, and the low-resolution image in the close-range is judged to be 1; the opposite happens when the generator is optimized.
S5, in the training strategy, because the super-resolution reconstruction is carried out by using the weak supervised learning, the training of the whole process is directly carried out by using a countermeasure generating mode, and the long-range images Fi are input into the generator network to obtain the reconstructed high-resolution long-range images Si and the low-resolution long-range images Fi recovered from Si. Inputting a randomly extracted near-field image Ni and a reconstructed far-field high-resolution result Si into a discriminator, respectively extracting and inputting corresponding feature maps by using a feature coding network of a CinCGAN discriminator, pooling the two feature maps with different sizes to the same size by using global posing, and outputting a discrimination result 0 or 1 by using a full-link layer, thereby alternately resisting training under the same epoch.
S6, using Adam as a generator and an optimizer of a discriminator, and using a ReduceLROnPlateau mode as a learning rate attenuation mode. In the loss function part, the loss function adopted by the generator comprises three parts: the method comprises the steps of resisting loss, cycle consistency loss and identity loss, wherein the resisting loss is responsible for improving the effect of generating a resisting training mode, the cycle consistency loss ensures the cycle consistency of the whole generator and improves the weak supervision learning effect, the identity loss is beneficial to restraining the color and the brightness of a generated high-resolution image, the network parameters are updated by back propagation after the loss is solved, and iterative optimization is repeatedly completed until the training is finished.
And S7, repeating the steps from S2 to S6, and performing iterative optimization until the training is finished, wherein Epoch represents the number of current training rounds, and epochs represents how many rounds are to be trained. The invention adopts the idea of generating confrontation in the training process, namely, the network part is defined as a generator, and a cross-resolution discriminator is designed at the same time, and the confrontation training is carried out alternately under the same epoch.
In summary, the weakly supervised super resolution reconstruction model and method based on the close and far views of the present invention performs super resolution reconstruction by using weakly supervised learning, performs training in a complete process by using a countermeasure generation mode, inputs the long-view image into the generator network to obtain a reconstructed high resolution long-view image and a low resolution long-view image recovered from the high resolution long-view image, and inputs the randomly extracted close-view domain image and the reconstructed long-view high resolution result into the discriminator to output the discrimination result, thereby achieving the purpose of realizing super resolution reconstruction of the long-view image by using rich detail information of the close-view image.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Modifications and variations can be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of the invention should be determined from the following claims.
Claims (10)
1. A long-and-short-range-scene-based weakly-supervised super-resolution reconstruction model comprises:
the data set construction module is used for constructing a far and near view resolution reconstruction data set for training, and the data in the data set takes a far view image and a near view image which is randomly extracted as a training sample;
the training sample loading module is used for loading a training sample;
the first generator network is used for reconstructing a long-range image Fi of the loaded training sample to obtain a long-range high-resolution image Si, inputting the obtained long-range high-resolution image Si into the discriminator and inputting the obtained long-range high-resolution image Si into the second generator network;
the second generator network is used for reconstructing the input long-range high-resolution image Si and outputting a long-range low-resolution image Fi, inputting the long-range image Fi, outputting the long-range high-resolution image Si by the first generator network G and outputting the long-range low-resolution image Fi by the second generator network to form a closed loop;
and the discriminator is used for discriminating the long-range high-resolution image Si reconstructed from the first generator network and the short-range image Ni of the training sample and outputting a discrimination result.
2. The weakly supervised super resolution reconstruction model based on close range view of claim 1, wherein: the training sample loading module loads a plurality of samples in batch, each training sample comprises a long-range view image and a randomly-extracted short-range view image, the size of each frame of picture is adjusted during loading, data enhancement is carried out by rotation and overturning, and normalization processing is carried out on the pixel value of each picture.
3. The weakly supervised super resolution reconstruction model based on close range view of claim 1, wherein: the discriminator respectively extracts and inputs corresponding feature maps by using a feature coding network of a CinCGAN discriminator, then uses a global pooling layer to pool two feature maps with different sizes to the same size, and finally uses a full connection layer to output a discrimination result of 0 or 1.
4. The weakly supervised super resolution reconstruction model based on close range view of claim 3, wherein: the model is trained in a complete process using the way antagonism is generated.
5. The weakly supervised super resolution reconstruction model based on close range view of claim 1, wherein: adam is used as an optimizer of each generator and discriminator, and a ReduceLROnPlateau mode is used as a learning rate attenuation mode.
6. A weak supervision super-resolution reconstruction method based on a far and near scene comprises the following steps:
step S1, loading a training sample, and preprocessing the loaded training sample during loading, wherein the loaded training sample comprises a long-range view image and a short-range view image;
step S2, inputting a long-range image Fi with a training sample loaded in an iterative mode into a first generator network to be reconstructed to obtain a long-range high-resolution image Si, inputting the long-range high-resolution image Si obtained through the first generator network into a second generator network, and outputting a long-range low-resolution image Fi obtained by reconstructing the long-range high-resolution image Si;
step S3, inputting the long-range high-resolution image Si reconstructed in the first generator network and the short-range image Ni of the training sample into a discriminator, and outputting a discrimination result;
step S4, calculating the discriminator loss, and updating the discriminator parameter by back propagation according to the loss;
step S5, calculating loss of each generator, and updating generator parameters by back propagation according to the obtained loss;
step S6, after each training, using the sample in the verification set to perform one round of verification, and reducing the learning rate of each generator and each discriminator according to the average loss of each generator and each discriminator in the verification process;
and step S7, repeating the steps S1-S6, and carrying out iterative optimization until the training is finished.
7. The method as claimed in claim 6, wherein in step S1, a plurality of training samples are loaded in batch, and the size of each frame of picture is adjusted during loading, and data enhancement is performed by rotation and inversion, and the pixel values of each picture are normalized.
8. The method for weakly supervised super resolution reconstruction based on close range view as claimed in claim 6, wherein: and the input long-range image Fi is used for outputting a long-range high-resolution image Si by the first generator network, and the second generator network is used for outputting a long-range low-resolution image Fi finally to form a closed loop.
9. The method for weakly supervised super resolution reconstruction based on close range view as claimed in claim 6, wherein: in step S3, the feature coding network of the CinCGAN discriminator is used to extract and input corresponding feature maps, then the two feature maps with different sizes are pooled to the same size by using the global pooling layer, and finally the discrimination result 0 or 1 is output by using the full connection layer.
10. The method for weakly supervised super resolution reconstruction based on close range view as claimed in claim 6, wherein: in step S6, it is determined whether the average loss of each generator is always reduced, and if the number of consecutive partitionings is kept constant, the generator learning rate is reduced by a set factor; if the average loss continuity probability of the discriminator is not decreased, the learning rate of the discriminator is decreased by a set factor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010820223.8A CN111915491A (en) | 2020-08-14 | 2020-08-14 | Weak supervision super-resolution reconstruction model and method based on distant and close scenes |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010820223.8A CN111915491A (en) | 2020-08-14 | 2020-08-14 | Weak supervision super-resolution reconstruction model and method based on distant and close scenes |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111915491A true CN111915491A (en) | 2020-11-10 |
Family
ID=73283180
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010820223.8A Pending CN111915491A (en) | 2020-08-14 | 2020-08-14 | Weak supervision super-resolution reconstruction model and method based on distant and close scenes |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111915491A (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
| CN109978762A (en) * | 2019-02-27 | 2019-07-05 | 南京信息工程大学 | A kind of super resolution ratio reconstruction method generating confrontation network based on condition |
| CN110136063A (en) * | 2019-05-13 | 2019-08-16 | 南京信息工程大学 | A kind of single image super resolution ratio reconstruction method generating confrontation network based on condition |
| CN110211045A (en) * | 2019-05-29 | 2019-09-06 | 电子科技大学 | Super-resolution face image method based on SRGAN network |
| CN110458758A (en) * | 2019-07-29 | 2019-11-15 | 武汉工程大学 | A kind of image super-resolution rebuilding method, system and computer storage medium |
| CN110689482A (en) * | 2019-09-18 | 2020-01-14 | 中国科学技术大学 | Face super-resolution method based on supervised pixel-by-pixel generation countermeasure network |
| CN110706157A (en) * | 2019-09-18 | 2020-01-17 | 中国科学技术大学 | Face super-resolution reconstruction method for generating confrontation network based on identity prior |
| KR102067629B1 (en) * | 2019-10-18 | 2020-01-17 | 국방과학연구소 | Learning method and apparatus for improved resolution of low resolution satellite images |
| CN110827201A (en) * | 2019-11-05 | 2020-02-21 | 广东三维家信息科技有限公司 | Generative confrontation network training method and device for high-dynamic-range image super-resolution reconstruction |
| US20200111194A1 (en) * | 2018-10-08 | 2020-04-09 | Rensselaer Polytechnic Institute | Ct super-resolution gan constrained by the identical, residual and cycle learning ensemble (gan-circle) |
| CN110992262A (en) * | 2019-11-26 | 2020-04-10 | 南阳理工学院 | Remote sensing image super-resolution reconstruction method based on generation countermeasure network |
| CN111179187A (en) * | 2019-12-09 | 2020-05-19 | 南京理工大学 | Single image rain removing method based on cyclic generation countermeasure network |
-
2020
- 2020-08-14 CN CN202010820223.8A patent/CN111915491A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
| US20200111194A1 (en) * | 2018-10-08 | 2020-04-09 | Rensselaer Polytechnic Institute | Ct super-resolution gan constrained by the identical, residual and cycle learning ensemble (gan-circle) |
| CN109978762A (en) * | 2019-02-27 | 2019-07-05 | 南京信息工程大学 | A kind of super resolution ratio reconstruction method generating confrontation network based on condition |
| CN110136063A (en) * | 2019-05-13 | 2019-08-16 | 南京信息工程大学 | A kind of single image super resolution ratio reconstruction method generating confrontation network based on condition |
| CN110211045A (en) * | 2019-05-29 | 2019-09-06 | 电子科技大学 | Super-resolution face image method based on SRGAN network |
| CN110458758A (en) * | 2019-07-29 | 2019-11-15 | 武汉工程大学 | A kind of image super-resolution rebuilding method, system and computer storage medium |
| CN110689482A (en) * | 2019-09-18 | 2020-01-14 | 中国科学技术大学 | Face super-resolution method based on supervised pixel-by-pixel generation countermeasure network |
| CN110706157A (en) * | 2019-09-18 | 2020-01-17 | 中国科学技术大学 | Face super-resolution reconstruction method for generating confrontation network based on identity prior |
| KR102067629B1 (en) * | 2019-10-18 | 2020-01-17 | 국방과학연구소 | Learning method and apparatus for improved resolution of low resolution satellite images |
| CN110827201A (en) * | 2019-11-05 | 2020-02-21 | 广东三维家信息科技有限公司 | Generative confrontation network training method and device for high-dynamic-range image super-resolution reconstruction |
| CN110992262A (en) * | 2019-11-26 | 2020-04-10 | 南阳理工学院 | Remote sensing image super-resolution reconstruction method based on generation countermeasure network |
| CN111179187A (en) * | 2019-12-09 | 2020-05-19 | 南京理工大学 | Single image rain removing method based on cyclic generation countermeasure network |
Non-Patent Citations (1)
| Title |
|---|
| JING NAN ET AL: "Image super-resolution reconstructing based on generative adversarial network", 《APPLIED OPTICS AND PHOTONICS CHINA (AOPC2019)》, pages 113420 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110135366B (en) | Shielded pedestrian re-identification method based on multi-scale generation countermeasure network | |
| CN108520503B (en) | Face defect image restoration method based on self-encoder and generation countermeasure network | |
| CN110378844B (en) | Image blind motion blur removing method based on cyclic multi-scale generation countermeasure network | |
| CN112507997B (en) | Face super-resolution system based on multi-scale convolution and receptive field feature fusion | |
| CN109035142B (en) | Satellite image super-resolution method combining countermeasure network with aerial image prior | |
| CN112001847A (en) | Method for generating high-quality image by relatively generating antagonistic super-resolution reconstruction model | |
| CN110473142B (en) | Single image super-resolution reconstruction method based on deep learning | |
| CN111915490A (en) | License plate image super-resolution reconstruction model and method based on multi-scale features | |
| CN109214989A (en) | Single image super resolution ratio reconstruction method based on Orientation Features prediction priori | |
| CN110363068B (en) | High-resolution pedestrian image generation method based on multiscale circulation generation type countermeasure network | |
| CN110827201A (en) | Generative confrontation network training method and device for high-dynamic-range image super-resolution reconstruction | |
| CN112215119B (en) | Small target identification method, device and medium based on super-resolution reconstruction | |
| CN111833277B (en) | Marine image defogging method with unpaired multi-scale mixed coding and decoding structure | |
| CN109949222A (en) | Image super-resolution rebuilding method based on grapheme | |
| CN115564649B (en) | Image super-resolution reconstruction method, device and equipment | |
| CN113379601A (en) | Real world image super-resolution method and system based on degradation variational self-encoder | |
| CN111383173A (en) | Baseline-based image super-resolution reconstruction method and system | |
| CN116563682A (en) | Attention scheme and strip convolution semantic line detection method based on depth Hough network | |
| Liu et al. | Facial image inpainting using multi-level generative network | |
| CN114283058A (en) | Image super-resolution reconstruction method based on countermeasure network and maximum mutual information optimization | |
| CN113592715A (en) | Super-resolution image reconstruction method for small sample image set | |
| CN112365405A (en) | Unsupervised super-resolution reconstruction method based on generation countermeasure network | |
| CN111814693A (en) | Marine ship identification method based on deep learning | |
| CN115880158B (en) | Blind image super-resolution reconstruction method and system based on variation self-coding | |
| CN111784583A (en) | Cyclic random super-resolution generation countermeasure network for precipitation graph |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |