CN111325671B - Network training method and device, image processing method and electronic equipment - Google Patents
Network training method and device, image processing method and electronic equipment Download PDFInfo
- Publication number
- CN111325671B CN111325671B CN201811529196.8A CN201811529196A CN111325671B CN 111325671 B CN111325671 B CN 111325671B CN 201811529196 A CN201811529196 A CN 201811529196A CN 111325671 B CN111325671 B CN 111325671B
- Authority
- CN
- China
- Prior art keywords
- network
- image
- self
- training
- learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012549 training Methods 0.000 title claims abstract description 97
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000005457 optimization Methods 0.000 claims abstract description 38
- 230000007246 mechanism Effects 0.000 claims abstract description 15
- 230000003042 antagnostic effect Effects 0.000 claims abstract description 5
- 239000004973 liquid crystal related substance Substances 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 17
- 238000000605 extraction Methods 0.000 claims description 9
- 238000011084 recovery Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 230000009471 action Effects 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention relates to a network training method, a network training device, an image processing method and electronic equipment. The method is applied to a generative countermeasure network, the generative countermeasure network comprising: generating a network and an antagonizing network, the method comprising: acquiring a sample image of the current iterative training; inputting the sample image into a generation network to obtain a restored image; determining self-step learning parameters representing the difficulty level of the sample image according to the restored image and the reference image corresponding to the sample image; and training the countermeasure network and the generation network according to the self-learning parameters and the self-learning optimization mechanism during the next iterative training. The training mode is adjusted by adding the self-walking learning parameters into the training network, the model is enabled to learn simple samples autonomously in priority in the initial training stage, and the priority of learning complex samples is gradually increased along with the increase of training times, so that the disturbance of complex samples (images with low quality) to the generated network can be solved, and the generalization capability and autonomous learning capability of the model are enhanced.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a network training method, a device, an image processing method and electronic equipment.
Background
The image processing includes: image denoising, deblurring, defogging, raindrop removal, low-illumination enhancement, boundary enhancement, and the like. Image processing technology has been a difficulty in research and application fields for many years, and due to the complexity and variability of real scenes, image degradation problems are often encountered in application, and the singleness of related image processing technology limits the use of the related image processing technology in engineering. In many cases, the conventional image processing technology is to process and fuse a plurality of single problems, and the rationality of the conventional image processing technology is not doubtful. However, it can be found through analysis that the single image processing problems are not independent of each other, for example, in the case that noise and blurring exist simultaneously, the preferential denoising can cause partial loss of image texture information, further influence the subsequent deblurring process, and the preferential deblurring can not only cause inaccurate image restoration but also amplify noise factors; in another example, the processing of the low-illumination enhancement technology is not just to adjust the distribution of the brightness information, and research and application of the technology need to ensure both enhancement of the brightness information and great suppression of noise information.
The traditional image processing technology carries out degradation modeling aiming at each single problem, and then carries out restoration operation by combining the statistical model and the image prior information. Enhancement is performed, for example, in image denoising using a Bayesian model and noise assumptions; in image deblurring, modeling is performed using a maximum posterior probability model and a heavy tail distribution. The models are well researched, but the problems of poor disturbance resistance, weak general-waffle capability and the like exist for complex scenes, so that the images cannot be well enhanced by using the models.
Disclosure of Invention
In view of the above, an embodiment of the present invention is to provide a network training method, a device, an image processing method, and an electronic apparatus, so as to effectively improve the above-mentioned problems.
Embodiments of the present invention are implemented as follows:
in a first aspect, an embodiment of the present invention provides a network training method applied to a generated type countermeasure network, where the generated type countermeasure network includes: generating a network and an antagonizing network, the method comprising: acquiring a sample image of the current iterative training; inputting the sample image into the generation network to obtain a restored image; determining a self-learning parameter representing the difficulty level of the sample image according to the restored image and the reference image corresponding to the sample image; and training the countermeasure network and the generating network according to the self-step learning parameters and a self-step learning optimization mechanism when the next iteration is trained, wherein the countermeasure network is optimized firstly by using a single alternate iteration optimization method when the next iteration is trained, and then the generating network is optimized, and the self-step learning parameters are updated when each iteration is trained until the iteration is ended.
According to the embodiment of the application, the training mode is adjusted by adding the self-walking learning parameters into the training network, the model is enabled to learn the simple sample preferentially and autonomously in the initial training stage, and the priority of learning the complex sample is gradually increased along with the increase of training times, so that the disturbance of the complex sample (the image with low quality) to the generated network can be solved, the generalization capability and the autonomous learning capability of the model are enhanced, and the model has strong robustness.
With reference to a possible implementation manner of the first aspect embodiment, training the countermeasure network and the generating network according to the self-learning parameters and a self-learning optimization mechanism includes: training the countermeasure network and the generation network by the following formula;
wherein v is [0,1 ]]For the self-learning parameters x, < >>
The reference image and the restored image G, D are the generation network and the countermeasure network, x-p, respectively r For the statistical distribution of the reference image, +.>
For the statistical distribution of the restored image, E (-) is desired,/->
Is a conditional loss function that adjusts the self-learning parameter.
With reference to a possible implementation manner of the first aspect embodiment, the optimization formula of the countermeasure network is as follows:
wherein (1)>
For the mixed distribution of the reference image and the restored image,>
and taking the value 0-1 as the weighted sum of the reference image and the restored image, wherein alpha is the parameter of the gradient regularization term.
With reference to a possible implementation manner of the first aspect embodiment, the optimization formula of the generating network is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
c 1 、w 1 、h 1 the number, width and height of channels of the restored image at pixel level, c 2 、w 2 、h 2 The number, width and height of channels, F (), at the feature level, respectively, of the restored image is the feature extraction network.
With reference to one possible implementation manner of the first aspect embodiment, determining a self-learning parameter characterizing a difficulty level of the sample image according to the restored image and a reference image corresponding to the sample image includes: determining a self-learning parameter characterizing the difficulty level of the sample image by the following formula;
wherein, the liquid crystal display device comprises a liquid crystal display device,
f (v, t) represents the regularization term of the self-learning parameter in the t-th iteration, v= (v) 1 ,v 2 ,v 3 ,...,v n-1 ,v n )，/>
c 1 、w 1 、h 1 The number, width and height of channels of the restored image at pixel level, c 2 、w 2 、h 2 The number, width and height of channels of the restored image at the feature level, F () is a feature extraction network; the self-step learning parameters are obtained according to an optimization theory:
wherein (1)>
t represents the t-th iteration, N represents the number of samples, N max ＝n，/>
Representing the largest conditional loss function value in the t-1 th iteration, epoch being the largest number of iterations, +.>
And the optimal self-step learning parameter is the optimal self-step learning parameter of the ith sample image.
In a second aspect, an embodiment of the present invention further provides an image processing method, including: acquiring an image to be processed; inputting the image to be processed into a generated network trained by the network training method according to the embodiment of the first aspect and/or with any one of the possible implementation manners of the embodiment of the first aspect, so as to obtain a restored image.
In a third aspect, an embodiment of the present invention further provides a network training apparatus, applied to a generative countermeasure, where the generative countermeasure network includes: generating a network and an antagonizing network, the apparatus comprising: the system comprises a sample image acquisition module, a restored image acquisition module, a determination module and a training module; the sample image acquisition module is used for acquiring a sample image of the current iterative training; the recovery image acquisition module is used for inputting the sample image into the generation network to obtain a recovery image; the determining module is used for determining a self-learning parameter representing the difficulty level of the sample image according to the restored image and the reference image corresponding to the sample image; and the training module is used for training the countermeasure network and the generating network according to the self-walking learning parameters and the self-walking learning optimization mechanism during the next iterative training, wherein during the training, the countermeasure network is optimized firstly by utilizing a single alternate iterative optimization method, then the generating network is optimized, and the self-walking learning parameters are updated until the iteration is ended during each iterative training.
With reference to a possible implementation manner of the second aspect embodiment, the training module is specifically configured to train the countermeasure network and the generating network by the following formula;
wherein v is [0,1 ]]For the self-learning parameters x, < >>
The reference image and the restored image G, D are the generation network and the countermeasure network, x-p, respectively r For the statistical distribution of the reference image, +.>
For the statistical distribution of the restored image, E (-) is desired,/->
Is a conditional loss function that adjusts the self-learning parameter.
With reference to a possible implementation manner of the second aspect embodiment, the optimization formula of the countermeasure network is as follows:
wherein (1)>
For the mixed distribution of the reference image and the restored image,>
and taking the value 0-1 as the weighted sum of the reference image and the restored image, wherein alpha is the parameter of the gradient regularization term.
With reference to a possible implementation manner of the second aspect embodiment, the optimization formula of the generated network is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
c 1 、w 1 、h 1 the number, width and height of channels of the restored image at pixel level, c 2 、w 2 、h 2 The number, width and height of channels, F (), at the feature level, respectively, of the restored image is the feature extraction network.
With reference to a possible implementation manner of the second aspect embodiment, the determining module is specifically configured to determine a self-learning parameter that characterizes a difficulty level of the sample image by using the following formula;
wherein, the liquid crystal display device comprises a liquid crystal display device,
f (v, t) represents the regularization term of the self-learning parameter in the t-th iteration, v= (v) 1 ,v 2 ,v 3 ,...,v n-1 ,v n )，/>
c 1 、w 1 、h 1 The number, width and height of channels of the restored image at pixel level, c 2 、w 2 、h 2 The number, width and height of channels of the restored image at the feature level, F () is a feature extraction network; the self-step learning parameters are obtained according to an optimization theory:
wherein (1)>
t represents the t-th iteration, N represents the number of samples, N max ＝n，/>
Representing the largest conditional loss function value in the t-1 th iteration, epoch being the largest number of iterations, +.>
And the optimal self-step learning parameter is the optimal self-step learning parameter of the ith sample image.
In a fourth aspect, an embodiment of the present invention further provides an electronic device, including: the device comprises a memory and a processor, wherein the memory is connected with the processor; the memory is used for storing programs; the processor is configured to invoke the program stored in the memory to perform the embodiments of the first aspect and/or the methods provided in connection with any possible implementation of the embodiments of the first aspect; or to perform the method provided by the embodiments of the second aspect described above.
In a fifth aspect, the present embodiments further provide a storage medium comprising a computer program which, when executed by a computer, performs the above-described embodiments of the first aspect and/or the method provided in connection with any one of the possible implementations of the embodiments of the first aspect; or to perform the method provided by the embodiments of the second aspect described above.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. The above and other objects, features and advantages of the present invention will become more apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the several views of the drawings. The drawings are not intended to be drawn to scale, with emphasis instead being placed upon illustrating the principles of the invention.
Fig. 1 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Fig. 2 shows a flowchart of an image processing method according to an embodiment of the present invention.
Fig. 3 shows a training schematic of a generated countermeasure network according to an embodiment of the present invention.
Fig. 4 shows a flowchart of a network training method according to an embodiment of the present invention.
Fig. 5 shows a schematic block diagram of a network training device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
In the description of the present invention, it should be noted that the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance. Furthermore, the term "and/or" in this application is merely an association relation describing an association object, and indicates that three relations may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone.
First embodiment
As shown in fig. 1, fig. 1 shows a block diagram of an electronic device 100 according to an embodiment of the present invention. The electronic device 100 includes: memory 120, memory controller 130, and processor 140. The components and structures of the electronic device 100 shown in fig. 1 are exemplary only and not limiting, as the electronic device 100 may have other components and structures as desired.
The memory 120, the memory controller 130, and the processor 140 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.
The Memory 120 may be, but is not limited to, a random access Memory (Random Access Memory, RAM), a Read Only Memory (ROM), a programmable Read Only Memory (Programmable Read-Only Memory, PROM), an erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc. The memory 120 is configured to store a program, that is, a program required for executing the image processing method or the network training method according to the present embodiment, and the processor 140 executes the program after receiving an execution instruction, and the method executed by the electronic device 100 defined by the flow disclosed in any embodiment of the present invention may be applied to the processor 140 or implemented by the processor 140. After the processor 140 receives the execution instruction, the processor 140 may execute the image processing method or the flow of the network training method after calling the program stored in the memory 120 through the bus.
The processor 140 may be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art.
In this embodiment of the present invention, the electronic device 100 may be, but is not limited to, a network server, a database server, a cloud server, etc.
Referring to fig. 2, an image processing method applied to the electronic device 100 according to an embodiment of the present invention is described below with reference to fig. 2.
Step S101: and acquiring an image to be processed.
A degraded image, such as a blurred image, to be processed is acquired. The degraded image may be, for example, an image of an image degradation phenomenon caused by a relative motion between the object and the camera during photographing, and such an image is generally represented as an image texture becoming blurred, and in severe cases, may cause loss of image information. Or such an image that the photographed image is unclear because the photographed light is dark. In addition, the degraded image may be a simulated blurred image synthesized by a machine, in addition to a true blurred image captured by a camera.
Step S102: and inputting the image to be processed into a generation network trained in advance to obtain a restored image.
The degraded image is input into a generating network trained by the network training method, and the enhanced restored image can be obtained under the action of the generating network.
The generating network may be a generating network commonly used at present, such as a residual network, a Skip network, a Unet network, etc., which is not limited in the present application. The generalization capability and the learning capability of the generating network can be improved by training the generating network in the following way, so that the generating network has stronger robustness, and further, a restored image with good enhancement effect can be obtained.
Aiming at the problems of single mode, weak generalization capability and the like of the traditional image enhancement model, the embodiment of the invention provides a sample which adopts a self-learning optimization mechanism to enhance the learning and generalization capability of a generating network, and introduces self-learning parameters into the generating network and a judging network in model training so as to lead the self-learning parameters to learn the sample to be trained independently. Fig. 3 is a schematic diagram of training a generating network by using a self-learning optimization mechanism according to an embodiment of the present invention.
Among them, self-learning (self-past learning) is a learning methodology that, in combination with the idea of cognitive science, carefully adaptively prioritizes learning simple, reliable examples, and then gradually transitions to learning for difficult examples. For example, children typically begin learning simple courses from a first grade and then gradually transition from a second grade, a third grade, etc. to difficult courses; for another example, in the image deblurring process, a simple blur kernel is learned first, and then a complex blur kernel (e.g., image blur due to rotation, etc.) is learned.
In training the optimized generation network, an countermeasure network, i.e., a discrimination network, needs to be connected in series behind the generation network, as shown in fig. 3. The network and the countermeasure network are generated as two players, the purpose of the network is to generate vivid images as much as possible, so that the countermeasure network cannot identify true or false, and the purpose of the countermeasure network is to distinguish whether the input image is from a true sample set or a false sample set as much as possible, wherein the closer the output value is to 1, the greater the possibility that the input image is from the true sample set, the closer the output value is to 0, the greater the possibility that the input image is from the false sample set, and the two can be reversed. In the embodiment of the present application, parameters of the countermeasure network are shown in table 1.
Table 1 (challenge network parameter settings)
| # | Layer(s) | Parameter dimension | Step size |
| 1 | Convolutional layer | 32x3x5x5 | 2 |
| 2 | Convolutional layer | 64x32x5x5 | 1 |
| 3 | Convolutional layer | 64x64x5x5 | 2 |
| 4 | Convolutional layer | 128x64x5x5 | 1 |
| 5 | Convolutional layer | 128x128x5x5 | 4 |
| 6 | Convolutional layer | 256x128x5x5 | 1 |
| 7 | Convolutional layer | 256x256x5x5 | 4 |
| 8 | Convolutional layer | 512x256x5x5 | 1 |
| 9 | Convolutional layer | 512x512x4x4 | 4 |
| 10 | Full connection layer | 512x1x1x1 | - |
The parameter dimensions in table 1, such as 32x3x5x5, 128x128x5x5, etc., have the following meanings, the first digits, such as 32, 64, 128, etc., represent the number of channels of the current layer feature, the second digits, such as 3, 32, 64, etc., represent the number of channels of the previous layer feature, and the last two digits, such as 5x5, 4x4, and 1x1, represent the size of the convolution kernel.
The training generating type countermeasure network, that is, the training generating network and the countermeasure network, may be referred to as a network training method shown in fig. 4. The steps involved will be described below in connection with fig. 4.
Step S201: and acquiring a sample image of the current iteration training.
And acquiring a sample image when the current iterative training generates the countermeasure network. The sample image can be a real blurred image shot by a camera or a simulated blurred image synthesized by a machine. The sample image obtained is the sample image at the time of the t-th substitution. Wherein, t is a positive integer, such as 1, 2, 3 and … ….
Step S202: and inputting the sample image into the generation network to obtain a restored image.
After the sample image of the current iterative training is obtained, for example, after the sample image of t times of substitution is obtained, the sample image is input into a generating network to obtain a restored image, namely, the sample image is input into the generating network, and the output characteristic of the generating network is the restored image.
Step S203: and determining a self-learning parameter representing the difficulty level of the sample image according to the restored image and the reference image corresponding to the sample image.
After a restored image corresponding to the sample image is obtained, determining a self-learning parameter v representing the difficulty level of the sample image according to a reference image corresponding to the sample image and the restored image during the current iterative training.
As an alternative embodiment, the magnitude of the self-learning parameter characterizing the difficulty level of the sample image may be given according to the complexity level of the restored image and the reference image corresponding to the sample image. The size of the self-learning parameter can be determined according to whether the scene in the reference image corresponding to the restored image and the sample image is single, whether the texture in the image is large or small, and the like. In this embodiment, the size of the self-learning parameter may be set in advance, and when in use, the self-learning parameter may be matched with the corresponding self-learning parameter.
As a further alternative embodiment, a self-learning parameter characterizing the difficulty level of the sample image may be determined by the following formula; wherein, the formula is as follows:
The self-step learning parameter v can be obtained according to the optimization theory as follows:
Wherein, the liquid crystal display device comprises a liquid crystal display device,
t represents the t-th iteration, N represents the number of samples, N max ＝n，/>
Representing the largest conditional loss function value in the t-1 th iteration, epoch is the largest number of iterations, e.g. 300 +.>
Optimal self-walking learning parameters for the ith sample image, wherein the optimal self-walking learning parameters
I.e. the self-learning parameter v, where i has a value from 1 to n.
It should be noted that, the difficulty level of the sample image is different, and the corresponding self-learning parameter v is also different. That is, in the training process, each sample image has a parameter which is learned by self-step; after initializing the self-learning parameters, the network can gradually adjust the value of the self-learning parameter v according to the self-learning capability so as to lead the self-learning parameter v to learn the sample to be trained.
Wherein the value of the self-learning parameter v is 0-1; the larger the value of the self-learning parameter v, the higher the difficulty level of characterizing the sample image.
Step S204: and training the countermeasure network and the generating network according to the self-learning parameters and a self-learning optimization mechanism when training is performed in the next iteration.
After obtaining the self-learning parameters representing the difficulty level of the sample image of the current iterative training, updating the previous self-learning parameters so as to train the countermeasure network and the generating network according to the self-learning parameters and a self-learning optimization mechanism when the next iterative training is performed. During training, the countermeasure network is optimized firstly by using a single alternate iterative optimization method, then the generating network is optimized, and the self-learning parameters are updated when each time of iterative training is performed until the iteration is finished, wherein the iteration times can be set as required, for example, 300 times.
Wherein the countermeasure network and the generation network can be trained by the following formula;
During training, the restored image (false sample set) and the reference image (true sample set) output by the generating network (which can be the generating network commonly used at present, such as a residual network, a Skip network, a Unet network and the like) are respectively input into the countermeasure network, and the countermeasure network and the generating network are trained by using a single alternate iterative optimization method until the iteration is finished. And during optimization, optimizing the countermeasure network first, and then optimizing the generation network. The method for optimizing the countermeasure network is characterized in that the generation network is temporarily not considered when the countermeasure network is optimized, and the countermeasure network is temporarily not considered when the generation network is optimized.
In the embodiment of the application, in order to enhance the learning and generalization capability of the generated network, a self-walking learning parameter is added into a conditional loss function to adjust a training mode, a model is allowed to learn a simple sample autonomously preferentially in the initial stage of training, and the priority of learning a complex sample is gradually increased along with the increase of training times.
Wherein, the optimization formula of the countermeasure network is as follows:
Wherein, the optimization formula of the countermeasure network means: the first term + the second term + the third term is minimized. Wherein the first term is:
the second item is: />
The third item is->
The term is a gradient canonical term.
The optimization formula for generating the network is as follows:
Wherein, the liquid crystal display device comprises a liquid crystal display device,
c 1 、w 1 、h 1 the number, width and height of channels of the restored image at pixel level, c 2 、w 2 、h 2 The number, width and height of channels, F (), at the feature level, respectively, of the restored image is the feature extraction network. The model is constrained at two levels of pixels and features by using a conditional loss function, and noise generation is restrained by the constraint at the pixel level while the noise is subjected to Gaussian distribution by the constraint at the feature level. Wherein (1)>
For pixel level constraint, +.>
Is a feature level constraint.
For ease of understanding, the following will exemplify, for example, initializing the values of parameters in the model when the generating network and the countermeasure network are trained by the training method, where the values of the self-learning parameters are initialized default values. And in the first iteration, inputting the 1 st sample image into a generating network with the self-learning parameter as a default value to obtain a first recovery image, determining the self-learning parameter representing the difficulty degree of the 1 st sample image according to the first recovery image and a reference image corresponding to the 1 st sample image, obtaining the self-learning parameter of a first value of the current iteration according to the evaluation formula of the self-learning parameter, and then replacing the previous self-learning parameter by the self-learning parameter of the first value, namely replacing the default value, so that the generating network and the countermeasure network are trained based on the self-learning parameter of the first value in the next iteration training. In the second iteration, the 2 nd sample image is input into the self-learning parameter generation network with the first value to obtain a second recovery image, for example, the self-learning parameter representing the difficulty level of the 2 nd sample image is determined according to the second recovery image and the reference image corresponding to the 2 nd sample image, the self-learning parameter of the second value of the current iteration can be obtained according to the evaluation formula of the self-learning parameter, and then the self-learning parameter of the second value is utilized to replace the previous self-learning parameter, namely the self-learning parameter of the first value is replaced, so that the generation network and the countermeasure network are trained based on the self-learning parameter of the second value in the next iteration training. Training the generating network and the countermeasure network based on the self-learning parameters determined in the second iteration in the third iteration; and training the generating network and the countermeasure network based on the self-learning parameters determined by the third iteration in the fourth iteration, and so on until the iteration is ended.
The training mode is adjusted by adding self-walking learning parameters into the training network, the model is enabled to learn simple samples autonomously in priority in the initial stage of training, and the priority of learning complex samples is gradually increased along with the increase of training times. Therefore, the disturbance of complex samples (images with low quality) to the generated network can be solved, the generalization capability of the model is enhanced, and the model has strong robustness. The training method shown in the embodiment of the invention is used for verification under various generation networks, the test result is shown in the condition of training by using the same model, and the self-learning optimization mechanism in the embodiment of the invention can averagely improve the peak signal-to-noise ratio (Peak Signal to Noise Ratio, PSNR) of the image by 0.5dB on the basis of the original model training.
The generating network and the judging network are trained by the method, so that the finally output graph of the generating network can enable the countermeasure network, namely the judging network, to be unrecognizable, and the final output value is about 0.5. That is, when the output value of the countermeasure network is about 0.5, the generation network is shown to be optimal, that is, the generation network is trained, and the trained generation network can be used for recovering the blurred image.
Second embodiment
The embodiment of the application also provides a network training device 200, as shown in fig. 4. The network training apparatus 200 includes: a sample image acquisition module 210, a restored image acquisition module 220, a determination module 230, a training module 240.
The sample image obtaining module 210 is configured to obtain a sample image of the current iterative training.
The restored image obtaining module 220 is configured to input the sample image into the generating network, and obtain a restored image.
The determining module 230 is configured to determine a self-learning parameter that characterizes the difficulty level of the sample image according to the restored image and a reference image corresponding to the sample image. The determining module 230 is specifically configured to determine a self-learning parameter that characterizes the difficulty level of the sample image according to the following formula;
The training module 240 is configured to train the countermeasure network and the generating network according to the self-learning parameters and the self-learning optimization mechanism when training is performed in the next iteration, wherein during training, the countermeasure network is optimized first and then the generating network is optimized by using an independent alternate iteration optimization method, and the self-learning parameters are updated until the iteration is completed when training is performed in each iteration. Wherein the training module 240 is specifically configured to train the countermeasure network and the generating network through the following formula;
wherein v is [0,1 ]]For the self-learning parameters x, < >>
The reference image and the restored image G, D are the generation network and the countermeasure network, x-p, respectively r For the statistical distribution of the reference image, +.>
For statistical distribution of the restored image, E (deg.) is desired,
is a conditional loss function that adjusts the self-learning parameter.
It should be noted that, in the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described as different from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
The network training device 200 according to the embodiment of the present invention has the same implementation principle and technical effects as those of the foregoing method embodiment, and for brevity, reference may be made to the corresponding content of the foregoing method embodiment where the device embodiment is not mentioned.
Third embodiment
The present application also provides a computer-readable storage medium, on which a computer program is stored, which when executed by a computer performs the steps of the method described in the first embodiment. The specific implementation may refer to a method embodiment, which is not described herein.
In particular, the storage medium can be a general-purpose storage medium, such as a removable disk, a hard disk, or the like, and when the program code on the storage medium is executed, the image processing method or the network training method shown in the above embodiment can be executed.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present invention may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a notebook computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes. It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A method of network training, applied to a generative countermeasure network, the generative countermeasure network comprising: generating a network and an antagonizing network, the method comprising:
acquiring a sample image of the current iterative training;
inputting the sample image into the generation network to obtain a restored image;
determining a self-learning parameter representing the difficulty level of the sample image according to the restored image and the reference image corresponding to the sample image;
training the countermeasure network and the generating network according to the self-step learning parameters and a self-step learning optimization mechanism when the next iteration training is performed, wherein during the training, the countermeasure network is optimized firstly by using a single alternate iteration optimization method, then the generating network is optimized, and the self-step learning parameters are updated when each iteration training is performed until the iteration is finished;
the formula of the self-learning optimization mechanism is as follows:
2. The method of claim 1, wherein the optimization formula for the countermeasure network is as follows:
3. The method of claim 2, wherein the optimization formula for generating the network is as follows:
4. A method according to any one of claims 1-3, wherein determining a self-learning parameter characterizing the ease of the sample image from the restored image and a reference image corresponding to the sample image comprises:
determining a self-learning parameter characterizing the difficulty level of the sample image by the following formula;
5. An image processing method, comprising:
acquiring an image to be processed;
inputting the image to be processed into a generated network trained by the network training method according to any one of claims 1-4 to obtain a restored image.
6. A network training device for use in a generative countermeasure network, the generative countermeasure network comprising: generating a network and an antagonizing network, the apparatus comprising:
the sample image acquisition module is used for acquiring a sample image of the current iterative training;
the recovery image acquisition module is used for inputting the sample image into the generation network to obtain a recovery image;
the determining module is used for determining a self-learning parameter representing the difficulty level of the sample image according to the restored image and the reference image corresponding to the sample image;
the training module is used for training the countermeasure network and the generating network according to the self-walking learning parameters and a self-walking learning optimization mechanism during the next iterative training, wherein during the training, the countermeasure network is optimized firstly by using a single alternate iterative optimization method, then the generating network is optimized, and the self-walking learning parameters are updated until the iteration is finished during each iterative training;
the formula of the self-learning optimization mechanism is as follows:
v∈[0,1]For the self-learning parameters, x,
The reference image and the restored image, respectively, G, D the generating network and the countermeasure network, respectively, x-p r For the statistical distribution of the reference image, +.>
For the statistical distribution of the restored image, E (-) is desired,/->
Is a conditional loss function that adjusts the self-learning parameter.
7. An electronic device, comprising: the device comprises a memory and a processor, wherein the memory is connected with the processor;
the memory is used for storing programs;
the processor is configured to invoke a program stored in the memory to perform the method of any of claims 1-4 or 5.
8. A storage medium comprising a computer program which, when executed by a computer, performs the method of any one of claims 1-4 or 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811529196.8A CN111325671B (en) | 2018-12-13 | 2018-12-13 | Network training method and device, image processing method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811529196.8A CN111325671B (en) | 2018-12-13 | 2018-12-13 | Network training method and device, image processing method and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111325671A CN111325671A (en) | 2020-06-23 |
| CN111325671B true CN111325671B (en) | 2023-07-04 |
Family
ID=71170654
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811529196.8A Active CN111325671B (en) | 2018-12-13 | 2018-12-13 | Network training method and device, image processing method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111325671B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114697709B (en) * | 2020-12-25 | 2023-06-06 | 华为技术有限公司 | Video transmission method and device |
| CN114926348B (en) * | 2021-12-22 | 2024-03-01 | 天翼数字生活科技有限公司 | Device and method for removing low-illumination video noise |
| CN116503923B (en) * | 2023-02-16 | 2023-12-08 | 深圳市博安智控科技有限公司 | Method and device for training face recognition model |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446927A (en) * | 2016-07-07 | 2017-02-22 | 浙江大学 | Self-paced reinforcement image classification method and system |
| CN108596267A (en) * | 2018-05-03 | 2018-09-28 | Oppo广东移动通信有限公司 | A kind of image rebuilding method, terminal device and computer readable storage medium |
| CN108764281A (en) * | 2018-04-18 | 2018-11-06 | 华南理工大学 | A kind of image classification method learning across task depth network based on semi-supervised step certainly |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10636141B2 (en) * | 2017-02-09 | 2020-04-28 | Siemens Healthcare Gmbh | Adversarial and dual inverse deep learning networks for medical image analysis |
-
2018
- 2018-12-13 CN CN201811529196.8A patent/CN111325671B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446927A (en) * | 2016-07-07 | 2017-02-22 | 浙江大学 | Self-paced reinforcement image classification method and system |
| CN108764281A (en) * | 2018-04-18 | 2018-11-06 | 华南理工大学 | A kind of image classification method learning across task depth network based on semi-supervised step certainly |
| CN108596267A (en) * | 2018-05-03 | 2018-09-28 | Oppo广东移动通信有限公司 | A kind of image rebuilding method, terminal device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111325671A (en) | 2020-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109325954B (en) | Image segmentation method and device and electronic equipment | |
| CN111028177B (en) | Edge-based deep learning image motion blur removing method | |
| CN110969589A (en) | Dynamic scene fuzzy image blind restoration method based on multi-stream attention countermeasure network | |
| CN111325671B (en) | Network training method and device, image processing method and electronic equipment | |
| CN112215773B (en) | Local motion deblurring method and device based on visual saliency and storage medium | |
| CN112417955B (en) | Method and device for processing tour inspection video stream | |
| CN110782397B (en) | Image processing method, generation type countermeasure network, electronic equipment and storage medium | |
| Feng et al. | URNet: A U-Net based residual network for image dehazing | |
| Zhang et al. | Deep motion blur removal using noisy/blurry image pairs | |
| Zhao et al. | Deep pyramid generative adversarial network with local and nonlocal similarity features for natural motion image deblurring | |
| CN110796624B (en) | Image generation method and device and electronic equipment | |
| CN110880001A (en) | Training method, device and storage medium for semantic segmentation neural network | |
| CN110717864B (en) | Image enhancement method, device, terminal equipment and computer readable medium | |
| CN110782398B (en) | Image processing method, generative countermeasure network system and electronic device | |
| Saleem et al. | A non-reference evaluation of underwater image enhancement methods using a new underwater image dataset | |
| Chang et al. | Beyond camera motion blur removing: How to handle outliers in deblurring | |
| CN111696064B (en) | Image processing method, device, electronic equipment and computer readable medium | |
| CN115358952B (en) | Image enhancement method, system, equipment and storage medium based on meta-learning | |
| Zhao et al. | Natural image deblurring based on ringing artifacts removal via knowledge-driven gradient distribution priors | |
| CN111028182A (en) | Image sharpening method and device, electronic equipment and computer-readable storage medium | |
| CN108898557B (en) | Image restoration method and apparatus, electronic device, computer program, and storage medium | |
| CN112102208B (en) | Underwater image processing system, method, apparatus, and medium with edge preservation | |
| Maik et al. | Blind deconvolution using maximum a posteriori (MAP) estimation with directional edge based priori | |
| CN114119377A (en) | Image processing method and device | |
| Zhou et al. | Prior-enlightened and motion-robust video deblurring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |