CN114240764A - Deblurring convolution neural network training method, device, equipment and storage medium - Google Patents
Deblurring convolution neural network training method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114240764A CN114240764A CN202111342163.4A CN202111342163A CN114240764A CN 114240764 A CN114240764 A CN 114240764A CN 202111342163 A CN202111342163 A CN 202111342163A CN 114240764 A CN114240764 A CN 114240764A
- Authority
- CN
- China
- Prior art keywords
- deblurring
- fuzzy
- training
- image
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012549 training Methods 0.000 title claims abstract description 269
- 238000000034 method Methods 0.000 title claims abstract description 64
- 238000013528 artificial neural network Methods 0.000 title abstract description 6
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 153
- 230000008447 perception Effects 0.000 claims abstract description 72
- 230000006870 function Effects 0.000 claims description 117
- 230000033001 locomotion Effects 0.000 claims description 56
- 238000012545 processing Methods 0.000 claims description 27
- 230000015654 memory Effects 0.000 claims description 25
- 230000011218 segmentation Effects 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- 238000002372 labelling Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 14
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 239000000463 material Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The application provides a method, a device, equipment and a storage medium for deblurring convolutional neural network training, wherein the method comprises the following steps: acquiring a fuzzy image training set, wherein the fuzzy image training set comprises a local fuzzy training set and a global fuzzy training set; constructing an initial deblurring convolutional neural network, which comprises a fuzzy area perception network and a deblurring network; the deblurring network comprises a fuzzy region perception attention module and a deblurring module; respectively training the fuzzy region perception network and the fuzzy region perception attention module by the local fuzzy training set, and inputting the fuzzy image training set into a deblurring module for training to obtain an intermediate deblurring convolution neural network; and alternately inputting the local fuzzy training set and the global fuzzy training set into the intermediate deblurring convolutional neural network for joint training to obtain the final deblurring convolutional neural network. The method enables the deblurring convolutional neural network to better meet the practical application scene, and improves the deblurring effect.
Description
Technical Field
The application relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, image processing and the like, and particularly relates to a deblurring convolutional neural network training method, device, equipment and storage medium.
Background
Image blur is generally a result of relative motion between the camera and the scene during the time the camera is exposed, and can be roughly classified into two categories: one is global blur, which mainly causes the camera to shake in the exposure time, forming relative motion with the shot picture, and making the shot image become blurred; the other is local blurring, which is mainly caused by the fact that a moving person or object exists in the shooting scene, so that relative motion is formed between the shooting scene and the camera, and the local blurring in the shot image is caused. At present, the optical anti-shake function of the photographic equipment is gradually mature, and the deblurring processing of a local blurred image becomes a wider blurred scene.
With the rapid development of deep learning and neural networks, the convolutional neural network has the capability of modeling a complex fuzzy kernel, and is widely used in an image deblurring task, so that the image deblurring effect is greatly improved. However, the existing convolutional neural network design and data set acquisition work mainly focuses on solving the problem of global blur, cannot be generalized to an actual use scene, and can cause the phenomena that the blur cannot be removed or a clear part is damaged. Meanwhile, due to the lack of a corresponding data set, the processing capability of the current work on the common phenomenon of local blurring cannot be evaluated.
Disclosure of Invention
Based on the above problems, the present application provides a method, an apparatus, a device and a storage medium for training a deblurring convolutional neural network to improve the deblurring processing effect.
According to a first aspect of the application, there is provided a method for training a deblurred convolutional neural network, comprising:
acquiring a fuzzy image training set, wherein the fuzzy image training set comprises a local fuzzy training set and a global fuzzy training set; wherein, the blurred image in the local blurring training set is obtained by calculating a background image and a foreground image block moving in a preset mode;
constructing an initial deblurring convolutional neural network, which comprises a fuzzy area perception network and a deblurring network; the deblurring network comprises a fuzzy region perception attention module and a deblurring module;
inputting the local fuzzy training set into the fuzzy area perception network, outputting a fuzzy area prediction result, and training the fuzzy area perception network according to the fuzzy area prediction result and a first loss function;
inputting the fuzzy image training set into the deblurring module, outputting a deblurring result, and training the deblurring module according to the deblurring result and a second loss function;
inputting the local fuzzy training set into the fuzzy region perception attention module, outputting an attention result, and training the fuzzy region perception attention module according to the attention result and a third loss function;
constructing an intermediate deblurring convolutional neural network according to the trained fuzzy region perception network, the deblurring module and the fuzzy region perception attention module;
and alternately inputting the local fuzzy training set and the global fuzzy training set into the intermediate deblurring convolutional neural network, outputting a prediction result, and performing combined training according to the prediction result and a fourth loss function to obtain a final deblurring convolutional neural network.
Wherein the local fuzzy training set comprises a plurality of the fuzzy images, and each fuzzy image is obtained by:
acquiring a clear image from the deblurring data set as a background image;
acquiring at least one image from a semantic segmentation data set, and extracting all foreground image blocks in each image;
acquiring a motion sequence with the length of n, wherein n is an odd number which is more than or equal to 3;
simultaneously moving all foreground image blocks in the at least one image on the background image according to the motion sequence to obtain a motion image of continuous frames;
and averaging the moving images of the continuous frames to obtain the blurred image.
In an embodiment of the present application, the acquiring a moving image of consecutive frames includes:
according to the motion sequence, moving all foreground image blocks in the at least one image on the background image once to obtain a combined image; wherein the combined image comprises all foreground image blocks and the background image in the at least one image;
and after the motion sequence is completed, taking a plurality of combined images as the motion images of the continuous frames.
In some embodiments of the present application, the number of the blur area awareness attention modules in the initial deblurring network is i, where i is an integer greater than or equal to 1; the calculation formula of the third loss function is as follows:
wherein L isBSAIn order to be a function of the third loss,
output the result of the attention of the module for perceiving attention for the ith blurred region, MsegAnd labeling information of the fuzzy region generated for the fuzzy image in the local fuzzy training set.
In some embodiments of the present application, when the local fuzzy training set is input to an intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network comprises the trained fuzzy area aware network and the trained deblurring network; when the global fuzzy training set is input to the intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network only comprises the deblurred module after training.
Performing joint training according to the prediction result and a fourth loss function to obtain a final deblurred convolutional neural network, wherein the joint training comprises:
acquiring a first prediction result when the local fuzzy training set is input to the intermediate deblurring convolutional neural network;
obtaining the fourth loss function; wherein the fourth loss function is a weighted sum of the first, second, and third loss functions;
and performing combined training according to the first prediction result and the fourth loss function to obtain a final deblurring convolutional neural network.
In addition, the performing joint training according to the prediction result and a fourth loss function to obtain a final deblurred convolutional neural network includes:
acquiring a second prediction result when the global fuzzy training set is input to the intermediate deblurring convolutional neural network;
obtaining the fourth loss function; wherein the fourth loss function is the second loss function;
and performing combined training according to the second prediction result and the fourth loss function to obtain a final deblurring convolutional neural network.
According to a second aspect of the present application, there is provided a deblurring convolutional neural network training device, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a fuzzy image training set, and the fuzzy image training set comprises a local fuzzy training set and a global fuzzy training set; wherein, the blurred image in the local blurring training set is obtained by calculating a background image and a foreground image block moving in a preset mode;
the first construction module is used for constructing an initial deblurring convolutional neural network, and comprises a fuzzy area perception network and a deblurring network; the deblurring network comprises a fuzzy region perception attention module and a deblurring module;
the first training module is used for inputting the local fuzzy training set into the fuzzy regional perception network, outputting a fuzzy regional prediction result and training the fuzzy regional perception network according to the fuzzy regional prediction result and a first loss function;
the second training module is used for inputting the blurred image training set into the deblurring module, outputting a deblurring result and training the deblurring module according to the deblurring result and a second loss function;
the third training module is used for inputting the local fuzzy training set into the fuzzy region perception attention module, outputting an attention result and training the fuzzy region perception attention module according to the attention result and a third loss function;
the second construction module is used for constructing an intermediate deblurring convolutional neural network according to the trained fuzzy region perception network, the deblurring module and the fuzzy region perception attention module;
and the fourth training module is used for inputting the local fuzzy training set and the global fuzzy training set into an intermediate deblurring convolutional neural network alternately, outputting a prediction result, and performing combined training according to the prediction result and a fourth loss function to obtain a final deblurring convolutional neural network.
In an embodiment of the present application, the local blur training set includes a plurality of the blurred images, and the obtaining module is configured to obtain each of the blurred images, and includes:
acquiring a clear image from the deblurring data set as a background image;
acquiring at least one image from a semantic segmentation data set, and extracting all foreground image blocks in each image;
acquiring a motion sequence with the length of n, wherein n is an odd number which is more than or equal to 3;
simultaneously moving all foreground image blocks in the at least one image on the background image according to the motion sequence to obtain a motion image of continuous frames;
and averaging the moving images of the continuous frames to obtain the blurred image.
In some embodiments of the present application, the obtaining module is further configured to:
according to the motion sequence, moving all foreground image blocks in the at least one image on the background image once to obtain a combined image; wherein the combined image comprises all foreground image blocks and the background image in the at least one image;
and after the motion sequence is completed, taking a plurality of combined images as the motion images of the continuous frames.
In some embodiments of the present application, the number of the blur area awareness attention modules in the initial deblurring network is i, where i is an integer greater than or equal to 1; the calculation formula of the third loss function is as follows:
wherein L isBSAIn order to be a function of the third loss,
output the result of the attention of the module for perceiving attention for the ith blurred region, MsegAnd labeling information of the fuzzy region generated for the fuzzy image in the local fuzzy training set.
In some embodiments of the present application, the fourth training module is to:
when the local fuzzy training set is input to an intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network comprises the trained fuzzy region awareness network and the trained deblurring network;
when the global fuzzy training set is input to the intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network only comprises the deblurred module after training.
In some embodiments of the present application, the fourth training module is to:
acquiring a first prediction result when the local fuzzy training set is input to the intermediate deblurring convolutional neural network;
obtaining the fourth loss function; wherein the fourth loss function is a weighted sum of the first, second, and third loss functions;
and performing combined training according to the first prediction result and the fourth loss function to obtain a final deblurring convolutional neural network.
In some embodiments of the present application, the fourth training module is further configured to:
acquiring a second prediction result when the global fuzzy training set is input to the intermediate deblurring convolutional neural network;
obtaining the fourth loss function; wherein the fourth loss function is the second loss function;
and performing combined training according to the second prediction result and the fourth loss function to obtain a final deblurring convolutional neural network.
According to a third aspect of the present application, there is provided a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to the first aspect of the present application.
According to a fourth aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of the first aspect of the present application.
According to the technical scheme of the embodiment of the application, the local blurred image is obtained through calculation of the foreground image block and the background image which move in the preset mode, so that a training set of the local blurred image is obtained, a new method is provided for obtaining the local blurred image, and the problem that a local blurred image data set is lacked at present is solved. In addition, a fuzzy area perception network and a fuzzy area perception attention module are introduced into the deblurring convolutional neural network, and a training method of combining a global fuzzy training set and a local fuzzy training set in a first-part mode is used, so that the model training efficiency is improved, the deblurring processing of a local fuzzy image and a global fuzzy image can be realized by the deblurring convolutional neural network, the image quality is greatly improved, and the scene applicability of the convolutional neural network to deblurring is improved.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart of a method for training a deblurred convolutional neural network according to an embodiment of the present disclosure;
fig. 2 is a flowchart of acquiring a blurred image according to an embodiment of the present application;
fig. 3 is a schematic diagram of acquiring a blurred image according to an embodiment of the present application;
FIG. 4 is a flowchart of joint training of a deblurred convolutional neural network according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a deblurring convolutional neural network in an embodiment of the present application;
FIG. 6 is a comparison graph of deblurring effect of the present application for actually acquired blurred images;
fig. 7 is a block diagram of a structure of a deblurring convolutional neural network training apparatus according to an embodiment of the present application.
Fig. 8 is a block diagram of a computer device according to an embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
It should be noted that local blurring of an image, that is, blurring of an image mainly caused by rapid movement of an object, is a common blurring scene to be solved, and as the optical anti-shake function of a photographing apparatus matures, a deblurring process of the local blurring of the image will be a requirement for a wider practical application scene.
At present, a deep learning method is adopted to carry out image deblurring processing, a clear image and a blurred image are required to be used as training data, a designed convolutional neural network is trained, the mean square error between a network output image and an actual clear image is used as a loss function, and network parameters are continuously adjusted until the training is finished. However, the involvement of the conventional convolutional neural network and the data acquisition mainly focus on the deblurring processing for solving the global blurred image, the image blur is mainly caused by the rapid motion of a camera, and a corresponding data set is also lacked for a local blurred scene, so that the phenomena that the blur cannot be removed or a clear part is damaged and the like occur in an actual application scene.
Based on the above problems, the present application provides a method, an apparatus, a device and a storage medium for training a deblurring convolutional neural network.
Fig. 1 is a flowchart of a method for training a deblurred convolutional neural network according to an embodiment of the present disclosure. It should be noted that the method for training a deblurring convolutional neural network according to the embodiment of the present application can be applied to a device for training a deblurring convolutional neural network according to the embodiment of the present application, and the device can be configured in a computer device. As shown in fig. 1, the method comprises the steps of:
In the embodiment of the present application, each training sample in the blurred image training set is a blurred image pair, that is, in the blurred image training set, a blurred image and a sharp image exist in a pair, and each blurred image and the sharp image are in a one-to-one correspondence relationship. In addition, in order to adapt the deblurred convolutional neural network to the actual scene, the blurred image training set includes both a local blur training set and a global blur training set. The global training set includes a large number of global blurred images and sharp images, and in the embodiment of the present application, the global training set may use an existing deblurred data set. The local blurred training set comprises a large number of local blurred images and sharp images, wherein the local blurred images are calculated by a background image and a foreground image block moving in a preset mode.
102, constructing an initial deblurring convolutional neural network, which comprises a fuzzy area perception network and a deblurring network; wherein the deblurring network includes a blurred region awareness attention module and a deblurring module.
It can be understood that in order to meet the practical application scenario of the network structure, a fuzzy region perception network and a fuzzy region perception attention module are introduced into the initial deblurring convolutional neural network. The fuzzy area sensing network is used for identifying fuzzy areas in the image and inputting output results of the fuzzy area sensing network into the deblurring network through convolution so as to obtain a clear picture. The fuzzy region perception attention module carries out key processing aiming at the fuzzy region based on the deblurring effect of the deblurring module. In addition, the backbone network structure of the initial deblurring convolutional neural network can be a commonly used U-Net structure, and comprises a plurality of convolutional layers and activation layers.
It should be noted that the number of the blur area awareness attention modules in the deblurring network is i, where i is an integer greater than or equal to 1, and the specific number may be determined according to an actual situation. Wherein, each fuzzy area perception attention module adopts a space attention mechanism, and the following formula is adopted:
MSA＝Sigmoid(Conv7*7(F)) (1)
FAtt＝F⊙MSA (2)
wherein M isSAFor each blurred region, the output of the active layer in the attention module is perceived, where F is the input feature tensor, Conv7*7() For convolution operation with convolution kernel size of 7 x 7, Sigmoid () is a normalized activation function, indicating a multiply by element operation, FAttAnd sensing the output result of the attention module for each fuzzy area.
Specifically, the fuzzy area perception attention module has the working principle that: and performing feature extraction on the input feature tensor through convolution operation, then normalizing through an activation function, equivalently obtaining each space weight value corresponding to the input feature, and performing point multiplication on a result and the input feature, thereby applying attention to the input feature. Therefore, when the network learns, the user can know which places are the fuzzy areas according to the weights, namely, the network can put more attention on the deblurring processing of the fuzzy areas, help the network learn main contents, avoid redundant expression and improve the deblurring processing effect of local fuzzy images.
And 103, inputting the local fuzzy training set into a fuzzy region perception network, outputting a fuzzy region prediction result, and training the fuzzy region perception network according to the fuzzy region prediction result and the first loss function.
Since the fuzzy area perception network is used to identify fuzzy areas in an image, in order to train the network in a targeted manner, a local fuzzy training set is used for training. And inputting the local blurred image into a blurred region perception network to obtain a blurred region prediction result. And calculating a loss value through a first loss function according to the fuzzy region prediction result, fuzzy region labels corresponding to the local fuzzy image and the region consistency, and thus continuously adjusting parameters of the fuzzy region perception network until the prediction result meets the expectation. Wherein, the first loss function may use a cross-entropy loss function, as shown in formula (3):
LLBP＝CrossEntropy(Mblur，Mseg) (3)
wherein L isLBPAs a first loss function, crossEncopy () is a pixel-level cross entropy loss function, MblurFor fuzzy region prediction results, MsegAnd marking information for the fuzzy region corresponding to the local fuzzy image.
And 104, inputting the fuzzy image training set into a deblurring module, outputting a deblurring result, and training the deblurring module according to the deblurring result and a second loss function.
It can be understood that the functions of the partial networks in the initial deblurring convolutional neural network are different, and in order to improve the training efficiency of the partial networks, in the embodiment of the present application, the corresponding training set is used for performing individual training on the partial networks.
The function of the deblurring module is to output a clear image according to the blurred image, and the deblurring module comprises the deblurring processing of the global blurred image and the deblurring processing of the local blurred image, so that the global blurred training set and the local blurred training set are both used as the training sets of the deblurring module. And inputting the blurred image training set into a deblurring module to obtain a deblurring result, namely a predicted clear image. And comparing the predicted clear image with a real clear image corresponding to the blurred image, and continuously adjusting the parameter value of the deblurring module according to the loss value calculated by the second loss function until the predicted clear image meets the expectation. Wherein the second loss function may be calculated using a conventional mean square error.
And 105, inputting the local fuzzy training set into a fuzzy region perception attention module, outputting an attention result, and training the fuzzy region perception attention module according to the attention result and a third loss function.
In the embodiment of the application, the fuzzy region perception attention module is used for focusing the focus of the network deblurring processing on the fuzzy region aiming at the local fuzzy image, helping the network to learn main contents and strengthening the effect of the fuzzy processing. That is, the blurred region perception attention module works primarily in locally blurred scenes, so the training for this attention module uses a locally blurred training set. And inputting the local fuzzy training set into a fuzzy region perception attention module to obtain an attention result. And obtaining a loss value of the prediction result according to the obtained attention result and the third loss function, thereby continuously adjusting the parameters of the attention module and realizing the training of the module.
It should be noted that, in the embodiment of the present application, the calculation formula of the third loss function is shown in formula (4):
and 106, constructing an intermediate deblurring convolutional neural network according to the trained fuzzy region perception network, the deblurring module and the fuzzy region perception attention module.
Namely, after the fuzzy area perception network, the deblurring module and the fuzzy area perception attention module in the initial deblurring convolutional neural network are trained respectively, an intermediate deblurring convolutional neural network is obtained.
And 107, alternately inputting the local fuzzy training set and the global fuzzy training set into the intermediate deblurring convolutional neural network, outputting a prediction result, and performing combined training according to the prediction result and a fourth loss function to obtain a final deblurring convolutional neural network.
It can be understood that the deblurring convolutional neural network provided in the application combines the fuzzy area perception network, the deblurring module and the fuzzy area perception attention module, and functions of all parts in the deblurring convolutional neural network are combined with each other in actual use to jointly act on deblurring processing of an image, so that a clear image corresponding to a local blurred image and a global blurred image is obtained. Therefore, after the fuzzy area perception network, the deblurring module and the fuzzy area perception attention module are independently trained respectively, the constructed intermediate deblurring convolutional neural network needs to be jointly trained, so that the obtained deblurring convolutional neural network achieves the expected deblurring effect.
In the embodiment of the application, in order to enable the finally obtained deblurring convolutional neural network to meet both the local fuzzy scene and the global fuzzy scene, the local fuzzy training set and the global fuzzy training set are alternately input into the intermediate deblurring convolutional neural network in the training process. Meanwhile, according to the prediction result and the real clear image, a loss value is calculated through a fourth loss function and is used for adjusting parameter values in the network structure, and therefore training of the deblurring convolutional neural network is achieved. And the fourth loss function is obtained by calculating the first loss function, the second loss function and the third loss function.
According to the deblurring convolutional neural network training method provided by the embodiment of the application, the local blurred image is obtained through calculation of the foreground image block and the background image which move in a preset mode, so that a training set of the local blurred image is obtained, a new method is provided for obtaining the local blurred image, and the problem that a local blurred image data set is lacked at present is solved. In addition, a fuzzy area perception network and a fuzzy area perception attention module are introduced into the deblurring convolutional neural network, and a training method of combining a global fuzzy training set and a local fuzzy training set in a first-part mode is used, so that the model training efficiency is improved, the deblurring processing of a local fuzzy image and a global fuzzy image can be realized by the deblurring convolutional neural network, the image quality is greatly improved, and the scene applicability of the convolutional neural network to deblurring is improved.
For the above embodiment of obtaining the local fuzzy training set, the present application proposes another embodiment. Fig. 2 is a flowchart of obtaining each blurred image in the embodiment of the present application, and as shown in fig. 2, the step of obtaining each blurred image includes:
In the embodiment of the present application, the deblurring data set may use a high-quality deblurring data set REDS, or may use other existing deblurring data sets, which is not limited in this application. Furthermore, the deblurred data set used in this step may be consistent with the deblurred data set used in the global fuzzy training set in the embodiments of the present application.
In the embodiment of the present application, the semantic segmentation data set used may be a high-quality semantic segmentation data set PASCAL VOC 2012, or may be another semantic segmentation data set. One or more images are randomly extracted from the semantic segmentation data set, and all foreground image blocks are extracted and used for being combined with the background image to simulate a local fuzzy scene.
It can be understood that in the simulation of the locally blurred scene, the foreground image block serves as a moving object in the scene, and the background image serves as the background of the shot scene. The motion of the foreground image block needs a preset motion mode, and the preset motion mode is a motion sequence. The motion sequence includes the random initial position of each foreground image block on the background picture and the motion method of each frame, and the length of the motion sequence is the number of times that each foreground image block needs to move.
In the embodiment of the present application, the motion method of each frame of the foreground image block may include the following categories: a movement or standstill in a unit of 1 pixel in the horizontal direction, a movement or standstill in a unit of 1 pixel in the vertical direction; zoom out by a factor of 0.1 or zoom in by a factor of 0.1 or still, clockwise or counterclockwise rotation by a factor of 1 or still, and other motion categories. When acquiring a motion sequence, a motion method for each frame is randomly selected from the above categories.
And step 204, simultaneously moving all foreground image blocks in at least one image on the background image according to the motion sequence to acquire a motion image of continuous frames.
In the embodiment of the present application, an implementation manner of acquiring a moving image of consecutive frames may be: according to the motion sequence, moving all foreground image blocks in at least one image on a background image once to obtain a combined image; wherein the combined image comprises all foreground image blocks and a background image in the at least one image; after the motion sequence is completed, the plurality of combined images are used as a moving image of consecutive frames.
In step 205, the moving images of consecutive frames are averaged to obtain a blurred image.
It will be appreciated that the locally blurred image is due to motion of the object, so averaging the motion images of successive frames is equivalent to applying motion to the image, resulting in a blurred image.
It should be noted that, in the embodiment of the present application, each blurred image in the local blur training set corresponds to a sharp image, and the sharp image uses an image after the middle motion in the moving image of consecutive frames. For example, if the length of the motion sequence is 7, it is described that the obtained continuous frame images are composed of 7 combined images, and in this case, the combined image obtained from the 4 th motion is taken as a sharp image corresponding to the obtained blurred image. It can thus also be stated why it is necessary to define the length of the motion sequence as an odd number equal to or greater than 3.
In addition, in the embodiment of the present application, since the semantic segmentation data set is used, the blurred region label corresponding to each generated blurred image can also be generated at the same time. In order to more intuitively explain the above-described procedure of obtaining a blurred image, the procedure is illustrated by way of example in fig. 3. As shown in fig. 3, (a) is 4 images of the obtained moving image of the continuous frames, and only 4 images of the moving image of the continuous frames are taken here as an example, (b) is a blurred image obtained by averaging the generated moving images of the continuous frames, and (C) is information for a blurred region corresponding to the blurred image.
According to the deblurring convolutional neural network training method provided by the embodiment of the application, a clear image is selected from the deblurring data set to serve as a background image, and a foreground image block extracted from an image in the semantic segmentation data set moves according to a motion sequence, so that simulation of a local fuzzy scene is achieved. A moving image of continuous frames is obtained through simulation of a local fuzzy scene, and the local fuzzy image is obtained through averaging, so that a new method is provided for obtaining the local fuzzy image, and the problem that a local fuzzy image data set is lacked at present is solved.
To further describe the training method of the deblurred convolutional neural network in detail, the joint training thereof is explained next. Fig. 4 is a flowchart of joint training of a deblurred convolutional neural network according to an embodiment of the present application, as shown in fig. 4, including the following steps:
and step 401, alternately inputting the local fuzzy training set and the global fuzzy training set into the intermediate deblurring convolutional neural network.
In the embodiment of the present application, the alternating input of the local fuzzy training set and the global fuzzy training set to the intermediate deblurring convolutional neural network means that the local fuzzy training set is input to the intermediate deblurring convolutional neural network for training, then the global fuzzy training set is input to the intermediate deblurring convolutional neural network for training, then the local fuzzy training set is input to the intermediate deblurring convolutional neural network, and so on. The local fuzzy training set may be used initially, or the global fuzzy training set may be used, which is not limited in this application.
In the embodiment of the application, when the local fuzzy training set is input to the intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network comprises a trained fuzzy area sensing network and a trained deblurring network; when the global fuzzy training set is input to the intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network only comprises the deblurring module after training.
The first prediction result comprises a prediction result obtained by the trained fuzzy regional awareness network and a prediction result of the trained deblurring network.
Wherein the fourth loss function is shown in equation (5):
L＝λ1LMSE+λ2LLBP+λ3LBSA (5)
wherein L is a fourth loss function, LMSEIs a first loss function, LLBPIs a second loss function, LBSAAs a third loss function, λ1、λ2And λ3Are coefficients determined by experiment.
And step 404, performing joint training according to the first prediction result and the fourth loss function.
It can be understood that parameter values in the network are continuously adjusted according to the first prediction result and the loss value obtained through the fourth loss function, so that the training of the intermediate deblurring convolutional neural network is realized, and the global fuzzy training set is input into the intermediate deblurring convolutional neural network instead until the first prediction result meets the expectation. If the global fuzzy training set is changed, the output result still can meet the expectation, which indicates that the current deblurring convolutional neural network is the final deblurring convolutional neural network.
And step 405, acquiring a second prediction result when the global fuzzy training set is input to the intermediate deblurring convolutional neural network.
Since the intermediate deblurring convolutional neural network only includes the deblurred module after training when the global fuzzy training set is input to the intermediate deblurring convolutional neural network, the current fourth loss function is equivalent to the second loss function, that is, the traditional mean square error is used.
It can be understood that parameter values in the network are continuously adjusted according to the second prediction result and the mean square error, so that the training of the intermediate deblurring convolutional neural network is realized, and the local fuzzy training set is input into the intermediate deblurring convolutional neural network instead until the second prediction result meets the expectation. If the local fuzzy training set is changed, the output result still can meet the expectation, which indicates that the current deblurring convolutional neural network is the final deblurring convolutional neural network.
And step 408, obtaining a final deblurring convolutional neural network.
According to the training method of the deblurring convolutional neural network provided by the embodiment of the application, each part in the deblurring convolutional neural network is trained independently and then jointly, so that the network training efficiency is improved, and the network training result is also improved. In addition, the local fuzzy training set and the global fuzzy training set are alternately input into the intermediate deblurring convolutional neural network for network training, so that the finally obtained deblurring convolutional neural network is suitable for both a local fuzzy scene and a global fuzzy scene, and meanwhile, the network structure and the loss function are flexibly matched aiming at different training sets, the aim of targeted training is achieved, the network training efficiency is further improved, and the image deblurring processing effect of the convolutional neural network is also improved.
Based on the above description of the embodiments, in order to make the structure of the deblurring convolutional neural network in some embodiments of the present application more intuitive, an example of fig. 5 will be shown. As shown in fig. 5, taking the synthesized locally blurred image as an example, the blurred image is input to the blurred region perception network to predict the blurred region, the feature tensor obtained by convolving the output result is input to the deblurring network, and a clear image is obtained through the combined action of the deblurring module and the blurred region perception attention module.
In order to prove the deblurring processing effect of the neural network on the local blurred image, a high-frame-rate camera is used in the method, and continuous frames acquired in a static state are combined into a picture, so that acquired data are closer to an actual shooting scene. Meanwhile, the acquired fuzzy image is input to the deblurring convolution neural network for testing, and compared with the deblurring processing effect of the existing method. Fig. 6 is a comparison of deblurring effects for actually acquired blurred images, wherein in each group of images, the leftmost image is the original blurred image, the middle image is the deblurring effect of the existing method, and the rightmost image is the deblurring effect of the deblurring convolutional neural network in the present application. By contrast, the deblurring convolutional neural network in the application can remarkably improve the local blurring of the image.
In order to implement the method, the embodiment of the application provides a deblurring convolutional neural network training device.
Fig. 7 is a block diagram of a structure of a deblurring convolutional neural network training apparatus according to an embodiment of the present application. As shown in fig. 7, the deblurring convolutional neural network training device includes:
an obtaining module 710, configured to obtain a fuzzy image training set, where the fuzzy image training set includes a local fuzzy training set and a global fuzzy training set; wherein, the blurred image in the local blurring training set is obtained by calculating a background image and a foreground image block which moves in a preset mode;
a first constructing module 720, configured to construct an initial deblurring convolutional neural network, which includes a fuzzy area-aware network and a deblurring network; the deblurring network comprises a fuzzy region perception attention module and a deblurring module;
the first training module 730 is used for inputting the local fuzzy training set into the fuzzy region perception network, outputting a fuzzy region prediction result, and training the fuzzy region perception network according to the fuzzy region prediction result and the first loss function;
the second training module 740 is configured to input the blurred image training set into the deblurring module, output a deblurring result, and train the deblurring module according to the deblurring result and a second loss function;
the third training module 750 is configured to input the local fuzzy training set into the fuzzy region awareness attention module, output an attention result, and train the fuzzy region awareness attention module according to the attention result and a third loss function;
a second construction module 760, configured to construct an intermediate deblurred convolutional neural network according to the trained fuzzy area sensing network, deblurring module, and fuzzy area sensing attention module;
and a fourth training module 770, configured to alternately input the local fuzzy training set and the global fuzzy training set to the intermediate deblurring convolutional neural network, output a prediction result, and perform joint training according to the prediction result and a fourth loss function to obtain a final deblurring convolutional neural network.
In this embodiment of the present application, the local blur training set includes a plurality of blurred images, and the obtaining module 710 is configured to obtain each blurred image, including:
acquiring a clear image from the deblurring data set as a background image;
acquiring at least one image from the semantic segmentation data set, and extracting all foreground image blocks in each image;
acquiring a motion sequence with the length of n, wherein n is an odd number which is more than or equal to 3;
simultaneously moving all foreground image blocks in at least one image on a background image according to a motion sequence to obtain a motion image of continuous frames;
the motion images of successive frames are averaged to obtain a blurred image.
In some embodiments of the present application, the obtaining module 710 is further configured to:
according to the motion sequence, moving all foreground image blocks in at least one image on a background image once to obtain a combined image; the combined image comprises all foreground image blocks and a background image in at least one image;
after the motion sequence is completed, the plurality of combined images are used as a moving image of consecutive frames.
In some embodiments of the present application, the number of the blur area awareness attention modules in the initial deblurring network is i, where i is an integer greater than or equal to 1; the calculation formula of the third loss function is shown in formula (4).
In some embodiments of the present application, the fourth training module 770 is configured to:
when the local fuzzy training set is input into the intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network comprises a trained fuzzy area perception network and a trained deblurring network;
when the global fuzzy training set is input into the intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network only comprises the deblurring module after training.
In some embodiments of the present application, the fourth training module 770 is configured to:
acquiring a first prediction result when a local fuzzy training set is input to an intermediate deblurring convolutional neural network;
obtaining a fourth loss function; wherein the fourth loss function is a weighted sum of the first loss function, the second loss function and the third loss function;
and performing combined training according to the first prediction result and the fourth loss function to obtain a final deblurring convolutional neural network.
In some embodiments of the present application, fourth training module 770 is further configured to:
acquiring a second prediction result when the global fuzzy training set is input into the intermediate deblurring convolutional neural network;
obtaining a fourth loss function; wherein the fourth loss function is the second loss function;
and performing combined training according to the second prediction result and the fourth loss function to obtain a final deblurred convolutional neural network.
According to the technical scheme of the embodiment of the application, the local blurred image is obtained through calculation of the foreground image block and the background image which move in the preset mode, so that a training set of the local blurred image is obtained, a new method is provided for obtaining the local blurred image, and the problem that a local blurred image data set is lacked at present is solved. In addition, a fuzzy area perception network and a fuzzy area perception attention module are introduced into the deblurring convolutional neural network, and a training method of combining a global fuzzy training set and a local fuzzy training set in a first-part mode is used, so that the model training efficiency is improved, the deblurring processing of a local fuzzy image and a global fuzzy image can be realized by the deblurring convolutional neural network, the image quality is greatly improved, and the scene applicability of the convolutional neural network to deblurring is improved.
To implement the above embodiments, the present application also provides a computer device and a computer-readable storage medium.
FIG. 8 is a block diagram of a computer device for implementing deblurred convolutional neural network training in accordance with an embodiment of the present application. Computer devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The computer device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices.
As shown in fig. 8, the computer apparatus includes: a memory 810, a processor 820, and a computer program 830 stored on the memory and executable on the processor. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system).
The memory 810 is a computer-readable storage medium provided herein. Wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of deblurring convolutional neural network training provided herein. The computer readable storage medium of the present application stores computer instructions for causing a computer to perform the method of deblurring convolutional neural network training provided herein.
The memory 810 is a computer readable storage medium, and can be used for storing software programs, computer executable programs, and modules, such as program instructions/modules corresponding to the method for training a deblurred convolutional neural network in the embodiment of the present application (for example, the obtaining module 710, the first constructing module 720, the first training module 730, the second training module 740, the third training module 750, the second constructing module 760, and the fourth constructing module 770 shown in fig. 7). The processor 820 executes the non-transitory software programs, instructions and modules stored in the memory 820 to execute various functional applications of the server and data processing, namely, to implement the method for training the deblurring convolutional neural network in the above method embodiment.
The memory 810 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of a computer device to deblur the convolutional neural network training method, and the like. Further, the memory 810 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 810 optionally includes memory located remotely from processor 820, which may be connected via a network to electronics used to implement the deblurring convolutional neural network training method. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The computer apparatus to deblur the convolutional neural network training method may further include: an input device 840 and an output device 850. The processor 820, memory 810, input device 840, and output device 850 may be connected by a bus or other means, such as by a bus connection in fig. 8.
The input device 840 may receive input numeric or character information and generate key signal inputs related to user settings and function controls of an electronic device used to implement the deblurred convolutional neural network training method, such as a touch screen, keypad, mouse, track pad, touch pad, pointer stick, one or more mouse buttons, track ball, joystick, or other input device. The output devices 850 may include a display device, auxiliary lighting devices (e.g., LEDs), tactile feedback devices (e.g., vibrating motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. A method for training a deblurring convolutional neural network, comprising:
acquiring a fuzzy image training set, wherein the fuzzy image training set comprises a local fuzzy training set and a global fuzzy training set; wherein, the blurred image in the local blurring training set is obtained by calculating a background image and a foreground image block moving in a preset mode;
constructing an initial deblurring convolutional neural network, which comprises a fuzzy area perception network and a deblurring network; the deblurring network comprises a fuzzy region perception attention module and a deblurring module;
inputting the local fuzzy training set into the fuzzy area perception network, outputting a fuzzy area prediction result, and training the fuzzy area perception network according to the fuzzy area prediction result and a first loss function;
inputting the fuzzy image training set into the deblurring module, outputting a deblurring result, and training the deblurring module according to the deblurring result and a second loss function;
inputting the local fuzzy training set into the fuzzy region perception attention module, outputting an attention result, and training the fuzzy region perception attention module according to the attention result and a third loss function;
constructing an intermediate deblurring convolutional neural network according to the trained fuzzy region perception network, the deblurring module and the fuzzy region perception attention module;
and alternately inputting the local fuzzy training set and the global fuzzy training set into the intermediate deblurring convolutional neural network, outputting a prediction result, and performing combined training according to the prediction result and a fourth loss function to obtain a final deblurring convolutional neural network.
2. The method of claim 1, wherein the local blur training set comprises a plurality of the blurred images, each of the blurred images being obtained by:
acquiring a clear image from the deblurring data set as a background image;
acquiring at least one image from a semantic segmentation data set, and extracting all foreground image blocks in each image;
acquiring a motion sequence with the length of n, wherein n is an odd number which is more than or equal to 3;
simultaneously moving all foreground image blocks in the at least one image on the background image according to the motion sequence to obtain a motion image of continuous frames;
and averaging the moving images of the continuous frames to obtain the blurred image.
3. The method of claim 2, wherein said obtaining a motion image of consecutive frames comprises:
according to the motion sequence, moving all foreground image blocks in the at least one image on the background image once to obtain a combined image; wherein the combined image comprises all foreground image blocks and the background image in the at least one image;
and after the motion sequence is completed, taking a plurality of combined images as the motion images of the continuous frames.
4. The method according to claim 1, wherein the number of the fuzzy area awareness attention modules in the initial deblurring network is i, where i is an integer greater than or equal to 1; the calculation formula of the third loss function is as follows:
wherein L isBSAIn order to be a function of the third loss,
output the result of the attention of the module for perceiving attention for the ith blurred region, MsegAnd labeling information of the fuzzy region generated for the fuzzy image in the local fuzzy training set.
5. The method of claim 1,
when the local fuzzy training set is input to an intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network comprises the trained fuzzy region awareness network and the trained deblurring network;
when the global fuzzy training set is input to the intermediate deblurring convolutional neural network, the intermediate deblurring convolutional neural network only comprises the deblurred module after training.
6. The method of claim 5, wherein the performing the joint training according to the prediction result and the fourth loss function to obtain the final deblurred convolutional neural network comprises:
acquiring a first prediction result when the local fuzzy training set is input to the intermediate deblurring convolutional neural network;
obtaining the fourth loss function; wherein the fourth loss function is a weighted sum of the first, second, and third loss functions;
and performing combined training according to the first prediction result and the fourth loss function to obtain a final deblurring convolutional neural network.
7. The method of claim 5, wherein the performing the joint training according to the prediction result and the fourth loss function to obtain the final deblurred convolutional neural network comprises:
acquiring a second prediction result when the global fuzzy training set is input to the intermediate deblurring convolutional neural network;
obtaining the fourth loss function; wherein the fourth loss function is the second loss function;
and performing combined training according to the second prediction result and the fourth loss function to obtain a final deblurring convolutional neural network.
8. A deblurring convolutional neural network training apparatus, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a fuzzy image training set, and the fuzzy image training set comprises a local fuzzy training set and a global fuzzy training set; wherein, the blurred image in the local blurring training set is obtained by calculating a background image and a foreground image block moving in a preset mode;
the first construction module is used for constructing an initial deblurring convolutional neural network, and comprises a fuzzy area perception network and a deblurring network; the deblurring network comprises a fuzzy region perception attention module and a deblurring module;
the first training module is used for inputting the local fuzzy training set into the fuzzy regional perception network, outputting a fuzzy regional prediction result and training the fuzzy regional perception network according to the fuzzy regional prediction result and a first loss function;
the second training module is used for inputting the blurred image training set into the deblurring module, outputting a deblurring result and training the deblurring module according to the deblurring result and a second loss function;
the third training module is used for inputting the local fuzzy training set into the fuzzy region perception attention module, outputting an attention result and training the fuzzy region perception attention module according to the attention result and a third loss function;
the second construction module is used for constructing an intermediate deblurring convolutional neural network according to the trained fuzzy region perception network, the deblurring module and the fuzzy region perception attention module;
and the fourth training module is used for inputting the local fuzzy training set and the global fuzzy training set into an intermediate deblurring convolutional neural network alternately, outputting a prediction result, and performing combined training according to the prediction result and a fourth loss function to obtain a final deblurring convolutional neural network.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, when executing the computer program, implementing the method of deblurring convolutional neural network training as defined in any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method of deblurring convolutional neural network training as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111342163.4A CN114240764B (en) | 2021-11-12 | 2021-11-12 | De-blurring convolutional neural network training method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111342163.4A CN114240764B (en) | 2021-11-12 | 2021-11-12 | De-blurring convolutional neural network training method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114240764A true CN114240764A (en) | 2022-03-25 |
| CN114240764B CN114240764B (en) | 2024-04-23 |
Family
ID=80749272
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111342163.4A Active CN114240764B (en) | 2021-11-12 | 2021-11-12 | De-blurring convolutional neural network training method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114240764B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114491146A (en) * | 2022-04-01 | 2022-05-13 | 广州智慧城市发展研究院 | Video image processing method suitable for video monitoring equipment |
| CN116823656A (en) * | 2023-06-27 | 2023-09-29 | 华东师范大学 | Image blind deblurring method and system based on frequency domain local feature attention mechanism |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150131898A1 (en) * | 2013-11-12 | 2015-05-14 | Microsoft Corporation | Blind image deblurring with cascade architecture |
| CN110909860A (en) * | 2018-09-14 | 2020-03-24 | 华为技术有限公司 | Method and device for initializing neural network parameters |
| CN111539884A (en) * | 2020-04-21 | 2020-08-14 | 温州大学 | Neural network video deblurring method based on multi-attention machine mechanism fusion |
| WO2021184894A1 (en) * | 2020-03-20 | 2021-09-23 | 深圳市优必选科技股份有限公司 | Deblurred face recognition method and system and inspection robot |
| CN113570516A (en) * | 2021-07-09 | 2021-10-29 | 湖南大学 | Image blind motion deblurring method based on CNN-Transformer hybrid self-encoder |
-
2021
- 2021-11-12 CN CN202111342163.4A patent/CN114240764B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150131898A1 (en) * | 2013-11-12 | 2015-05-14 | Microsoft Corporation | Blind image deblurring with cascade architecture |
| CN110909860A (en) * | 2018-09-14 | 2020-03-24 | 华为技术有限公司 | Method and device for initializing neural network parameters |
| WO2021184894A1 (en) * | 2020-03-20 | 2021-09-23 | 深圳市优必选科技股份有限公司 | Deblurred face recognition method and system and inspection robot |
| CN111539884A (en) * | 2020-04-21 | 2020-08-14 | 温州大学 | Neural network video deblurring method based on multi-attention machine mechanism fusion |
| CN113570516A (en) * | 2021-07-09 | 2021-10-29 | 湖南大学 | Image blind motion deblurring method based on CNN-Transformer hybrid self-encoder |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114491146A (en) * | 2022-04-01 | 2022-05-13 | 广州智慧城市发展研究院 | Video image processing method suitable for video monitoring equipment |
| CN116823656A (en) * | 2023-06-27 | 2023-09-29 | 华东师范大学 | Image blind deblurring method and system based on frequency domain local feature attention mechanism |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114240764B (en) | 2024-04-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7093886B2 (en) | Image processing methods and devices, electronic devices and storage media | |
| US10755173B2 (en) | Video deblurring using neural networks | |
| CN109670558B (en) | Digital image completion using deep learning | |
| Zhang et al. | Unifying motion deblurring and frame interpolation with events | |
| US9430817B2 (en) | Blind image deblurring with cascade architecture | |
| Huang et al. | Video super-resolution via bidirectional recurrent convolutional networks | |
| CN110140147B (en) | Video frame synthesis with deep learning | |
| WO2020063475A1 (en) | 6d attitude estimation network training method and apparatus based on deep learning iterative matching | |
| CN111753961B (en) | Model training method and device, prediction method and device | |
| CN114240764A (en) | Deblurring convolution neural network training method, device, equipment and storage medium | |
| CN114008663A (en) | Real-time video super-resolution | |
| CN109903315B (en) | Method, apparatus, device and readable storage medium for optical flow prediction | |
| CN111739005B (en) | Image detection method, device, electronic equipment and storage medium | |
| Zhang et al. | A flexible recurrent residual pyramid network for video frame interpolation | |
| US10977549B2 (en) | Object animation using generative neural networks | |
| US11641446B2 (en) | Method for video frame interpolation, and electronic device | |
| Luo et al. | Bokeh rendering from defocus estimation | |
| CN112308145A (en) | Classification network training method, classification device and electronic equipment | |
| Mai et al. | Deep unrolled low-rank tensor completion for high dynamic range imaging | |
| CN108520532B (en) | Method and device for identifying motion direction of object in video | |
| Won et al. | Learning depth from focus in the wild | |
| CN112053362B (en) | Image segmentation method, device, electronic equipment and storage medium | |
| Xu et al. | Sharp image estimation from a depth-involved motion-blurred image | |
| CN113706390A (en) | Image conversion model training method, image conversion method, device and medium | |
| Qi | A multi-path attention network for non-uniform blind image deblurring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |