CN111028177B - Edge-based deep learning image motion blur removing method - Google Patents
Edge-based deep learning image motion blur removing method Download PDFInfo
- Publication number
- CN111028177B CN111028177B CN201911275632.8A CN201911275632A CN111028177B CN 111028177 B CN111028177 B CN 111028177B CN 201911275632 A CN201911275632 A CN 201911275632A CN 111028177 B CN111028177 B CN 111028177B
- Authority
- CN
- China
- Prior art keywords
- image
- edge
- loss function
- multiplied
- deblurring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000013135 deep learning Methods 0.000 title claims abstract description 22
- 230000006870 function Effects 0.000 claims abstract description 33
- 230000009466 transformation Effects 0.000 claims abstract description 21
- 238000012549 training Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims abstract description 11
- 239000000284 extract Substances 0.000 claims abstract description 4
- 230000008447 perception Effects 0.000 claims abstract description 4
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 14
- 230000004913 activation Effects 0.000 claims description 9
- 238000010586 diagram Methods 0.000 claims description 6
- 238000005065 mining Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 238000012937 correction Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 9
- 230000010354 integration Effects 0.000 abstract description 7
- 238000012360 testing method Methods 0.000 description 6
- 238000005457 optimization Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004091 panning Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20201—Motion blur correction
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to an image restoration technology, in particular to a deep learning image motion blur removal method based on edges, which comprises the steps of utilizing a trained HED network to extract edges from a blurred image, and then using a convolution layer to extract edge characteristic information for guiding a motion blur removal process; the deblurring backbone network extracts multi-scale feature information from the blurred image, the spatial feature transformation layer is used for integrating the image features and the edge features on each scale, and the decoding part gradually recovers the potential clear image from the deepest image features; the fuzzy-clear image pair is used as a training sample set, a total loss function is defined by the sum of a mean square error loss function and a perception loss function, and the deblurring backbone network is trained by using the total loss function until the optimal precision is converged; and inputting the motion blurred image into a trained deblurring backbone network to obtain a deblurred result. The method realizes the effective integration of the image features and the edge features, and has remarkable deblurring effect.
Description
Technical Field
The invention belongs to the technical field of image restoration, and particularly relates to a deep learning image motion blur removing method based on edges.
Background
In the photographing process, the imaging device and the scene object generate motion blur due to relative motion, and the obtained image loses important detail information. The process of recovering a potentially sharp image from a blurred image that has been degraded is called deblurring. The motion blur removal can recover clear edges from blurred images caused by camera shake, vehicles which fly in a scene and the like, not only can improve visual perception quality, but also is beneficial to subsequent high-level application such as character recognition, target detection and the like, so the motion blur removal has higher research value and application prospect.
Existing image deblurring algorithms can be generally divided into conventional deblurring methods based on energy optimization and deblurring methods based on deep learning. In the conventional deblurring method based on energy optimization, the method can be further subdivided into global consistent deblurring and global non-consistent deblurring.
In conventional approaches, a motion blurred image may be modeled as a convolution of a blur kernel with a sharp image, followed by the addition of additive noise. The deblurring method based on energy optimization comprises two stages of fuzzy core estimation and image deconvolution, wherein a degradation model of a motion fuzzy image is analyzed in the fuzzy core estimation stage, an energy equation is established by combining prior statistical knowledge of a fuzzy core and a clear image, and a fuzzy core estimation value is obtained by solving the minimum value of the equation; after the fuzzy core is acquired, the degradation model and the priori knowledge of the clear image are combined for modeling and solving to obtain the clear image estimated value. Global consistent blur is typically caused by in-plane panning that occurs when a camera shoots a static scene, where blur kernels are shared in full view, an image pyramid can be built, and blur kernels can be restored from coarse to fine. The global inconsistent blur causes are complex, including camera rotation, dynamic targets in static scenes, depth of field changes, etc., and it is generally considered that each small region in the blurred image shares a blur kernel, and a linear blur kernel library is usually built to fit the blur kernels of the small regions. The global consistent deblurring effect is good, but the assumption of consistent blurring is over-idealized; the incongruous blurring with complex causes is closer to the real world, and has more practical application value for the research of the incongruous blurring, but modeling and solving are complex under the framework of the traditional method, and the effect is not very satisfactory.
In recent years, the deep neural network has strong learning ability in the field of computer vision, and has a lot of achievements in the field of image motion blur removal. The deep learning method is based on data driving, which does not strictly distinguish between globally consistent deblurring and globally non-consistent deblurring. The method is characterized in that the estimation value of the fuzzy kernel is obtained by utilizing the strong characteristic expression capability of the network to learn, and then the traditional method is used for deconvolution of the image, but the deblurring effect is not improved greatly. The subsequent end-to-end deblurring network framework learns a mapping from a blurred image to a clear image, and compared with the traditional energy optimization method, the existing non-uniform deblurring method based on deep learning has greatly improved model establishment, model solving and deblurring effects, but the problem that deblurring is not thorough still exists at the edge of the image.
Disclosure of Invention
The invention aims to provide a deep learning image motion blur removing method taking edge information as auxiliary information.
In order to achieve the above purpose, the invention adopts the following technical scheme: an edge-based deep learning image motion blur removing method comprises the following steps:
step 1, extracting edges from a blurred image by using a trained HED network, and then extracting edge characteristic information for guiding a motion deblurring process by using a convolution layer;
step 2, the deblurring backbone network extracts multi-scale feature information from the blurred image, the spatial feature transformation layer is used for integrating the image features and the edge features on each scale, and the decoding part gradually recovers the potential clear image from the deepest image features;
step 3, taking the fuzzy-clear image pair as a training sample set, defining the sum of a mean square error loss function and a perception loss function as a total loss function, and training the deblurring main network by using the total loss function until the deblurring main network converges to the optimal precision;
and step 4, inputting the motion blurred image into the deblurred backbone network trained in the step 3 to obtain a deblurred result.
In the above-mentioned edge-based deep learning image motion blur removing method, the obtaining of the edge feature information in step 1 includes the following sub-steps:
step 1.1, acquiring a blurred image edge map; inputting a color blurred image with the size W multiplied by H multiplied by 3 into an HED network loaded with pre-training weights to obtain an edge map with the size W multiplied by H multiplied by 1, wherein W is the width of an original map, and H is the height of the original map;
step 1.2, excavating deep characteristic information of an edge map; taking the edge map output in the step 1.1 as input, extracting high-level edge characteristic information from the fuzzy edge through a series of convolution and nonlinear activation operations: the convolution kernel size of the first convolution is 1 multiplied by 1, and the convolution kernel size of the subsequent four convolutions is 3 multiplied by 3, so that the spatial resolution of the image is kept unchanged in the whole process; nonlinear activation adopts a hole correction linear unit, and finally high-dimensional edge characteristic information with the size of W multiplied by H multiplied by 128 is output.
In the above-mentioned edge-based deep learning image motion blur removing method, the implementation of step 2 includes the following sub-steps:
step 2.1, extracting the characteristics of the blurred image; the method comprises the steps of inputting a color blurred image with the size W multiplied by H multiplied by 3 into a convolution layer formed by convolution and nonlinear activation, wherein the encoding stage can be divided into 4 processing blocks, the characteristic diagram of each block is respectively of the size kW multiplied by kH multiplied by l, W is the width of an original diagram, and H is the height of the original diagram; k=1, 0.5,0.25, l=32/k;
step 2.2, integrating the fuzzy image features and the edge information in different scales; integrating the edge characteristics output in the step 1.2 and the image characteristics of the current scale obtained in the step 2.1 by adopting a spatial characteristic transformation residual block, wherein the spatial characteristic transformation residual block comprises a spatial characteristic transformation layer, a convolution layer, a spatial characteristic transformation layer and a convolution layer;
step 2.3, further mining the fuzzy image characteristics; the method comprises the steps of combining hole convolutions with different hole rates, and increasing receptive fields so as to further mine characteristic information, wherein the characteristic information comprises 2 serial-hole convolutions residual blocks and 1 parallel-hole convolutions residual blocks;
and 2.4, gradually reconstructing an image after deblurring from the deep image features.
In the above-mentioned edge-based deep learning image motion blur removing method, the implementation of step 3 includes the following sub-steps:
step 3.1, respectively defining the mean square error loss function L mse And a perceptual loss function L p ：
Wherein I is c And I d Respectively a real clear image and a deblurred image; m, n represent the horizontal and vertical index of the image; phi (phi) i,j VGG19 feature map loaded with weights pre-trained on ImageNet, which is the jth convolution before the ith max pooling layer, W i,j And H i,j Is the size of the feature map, i=3, j=3 is usually set;
step 3.2, defining a total loss function:
L total ＝L mse +λ×L p ，
wherein λ is the weight of the perceptual loss function, set to 0.01;
using the total loss function L total The network is trained until the entire network converges to an optimal accuracy.
The invention has the beneficial effects that: 1. and the characteristic learning and generalization capability is strong. By using a deep learning method based on a convolutional neural network, an end-to-end model is trained, so that a motion blurred image is input, and a clear image with the same resolution as the input image can be obtained. The process does not need to give manually designed features in advance, and the network itself can learn the required features from training data and reasonably utilize the features, so that the method has better generalization capability and can also have stable performance even under a severe fuzzy scene.
2. The network structure is simple and easy to train. The deblurring backbone network is single-scale, no additional sub-networks (such as edge extraction networks) need to be trained, only the existing edge extraction networks are used for acquiring the edge, and the guiding information provided by the edge is effectively utilized through the edge feature and image feature integration module. Therefore, the network designed by the invention has simple structure and is easy to train.
3. The deblurring precision is improved, and the effect at the edge is obvious. By introducing edge information and effectively integrating the edge information and the image information on the feature level through the spatial feature transformation layer, the deblurring effect is obviously improved, especially at the edge.
Drawings
FIG. 1 is a flow chart of a method for edge deep learning image motion blur removal according to one embodiment of the present invention;
FIG. 2 (a) is a motion blur removal backbone network in the architecture of an edge-based deep learning motion blur removal network according to one embodiment of the present invention;
FIG. 2 (b) is an edge-based feature integration module in the architecture of an edge-based deep learning de-motion blur network according to one embodiment of the present invention;
FIG. 2 (c) is a block of parallel-hole convolution residuals in an architecture of an edge-based deep learning de-motion blur network according to one embodiment of the invention;
FIG. 3 (a) is a blurred image of an experiment on a GOPRO test dataset in accordance with one embodiment of the present invention;
FIG. 3 (b) is a deblurred image of an experiment on a GOPRO test dataset according to one embodiment of the invention;
FIG. 4 (a) is a first blurred image on the Stereo Blur Dataset test data set in accordance with one embodiment of the invention;
FIG. 4 (b) is a first deblurred image on a Stereo Blur Dataset test dataset according to an embodiment of the invention;
FIG. 4 (c) is a second blurred image on the Stereo Blur Dataset test data set in accordance with one embodiment of the invention;
fig. 4 (d) is a second deblurred image on the Stereo Blur Dataset test dataset according to an embodiment of the invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Considering the performance deficiency of the end-to-end non-uniform deblurring method based on deep learning at the edge and the feasibility of reducing the solution space by introducing image priori knowledge to obtain an effective deblurring result by the traditional deblurring method, the embodiment provides a deep learning image deblurring method taking edge information as auxiliary information, so that the deblurring effect at the edge is further improved. Compared with a multi-scale deblurring network architecture, the embodiment only uses a single-scale network architecture, so that the complexity and the parameter quantity of the network are greatly reduced. The present embodiment introduces edge information to make the deblurring process more focused on edge regions than most single-scale deblurring network architectures. In contrast to existing single-scale deblurring network architectures that take into account edge information, the present embodiment directly uses existing edge extraction networks to derive edges and integrates image and edge information at the feature level with novel spatial feature transformations.
The embodiment is realized by the following technical scheme, namely, an edge-based deep learning image motion blur removing method, the whole network structure is shown in fig. 2 (a), 2 (b) and 2 (c), and the method mainly comprises two modules, namely, an edge information preprocessing and motion blur removing backbone network, and if not specifically described, the convolution kernels used in the embodiment are all 3×3 in size. The specific steps of motion blur removal are as follows:
step one: extracting edges from the blurred image using a trained HED (Holistcally-Nested Edge Detection) network, and then extracting edge feature information guiding a motion deblurring process using a convolution layer;
step two: the image deblurring backbone network of the encoding-decoding structure extracts multi-scale feature information from the blurred image, the spatial feature transformation layer is used for integrating the image features and the edge features on each scale, and the decoding part gradually recovers potential clear images from the deepest image features;
step three: the fuzzy-clear image pair is used as a training sample set, the sum of the mean square error loss function and the perception loss function is defined as a total loss function, and the deblurring main network is trained by using the total loss function until the deblurring main network converges to the optimal precision;
step four: and inputting the motion blurred image into a trained network to obtain a deblurred result.
In specific implementation, as shown in fig. 2 (a), fig. 2 (b) and fig. 2 (c), the edge-based deep learning image deblurring network frame includes the following steps, as shown in fig. 1:
s1, acquiring edge characteristic information, which comprises the following substeps:
s1.1, acquiring a blurred image edge map. The HED (Holisically-Nested Edge Detection) network is a neural network framework for extracting edges of images, which can output an edge probability map of the same resolution from the input color image once training is completed, with pixel values between 0 and 1. And inputting the color blurred image with the size W multiplied by H multiplied by 3 into the HED network loaded with the pre-training weight to obtain a W multiplied by H multiplied by 1 edge map, wherein W represents the width of the original map and H represents the height of the original map.
S1.2, mining deep characteristic information of the edge map. Taking the edge output in the step S1.1 as an input, extracting high-level edge characteristic information from the fuzzy edge through a series of convolution and nonlinear activation operations: the convolution kernel size of the first convolution is 1 multiplied by 1, and the convolution kernel size of the subsequent four convolutions is 3 multiplied by 3, so that the spatial resolution of the image is kept unchanged in the whole process; nonlinear activation adopts a hole correction linear unit (leakyReLU) and finally outputs high-dimensional edge characteristic information with the size of w×h×128.
S2, the deblurring backbone network realizes end-to-end deblurring of the input blurred image, and as shown in FIG. 2 (a), the deblurring method comprises the following sub-steps:
s2.1, extracting the characteristics of the blurred image. The color blurred image of size w×h×3 is input to a convolution layer composed of convolution and nonlinear activation, and the encoding stage can be divided into 4 processing blocks, each block having a feature map size kw×kh×l, (k=1, 0.5,0.25, l=32/k), respectively.
S2.2, integrating the image features and the edge information in different scales. And integrating the edge feature obtained in the step S1.2 and the image feature of the current scale obtained in the step S2.1 by adopting a spatial feature transformation residual block, wherein the spatial feature transformation residual block comprises a spatial feature transformation layer, a convolution layer, a spatial feature transformation layer and a convolution layer. As shown in fig. 2 (b).
Taking the 1 st time of spatial feature transformation residual operation on the original image resolution scale as an example for explanation, wherein k=1, the image feature size is w×h×32, (1) firstly, a spatial feature transformation layer is used for respectively obtaining gain parameters gamma and deviation parameters beta for adjusting the image feature from edge features through 2 times of convolution layer operation, the spatial resolution of the two parameters is consistent with the channel number and the current image feature size, and each convolution layer comprises 2 times of convolutions: the 1 st convolution input is W×H×32, and W×H×32 is output at the original resolution; the 2 nd convolution input is w×h×32, and w×h×l (k=1, l=32/k) is output. Then the gain parameter gamma and the image feature are multiplied pixel by pixel, and the deviation parameter beta and the image feature output in the last step are added pixel by pixel to obtain the adjusted image feature, so that the deblurring effect at the edge is more concerned. (2) A convolution operation is performed with a constant number of channels and spatial resolution. (3) As in (1), new gain parameters γ and deviation parameters β are learned to further adjust the image characteristics. (4) And (3) performing convolution operation with unchanged channel number and spatial resolution, as in (2). And adding the input current scale image features and the features after the spatial feature transformation residual operation, thus completing the integration operation of the image features and the edge information at one time. Performing integration operation of image features and edge information for 3 times at original image resolution (k=1 at this time); then downsampling the image features to 1/2 (k=0.5 at this time) by a convolution operation with a step length of 2, wherein the size is kW×kH×64, reducing the edge features to 1/2 of the original image by a convolution layer with a step length of 1 and a convolution layer with a step length of 2, and performing 3 times of image feature and edge information integration operation on the resolution; the image features are then downsampled again by a factor of 1/2 (where k=0.25) by a convolution operation of step size 2, with a size kW x kH x 128, and the edge features are reduced to 1/4 of the original size by a convolution layer of step size 2 twice, again at this resolution, 3 image feature and edge information integration operations are performed.
S2.3, further mining the blurred image features. The hole convolutions with different hole rates are combined to increase the receptive field and thereby mine more characteristic information, including 2 serial-hole convolutions residual blocks and 1 parallel-hole convolutions residual blocks. The standard residual error module adds the result of the convolution-nonlinear activation-convolution operation of the input and the input to obtain the output, and the serial-hole convolution residual error block and the parallel-hole convolution are all changes on the standard residual error: the 2 serial-hole convolution residual blocks change the original hole rate 1 of the 1 st convolution to 2 and 3 respectively, and the input and output are kW×kH×128 (k=0.25). Parallel-hole convolution as shown in fig. 2 (c), the input is subjected to a hole convolution operation with hole ratios of 1,2,3,4 in a parallel manner, and the input and output are kw×kh×128 (k=0.25); the results of the 4 convolution operations are then combined together by concatenation in the channel dimension, where the image feature size is kW x kH x 512 (k=0.25); then the number of image feature channels is changed to 128 by a convolution operation with a void ratio of 1, and the input and output are kw×kh×512 (k=0.25) and kw×kh×128 (k=0.25), respectively; and finally, adding the result of the operation to the input.
S2.4, gradually reconstructing an image after deblurring from the deep image features. Firstly, processing image features by adopting the operation of a residual error module of 1 time convolution and 3 times on the lowest resolution scale kW multiplied by kH multiplied by 128 (k=0.25); then upsampling the image features to kW x kH x 64 (k=0.5) by transposed convolution followed by the same operation of 1 convolution plus 3 residual modules, except that the input of the convolution is the value of the current scale image feature and the corresponding scale image feature of the encoded portion in series in the channel dimension; likewise, the image features are then up-sampled to kw×kh×32 (k=1) by a transposed convolution, followed by 1 convolution plus 3 operations of the residual block, the input of the convolution still being the value of the concatenation of the image features of the current scale and the image features of the corresponding scale of the encoded portion in the channel dimension; then changing the image feature of kw×kh×32 (k=1) to a color image of w×h×3 by a convolution operation, which represents the difference between the blurred image and the clear image learned by the network; and finally, adding the input and the difference map to obtain a final deblurring image.
S3, defining a loss function, training the network based on a training sample set, and comprising the following substeps:
s3.1 using a mean square error loss function L mse Guaranteeing the fidelity of the deblurring result, using the perceptual loss function L p To improve the quality at the details of the deblurred results, the two loss functions are defined as follows:
wherein I is c And I d Respectively a real clear image and a deblurred image; m, n represent the horizontal and vertical index of the image; phi (phi) i,j VGG19 feature map loaded with weights pre-trained on ImageNet, which is the jth convolution before the ith max pooling layer, W i,j And H i,j Is the size of the feature map, i=3, j=3 is usually set.
S3.2, defining a total loss function,
L total ＝L mse +λ×L p ，
where λ is the weight of the perceptual loss function, which is typically set to 0.01. Using the total loss function L total The network is trained until the entire network converges to an optimal accuracy.
S4, inputting the motion blurred image into a trained network in the S3 to obtain a deblurring result with more thorough blur removal.
In this embodiment, the deblurring is performed on a part of experimental data, and the results are shown in fig. 3 (a), fig. 3 (b), fig. 4 (a), fig. 4 (b), fig. 4 (c), and fig. 4 (d), so that it can be seen that the present embodiment can stably and accurately deblur blurred images with different blur degrees.
It should be understood that parts of the specification not specifically set forth herein are all prior art.
While particular embodiments of the present invention have been described above with reference to the accompanying drawings, it will be understood by those skilled in the art that these are by way of example only, and that various changes and modifications may be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is limited only by the appended claims.
Claims (1)
1. The edge-based deep learning image motion blur removing method is characterized by comprising the following steps of:
step 1, extracting edges from a blurred image by using a trained HED network, and then extracting edge characteristic information for guiding a motion deblurring process by using a convolution layer;
step 2, the deblurring backbone network extracts multi-scale feature information from the blurred image, the spatial feature transformation layer is used for integrating the image features and the edge features on each scale, and the decoding part gradually recovers the potential clear image from the deepest image features;
step 3, taking the fuzzy-clear image pair as a training sample set, defining the sum of a mean square error loss function and a perception loss function as a total loss function, and training the deblurring main network by using the total loss function until the deblurring main network converges to the optimal precision;
step 4, inputting the motion blurred image into the deblurred backbone network trained in the step 3 to obtain a deblurred result;
the obtaining of the edge characteristic information in the step 1 comprises the following substeps:
step 1.1, acquiring a blurred image edge map; inputting a color blurred image with the size W multiplied by H multiplied by 3 into an HED network loaded with pre-training weights to obtain an edge map with the size W multiplied by H multiplied by 1, wherein W is the width of an original map, and H is the height of the original map;
step 1.2, excavating deep characteristic information of an edge map; taking the edge map output in the step 1.1 as input, extracting high-level edge characteristic information from the fuzzy edge through a series of convolution and nonlinear activation operations: the convolution kernel size of the first convolution is 1 multiplied by 1, and the convolution kernel size of the subsequent four convolutions is 3 multiplied by 3, so that the spatial resolution of the image is kept unchanged in the whole process; the nonlinear activation adopts a hole correction linear unit, and finally outputs high-dimensional edge characteristic information with the size of W multiplied by H multiplied by 128;
the implementation of step 2 comprises the following sub-steps:
step 2.1, extracting the characteristics of the blurred image; the method comprises the steps of inputting a color blurred image with the size W multiplied by H multiplied by 3 into a convolution layer formed by convolution and nonlinear activation, wherein the encoding stage can be divided into 4 processing blocks, and the characteristic diagram of each block is respectively kW multiplied by kH multiplied by l, wherein W is the width of an original diagram, and H is the height of the original diagram; k=1, 0.5,0.25, l=32/k;
step 2.2, integrating the fuzzy image features and the edge information in different scales; integrating the edge characteristics output in the step 1.2 and the image characteristics of the current scale obtained in the step 2.1 by adopting a spatial characteristic transformation residual block, wherein the spatial characteristic transformation residual block comprises a spatial characteristic transformation layer, a convolution layer, a spatial characteristic transformation layer and a convolution layer;
step 2.3, further mining the fuzzy image characteristics; the method comprises the steps of combining hole convolutions with different hole rates, and increasing receptive fields so as to further mine characteristic information, wherein the characteristic information comprises 2 serial-hole convolutions residual blocks and 1 parallel-hole convolutions residual blocks;
step 2.4, gradually reconstructing an image with blur removed from deep image features;
the implementation of step 3 comprises the following sub-steps:
step 3.1, respectively defining the mean square error loss function L mse And a perceptual loss function L p ：
Wherein I is c And I d Respectively a real clear image and a deblurred image; m, n represent the horizontal and vertical index of the image; phi (phi) i,j VGG19 feature map loaded with weights pre-trained on ImageNet, which is the jth convolution before the ith max pooling layer, W i,j And H i,j Is the size of the feature map, i=3, j=3 is usually set;
step 3.2, defining a total loss function:
L total ＝L mse +λ×L p ，
wherein λ is the weight of the perceptual loss function, set to 0.01;
using the total loss function L total The network is trained until the entire network converges to an optimal accuracy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911275632.8A CN111028177B (en) | 2019-12-12 | 2019-12-12 | Edge-based deep learning image motion blur removing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911275632.8A CN111028177B (en) | 2019-12-12 | 2019-12-12 | Edge-based deep learning image motion blur removing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111028177A CN111028177A (en) | 2020-04-17 |
| CN111028177B true CN111028177B (en) | 2023-07-21 |
Family
ID=70206421
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911275632.8A Active CN111028177B (en) | 2019-12-12 | 2019-12-12 | Edge-based deep learning image motion blur removing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111028177B (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583143A (en) * | 2020-04-30 | 2020-08-25 | 广州大学 | Complex image deblurring method |
| CN111695421B (en) * | 2020-04-30 | 2023-09-22 | 北京迈格威科技有限公司 | Image recognition method and device and electronic equipment |
| CN111986102B (en) * | 2020-07-15 | 2024-02-27 | 万达信息股份有限公司 | Digital pathological image deblurring method |
| CN111815536B (en) * | 2020-07-15 | 2022-10-14 | 电子科技大学 | Motion blur restoration method based on contour enhancement strategy |
| CN112488946B (en) * | 2020-12-03 | 2024-04-09 | 重庆邮电大学 | Single-scale motion blurred image frame restoration method for cab environment |
| CN112634153B (en) * | 2020-12-17 | 2023-10-20 | 中山大学 | Image deblurring method based on edge enhancement |
| CN112465730A (en) * | 2020-12-18 | 2021-03-09 | 辽宁石油化工大学 | Motion video deblurring method |
| CN112767277B (en) * | 2021-01-27 | 2022-06-07 | 同济大学 | Depth feature sequencing deblurring method based on reference image |
| CN112991194B (en) * | 2021-01-29 | 2022-06-24 | 电子科技大学 | Infrared thermal wave image deblurring method based on depth residual error network |
| CN113205464B (en) * | 2021-04-30 | 2023-05-05 | 作业帮教育科技（北京）有限公司 | Image deblurring model generation method, image deblurring method and electronic equipment |
| CN113191984B (en) * | 2021-05-24 | 2023-04-18 | 清华大学深圳国际研究生院 | Deep learning-based motion blurred image joint restoration and classification method and system |
| CN114187191B (en) * | 2021-11-20 | 2024-02-27 | 西北工业大学 | Image deblurring method based on high-frequency-low-frequency information fusion |
| CN114359082B (en) * | 2021-12-24 | 2023-01-06 | 复旦大学 | Gastroscope image deblurring algorithm based on self-built data pair |
| CN114549361B (en) * | 2022-02-28 | 2023-06-30 | 齐齐哈尔大学 | Image motion blur removing method based on improved U-Net model |
| CN114998156B (en) * | 2022-06-30 | 2023-06-20 | 同济大学 | Image motion deblurring method based on multi-patch multi-scale network |
| CN117593591B (en) * | 2024-01-16 | 2024-05-31 | 天津医科大学总医院 | Tongue picture classification method based on medical image segmentation |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198319A (en) * | 2013-04-11 | 2013-07-10 | 武汉大学 | Method of extraction of corner of blurred image in mine shaft environment |
| CN103761710A (en) * | 2014-01-08 | 2014-04-30 | 西安电子科技大学 | Image blind deblurring method based on edge self-adaption |
| WO2018045602A1 (en) * | 2016-09-07 | 2018-03-15 | 华中科技大学 | Blur kernel size estimation method and system based on deep learning |
| CN107871310A (en) * | 2017-10-26 | 2018-04-03 | 武汉大学 | A kind of single image for being become more meticulous based on fuzzy core is blind to go motion blur method |
| CN109035149A (en) * | 2018-03-13 | 2018-12-18 | 杭州电子科技大学 | A kind of license plate image based on deep learning goes motion blur method |
| CN109087256A (en) * | 2018-07-19 | 2018-12-25 | 北京飞搜科技有限公司 | A kind of image deblurring method and system based on deep learning |
| CN110033415A (en) * | 2019-03-20 | 2019-07-19 | 东南大学 | A kind of image deblurring method based on Retinex algorithm |
| CN110060215A (en) * | 2019-04-16 | 2019-07-26 | 深圳市商汤科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN110378844A (en) * | 2019-06-14 | 2019-10-25 | 杭州电子科技大学 | Motion blur method is gone based on the multiple dimensioned Image Blind for generating confrontation network is recycled |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9076205B2 (en) * | 2012-11-19 | 2015-07-07 | Adobe Systems Incorporated | Edge direction and curve based image de-blurring |
| US9430817B2 (en) * | 2013-11-12 | 2016-08-30 | Microsoft Technology Licensing, Llc | Blind image deblurring with cascade architecture |
| US10360494B2 (en) * | 2016-11-30 | 2019-07-23 | Altumview Systems Inc. | Convolutional neural network (CNN) system based on resolution-limited small-scale CNN modules |
| US11593632B2 (en) * | 2016-12-15 | 2023-02-28 | WaveOne Inc. | Deep learning based on image encoding and decoding |
-
2019
- 2019-12-12 CN CN201911275632.8A patent/CN111028177B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198319A (en) * | 2013-04-11 | 2013-07-10 | 武汉大学 | Method of extraction of corner of blurred image in mine shaft environment |
| CN103761710A (en) * | 2014-01-08 | 2014-04-30 | 西安电子科技大学 | Image blind deblurring method based on edge self-adaption |
| WO2018045602A1 (en) * | 2016-09-07 | 2018-03-15 | 华中科技大学 | Blur kernel size estimation method and system based on deep learning |
| CN107871310A (en) * | 2017-10-26 | 2018-04-03 | 武汉大学 | A kind of single image for being become more meticulous based on fuzzy core is blind to go motion blur method |
| CN109035149A (en) * | 2018-03-13 | 2018-12-18 | 杭州电子科技大学 | A kind of license plate image based on deep learning goes motion blur method |
| CN109087256A (en) * | 2018-07-19 | 2018-12-25 | 北京飞搜科技有限公司 | A kind of image deblurring method and system based on deep learning |
| CN110033415A (en) * | 2019-03-20 | 2019-07-19 | 东南大学 | A kind of image deblurring method based on Retinex algorithm |
| CN110060215A (en) * | 2019-04-16 | 2019-07-26 | 深圳市商汤科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN110378844A (en) * | 2019-06-14 | 2019-10-25 | 杭州电子科技大学 | Motion blur method is gone based on the multiple dimensioned Image Blind for generating confrontation network is recycled |
Non-Patent Citations (3)
| Title |
|---|
| Motion Blur Kernel Estimation via Deep Learning;Xiangyu Xu etal;IEEE Transactions on Image Processing;全文 * |
| 基于快速卷积神经网络的图像去模糊;任静静;方贤勇;陈尚文;汪粼波;周健;;计算机辅助设计与图形学学报(第08期);全文 * |
| 结合深度残差学习和感知损失的图像去噪;吴从中;陈曦;季栋;詹曙;;中国图象图形学报(第10期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111028177A (en) | 2020-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111028177B (en) | Edge-based deep learning image motion blur removing method | |
| Yue et al. | Supervised raw video denoising with a benchmark dataset on dynamic scenes | |
| CN111915530B (en) | End-to-end-based haze concentration self-adaptive neural network image defogging method | |
| Li et al. | Learning a deep dual attention network for video super-resolution | |
| CN111709895A (en) | Image blind deblurring method and system based on attention mechanism | |
| US20240062530A1 (en) | Deep perceptual image enhancement | |
| CN111091503A (en) | Image out-of-focus blur removing method based on deep learning | |
| CN112241939B (en) | Multi-scale and non-local-based light rain removal method | |
| Li et al. | A maximum a posteriori estimation framework for robust high dynamic range video synthesis | |
| CN116681584A (en) | Multistage diffusion image super-resolution algorithm | |
| CN114463218A (en) | Event data driven video deblurring method | |
| Zhang et al. | Deep motion blur removal using noisy/blurry image pairs | |
| Zhao et al. | Deep pyramid generative adversarial network with local and nonlocal similarity features for natural motion image deblurring | |
| CN117274059A (en) | Low-resolution image reconstruction method and system based on image coding-decoding | |
| Huang et al. | Deep Gaussian scale mixture prior for image reconstruction | |
| Chen et al. | Attention-based broad self-guided network for low-light image enhancement | |
| Guo et al. | Image blind deblurring using an adaptive patch prior | |
| CN116208812A (en) | Video frame inserting method and system based on stereo event and intensity camera | |
| Han et al. | MPDNet: An underwater image deblurring framework with stepwise feature refinement module | |
| CN110648291B (en) | Unmanned aerial vehicle motion blurred image restoration method based on deep learning | |
| CN113902647A (en) | Image deblurring method based on double closed-loop network | |
| CN114511487A (en) | Image fusion method and device, computer readable storage medium and terminal | |
| CN115393491A (en) | Ink video generation method and device based on instance segmentation and reference frame | |
| Zhao et al. | Single image super-resolution via blind blurring estimation and anchored space mapping | |
| CN108665412B (en) | Method for performing multi-frame image super-resolution reconstruction by using natural image priori knowledge |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |