CN109598695B - No-reference image fuzzy degree estimation method based on deep learning network - Google Patents

No-reference image fuzzy degree estimation method based on deep learning network Download PDF

Info

Publication number
CN109598695B
CN109598695B CN201710909377.2A CN201710909377A CN109598695B CN 109598695 B CN109598695 B CN 109598695B CN 201710909377 A CN201710909377 A CN 201710909377A CN 109598695 B CN109598695 B CN 109598695B
Authority
CN
China
Prior art keywords
network
fuzzy
clear
training
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710909377.2A
Other languages
Chinese (zh)
Other versions
CN109598695A (en
Inventor
岳涛
索津莉
袁飞
黄华
陈鑫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University
Original Assignee
Nanjing University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University filed Critical Nanjing University
Priority to CN201710909377.2A priority Critical patent/CN109598695B/en
Publication of CN109598695A publication Critical patent/CN109598695A/en
Application granted granted Critical
Publication of CN109598695B publication Critical patent/CN109598695B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The invention discloses a no-reference image fuzzy degree estimation method based on a deep learning network. The method comprises the following specific steps: (1) preprocessing the image to generate training data; (2) Training a dual-channel clear/fuzzy perception network by using clear and fuzzy image blocks, extracting clear and fuzzy features and reconstructing input; (3) The features extracted by the clear/fuzzy perception network are used as the input of the joint perception network, the fuzzy image blocks are used for training the joint perception network, and the fuzzy essential features are obtained; (4) Training a nonlinear feature mapping and regression network, and mapping the essential features in the step (3) to a fuzzy degree; and (5) fine-tuning the whole network and optimizing all parameters. The method has a good effect on the fuzzy degree estimation accuracy of the non-reference slight fuzzy, and can be effectively applied to the aspects of fuzzy detection, depth estimation, field depth editing, defogging and the like.

Description

No-reference image fuzzy degree estimation method based on deep learning network
Technical Field
The invention relates to the field of computational photography and deep learning, in particular to a reference-free image blur degree estimation method based on a deep learning network.
Background
The availability of digital cameras and networks has prompted the rapid development of photography. A large number of pictures are from non-professional human hands and are distorted, especially blurred, due to incorrect setting of the camera parameters. Blurring is a common degradation in natural images, and although it affects the visual quality and continuous processing of images, it provides rich clues for solving many visual problems and applications (e.g. blur removal, refocusing, depth estimation, segmentation). An accurate estimate of the degree of blurring is important. However, it is very tricky to estimate the non-uniform blur level from a single natural picture. Many existing ambiguity estimation methods are based on manual description features, which are mostly unreasonable or difficult to distinguish in a specific scene. Therefore, better feature extraction is required for reference-free image blur degree estimation.
Deep learning learns more useful features by constructing a machine learning model with a plurality of hidden layers and massive training data, thereby finally improving the accuracy of classification or prediction. By means of layer-by-layer feature transformation, the feature representation of the sample in the original space is transformed to a new feature space, so that classification or prediction is easier. Compared with a method for constructing the features by using manual rules, the method for constructing the features by using the big data to learn the features can depict rich intrinsic information of the data.
Therefore, how to construct and train an effective deep learning network to estimate the image blur degree is a current research direction.
Disclosure of Invention
Aiming at the defects of the existing fuzzy degree estimation method, the invention aims to provide a no-reference image fuzzy degree estimation method based on a deep learning network.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
a no-reference image fuzzy degree estimation method based on a deep learning network comprises the following steps:
step 1, generating training data: the method comprises the steps of down-sampling an image, intercepting a structured or textured clear image block in the image, and convolving the clear image block by using a Gaussian blur kernel to generate a blur image block;
step 2, respectively training two sparse self-encoders of a clear perception network and a fuzzy perception network by using the clear image blocks and the fuzzy image blocks; respectively extracting clear and fuzzy effective characteristics, and decoding and reconstructing the input;
step 3, taking the features extracted by the clear perception network and the fuzzy perception network as the input of the joint perception network, training the joint perception network by using the fuzzy image blocks, and acquiring fuzzy essential features;
step 4, training a nonlinear feature mapping and regression network, and mapping the essential features in the step 3 to a fuzzy degree;
and 5, finely adjusting the whole network and optimizing all parameters.
The invention uses a deep neural network comprising three layers of sub-networks: a two-channel clear/fuzzy perception network, a clear fuzzy joint perception network, a nonlinear feature mapping and a regression network. By adopting a learning strategy of layered training and utilizing a large amount of training data, the internal parameters of the deep network can be obtained gradually from front to back and used as prior information of the training data, so that the convergence speed of the network parameters can be greatly accelerated. Compared with the prior art, the method has a good effect on the fuzzy degree estimation accuracy of the non-reference slight fuzzy, and can be effectively applied to the aspects of fuzzy detection, depth estimation, field depth editing, defogging and the like.
Drawings
FIG. 1 is a deep learning network architecture for the method of the present invention;
FIG. 2 is a flow chart of the method of the present invention.
Detailed Description
The invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
Referring to fig. 1 and fig. 2, a method for estimating a degree of blur of a non-reference image based on a deep learning network according to this embodiment includes the following specific steps:
step 1, generating training data: converting the color image into a gray image; down-sampling the image, and reserving the most remarkable characteristics; and intercepting image blocks with structures or textures in the image, wherein the size of the image blocks is 13 × 13, and convolving the clear image blocks by using a Gaussian blur kernel to generate blurred image blocks.
Step 2, respectively training two sparse self-encoders of a clear perception network and a fuzzy perception sub-network by using the clear image blocks and the image blocks with fixed fuzziness; respectively extracting clear and fuzzy effective characteristics, and decoding and reconstructing the input.
Two fully-connected three-layer networks are established, including an input layer, a hidden layer and an output layer. In this embodiment, the number of input layer nodes is 169, the number of hidden layer nodes is 100, and the number of output layer nodes is 169. For the input image block x, the features extracted by the hidden layer and the output layer reconstruction are respectively
h＝f(W 1 *x+b 1 )
x'＝f(W 2 *h+b 2 )
Where f (x) = 1/(1 + exp (-x)) is a non-linear function, W 1 ，W 2 Represents a weight, b 1 ，b 2 Representing an offset.
Adjusting W using back-propagation methods with reduced reconstruction errors 1 ，W 2 And b 1 ，b 2 The value of (c). In order to extract the most effective features, activation of the hidden unit is limited and made sparse. A regularization term is added to penalize deviations of the hidden unit activation level from a low value. Therefore, the training of the network is equivalent to the following optimization problem:
Figure BDA0001424498850000031
SP(ρ||ρ j )＝ρlog(ρ/ρ j )+(1-ρ)log((1-ρ)/(1-ρ j ))
where x is i And x i ' represents an input image block and an output reconstruction result, respectively, p represents the number of input image blocks, and SP (ρ | | ρ |, ρ) j ) Representing a sparse penalty term, p representing the average activation of the hidden unit, p j Representing the level of sparsity, beta controls the weight of the sparsity penalty term.
Given a clear image block x S And blurred image block x B As input, the extracted sharp and fuzzy features are respectively
Figure BDA0001424498850000032
Figure BDA0001424498850000033
Decoded reconstruction into
Figure BDA0001424498850000034
Figure BDA0001424498850000035
Parameter(s)
Figure BDA0001424498850000036
Representing clearly aware networks (SPN), parameters/>
Figure BDA0001424498850000037
Representing a fuzzy-aware network (BPN), two sub-networks were trained with two different sets of data.
And 3, taking the features extracted by the clear/fuzzy sensing network as the input of the joint sensing network, and training the joint sensing network by using the image blocks with variable fuzziness to obtain fuzzy essential features.
Combining a pre-trained Sharp Perception Network (SPN) and a fuzzy perception network (BPN) together to extract the intrinsic features in the fuzzy, which is a nonlinear mapping of different features of the sharp and the fuzzy. Defining an output of a first layer of a joint awareness network (SBBN) as
Figure BDA0001424498850000038
The operation of the joint awareness network (SBBN) is thus
Figure BDA0001424498850000039
Where h is 3 Is the output vector of the output layer of the joint-aware network, W 3 Represents a weight, b 3 Indicating the bias. The joint awareness network output layer contains 169 nodes.
To train a joint awareness network, parameters are fixed
Figure BDA00014244988500000310
By reducing h 3 And y R Inter loss, optimizing the remaining parameters (W) 3 ,b 3 ). The loss function is:
Figure BDA00014244988500000311
where n is the number of training image blocks, y R Is the residue between the blurred input and the original sharp image patch, representing the lost information between themAnd (4) information. The loss function is minimized by the back propagation algorithm.
And 4, training a nonlinear feature mapping and regression network, and mapping the essential features in the step 3 to the fuzzy degree.
The multi-layer neural network implements nonlinear feature mapping and regression, in which each feature mapping layer operates as:
h i ＝f(W i *h i-1 +b i )
where i =4,5,h i Is the output vector of the feature mapping layer. The number of nodes of the two feature mapping layers is 100 and 50 respectively. The final regression layer is:
D B ＝max(0,W 6 h 5 +b 5 )
relu is used here as the activation function.
Training the whole network, estimating all network parameters W i ,b i I =1,2, \ 8230;, 6, parameter { W } 1 ,W 2 ,W 3 ,b 1 ,b 2 ,b 3 And initialized to the corresponding pre-trained parameters, and the rest parameters are initialized immediately.
And 5, fine-tuning the whole network and optimizing all parameters. The back propagation algorithm is used to calculate the gradients of all network layers, since the algorithm is applicable to any number of layers of the network.

Claims (4)

1. A method for estimating the degree of blurring of a reference-free image based on a deep learning network is characterized by comprising the following steps:
step 1, generating training data: the method comprises the steps of down-sampling an image, intercepting a structured or textured clear image block in the image, and convolving the clear image block by using a Gaussian blur kernel to generate a blur image block;
step 2, respectively training two sparse self-encoders of a clear perception network and a fuzzy perception network by using the clear image blocks and the fuzzy image blocks; respectively extracting clear and fuzzy effective characteristics, and decoding and reconstructing the input;
step 3, taking the features extracted by the clear perception network and the fuzzy perception network as the input of the joint perception network, training the joint perception network by using the fuzzy image blocks, and acquiring fuzzy essential features;
step 4, training a nonlinear feature mapping and regression network, and mapping the essential features in the step 3 to a fuzzy degree;
and 5, fine-tuning the whole network and optimizing all parameters.
2. The method for estimating the degree of blur of the reference-free image based on the deep learning network as claimed in claim 1, wherein in the step 2, the specific process of extracting the sharp and blurred effective features and performing decoding reconstruction on the input is as follows:
establishing two fully-connected three-layer networks comprising an input layer, a hidden layer and an output layer; for the input image block x, the features extracted by the hidden layer and the output layer reconstruction are respectively
h＝f(W 1 *x+b 1 )
x'＝f(W 2 *h+b 2 )
Where f (x) = 1/(1 + exp (-x)) is a non-linear function, W 1 ，W 2 Represents a weight, b 1 ，b 2 Represents an offset;
adjusting the weights W using back-propagation methods with reduced reconstruction errors 1 、W 2 And offset b 1 、b 2 A value of (d); the training of the network is equivalent to the following optimization problem:
Figure FDA0003921937630000011
SP(ρ||ρ j )＝ρlog(ρ/ρ j )+(1-ρ)log((1-ρ)/(1-ρ j ))
where x i And x i ' represents an input image block and an output reconstruction result, respectively, p represents the number of input image blocks, and SP (ρ | | ρ |, ρ) j ) Denotes a sparse penalty term, p denotes the mean activation of the hidden unit, p j Representing the level of sparsity, beta controls the weight of a sparse penalty term;
given a clear image block x S And blurred image block x B As input, the extracted sharp and fuzzy features are:
Figure FDA0003921937630000012
Figure FDA0003921937630000013
the decoding reconstruction is respectively as follows:
Figure FDA0003921937630000021
Figure FDA0003921937630000022
parameter W 1 S ,
Figure FDA0003921937630000023
Representing a clearly aware network, parameter W 1 B ,/>
Figure FDA0003921937630000024
Representing a fuzzy-aware network, two sub-networks were trained with two different sets of data.
3. The method for estimating the degree of blur of the reference-free image based on the deep learning network as claimed in claim 2, wherein the specific process of the step 3 is as follows:
combining the pre-trained clear perception network and the fuzzy perception network together to extract the inherent characteristics in the fuzzy, which is a nonlinear mapping of different clear and fuzzy characteristics, and defining the output of the first layer of the combined perception network as
Figure FDA0003921937630000025
Thus the operation of the joint awareness network is
Figure FDA0003921937630000026
Where h is 3 Is the output layer of the joint-aware network, W 3 Represents a weight, b 3 Represents a bias;
to train the joint awareness network, the parameter W is fixed 1 S ,
Figure FDA0003921937630000027
W 1 B And &>
Figure FDA0003921937630000028
By reducing h 3 And y R Inter loss, optimizing the residual parameter W 3 And b 3 (ii) a Wherein the loss function is:
Figure FDA0003921937630000029
where n is the number of training image blocks, real output y R Is the residue between the fuzzy input and the original sharp image block, representing the lost information between them; the loss function is minimized by the back propagation algorithm.
4. The method for estimating the degree of blur of the reference-free image based on the deep learning network as claimed in claim 3, wherein the specific process of the step 4 is as follows:
the multi-layer neural network implements nonlinear feature mapping and regression, in which each feature mapping layer operates as:
h i ＝f(W i *h i-1 +b i )
where i =4,5,h i Is the output vector, W, of each feature mapping layer i 、b i Weights and biases of the feature mapping layers, respectively;
the final regression layer is:
D B ＝max(0,W 6 h 5 +b 5 )
relu is used here as the activation function;
training the whole network, estimating all network parameters W i ,b i }, i =1, 2.., 6, parameter { W } 1 ,W 2 ,W 3 ,b 1 ,b 2 ,b 3 And, initializing to the corresponding pre-trained parameters, and then initializing the rest parameters.
CN201710909377.2A 2017-09-29 2017-09-29 No-reference image fuzzy degree estimation method based on deep learning network Active CN109598695B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710909377.2A CN109598695B (en) 2017-09-29 2017-09-29 No-reference image fuzzy degree estimation method based on deep learning network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710909377.2A CN109598695B (en) 2017-09-29 2017-09-29 No-reference image fuzzy degree estimation method based on deep learning network

Publications (2)

Publication Number Publication Date
CN109598695A CN109598695A (en) 2019-04-09
CN109598695B true CN109598695B (en) 2023-04-07

Family

ID=65955301

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710909377.2A Active CN109598695B (en) 2017-09-29 2017-09-29 No-reference image fuzzy degree estimation method based on deep learning network

Country Status (1)

Country Link
CN (1) CN109598695B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110033848B (en) * 2019-04-16 2021-06-29 厦门大学 Three-dimensional medical image z-axis interpolation method based on unsupervised learning
CN110517203B (en) * 2019-08-30 2023-06-23 山东工商学院 Defogging method based on reference image reconstruction
CN111526357B (en) * 2020-04-14 2021-06-29 艾瑞思检测技术(苏州)有限公司 Display card interface machine testing method based on PCA learning
CN113436137A (en) * 2021-03-12 2021-09-24 北京世纪好未来教育科技有限公司 Image definition recognition method, device, equipment and medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9218648B2 (en) * 2009-10-27 2015-12-22 Honeywell International Inc. Fourier domain blur estimation method and system
CN104680491B (en) * 2015-02-28 2016-03-30 西安交通大学 A kind of image nonuniform motion ambiguity removal method based on deep neural network
CN106971378A (en) * 2016-08-23 2017-07-21 上海海洋大学 A kind of removing rain based on single image method based on depth denoising self-encoding encoder

Also Published As

Publication number Publication date
CN109598695A (en) 2019-04-09

Similar Documents

Publication Publication Date Title
CN108986050B (en) Image and video enhancement method based on multi-branch convolutional neural network
CN109598695B (en) No-reference image fuzzy degree estimation method based on deep learning network
CN110008817B (en) Model training method, image processing method, device, electronic equipment and computer readable storage medium
CN106204468B (en) A kind of image de-noising method based on ReLU convolutional neural networks
CN110969589B (en) Dynamic scene blurred image blind restoration method based on multi-stream annotating countermeasure network
CN109118445B (en) Underwater image enhancement method based on multi-branch generation countermeasure network
CN109859288B (en) Image coloring method and device based on generation countermeasure network
Jiang et al. Underwater image enhancement with lightweight cascaded network
CN110148088B (en) Image processing method, image rain removing method, device, terminal and medium
Wang et al. Joint iterative color correction and dehazing for underwater image enhancement
CN113284061B (en) Underwater image enhancement method based on gradient network
CN111047543A (en) Image enhancement method, device and storage medium
CN110807757A (en) Image quality evaluation method and device based on artificial intelligence and computer equipment
CN109685772B (en) No-reference stereo image quality evaluation method based on registration distortion representation
CN112288632A (en) Single image super-resolution method and system based on simplified ESRGAN
CN111179196A (en) Multi-resolution depth network image highlight removing method based on divide-and-conquer
Saleem et al. A non-reference evaluation of underwater image enhancement methods using a new underwater image dataset
CN113034388B (en) Ancient painting virtual repair method and construction method of repair model
CN112200752B (en) Multi-frame image deblurring system and method based on ER network
CN116664454B (en) Underwater image enhancement method based on multi-scale color migration parameter prediction
CN112634168A (en) Image restoration method combined with edge information
CN115965844B (en) Multi-focus image fusion method based on visual saliency priori knowledge
CN115018726A (en) U-Net-based image non-uniform blur kernel estimation method
CN114881879A (en) Underwater image enhancement method based on brightness compensation residual error network
CN113870162A (en) Low-light image enhancement method integrating illumination and reflection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant