CN112215773B - Local motion deblurring method and device based on visual saliency and storage medium - Google Patents
Local motion deblurring method and device based on visual saliency and storage medium Download PDFInfo
- Publication number
- CN112215773B CN112215773B CN202011084727.4A CN202011084727A CN112215773B CN 112215773 B CN112215773 B CN 112215773B CN 202011084727 A CN202011084727 A CN 202011084727A CN 112215773 B CN112215773 B CN 112215773B
- Authority
- CN
- China
- Prior art keywords
- image
- saliency
- fuzzy
- map
- foreground
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000000007 visual effect Effects 0.000 title claims abstract description 18
- 238000003860 storage Methods 0.000 title claims abstract description 15
- 230000011218 segmentation Effects 0.000 claims abstract description 22
- 238000005295 random walk Methods 0.000 claims abstract description 21
- 238000001514 detection method Methods 0.000 claims abstract description 20
- 238000001914 filtration Methods 0.000 claims abstract description 20
- 238000005457 optimization Methods 0.000 claims abstract description 20
- 230000006870 function Effects 0.000 claims description 14
- 230000003044 adaptive effect Effects 0.000 claims description 10
- 238000003709 image segmentation Methods 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 6
- 230000003321 amplification Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 5
- 238000009499 grossing Methods 0.000 claims description 5
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 5
- 239000013598 vector Substances 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 4
- 230000035945 sensitivity Effects 0.000 claims description 3
- 238000004422 calculation algorithm Methods 0.000 description 24
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000002068 genetic effect Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011840 criminal investigation Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 125000001475 halogen functional group Chemical group 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20201—Motion blur correction
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a local motion deblurring method, a device and a storage medium based on visual saliency, wherein the method comprises the following steps: detecting a saliency map of the fuzzy region by adopting the popular regularization random walk saliency, and using a detection result for marking the fuzzy region in the image; performing binarization operation on the saliency map by adopting an Otsu method based on heredity, segmenting a fuzzy foreground and a clear background into different map layers, and acquiring a foreground and background segmentation binarization image; substituting the obtained binary image into an image deblurring model based on an MAP framework for optimization, and iteratively estimating an initial fuzzy kernel and a potential image; listing the priors of the initial fuzzy kernel and the potential image, and solving the initial fuzzy kernel and the potential image based on iterative weighted least square; and further optimizing a solution result by adopting improved self-adaptive guide filtering, and keeping the edge information of the image.
Description
Technical Field
The invention relates to an image motion deblurring technology, belongs to the field of image processing, and particularly relates to a local motion deblurring method and device based on visual saliency and a storage medium.
Background
The motion blur image is a blur generated by relative motion between a camera and a moving object during an exposure time, and the blur of the image causes that a target object to be observed cannot be seen clearly or valuable information is acquired from the image. Moreover, the shooting process is short and is not easy to copy and reproduce, and in many cases, the shooting cannot be performed again to obtain clear images, so that the research on the motion blur restoration technology becomes particularly important, and the motion blur restoration technology is applied to the fields of road video monitoring, industrial production, criminal investigation, astronomical observation, military satellite tracking and the like.
In recent years, motion-blurred image restoration methods have been greatly developed, but such methods still have certain limitations to the problem of locally blurred image restoration. Because the imaging processes of the foreground and the background are different, a globally consistent fuzzy model cannot well model the image forming process, and because the rapid movement of an object can cause the fuzzy to change suddenly, some non-consistent deblurring methods based on camera shake modeling cannot well process the local motion fuzzy problem. There are also some existing methods to obtain additional information of moving objects by building a hybrid camera system, but such methods require elaborate hardware support.
In addition, some image segmentation-based local motion blurred image restoration methods are available, which depend on the segmentation quality to a great extent, and if the segmentation is not accurate, a good image restoration effect cannot be obtained, so that various requirements in practical application cannot be met.
Disclosure of Invention
The invention provides a local motion deblurring method, a device and a storage medium based on visual saliency, which improve the definition of an image, recover a clear image with higher quality, effectively enhance the contrast and edge details of the image, and solve the problem of losing texture information of a blurred region in the image, and are described in detail as follows:
in a first aspect, a method for local motion deblurring based on visual saliency, the method comprising:
detecting a saliency map of the fuzzy region by adopting popular regularization random walk saliency, and using a detection result to mark the fuzzy region in the image;
performing binarization operation on the saliency map by adopting an Otsu method based on heredity, segmenting a fuzzy foreground and a clear background into different map layers, and acquiring a foreground and background segmentation binarization image;
substituting the obtained binary image into an image deblurring model based on an MAP framework for optimization, and iteratively estimating an initial fuzzy core and a potential image;
listing the priors of the initial fuzzy kernel and the potential image, and solving the initial fuzzy kernel and the potential image based on iterative weighted least square;
and further optimizing a solution result by adopting improved self-adaptive guide filtering, and keeping the edge information of the image.
In an implementation manner, the detecting the saliency map of the fuzzy region by using the popular regularization random walk saliency is specifically:
optimizing the influence generated by the boundary according to a means of positioning and eliminating an error boundary to obtain a background significance value;
optimizing a pure background query according to the significance estimation of the foreground query;
and based on the random walk model, proposing fitting constraints for inheriting the significance value of the foreground query to obtain a final significance map.
In an implementation manner, the bringing the obtained binarized image into an image deblurring model based on a MAP framework for optimization specifically includes:
introducing an image segmentation term l i The image is divided into different layers, and fuzzy kernel estimation and recovery of a clear latent image are carried out on each layer.
In one implementation, the improved adaptive guided filtering is specifically:
constructing a guide image by using a weighted least squares filter WLS:
wherein, WLS (G) v Guide image, G, constructed for weighted least squares filter WLS v For a guide image corresponding to a spatial position v of an image pixel, I v The input image corresponding to the spatial position v of the image pixel, v represents the spatial position of the image pixel (x, y), the index tau determines the sensitivity of the gradient change of the input image I at the pixel v (x, y), eta is a smoothing term parameter,
and
first order partial derivatives of G in the x and y directions, respectively, indicating the degree of steepness of the image; tau is x,v And τ y,v A weight coefficient for refinement;
according to a local linear model between the guide image and the output image, incorporating the average value of the local variance of all pixels into a cost function of the guide filtering; an adaptive amplification factor beta is introduced to suppress noise.
In a second aspect, a local motion deblurring apparatus based on visual saliency, the apparatus comprising:
the detection and marking module is used for detecting a saliency map of the fuzzy area by adopting the popular regularization random walk saliency and using a detection result to mark the fuzzy area in the image;
the first acquisition module is used for carrying out binarization operation on the saliency map by adopting a heredity-based Otsu method, segmenting the fuzzy foreground and the clear background into different map layers and acquiring a foreground and background segmentation binarization image;
the first optimization module is used for substituting the obtained binary image into an image deblurring model based on an MAP framework for optimization, and iteratively estimating an initial fuzzy kernel and a potential image;
the second acquisition module is used for listing the initial fuzzy kernel and the prior of the potential image and solving the initial fuzzy kernel and the potential image based on iterative weighted least square;
and the second optimization module is used for further optimizing the solution result by adopting improved self-adaptive guide filtering and maintaining the edge information of the image.
In one implementation, the detection and labeling module includes:
the detection unit is used for optimizing the influence generated by the boundary according to the means of positioning and eliminating the error boundary and acquiring a background significance value; optimizing a simple background query according to the saliency estimate of the foreground query;
and the fitting constraint unit is used for proposing fitting constraint based on the random walk model, and is used for inheriting the significance value of the foreground query to obtain a final significance map.
In one implementation, the first optimization module includes:
a segmentation and restoration unit for introducing an image segmentation term l i Dividing the image into differentAnd performing fuzzy kernel estimation and clear latent image recovery on each layer.
In a third aspect, an apparatus for local motion deblurring based on visual saliency, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform the method steps of the first aspect.
In a fourth aspect, a computer-readable storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method steps of the first aspect.
The technical scheme provided by the invention has the beneficial effects that:
1. aiming at solving problems of removing a fuzzy core and a potential image in a fuzzy frame through maximum posterior, the method adopts an effective iterative weighted least square algorithm to optimize the solution of an object motion fuzzy estimation model, namely, on the basis of solving the fuzzy core through a Laplace prior and solving a clear latent image through a sparse image gradient prior, the problems of inaccurate optimal solution and low algorithm efficiency existing in the original algorithm are further solved;
2. the invention adopts the improved adaptive guided filtering algorithm to solve the problem that the texture edge details are difficult to recover in serious motion blur, adaptively inhibits noise by keeping the edge information of the image, further improves the definition of the image and enriches the detail information of the image.
Drawings
FIG. 1 is a flowchart of a local motion deblurring method based on visual saliency according to the present invention;
FIG. 2 is another flow chart of a local motion deblurring method based on visual saliency according to the present invention;
FIG. 3 is a schematic diagram of a locally motion blurred image;
FIG. 4 is a schematic diagram of the deblurred target image of FIG. 3;
FIG. 5 is a schematic illustration of another locally motion blurred image;
FIG. 6 is a schematic diagram of the deblurred target image of FIG. 5;
FIG. 7 is a schematic illustration of another locally motion blurred image;
FIG. 8 is a schematic diagram of the deblurred target image of FIG. 7;
FIG. 9 is a schematic structural diagram of a local motion deblurring apparatus based on visual saliency according to the present invention;
FIG. 10 is a schematic structural diagram of a detection and labeling module;
fig. 11 is another structural schematic diagram of a local motion deblurring apparatus based on visual saliency according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below.
Referring to fig. 1, an embodiment of the present invention provides a local motion deblurring method based on visual saliency, including the following steps:
step 101: detecting a saliency map of the fuzzy region by adopting the popular regularization random walk saliency, and using a detection result for marking the fuzzy region in the image;
in a local blurred image caused by object motion, the blur kernel of each region is usually inconsistent, so that the blur kernel needs to be solved hierarchically, and the solved blur kernel is used for recovering a clear image, so that the embodiment of the invention detects the saliency map of the blurred region by using the popular regularization random walk saliency to mark the blurred region in the image.
Step 102: carrying out binarization operation on the saliency map by Ostu (Otsu method) threshold segmentation based on a genetic algorithm, segmenting a fuzzy foreground and a clear background into different image layers, and further acquiring a foreground segmentation binarization image and a background segmentation binarization image;
in the subsequent steps, the fuzzy kernel is estimated through different layers, namely, the estimation accuracy of the fuzzy kernel is improved through the operation.
The Ostu is an algorithm for determining a binarization segmentation threshold of an image, and is also called as a maximum inter-class variance method, which is well known to those skilled in the art, and is not described in detail in the embodiments of the present invention.
Step 103: the obtained binary image is brought into an image deblurring model based on an MAP framework for optimization, namely a fuzzy kernel and a potential image are iteratively estimated from the fuzzy image;
in the image deblurring model based on the MAP framework, the segmentation algorithm can effectively guide motion blur estimation, and meanwhile, the motion blur estimation is in turn helpful for segmentation estimation, so that local blur in an image can be better removed. And taking the potential image and the fuzzy kernel obtained in the step as intermediate results, further obtaining a final accurate fuzzy kernel through the processing of subsequent steps, and further outputting a recovered clear image.
The optimized maximum a posteriori deblurring frame is well known to those skilled in the art, and the embodiment of the present invention will not be described in detail herein.
Step 104: respectively listing the priors of the initial fuzzy kernel and the potential image by adopting a Laplace prior and a sparse image gradient prior, and further solving the initial fuzzy kernel and the potential image based on a prior value and an iterative weighted least square algorithm;
aiming at the solving problem of the fuzzy core and the potential image in the maximum posterior fuzzy frame, an effective iteration weighted least square algorithm is adopted to solve the image deblurring model, namely Laplace prior and sparse image gradient prior are used for respectively defining and solving the prior of the initial fuzzy core and the potential image, then an iteration weighted minimum two-component algorithm is used for further optimizing and solving the image deblurring model, and the problems that the optimal solution is inaccurate and the efficiency of the algorithm is low in the original algorithm are further solved.
Step 105: and (3) further optimizing the solution result of the step (104) by adopting improved adaptive guided filtering, wherein the improved adaptive guided filtering is used for solving the problem of detail blurring at texture edges when motion blurring is serious, and by keeping the edge information of the image, noise is suppressed in a self-adaptive manner, the definition of the image is further improved, and the detail information of the image is enriched.
In summary, in the embodiments of the present invention, through the steps 101 to 105, the definition of the image is improved, a clear image with higher quality is restored, the contrast and edge details of the image are effectively enhanced, and the problem of texture information loss in a blurred region in the image is solved.
In the following, with reference to fig. 2 and a specific calculation formula, a local motion blur restoration method based on significance in the foregoing embodiment is detailed and expanded, an experimental object adopted by the present invention is a local motion blur image with different blur degrees, and for the problem of information loss in the blur image, the method includes the following steps:
step 201: detecting a saliency map of the blurred region by adopting the popular regularization random walk saliency, wherein the saliency map is used for marking the blurred region in the image;
in a local blurred image caused by object motion, blurred kernels are usually inconsistent and need to be solved hierarchically, so that the method and the device adopt the popular regularization random walk saliency to detect the saliency map of the blurred region so as to mark the blurred region in the image.
Before background saliency detection is performed, the impact of the boundaries is optimized by means of locating and eliminating false boundaries. Based on the popularity ranking regularization framework, those boundaries with the smallest probability of belonging to the background are deleted, and a saliency estimate is generated from the background query, with the popularity ranking function as follows:
wherein, f * For values of popularity ranking functions, d ii ,d jj The diagonal elements of the ith, j-th row of the degree matrix of the local graph (which is a term of art and is not described herein), and the parameter α is the balance between the control smoothing constraint (the first term before the plus sign) and the fitting constraint (the second term after the plus sign); f. of i Is the rank value of the i node, f j Is a j sectionRank value of points, ω ij Is a weight matrix of a graph edge, n is the number of elements, y is an index vector, defined as y = [ y = 1 ,…,y n ] T Then by pixel-by-pixel multiplication, the background saliency is found:
where l is the total number of pixels in the target area, n is the total number of super pixels, S l (i) Significance value based on the foreground, f l * (i) Calculated from equation (1), where l corresponds to the three boundary positions after elimination of the error boundary, S step1 (i) Is a one-time ranked background significance estimate.
But a mere background query is sometimes inaccurate for fully describing foreground information, especially when salient object structures are complex and similar to background layout aspects. In view of this, the following foreground query-based significance estimation, ranking function, is proposed
It can be calculated directly from equation (1) and treated as a foreground significance estimate as follows:
S step2 (i)＝f(i),i＝1,...,n, (3)
wherein S is step2 (i) For a quadratic ranking foreground significance estimate, f (i) is the ranking function estimate for the i superpixels.
While a popular regularized random walk algorithm is utilized to generate a pixel-level saliency map from saliency estimates for the background and foreground based superpixels.
The embodiment of the invention provides a fitting constraint based on a random walk model:
where Dir is the Dirichlet integral and Y is a one-by-oneAn indication vector of the pixel inherits the step S step2 The value of (c). The regularized random walk ordering is computed on a pixel level, so p k And Y are both N × 1 vectors, L is an N × N matrix, where N is the total number of pixels in the image, T is the transpose of the matrix, k is 1 or 2, where k =1 corresponds to the background label and k =2 corresponds to the foreground label.
Step 202: carrying out binarization operation on the saliency map by adopting an Ostu threshold segmentation algorithm based on a genetic algorithm;
in the specific implementation, after the saliency map is obtained, an Ostu threshold segmentation algorithm based on a genetic algorithm is adopted to carry out binarization operation on the saliency map, wherein white represents a salient region part and a non-salient region corresponding to black is represented by 0.
The traditional Ostu algorithm processes the gray information of an image, and the spatial information between a pixel and a neighborhood is not processed, so that the image is easy to cause segmentation errors when being interfered by noise or other external factors, and the calculation step is complicated. The genetic algorithm has the characteristics of global search and parallelism in the operation process, intelligent preferential treatment can be carried out on the threshold, and the processing efficiency of image segmentation is improved to a great extent. Therefore, the embodiment of the invention combines the genetic algorithm to practically improve the image segmentation efficiency by seeking the optimal Ostu threshold, eliminates noise by using morphological operation and optimizes the segmentation result.
Step 203: estimating a latent image X and a fuzzy kernel K from a fuzzy image B, optimizing an image deblurring model based on a MAP (maximum a posteriori) frame, and introducing an image segmentation item l i Dividing the image into different layers, and performing fuzzy kernel estimation and latent image X recovery on each layer;
the above proposed optimized image deblurring model is:
wherein m represents the number of division pattern layers, l i A binary image representing the ith layer and havingSame size, K i Is a fuzzy core corresponding to the ith layer and simultaneously satisfies
p is the probability sign, p (X) is the prior probability of the latent image X, p (K) is the prior probability of the blur kernel K, p (l) i |K i X) is an introduction of a partition term l i Derived prior probability, p (K) i ) Is the prior probability of the fuzzy core corresponding to the ith layer, p (B | K, X) is the probability of estimating the latent image X and the fuzzy core K from the fuzzy image B, and p (B | l) i ,K i And X) is the likelihood probability.
The key of the optimized image deblurring model is to solve a latent image X and a blur kernel K from the image deblurring model.
Step 204: iterative solution of potentially sharp images X and blur kernel K by alternating minimization formulation i And an effective iterative weighted least square (IRLS) algorithm is introduced to carry out optimization solution on the image deblurring model;
based on the above discussion, embodiments of the present invention entail defining a clear latent image X and a blur kernel K i A priori p (X) and p (K) i ). I.e. using a sparse image gradient prior to define the prior p (X) of the sharp latent image X, and using a Laplace prior to define the blur kernel K i A priori p (K) i )：
Wherein,
and
respectively representing differential operators in x and y directions; λ and γ are weight parameters; z X And Z K Is a regularization term, X u Is pixel spaceClear latent image of the intermediate position u, K iu And (4) a fuzzy kernel corresponding to the spatial position u of the pixel of the ith layer, wherein u is the spatial position of one pixel.
And estimating the intermediate latent image X by using the fuzzy kernel K of the last iteration:
wherein l iu Is a binary image corresponding to the pixel space position u of the ith image layer, X u A sharp latent image at pixel spatial position u.
Solving the following formula by adopting an IRLS algorithm:
wherein, ω is du ＝|(x*K i -B) u | -1 ,
t denotes the iteration index, u denotes the spatial position of each pixel, l iu And obtaining a binary image corresponding to the spatial position u of the ith layer pixel.
Given the intermediate latent image, the blur kernel is estimated. Since better results can be obtained with kernel estimation based on image gradients, the blur kernel K is estimated by replacing the image intensities with image derivatives in the data fitting terms and removing small gradient values i Estimated as:
wherein,
is a gradient operator.
Similar to the latent image X solution, the IRLS algorithm is used to solve the following equation:
wherein,
ω u ＝|K iu | -1 。
step 205: when motion blur is serious, an improved adaptive guided filtering algorithm is adopted to further enhance the image, improve the definition of the image and enrich the detail information of the image.
A guided filter is a filter that enhances image details. The filter carries out filtering processing on an input image through a guide image, the guide image in the filtering process is marked as G, the input image is marked as I, a filtering output image is marked as Q, and a window omega with a pixel r as a center is assumed to be arranged in r A local linear model exists between the intermediate guide image G and the output image Q as follows:
wherein, a r And b r Is window omega r Linear coefficient of (2).
Firstly, in order to further enhance the details of the image and sufficiently obtain the edge features of the guide image, the embodiment of the invention adopts a weighted least square filter WLS to construct the guide image of the image, as follows:
wherein, WLS (G) v Guide image, G, constructed for a weighted least squares filter WLS v For a guide image corresponding to a spatial position v of an image pixel, I v The index τ determines the input image for the spatial position v of the image pixel, v representing the spatial position of the image pixel (x, y)The sensitivity of the gradient change of the input image I at the pixel point v (x, y), eta is a smoothing term parameter,
and
first order partial derivatives of G in the x and y directions, respectively, indicating the degree of steepness of the image; tau is x,v And τ y , v The weighting coefficients are refined, the edges of the image I can be determined, and the edges with different degrees of steepness have different weighting coefficients.
Second, the guided image filtering, while having good edge preserving properties, is susceptible to halo artifacts near the edges. The average of the local variance of all pixels is therefore incorporated into the cost function of the guided filtering to accurately preserve the edges. The average of the local variance of all pixels is defined as:
where N is the number of pixels of the guide image;
is the average of the standard deviation of the sample,
is to guide the image in the window omega r The local variance in (c). The minimum solution to the cost function in the window is therefore:
where ε is a regularization term smoothing parameter used to prevent a r Too large.
Finally, the noise in the background tends to be amplified due to the amplification factor of detail layer J. In general, the amplification factor of the detail layer is set to a fixed value, but noise is amplified while the image is enhanced, so that an adaptive amplification factor β is introduced into the detail layer to suppress noise and improve detail. Multiplying the detail layer by a magnification factor β:
J'＝β·J＝β·(G-Q) (15)
where J' is the enhanced detail layer. When the value of β is small, detail will be suppressed. On the other hand, when the value of β is large, noise is amplified. Therefore, while noise suppression can enhance details, β is set as:
the final output image is therefore: f = Q + J',
is window omega r The average of the linear coefficients a in (1).
Based on the same inventive concept, as an implementation of the above method, referring to fig. 9, an embodiment of the present invention further provides a local motion deblurring apparatus based on visual saliency, including:
the detection and marking module 1 is used for detecting a saliency map of the fuzzy area by adopting popular regularization random walk saliency, and using a detection result to mark the fuzzy area in the image;
the first acquisition module 2 is used for performing binarization operation on the saliency map by adopting an Otsu method based on heredity, segmenting the fuzzy foreground and the clear background into different map layers, and acquiring a foreground and background segmentation binarization image;
the first optimization module 3 is used for substituting the obtained binary image into an image deblurring model based on an MAP framework for optimization, and iteratively estimating an initial fuzzy kernel and a potential image;
the second acquisition module 4 is used for listing the initial fuzzy kernel and the prior of the potential image and solving the initial fuzzy kernel and the potential image based on iterative weighted least square;
and the second optimization module 5 is used for further optimizing the solution result by adopting improved adaptive guided filtering and maintaining the edge information of the image.
In a specific implementation, referring to fig. 10, the detection and labeling module 1 includes:
the detection unit 11 is used for optimizing the influence generated by the boundary according to the means of positioning and eliminating the error boundary and acquiring a background significance value; optimizing a simple background query according to the saliency estimate of the foreground query;
the fitting constraint unit 12 is configured to propose fitting constraints based on the random walk model, and inherit the saliency value of the foreground query to obtain a final saliency map;
a marking unit 13 for using the detection result for marking the blurred region in the image.
In one implementation, the first optimization module 2 includes:
a segmentation and restoration unit for introducing an image segmentation term l i The image is divided into different layers, and fuzzy kernel estimation and recovery of a clear latent image are carried out on each layer.
It should be noted that the device description in the above embodiments corresponds to the description of the method embodiments, and the embodiments of the present invention are not described herein again.
The execution main bodies of the modules and units can be computers, single-chip microcomputers, microcontrollers and other devices with calculation functions, and in specific implementation, the execution main bodies are not limited in the embodiment of the invention and are selected according to requirements in practical application.
Based on the same inventive concept, an embodiment of the present invention further provides a local motion deblurring apparatus based on visual saliency, referring to fig. 11, the apparatus including: a processor 6 and a memory 7, the memory 7 having stored therein program instructions, the processor 6 calling upon the program instructions stored in the memory 7 to cause the apparatus to perform the following method steps in an embodiment:
detecting a saliency map of the fuzzy region by adopting the popular regularization random walk saliency, and using a detection result for marking the fuzzy region in the image;
performing binarization operation on the saliency map by adopting an Otsu method based on heredity, segmenting a fuzzy foreground and a clear background into different map layers, and acquiring a foreground and background segmentation binarization image;
substituting the obtained binary image into an image deblurring model based on an MAP framework for optimization, and iteratively estimating an initial fuzzy core and a potential image;
listing the priors of the initial fuzzy kernel and the potential image, and solving the initial fuzzy kernel and the potential image based on iterative weighted least square;
and further optimizing a solution result by adopting improved self-adaptive guide filtering, and keeping the edge information of the image.
It should be noted that the device description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the present invention are not described herein again.
The execution main bodies of the processor and the memory can be devices with calculation functions such as a computer, a single chip microcomputer and a microcontroller, and in the specific implementation, the execution main bodies are not limited in the embodiment of the invention and are selected according to the requirements in practical application.
The data signal is transmitted between the memory 7 and the processor 6 through the bus 8, which is not described in detail in the embodiment of the present invention.
Based on the same inventive concept, embodiments of the present invention further provide a computer-readable storage medium, where the storage medium includes a stored program, and when the program runs, the apparatus on which the storage medium is located is controlled to execute the method steps in the foregoing embodiments.
The computer readable storage medium includes, but is not limited to, flash memory, hard disk, solid state disk, and the like.
It should be noted that, descriptions of the readable storage medium in the above embodiments correspond to descriptions of the method in the embodiments, and details of the embodiments of the present invention are not repeated herein.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the invention are brought about in whole or in part when the computer program instructions are loaded and executed on a computer.
The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on or transmitted over a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium or a semiconductor medium, etc.
In the embodiment of the present invention, except for the specific description of the model of each device, the model of other devices is not limited as long as the device can perform the above functions.
Those skilled in the art will appreciate that the drawings are only schematic illustrations of preferred embodiments, and the above-described embodiments of the present invention are merely provided for description and do not represent the merits of the embodiments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (6)
1. A method for local motion deblurring based on visual saliency, the method comprising:
detecting a saliency map of the fuzzy region by adopting the popular regularization random walk saliency, and using a detection result for marking the fuzzy region in the image;
performing binarization operation on the saliency map by adopting an Otsu method based on heredity, segmenting a fuzzy foreground and a clear background into different map layers, and acquiring a foreground and background segmentation binarization image;
substituting the obtained binary image into an image deblurring model based on an MAP framework for optimization, and iteratively estimating an initial fuzzy kernel and a potential image;
listing the priors of the initial fuzzy core and the potential image, and solving the initial fuzzy core and the potential image based on iterative weighted least square;
further optimizing the solution result by adopting improved self-adaptive guided filtering, and keeping the edge information of the image;
the saliency map for detecting the fuzzy area by adopting the popular regularization random walk saliency is specifically as follows:
optimizing the influence generated by the boundary according to a means of positioning and eliminating an error boundary to obtain a background significance value;
optimizing a simple background query according to the saliency estimate of the foreground query;
based on a random walk model, proposing fitting constraints for inheriting a significance value of foreground query to obtain a final significance map;
background significance value:
where l is the total number of pixels in the target area, n is the total number of super pixels, S l (i) Significance value based on the foreground, f l * (i) For ranking function values of popularity, S step1 (i) The background significance estimated value is ranked once;
significance estimation of foreground query:
S step2 (i)＝f(i),i＝1,...,n,
wherein S is step2 (i) F (i) is the estimated value of the ranking function of the i superpixels;
the fitting constraint is as follows:
wherein Dir is Dirichlet integral, Y is a pixel-by-pixel indication vector, and the step S is inherited step2 Value of (a), p k And Y are both N × 1 vectors, L is an N × N matrix, NIs the total number of pixels in the image, T is the transpose of the matrix, k is 1 or 2;
the bringing of the obtained binary image into an image deblurring model based on a MAP framework for optimization specifically comprises the following steps:
introducing an image segmentation term l i Dividing the image into different layers, and performing fuzzy kernel estimation and clear latent image restoration on each layer;
the improved adaptive guided filtering is specifically as follows:
constructing a guide image by using a weighted least squares filter WLS:
wherein, WLS (G) v Guide image, G, constructed for a weighted least squares filter WLS v For a guide image corresponding to a spatial position v of an image pixel, I v The input image corresponding to the spatial position v of the image pixel, v represents the spatial position of the image pixel (x, y), the index tau determines the sensitivity of the gradient change of the input image I at the pixel v (x, y), eta is a smoothing term parameter,
and
first order partial derivatives of G in the x and y directions, respectively, indicating the degree of steepness of the image; tau is x,v And τ y,v A weight coefficient for refinement;
according to a local linear model between the guide image and the output image, the average value of local variances of all pixels is incorporated into a cost function of the guide filtering, and an adaptive amplification factor beta is introduced to suppress noise.
2. An apparatus for local motion deblurring based on visual saliency, characterized in that the apparatus is configured to perform the local motion deblurring method of claim 1, the apparatus comprising:
the detection and marking module is used for detecting a saliency map of the fuzzy area by adopting the popular regularization random walk saliency and using a detection result to mark the fuzzy area in the image;
the first acquisition module is used for performing binarization operation on the saliency map by adopting an Otsu method based on heredity, segmenting a fuzzy foreground and a clear background into different map layers and acquiring a foreground and background segmentation binarization image;
the first optimization module is used for substituting the obtained binary image into an image deblurring model based on an MAP framework for optimization, and iteratively estimating an initial fuzzy kernel and a potential image;
the second acquisition module is used for listing the initial fuzzy kernel and the prior of the potential image and solving the initial fuzzy kernel and the potential image based on iterative weighted least square;
and the second optimization module is used for further optimizing the solution result by adopting improved self-adaptive guide filtering and maintaining the edge information of the image.
3. The apparatus of claim 2, wherein the detection and labeling module comprises:
the detection unit is used for optimizing the influence generated by the boundary according to the means of positioning and eliminating the error boundary and acquiring a background significance value; optimizing a simple background query according to the saliency estimate of the foreground query;
and the fitting constraint unit is used for proposing fitting constraint based on the random walk model, and is used for inheriting the significance value of the foreground query to obtain a final significance map.
4. The apparatus of claim 2, wherein the first optimization module comprises:
a segmentation and restoration unit for introducing an image segmentation term l i Dividing the image into different layers, performing fuzzy kernel estimation and clear latent image on each layerRecovery of (3).
5. An apparatus for local motion deblurring based on visual saliency, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling upon the program instructions stored in the memory to cause the apparatus to perform the method steps of claim 1.
6. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method steps of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011084727.4A CN112215773B (en) | 2020-10-12 | 2020-10-12 | Local motion deblurring method and device based on visual saliency and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011084727.4A CN112215773B (en) | 2020-10-12 | 2020-10-12 | Local motion deblurring method and device based on visual saliency and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112215773A CN112215773A (en) | 2021-01-12 |
| CN112215773B true CN112215773B (en) | 2023-02-17 |
Family
ID=74054521
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011084727.4A Active CN112215773B (en) | 2020-10-12 | 2020-10-12 | Local motion deblurring method and device based on visual saliency and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112215773B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034407B (en) * | 2021-04-27 | 2022-07-05 | 深圳市慧鲤科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN113177890B (en) * | 2021-04-27 | 2023-04-18 | 深圳市慧鲤科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN113284080A (en) * | 2021-06-17 | 2021-08-20 | Oppo广东移动通信有限公司 | Image processing method and device, electronic device and storage medium |
| CN113313655B (en) * | 2021-06-28 | 2022-09-23 | 合肥工业大学 | Blind image deblurring method based on saliency mapping and gradient cepstrum technology |
| CN113706572B (en) * | 2021-08-25 | 2023-10-20 | 北京大学 | End-to-end panoramic image segmentation method based on query vector |
| CN114037710A (en) * | 2021-10-13 | 2022-02-11 | 北京百度网讯科技有限公司 | Image segmentation method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514582A (en) * | 2012-06-27 | 2014-01-15 | 郑州大学 | Visual saliency-based image deblurring method |
| CN104766287A (en) * | 2015-05-08 | 2015-07-08 | 哈尔滨工业大学 | Blurred image blind restoration method based on significance detection |
| CN108154488A (en) * | 2017-12-27 | 2018-06-12 | 西北工业大学 | A kind of image motion ambiguity removal method based on specific image block analysis |
| CN108564528A (en) * | 2018-04-17 | 2018-09-21 | 福州大学 | A kind of portrait photo automatic background weakening method based on conspicuousness detection |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8750645B2 (en) * | 2009-12-10 | 2014-06-10 | Microsoft Corporation | Generating a composite image from video frames |
| US8928772B2 (en) * | 2012-09-21 | 2015-01-06 | Eastman Kodak Company | Controlling the sharpness of a digital image |
| US9607391B2 (en) * | 2015-08-04 | 2017-03-28 | Adobe Systems Incorporated | Image object segmentation using examples |
-
2020
- 2020-10-12 CN CN202011084727.4A patent/CN112215773B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514582A (en) * | 2012-06-27 | 2014-01-15 | 郑州大学 | Visual saliency-based image deblurring method |
| CN104766287A (en) * | 2015-05-08 | 2015-07-08 | 哈尔滨工业大学 | Blurred image blind restoration method based on significance detection |
| CN108154488A (en) * | 2017-12-27 | 2018-06-12 | 西北工业大学 | A kind of image motion ambiguity removal method based on specific image block analysis |
| CN108564528A (en) * | 2018-04-17 | 2018-09-21 | 福州大学 | A kind of portrait photo automatic background weakening method based on conspicuousness detection |
Non-Patent Citations (3)
| Title |
|---|
| "Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation";Zeev Farbman et al;《ACM Transactions on Graphicds(TOG)》;20080831;第3小节 * |
| "Reversion Correction and Regularized Random Walk Ranking for Saliency Detection";Yuchen Yuan et a;《IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 3, MARCH 2018》;20171012;第1312页左栏第14行至右栏第3行，第III小节 * |
| "Soft-Segmentation Guided Object Motion Deblurring";Jinshan Pan et al;《2016 IEEE Conference on Computer Vision and Pattern Recognition》;20161216;第3小节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112215773A (en) | 2021-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112215773B (en) | Local motion deblurring method and device based on visual saliency and storage medium | |
| Hu et al. | Good regions to deblur | |
| Couzinie-Devy et al. | Learning to estimate and remove non-uniform image blur | |
| Li et al. | SPM-BP: Sped-up PatchMatch belief propagation for continuous MRFs | |
| Huang et al. | Joint blur kernel estimation and CNN for blind image restoration | |
| CN111047544B (en) | Saturated image deblurring method based on nonlinear degradation model | |
| CN109685830B (en) | Target tracking method, device and equipment and computer storage medium | |
| Ma et al. | Shape-from-focus reconstruction using nonlocal matting Laplacian prior followed by MRF-based refinement | |
| CN113793285B (en) | Ultrafast restoration method and system for pneumatic optical effect target twin image | |
| WO2023115801A1 (en) | Point-by-point correction and restoration method and system for large field-of-view degraded image having aero-optical effect | |
| Patil et al. | End-to-end recurrent generative adversarial network for traffic and surveillance applications | |
| WO2017135120A1 (en) | Computationally efficient frame rate conversion system | |
| Zhang et al. | Improved image deblurring based on salient-region segmentation | |
| CN116486288A (en) | Aerial target counting and detecting method based on lightweight density estimation network | |
| CN107451595A (en) | Infrared image salient region detection method based on hybrid algorithm | |
| WO2022233252A1 (en) | Image processing method and apparatus, and computer device and storage medium | |
| Wang et al. | PFDN: Pyramid feature decoupling network for single image deraining | |
| Chen et al. | Visual depth guided image rain streaks removal via sparse coding | |
| Wang et al. | An efficient remote sensing image denoising method in extended discrete shearlet domain | |
| Kryjak et al. | Real-time implementation of foreground object detection from a moving camera using the vibe algorithm | |
| Wang et al. | RT-Deblur: Real-time image deblurring for object detection | |
| Zheng et al. | Memory-efficient multi-scale residual dense network for single image rain removal | |
| Zhao et al. | Natural image deblurring based on ringing artifacts removal via knowledge-driven gradient distribution priors | |
| Su et al. | Restoration of turbulence-degraded images using the modified convolutional neural network | |
| Zhou et al. | Single image dehazing based on weighted variational regularized model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |