CN115578289A - Defocused image deblurring method based on boundary neighborhood gradient difference - Google Patents

Abstract

The invention discloses a method for deblurring a defocused image based on a boundary neighborhood gradient difference, and belongs to the technical field of computer vision. In view of the fact that the existing defocused image deblurring method cannot accurately obtain the blur amount of the boundary position of the defocused image for static scenes with multiple depth layers, the present invention makes full use of the relationship between the boundary neighborhood gradient difference and the blur amount to accurately obtain the defocused image The amount of blurring at the boundary position, so as to solve the problem of boundary ringing artifacts in the deblurring result; the ability of the non-blind deconvolution algorithm to retain image detail information is not strong enough to cause the loss of detail information in the deblurring result. The present invention designs a non-blind convolution algorithm by combining the discrete fuzzy amount selection strategy and sparse prior to strengthen the ability of the non-blind deconvolution algorithm to retain image detail information, and solves the problem of loss of detail information in the deblurring results. It can be used for deblurring of defocused images in static scenes with multiple depth layers.

Description

A Defocused Image Deblurring Method Based on Boundary Neighborhood Gradient Difference

technical field

The invention relates to a method for deblurring a defocused image based on a boundary neighborhood gradient difference, and belongs to the field of computer vision.

Background technique

In real life, defocused images can be seen everywhere. The so-called defocus means that the image becomes unclear and blurred, and the blurred image will make people unable to see the objects in the picture and lose detailed information, which limits the accuracy of the image in precision instruments. Further applications in areas such as flaw detection and augmented reality.

In order to improve the efficiency of using the defocused image, it is necessary to remove the defocus in the defocused image to sharpen the image. The existing traditional defocused image deblurring method obtains the blur amount by estimating the blur degree of the pixel, and then uses the point spread function represented by the Gaussian function and the blur amount to calculate the blur kernel. Finally, the blur kernel and the defocused image are input into the non-blind deconvolution algorithm to perform the deblurring operation.

For static scenes with multiple depth layers, existing defocused image deblurring methods cannot accurately obtain the amount of blur at the boundary of the defocused image, and the amount of blur represents the degree of blurring of pixels. Inaccurate estimation of the amount of defocus blur at the boundary will lead to boundary ringing artifacts in the deblurring results. Boundary ringing artifacts refer to oscillations at places where the gray level changes sharply. In addition, the prior knowledge of the non-blind deconvolution algorithm in the defocused image deblurring method is not strong enough. The prior knowledge refers to some known features and characteristics in the image, which can be used to constrain the non-blind deconvolution algorithm so that The deblurring result can continuously approach the real clear image. The lack of prior knowledge will lead to the non-blind deconvolution algorithm's ability to recover image detail information is not strong enough, and the problem of detail information loss will appear in the final deblurring result.

In conclusion, both the existence of boundary ringing artifacts and the lack of prior knowledge will affect the deblurring process of defocused images, and finally lead to poor deblurring effect and insufficient image clarity.

Contents of the invention

In order to solve the problems of poor deblurring effect and insufficient image definition in the current defocused image deblurring method in the face of static scenes with multiple depth layers, the present invention provides a defocused image deblurring method. The defocused image deblurring method includes:

Step 1: Obtain the boundary of the defocused image, and obtain the gradient difference of the boundary neighborhood at the boundary position;

Step 2: using the gradient difference value of the boundary neighborhood obtained in the step 1 to obtain the blur amount at the boundary position, thereby obtaining a sparse fuzzy map;

Step 3: Interpolating the sparse fuzzy map obtained in step 2 to obtain an interpolated fuzzy map;

Step 4: Use the interpolation blur map obtained in step 3 to perform blur detection on the defocused image and calculate a blur ratio, and when the blur ratio is greater than a preset blur ratio threshold, perform blur detection on the defocused image deblurring;

Step 5: For the image that needs to be deblurred in step 4, use the interpolation blur map in step 3 to obtain a blur kernel, and then perform a deblurring operation in combination with a non-blind deconvolution algorithm.

Optionally, the step 1 includes:

Use the scale-consistent boundary detection algorithm to obtain the boundary of the defocused image, and after obtaining the boundary position, calculate the gradient in the neighborhood centered on each boundary position in the original image, and then calculate the maximum gradient and minimum gradient in the neighborhood Difference gets the gradient difference at the boundary position.

Optionally, the step 2 includes:

Step 2.1: Obtain the relationship between the gradient difference of the boundary neighborhood and the blur amount:

Model the defocusing process of the image, as shown in formula (1):

表示卷积操作，所述模糊核对所述清晰图像做卷积操作再加上所述噪声就会得到所述散焦图像；Wherein, I represents the defocused image, k represents the blur kernel, x represents the clear image, n represents the noise,

Represents a convolution operation, the blur kernel performs a convolution operation on the clear image and adds the noise to obtain the defocused image;

The blur kernel is obtained by a point spread function and a blur quantity, and the point spread function is represented by a Gaussian function, as shown in formula (2):

Among them, σ represents the amount of blur, the amount of blur represents the degree of blur of the pixel, and (x, y) represents the coordinates of the pixel;

The boundary l(x, y) of the clear image is modeled as shown in formula (3):

l(x,y)=a*u(x,y)+b (3) where a is the amplitude, b is the offset, u(x,y) is the step function;

Define the boundary neighborhood gradient difference as shown in formula (4):

表示边界邻域中的梯度，也就是对边界求导，求导结果如式(5)所示：Among them, GD(x, y) is the gradient difference value of the boundary neighborhood, (x, y)' represents the neighborhood centered on (x, y), and J(x, y) represents the boundary of the defocused image ,

Represents the gradient in the boundary neighborhood, that is, deriving the boundary, and the derivation result is shown in formula (5):

The boundary position is at (x,y)=(0,0), so the gradient difference of the boundary neighborhood is shown in formula (6):

According to formula (6), the relationship between the boundary neighborhood gradient difference and the fuzzy amount is derived, as shown in formula (7):

Step 2.2: Obtain the blur amount at the boundary according to the relationship between the boundary neighborhood gradient difference and the blur amount and the boundary neighborhood gradient difference obtained in step 1, thereby obtaining a sparse fuzzy map.

Optionally, the process of calculating the blur ratio in the step 4 includes:

Setting a threshold of blur amount, judging whether the blur amount of each pixel in the interpolation blur map is greater than the threshold of blur amount, when the blur amount of a pixel is greater than the threshold, the pixel is set as blurred, Otherwise it is non-ambiguous;

Divide the interpolation fuzzy image into fuzzy areas and non-fuzzy areas according to the judgment results, and calculate the ratio of the number of pixels in the fuzzy area to the number of pixels in the entire image, that is, the blur ratio.

Optionally, the step 5 includes:

Step 5.1: Perform discrete fuzzy quantity selection on the interpolated fuzzy graph obtained in step 3, and obtain n fuzzy quantities σ ₁ , σ ₂ ...σ _n by step q; use these n fuzzy quantities to obtain n by using the point spread function A fuzzy kernel, the point spread function is represented by a Gaussian function, and the standard deviation in the Gaussian function represents the amount of blurring;

Step 5.2: Use the non-blind deconvolution algorithm based on sparse prior and n blur kernels to carry out n non-blind deconvolution operations on the defocused image to obtain n deblurred images; each blur kernel corresponds to a blur amount , when the defocused image is deblurred with the nth blurring kernel, firstly obtain the position of the blurring amount larger than the nth blurring amount in the blur map, and extract the pixels corresponding to these positions in the deblurring result to obtain nth deblurred result;

The above operations are performed n times, and finally n deblurred images are obtained;

Step 5.3: Add the pixel values of the corresponding positions of the n deblurred images in step 5.2, and finally divide each pixel value by 255 to obtain an all-focus image, that is, a completely clear image.

Optionally, the step 3 uses a KNN matting interpolation algorithm to interpolate the sparse fuzzy map.

Optionally, the size of the neighborhood is 11×11.

Optionally, the image used for processing is at least one of the following: blurred face image; blurred person image; blurred scene image; blurred vehicle image; blurred animal image; and blurred plant image.

A second object of the present invention is to provide an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores information executable by the at least one processor. instructions, the instructions are executed by the at least one processor, so that the at least one processor can execute the above-mentioned method for deblurring a defocused image.

A third object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions are used to make the computer execute the above-mentioned method for deblurring a defocused image.

The beneficial effects of the present invention are:

The defocused image deblurring method of the present invention makes full use of the relationship between the boundary neighborhood gradient difference and the blurring amount to accurately obtain the blurring amount at the boundary position of the defocused image, thereby solving the problem of boundary ringing artifacts in the deblurring result . Aiming at the problem that the ability of the non-blind deconvolution algorithm to retain image detail information is not strong enough to cause the loss of detail information in the deblurring result, the present invention combines the discrete fuzzy amount selection strategy and sparse prior to design a non-blind deconvolution algorithm to Strengthen the ability of the non-blind deconvolution algorithm to retain image detail information, and solve the problem of loss of detail information in the deblurring result.

Compared with the existing methods for deblurring defocused images, the present invention can not only effectively solve the problem of boundary ringing artifacts, but also avoid loss of image detail information, and effectively improve the clarity of defocused images after deblurring.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a defocused image input in Embodiment 2 of the present invention.

FIG. 2 shows a boundary map obtained by performing boundary detection on FIG. 1 by using a scale-consistent boundary detection algorithm according to Embodiment 2 of the present invention.

FIG. 3 shows a boundary neighborhood gradient difference map calculated for the boundary in FIG. 1 using the boundary position in FIG. 2 according to Embodiment 2 of the present invention.

FIG. 4 shows the blur map at the boundary position obtained by using the boundary neighborhood gradient difference in FIG. 3 according to Embodiment 2 of the present invention.

FIG. 5 shows a map of the blur amount at each pixel position obtained by using the blur amount at the boundary position in FIG. 4 and the KNN matting interpolation algorithm according to Embodiment 2 of the present invention.

FIG. 6 shows a final deblurring result diagram obtained according to the obtained blur amount and the non-blind deconvolution algorithm according to Embodiment 2 of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the implementation manner of the present invention will be further described in detail below in conjunction with the accompanying drawings.

Embodiment one:

This embodiment provides a method for deblurring a defocused image, including:

Step 1: Obtain the boundary of the defocused image, and obtain the gradient difference of the boundary neighborhood at the boundary position;

Step 2: using the gradient difference value of the boundary neighborhood obtained in the step 1 to obtain the blur amount at the boundary position, thereby obtaining a sparse fuzzy map;

Step 3: Interpolating the sparse fuzzy map obtained in step 2 to obtain an interpolated fuzzy map;

Step 4: Use the interpolation blur map obtained in step 3 to perform blur detection on the defocused image and calculate a blur ratio, and when the blur ratio is greater than a preset blur ratio threshold, perform blur detection on the defocused image deblurring;

Step 5: For the image that needs to be deblurred in step 4, use the interpolation blur map in step 3 to obtain a blur kernel, and then perform a deblurring operation in combination with a non-blind deconvolution algorithm.

Embodiment two:

The present embodiment provides a defocused image deblurring method based on boundary neighborhood gradient difference, comprising the following steps:

Step 1: Use the scale-consistent boundary detection algorithm to obtain the boundary of the defocused image. The scale-consistent boundary detection algorithm comes from the article Edge-Based Defocus Blur Estimation With Adaptive Scale Selection. After obtaining the boundary position, find each boundary position in the original image is the gradient in the 11x11 neighborhood of the center, and then make a difference between the maximum gradient and the minimum gradient in the neighborhood to obtain the gradient difference at the boundary position.

Step 2: Use the boundary neighborhood gradient difference obtained in step 1 to obtain the blur amount at the boundary position, so as to obtain a sparse fuzzy map. The specific steps are as follows:

Step 2.1: Obtain the relationship between the boundary neighborhood gradient difference and the blur amount.

Model the defocusing process of the image, as shown in formula (1):

表示卷积操作，模糊核对清晰图像做卷积操作再加上噪声就会得到散焦图像。模糊核由点扩散函数和模糊量获得，点扩散函数由高斯函数表示，如式(2)所示：Among them, I represents the defocused image, k represents the blur kernel, x represents the clear image, n represents the noise,

Represents a convolution operation. The blur kernel performs a convolution operation on a clear image and adds noise to obtain a defocused image. The blur kernel is obtained by the point spread function and the blur quantity, and the point spread function is represented by a Gaussian function, as shown in formula (2):

Among them, σ represents the amount of blur, the amount of blur represents the degree of blur of the pixel, and (x, y) represents the coordinates of the pixel.

The boundary l(x,y) of the clear image is modeled as shown in formula (3):

l(x,y)=a*u(x,y)+b (3)

where a is the amplitude, b is the offset, and u(x,y) is the step function.

Define the boundary neighborhood gradient difference as shown in formula (4):

表示边界邻域中的梯度，也就是对边界求导。求导结果如式(5)所示：It is found through experiments that the gradient minimum in the boundary neighborhood is very small and close to zero, so the minimum value part is omitted in the formula derivation. Among them, GD(x, y) is the gradient difference value of the boundary neighborhood, (x, y)' represents the neighborhood centered on (x, y), J(x, y) represents the defocused image boundary,

Represents the gradient in the boundary neighborhood, that is, deriving the boundary. The derivation result is shown in formula (5):

The boundary position is at (x,y)=(0,0), so the gradient difference of the boundary neighborhood is shown in formula (6):

According to formula (6), the relationship between the boundary neighborhood gradient difference and the fuzzy amount can be deduced, as shown in formula (7):

Step 2.2: According to the relationship between the boundary neighborhood gradient difference and the fuzzy amount and the boundary neighborhood gradient difference obtained in step 1, obtain the fuzzy amount at the boundary, thereby obtaining a sparse fuzzy map.

Step 3: The sparse blur map only includes the blur amount at the boundary position. In order to obtain the blur amount of the remaining pixels, this embodiment uses the KNN matting interpolation algorithm to interpolate the sparse blur map obtained in step 2, thereby obtaining a blur value that includes all pixels. Amount of interpolated blur maps.

Step 4: Based on the interpolation blur map obtained in step 3 and the set blur threshold, divide the defocused image into blurred areas and non-blurred areas. When the blurring amount of the pixel is greater than the blurring threshold, the pixel is set as blurred, otherwise it is not blurred. Perform the above operations for each pixel, so that the image can be divided into blurred areas and non-blurred areas. Calculate the ratio of the number of pixels in the blurred area to the number of pixels in the entire image, that is, the blur ratio. When the blur ratio is greater than a certain value, the image is considered to be blurred and needs to be deblurred, otherwise it is not needed.

Step 5: For the image that needs to be deblurred in step 4, use the interpolated blur map in step 3 to obtain the blur kernel, and then perform the deblurring operation in combination with the non-blind deconvolution algorithm. Specific steps are as follows:

Step 5.1: Perform discrete blurring selection on the interpolation blur map obtained in step 3 to obtain n blurring quantities. The discrete blurring selection strategy comes from the article Spatially-Varying Out-Of-Focus Image DeblurringWith L1-2Optimization And A Guided Blur Map. The n blur quantities are used to obtain n blur kernels by using the point spread function. The point spread function in this embodiment is represented by a Gaussian function, and the standard deviation in the Gaussian function represents the blur quantity.

Step 5.2: Use the non-blind deconvolution algorithm based on sparse prior and n blur kernels to perform n non-blind deconvolution operations on the defocused image to obtain n deblurred images. The non-blind deconvolution algorithm comes from the article Image and Depth from a Conventional Camera with a Coded Aperture. Each blur kernel corresponds to a blur amount. When the nth blur kernel is used to deblur the defocused image, first obtain the pixel position of the blur amount larger than the nth blur amount in the blur map, and deblur the nth The pixel values at these pixel positions in the image are extracted, and the remaining pixels are discarded to obtain the nth deblurred image. The deblurred image only contains the values of some pixel positions, and the values of other pixel positions are 0. The blur map is a grayscale image, and the value of each pixel position is the amount of blur. The blur map corresponds to the defocused image, and each pixel position in the defocused image corresponds to the pixel position in the blur map. The blur amount of a certain pixel position in the blur map is the blur amount of the corresponding pixel position in the defocused image. The above operations are performed n times, and finally n deblurred images are obtained.

Step 5.3: Add the pixel values of the n deblurred images in step 5.2, and finally divide each pixel value by 255 to obtain an all-focus image, that is, a completely clear image.

The environment for realizing the system in this embodiment is as follows:

Table 1: System hardware configuration table

Table 2: System software configuration table

software Related Information operating system Windows 10 64 bit Matlab Matlab R2021a

The experimental results are shown in the accompanying drawings.

Figure 1: The input defocused image, that is, the image that needs to be processed to make it sharp. It can be seen in the figure that the degree of blur is the highest on the left, and the degree of blur decreases from left to right.

Figure 2: The boundary map obtained by using the scale-consistent boundary detection algorithm. It is a binary map, that is, the pixel values are only 0 and 1, 1 represents white, and 0 represents black. The white pixels represent the boundary positions.

Figure 3: The gradient difference map of the boundary neighborhood, which shows the gradient difference of each boundary position. In the method of this embodiment, it is found that the gradient difference of the boundary neighborhood is inversely proportional to the amount of blur, that is, the more blurred the boundary position The smaller the gradient difference, the clearer the boundary position, the larger the gradient difference. Figure 1 also describes that the left side of the input defocused image is the most blurred, and the degree of blurring decreases from left to right. In the boundary neighborhood gradient difference diagram, it can be seen that the gradient difference at the left boundary position is small, and the gradient difference at the right boundary position is relatively large. Thus, the method of this embodiment is also verified. The brighter places in the figure represent larger values, and the darker places have smaller values.

Figure 4: Sparse blur map, which only includes the blur amount at the boundary position. The blur amount indicates the blur degree of the pixel. In this figure, it can be seen that the blur amount at the left boundary position is larger, and the blur amount at the right boundary position Smaller, which also corresponds to the description in Figure 1. The brighter places in the figure represent larger values, and the darker places have smaller values.

Figure 5: Blur map, which represents the blur amount of each pixel in the defocused image. It is a grayscale map, where the brighter the place is, the higher the degree of blurring is, and the darker the place is, the lower the degree of blurring is.

Figure 6: The deblurred image, that is, the image that has become clearer. It can be seen from the figure that compared with Figure 1, the right side of the figure is obviously clearer, and overall there is no boundary ringing artifact and it is completely preserved Image details.

Part of the steps in the embodiments of the present invention can be realized by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disk or a hard disk.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (10)

1. A method of deblurring a defocused image, the method comprising:

step 1: solving a boundary of the defocused image, and solving a gradient difference value of a boundary neighborhood at the position of the boundary;

step 2: obtaining a fuzzy quantity at the boundary position by using the gradient difference value of the boundary neighborhood obtained in the step 1, thereby obtaining a sparse fuzzy graph;

and 3, step 3: interpolating the sparse fuzzy graph obtained in the step 2 to obtain an interpolated fuzzy graph;

and 4, step 4: carrying out fuzzy detection on the defocused image by using the interpolation fuzzy image obtained in the step 3 and calculating a fuzzy ratio, and when the fuzzy ratio is greater than a preset fuzzy ratio threshold value, deblurring the defocused image;

and 5: for the image needing to be deblurred in the step 4, obtaining a blur kernel by using the interpolation blur graph in the step 3, and then performing a deblurring operation by combining a non-blind deconvolution algorithm.

2. The method of deblurring a defocused image as claimed in claim 1, wherein said step 1 comprises:

and solving the boundary of the defocused image by using a scale-consistent boundary detection algorithm, solving the gradient in a neighborhood taking each boundary position as a center in the original image after the boundary position is obtained, and then performing difference on the maximum gradient and the minimum gradient in the neighborhood to obtain the gradient difference at the boundary position.

3. The method of claim 1, wherein the step 2 comprises:

step 2.1: obtaining a relation between the boundary neighborhood gradient difference value and the fuzzy quantity:

modeling the defocusing process of the image, as shown in equation (1):

wherein I represents the defocused image, k represents a blur kernel, x represents a sharp image, n represents noise,

representing convolution operation, wherein the fuzzy core performs convolution operation on the clear image and the noise is added to obtain the defocused image;

the fuzzy kernel is obtained by a point spread function and a fuzzy quantity, the point spread function is represented by a Gaussian function, and the formula (2) is as follows:

wherein σ represents a blurring amount representing a blurring degree of the pixel, and (x, y) represents a pixel coordinate;

the boundary l (x, y) of the sharp image is modeled as shown in formula (3):

l(x,y)＝a*u(x,y)+b (3)

where a is amplitude, b is offset, and u (x, y) is a step function;

defining the boundary neighborhood gradient difference value is shown as formula (4):

wherein GD (x, y) is the boundary neighborhood gradient difference, (x, y)' denotes a neighborhood centered at (x, y), J (x, y) denotes the defocused image boundary,

the gradient in the neighborhood of the boundary is represented, i.e. the boundary is differentiated, and the result of the derivation is shown in equation (5):

the boundary position is (x, y) = (0, 0), so the boundary neighborhood gradient difference is as shown in equation (6):

deriving a relation between the boundary neighborhood gradient difference and the fuzzy quantity according to equation (6), as shown in equation (7):

step 2.2: and (3) solving the fuzzy quantity at the boundary according to the relation between the boundary neighborhood gradient difference and the fuzzy quantity and the boundary neighborhood gradient difference obtained in the step (1), thereby obtaining a sparse fuzzy graph.

4. The method of claim 1, wherein the step 4 of calculating the blur ratio comprises:

setting a fuzzy quantity threshold, judging whether the fuzzy quantity of each pixel point in the interpolation fuzzy graph is greater than the fuzzy quantity threshold, when the fuzzy quantity of the pixel point is greater than the threshold, setting the pixel point as fuzzy, otherwise, setting the pixel point as non-fuzzy;

and dividing the interpolation fuzzy graph into a fuzzy region and a non-fuzzy region according to the judgment result, and calculating the proportion of the number of pixels in the fuzzy region to the number of pixels in the whole image, namely a fuzzy ratio.

5. The method of claim 1, wherein the step 5 comprises:

step 5.1: selecting discrete fuzzy quantity for the interpolation fuzzy graph obtained in the step 3, and obtaining n fuzzy quantities sigma by stepping q ₁ 、σ ₂ …σ _n (ii) a Obtaining n fuzzy kernels by using the n fuzzy quantities and utilizing a point spread function, wherein the point spread function is expressed by a Gaussian function, and a standard deviation in the Gaussian function expresses the fuzzy quantities;

step 5.2, using a non-blind deconvolution algorithm based on sparse prior and n fuzzy cores to perform n times of non-blind deconvolution operations on the defocused image to obtain n deblurred images; each fuzzy core is corresponding to a fuzzy quantity, when the nth fuzzy core is used for deblurring a defocused image, the positions of the fuzzy quantities which are larger than the nth fuzzy quantity in the fuzzy image are firstly obtained, and pixels in deblurring results corresponding to the positions are extracted to obtain the nth deblurring result;

executing the operations for n times to finally obtain n deblurred images;

and 5.3, adding the pixel values of the corresponding positions of the n deblurred images obtained in the step 5.2, and finally dividing each pixel value by 255 to obtain a full-focus image, namely a completely clear image.

6. The method according to claim 1, wherein the step 3 interpolates the sparse blur map by using a KNN matching interpolation algorithm.

7. The method of claim 2, wherein the size of the neighborhood is 11x 11.

8. The defocused image deblurring method of any of claims 1-7, wherein the image for processing is at least one of: blurring a face image; blurring the image of the character; blurring a scene image; blurring the vehicle image; blurring an animal image; and blurring the plant image.

9. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the defocused image deblurring method of any of claims 1-8.

10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the defocused image deblurring method of any one of claims 1-8.

Images