KR102563750B1 - Method and Device of Image Deblurring - Google Patents

Abstract

An image de-blurring method includes obtaining a blurred image to be processed and a DVS event set recorded by a DVS sensor while the blurred image is exposed, and de-blurring the blurred image based on the DVS event set. It includes the steps of performing A method for improving image deblurring effect is disclosed.

Description

Method and device for removing image blurring {Method and Device of Image Deblurring}

Embodiments described below relate to image processing technology, and to a method and apparatus for removing image blurring.

When taking a picture using a camera, the image may be blurred due to various reasons, such as hand shaking. In general, in order to remove or reduce image blurring due to camera movement, a blurring image is obtained by performing a deblurring process on the formed blurring image.

Existing blurring removal methods have the effect of removing blurring only for a specific image because blurring is removed using a deconvolution method. When the cause of blurring in a clear image is unclear, the method of removing blurring using deconvolution itself may be a problem. Since the cause of the image blur is not known, most methods set a target value for the image to be solved, and if the actual image does not match the set image value, the blurring removal method may not have a great effect. .

An image de-blurring method includes obtaining a blurred image to be processed and a DVS event set recorded by a DVS sensor while the blurred image is exposed, and de-blurring the blurred image based on the DVS event set. It includes the steps of performing

The performing of deblurring processing on the blurring image based on the DVS event set may include: estimating a DVS edge estimation image based on the DVS event set; It may include performing deblurring processing.

The method of performing de-blurring processing based on the DVS edge estimation image includes a method of aligning the blurred image with the DVS edge estimation image and performing de-blurring processing based on a result of the alignment, the blurring image is aligned with the DVS edge estimation image, de-blurring processing is performed based on the alignment result, the de-blurring processing result is less than a predetermined de-blurring processing value, and the number of de-blurring processing is the maximum iteration If it is less than the number of times, an average edge diagram is determined based on the deblurring processing result, and the deblurring processing result satisfies a predetermined deblurring processing value using the average edge diagram, or A method of rearranging the blurring image with the DVS edge estimation image until the number of deblurring processing reaches the maximum number of iterations, and outputting the deblurring processing result as a deblurring result, the blurring image Align the DVS edge estimation image, perform de-blurring processing based on the alignment result, and determine an average edge diagram based on the de-blurring processing result when the number of de-blurring processing is less than the maximum iteration number; , Based on the average edge diagram, rearranging the blurred image to the DVS edge estimation image until the maximum number of iterations is reached, and outputting the blurring processing result as a deblurring result, and the blurring An image is aligned with the DVS edge estimation image, deblurring processing is performed based on the alignment result, and the deblurring processing result is less than a predetermined deblurring processing value and the number of deblurring processing reaches the maximum iteration number. an average edge diagram is determined based on the deblurring processing result, and based on the average edge diagram, the deblurring processing result satisfies a predetermined deblurring processing value or the number of deblurring processing is At least one of methods of aligning the blurred image with the DVS edge estimation image again until the maximum number of iterations is reached and outputting the blurring processing result as a deblurring result may be included.

The performing of de-blurring processing on the blurred image based on the DVS event set includes estimating a camera motion trajectory and a DVS edge estimation image while the blurred image is exposed based on the DVS event set. and performing the de-blurring process based on the camera motion trajectory and the DVS edge estimation image.

The method of performing blurring removal processing based on the camera motion trajectory and the DVS edge estimation image includes aligning the blurred image to the DVS edge estimation image, performing blurring removal processing based on the alignment result, and blurring When the deblurring processing result is smaller than the predetermined deblurring processing value, determining an average edge diagram on the camera motion trajectory direction based on the deblurring processing result, and performing deblurring processing based on the average edge diagram. A method of rearranging the blurred image to the DVS edge estimation image until a result satisfies a predetermined de-blurring processing value, and outputting the blurring processing result as a de-blurring result; Align the edge estimation image, perform de-blurring processing based on the alignment result, and if the set maximum number of iterations is not achieved, determine an average edge diagram in the direction of the camera motion trajectory based on the de-blurring processing result A method of rearranging the blurred image to the DVS edge estimation image using the average edge diagram until a set maximum number of iterations is reached, and outputting the blurring processing result as a deblurring result, and the blurring An image is aligned with the DVS edge estimation image, deblurring processing is performed based on the alignment result, and the deblurring processing result is less than a predetermined deblurring processing value and the number of deblurring processing is less than the maximum iteration number. an average edge diagram in the direction of the camera motion trajectory is determined based on the deblurring process result, and based on the average edge diagram, the deblurring process result satisfies a preset deblurring process value or blurring It may include at least one of a method of rearranging the blurring image with the DVS edge estimation image until the number of iterations of the deblurring process reaches a maximum number of iterations, and outputting the result of the blurring process as a deblurring result.

In the performing of the de-blurring process based on the result of the alignment, the de-blurring process is performed on the currently aligned blurring image using a deconvolution transform based on the currently aligned DVS edge estimation image. and making the sharp image obtained in the current iteration step the deblurring process result, when the difference in sharpness between the images in the current iteration step and the previous iteration step is less than a predetermined threshold value, a predetermined deblurring process value. and determining that the sharpness difference is smaller than a predetermined deblurring processing value when the sharpness difference is greater than the threshold value.

In the step of estimating the camera motion trajectory based on the DVS event set, the exposure time is divided into N time slices according to time order, and DVS events located in the same time slice among the DVS event sets are divided into independent images. and determining a camera motion trajectory during exposure based on images formed in the N time slices. The method for determining the camera motion trajectory may include an image formed every two consecutive time slices. The displacement of the camera in two consecutive time slices is determined based on the positional relationship of the DVS event, and the displacements of the camera for each two consecutive time slices during the exposure are all ordered in time order. and forming a camera motion trajectory during the exposure. In the estimating the DVS edge estimation image based on the DVS event set, the exposure time may be divided into N time slices according to a time sequence. segmentation, form an image incorporating DVS events in the same time slice among the DVS event set, align and overlay images formed in all time slices, and create a skeleton diagram of the overlapped image in the DVS event set. A step of calculating an edge estimation image may be included.

을 계산하고, 상기 (x,y)는 이미지 중 픽셀 포인트의 2D 좌표이고, A (x,y) 및 B (x,y)는 각각 이미지 A 및 B 의 픽셀 포인트의 값이고,

는 이미지B를 이미지A에 정렬할 경우 필요한 2D 변위 값이고,

는

가 최소일 때의 독립변수이고, 계산된

에 기초하여 이미지 B 및/또는 이미지A의 변위를 결정하고, 이미지 A 및 B의 정렬을 포함할 수 있다.The method of aligning the images of any two time slices is, for images A and B of any two time slices,

where (x,y) is the 2D coordinate of a pixel point in the image, A (x,y) and B (x,y) are the values of the pixel point in images A and B, respectively;

is the 2D displacement value required to align image B to image A,

Is

is the independent variable when is the minimum, and the calculated

Determine the displacement of image B and/or image A based on , and may include alignment of images A and B.

를 계산하고, 차례에 따라 늦은 시간의 이미지를 앞선 시간의 이미지에 정렬하고 중첩하는 방식 또는 2개의 연속된 타임 슬라이스마다 형성된 이미지들에 대해

를 계산하고, 차례에 따라 앞선 시간의 이미지를 늦은 시간의 이미지에 정렬하고 중첩하는 방식 중 적어도 하나를 포함할 수 있다.The method of aligning and overlapping images formed in every time slice is for images formed every two consecutive time slices.

For the method of calculating , and aligning and overlapping the image of the late time with the image of the earlier time according to the order or for the images formed every two consecutive time slices

It may include at least one of the methods of calculating , arranging and overlapping an image of an earlier time with an image of a later time according to the order.

일 수 있고, 상기 은 매트릭스 x를 벡터 형식으로 표시한 것을 의미할 수 있고, 는 경도(gradient)의 연산을 의미할 수 있으며, I는 블러링 제거 처리 후의 선명도를 의미할 수 있고, C는 현재 정렬된 블러링 이미지일 수 있고, E는 현재 정렬된 DVS 엣지 추정 영상일 수 있고,

는 벡터의 2놈(norm)을 계산한다는 의미일 수 있고,

는 벡터의 1놈을 계산하는 의미일 수 있고,

및

는 미리 정해진 2개의 가중치(weight value)일 수 있고,

는

가 최소값일 때의 독립변수를 의미할 수 있다.When performing the de-blurring process, the blurring kernel of the method of calculating the blurring kernel is

may be, and the may mean that the matrix x is expressed in vector form, may mean gradient operation, I may mean sharpness after deblurring processing, C may be a currently aligned blurring image, and E may be a currently aligned DVS edge estimation image. there is,

may mean calculating the 2 norm of the vector,

may mean calculating one norm of a vector,

and

May be two predetermined weight values,

Is

may mean an independent variable when is the minimum value.

The method for determining the average edge diagram is the result of calculating the edge diagram in each segment direction of the camera movement trajectory, superimposing all the edge diagrams, and then calculating the skeleton diagram for the sharpness of the image in the current iteration step. It may include a method of making the average edge diagram.

An image deblurring device includes an acquisition module and an image deblurring module, wherein the acquisition module acquires a blurring image to be processed and a set of DVS events recorded by a DVS sensor while the blurring image is exposed; A deblurring module performs deblurring processing on the blurring image based on the set of DVS events.

Estimating the camera motion trajectory, estimating the edge of the blurring image using the motion trajectory, estimating the DVS edge using the DVS event, and converting the blurring image or the edge estimation image of the estimated blurring image to the DVS edge estimation result By aligning and performing deblurring processing, the deblurring effect can be improved.

The de-blurring test was performed with the de-blurring method and the conventional de-blurring method, respectively, and the results were compared. 10 compares the effect of the de-blurring method according to an embodiment with the effect of the existing technology. As shown in FIG. 10 , the deblurring effect can be further increased.

1 is a flowchart illustrating a method for removing image blurring according to an exemplary embodiment.
Fig. 2 is a schematic diagram showing the relationship between the image formation of a DVS event set and the image formed within each time slice.
Figure 3 is a schematic diagram showing the displacement of a camera within two consecutive time slices.
Figure 4 is a schematic diagram of performing DVS edge estimation.
5 is a schematic diagram showing alignment processing of RGB-DVS images.
6 is a schematic diagram of yielding a clear image.
7 is a schematic diagram illustrating the creation of an averaged edge diagram.
8 is a schematic diagram showing the basic structure of an image deblurring removal device.
Fig. 9 is a schematic diagram showing the structure of an image deblurring module in an image deblurring device;
10 shows a result of performing an image blurring removal method and a conventional image blurring removal method.

The DVS sensor is a high-speed camera with microsecond time resolution and can record the output DVS, and this DVS time can represent all changes that occur in the camera lens in microsecond time. In general, since intra-lens change occurs at the edge of an image, if the DVS sensor is fixed to a general camera, a general camera operating at high speed can obtain a relatively clear edge image. the examples. Blurring removal processing is performed on a blurring image captured by a general camera using a relatively clear edge image that can be obtained through a DVS sensor. Therefore, the deblurring effect can be improved.

A basic image deblurring removal method includes the following steps.

Step 1: Obtain a blurred image to be processed and a set of DVS events recorded by the DVS sensor while the blurred image is exposed.

Step 2: Perform deblurring processing on the blurred image based on the DVS event set.

In step 2, a DVS edge estimate image is estimated based on the DVS event set, and blurring removal processing is performed based on the DVS edge estimate image, or a camera motion trajectory and a DVS edge estimate image are estimated based on the DVS event set. and deblurring is performed based on the camera motion trajectory and the DVS edge estimation image.

1 is a flowchart illustrating a method for removing image blurring according to an exemplary embodiment. As shown in Fig. 1, the method includes the following steps.

Step 101 obtains a blurred image to be processed and a set of DVS events recorded by a DVS sensor while the blurred image is exposed. It is possible to obtain an RGB blurred image taken by a general camera, fix the DVS sensor to the general camera, and obtain a recording of the same angle as the general camera. In addition, the recorded DVS events may reflect the motion of the shooting camera. When a DVS event set of a blurring image is obtained during exposure, a camera movement trajectory during the corresponding exposure time may be reflected.

Step 102 estimates a camera motion trajectory and a DVS edge estimation image within an exposure time based on the DVS event set. The exposure time can be divided into N time slices according to the time sequence. At this time, N is a preset positive integer and can be changed as needed. It is not limited to what is described in the specification. A set of DVS events during exposure includes a plurality of DVS events. DVS events each indicate a time. Based on the time representation of DVS events according to the divided time slices, DVS events located in the same time slice form images independently. An arbitrary method may be used for independently forming an image, and the arbitrary method is not limited. Figure 2 shows the relationship between the image formation of a set of DVS events and the image formed within each time slice.

First of all, the generation of the blurring image and the role of the image formed by the DVS event are analyzed. Assuming that the photographed subject is in a still state while being exposed to the camera, blurring of the image is caused by the movement of the photographed subject in which the camera itself is in a relatively still state. Based on this, the images formed for each time slice are all It corresponds to a relatively clear edge image in a situation where the camera moves relatively quickly within the time range. Since the images in different time slices have the same shooting target, the outline of the edge image is also the same. However, in the edge image, a relative displacement occurs in the background, and this displacement is a relative motion trajectory of the camera between different time slices. In addition, images formed at different time slices refer to the same edge image at different viewpoints, but the edge of each image may not be accurate or perfect due to an effect such as noise. Therefore, a relatively sharper edge diagram can be obtained by restoring images in all N time slices to the same viewpoint based on the movement trajectory of the camera and overlaying the N images of the same viewpoint.

In summary, images formed in two different time slices will have the same shape, and a change in the positional relationship of the shapes in the entire image means the linear motion relationship of the camera in the two time slices. If one of the images is subjected to reverse motion according to the motion relation, an image within a time slice where another image is located can be obtained. By overlapping these two images, an edge diagram with a higher effect can be obtained.

Based on the analyzed contents, a camera motion trajectory and a DVS edge estimation image during exposure are determined based on images formed in N time slices.

Specifically, the method of determining the camera movement trajectory during exposure determines the displacement of the camera within two consecutive time slices based on the positional relationship of the DVS events, and determines the camera movement trajectory during exposure every two consecutive time slices. We connect all the displacements in time sequence and construct the trajectory of the camera movement during the exposure. Figure 3 shows the displacement of the camera within the two consecutive time slices.

In one embodiment, the method for determining the displacement of the camera within two consecutive time slices can be calculated using the formula below for images A and B of any two consecutive time slices.

（1）

(One)

는 카메라가 이미지B의 시점에서 이미지A의 시점까지의 2D 변위를 의미하고, 상기 2D 변위에 기초하여 도3에 도시한 바와 같은 운동 벡터를 획득할 수 있다. 여기서 2개의 연속된 타임 슬라이스에서 카메라는 직선 운동을 한다고 가정한다. 모두 2개의 연속된 타임 슬라이스에서의 운동 벡터를 시간 순서에 따라 연결하면 노출되는 동안의 카메라 운동 궤적을 획득할 수 있다.(x,y) is the 2D coordinate of the pixel point in the image, A (x,y) and B (x,y) are the values of the pixel point in images A and B, respectively;

Means the 2D displacement of the camera from the viewpoint of image B to the viewpoint of image A, and a motion vector as shown in FIG. 3 can be obtained based on the 2D displacement. Here, it is assumed that the camera moves in a straight line in two consecutive time slices. The camera motion trajectory during exposure can be obtained by connecting the motion vectors in two successive time slices according to the time sequence.

As shown in FIG. 4, in the method of determining the DVS edge estimation image, images formed in all time slices are aligned and overlapped, and skeleton diagrams of the overlapped images are calculated as the DVS edge estimation image.

를 계산하고, 상기 2D 변위 값은 이미지B를 이미지A에 정렬할 경우 필요한 2D 변위 값을 의미한다. 이미지B 변위

를 이미지A에 정렬시키거나, 이미지A 변위

를 이미지B 에 정렬시키거나, 이미지A와 이미지B에 대해 동시에 변위를 수행하고, 이미지A와 이미지B 간의 다른 타임 슬라이스에 정렬시킬 수 있다.When performing alignment, either align images of all time slices to the same time slice and merge and overlap, or align images of one time slice to another time slice and overlap images of the two time slices, and overlap It is inferred by aligning the results to different time slices. Here, the 2D displacement value between images of two arbitrary time slices using the above formula (1)

is calculated, and the 2D displacement value means a 2D displacement value required when aligning image B with image A. image B displacement

align to imageA, or imageA displacement

can be aligned to image B, or the displacement can be performed on image A and image B simultaneously, and aligned to different time slices between image A and image B.

를 계산하고, 시간 순서에 따라 뒤/앞 시간의 이미지를 앞/뒤 시간의 이미지에 정렬하고 중첩한다. 이런 방법으로 추정의 정확성을 향상시킬 수 있다. "/"는 "또는"을 의미한다.In addition, when calculating the displacement using equation (1), it is assumed that the movement between two time slices is a linear motion, but the estimation becomes inaccurate as the error between the linear motion and the actual motion increases as the time interval actually increases. Thus, when aligning and overlapping images, every two consecutive time slices

Calculate , and arrange and superimpose the images of the previous/previous time with the image of the previous/posterior time according to the time sequence. In this way, the accuracy of estimation can be improved. "/" means "or".

After performing step 102, image blurring is removed iteratively.

Step 103 aligns the obtained blurred image to the DVS edge estimation image, and obtains the aligned blurring image and the aligned DVS edge estimation image.

As shown in FIG. 5, the RGB-DVS image alignment process first performs distortion removal on each of the RGB image and the DVS edge estimation image. Then, the corresponding key points of the two images are acquired and an affine transformation model is calculated to align the two images. Distortion checking, key point determination, and image alignment processing through a model are all conventional techniques, and thus detailed descriptions thereof are omitted here. The image alignment may be performed by including the distortion elimination process in step 103 and directly determining key points without performing the distortion elimination process in consideration of complexity and the like. When performing alignment, an RGB image may be aligned with a DVS edge estimation image or a DVS edge estimation image may be aligned with an RGB image. A better effect can be obtained by aligning the RGB image to the DVS edge estimation image.

In addition, when performing the first iteration, the alignment process is performed using the RGB blurring image and the DVS edge estimation image directly. When the next iteration is performed, the matching precision between RGB and DVS can be improved by aligning the RGB blurring image and the DVS edge estimation image based on the RGB edge estimation image determined at the time of the previous iteration.

In step 104, based on the currently aligned DVS edge estimation image, a de-blurring process is performed on the aligned blurring image by using a deconvolution transform, and a clear image of the current iteration step is obtained. Acquire

The processing of the image deblurring module will estimate a blurring kernel of the image, and use deconvolution transformation to realize deblurring.

The blurring kernel k can be calculated using the formula below.

는 벡터의 2놈(norm)을 계산한다는 의미이고,

는 벡터의 1놈을 계산한다는 의미이고,

및

는 미리 정해진 2개의 가중치(weight value)이고,

는

가 최소일 때의 독립변수를 의미한다.remind means that the matrix x is expressed in vector form, Means the operation of the gradient, I is the sharp image after deblurring processing, C is the currently aligned blurring image, E is the currently aligned DVS edge estimation image,

means to calculate the 2 norm of the vector,

means to calculate one norm of the vector,

and

is two predetermined weight values,

Is

is the independent variable when is the minimum.

After obtaining the blurring kernel k, a clear RGB image I can be obtained using the following formula.

는 벡터의 2놈(norm)를 계산한다는 의미이고,

는 벡터의 1(norm)를 계산한다는 의미이며, B'는 현재 정렬된 블러링 이미지를 의미한다.remind means matrix x in vector form,

means to calculate the 2 norm of the vector,

means to calculate the 1 (norm) of the vector, and B' means the currently aligned blurring image.

When the number of times the blurring kernel and the process of obtaining a clear image reach a predetermined maximum number or the acquired clear image no longer changes, I obtained through calculation is determined as the clear image obtained in the corresponding iterative step. do.

In step 104, the method of calculating I is the same as the existing technology, and when calculating the blurring kernel k, it is performed based on the aligned DVS edge estimation image E. Therefore, by using the image E when calculating the blurring kernel k, the image blurring situation can be fully reflected, and the calculated sharp image I is more similar to the original screen.

In step 104, deblurring processing is performed based on the DVS edge estimation image. Within the processable range, the following steps may be further performed. It is possible to optimize the deblurring processing result through an iterative process.

In step 105, it is determined whether the sharpness of the images in the current iteration step and the previous iteration step match. can be terminated If the determination results do not match, step 106 may be performed.

The fact that there is no change in the clear image that is repeated two times before and after means that the target deblurring value is already satisfied. When there is no change in the clear image, the clear image may be output and the deblurring process may be terminated. If there is a change in the sharp image, the next iteration step can be performed by calculating the edge diagram of the RGB. The method of determining whether the images that are repeated twice before and after is the same is as follows: If the sharpness error between the image of the current iteration step and the image of the previous iteration step is smaller than a preset threshold value, the clear images that are repeated twice are the same, otherwise Otherwise, it can be determined that the images performed twice are different.

In step 106, an average edge diagram is determined based on the sharp image obtained in the current step, and the process returns to step 103 again. In the next iteration, the RGB blurred image and the DVS edge estimation image are aligned based on the average edge diagram.

The processing of step 106 is used to generate a directed edge diagram of the RGB image, which is referred to herein as an average edge diagram. Specifically, based on the sharp image estimated during the current iteration step, an edge diagram for a direction set by the clear image is obtained, and an average edge diagram is obtained based on a plurality of edge diagrams. The set direction may be a randomly designated direction or a direction of a camera motion trajectory.

Then, the RGB image and the DVS edge estimation image are aligned based on the mean edge diagram. Specifically, the displacement of the alignment can be obtained by comparing the average edge diagram with the DVS edge estimation image during the next iterative step, and by moving and aligning the RGB blurring image and/or the DVS edge estimation image, the RGB and DVS images can improve the matching accuracy.

When the direction is set to the direction of the camera movement trajectory, the method for generating the average edge diagram is as follows. Segment the trajectory of the camera movement during exposure, calculate the edge diagram for the sharp image acquired in the current iteration step in each segment direction, superimpose all the acquired edge diagrams, and calculate the skeleton diagram to achieve this. As an average edge diagram.

Segmentation of camera motion trajectories may be performed according to motion trajectories generated during operation 102 . A method of generating an averaged edge diagram is shown in FIG. 7 .

Above, an embodiment of the image blurring removal method has been described. In the iterative process, if the clear images obtained twice repeatedly are the same, the repetition is terminated, and the final processing result for deblurring is obtained. More simply, set the maximum number of iterations, repeat until the maximum number of iterations is reached, and when the number of iterations reaches the maximum number of iterations, terminate the iteration process and obtain the final processing result for deblurring. . Alternatively, the method of comparing the sharp images obtained by repeating the previous and previous two times is used together with the method of setting the maximum number of iterations, and if one condition is satisfied among them, the iteration process is terminated, and otherwise, the iteration is performed. The maximum number of iterations may be set based on actual needs and equipment processing performance.

During the deblurring process, a motion trajectory of a camera may be estimated using a DVS event recorded by a DVS sensor, and an edge of a blurred image may be estimated using the motion trajectory. In addition, a DVS edge is estimated for the DVS event, and after aligning the blurring image or edge estimation image with the DVS edge estimation result, deblurring processing is performed. Through the above processing, it is possible to improve the deblurring effect by providing a more realistic motion hypothesis using the camera motion trajectory and the DVS edge estimation image.

According to an embodiment, an image de-blurring removal device is further provided, and the image blur removal device may be used in the implementation of the de-blurring method shown in FIG. 1 . As shown in FIG. 8 , the device includes an acquisition module and an image deblurring module.

The acquisition module obtains a blurred image to be processed and a set of DVS events recorded by a DVS sensor while the blurred image is exposed, and the deblurring module obtains a set of blurring images based on the set of DVS events. Perform deblurring processing.

As shown in FIG. 9 , the image deblurring module may include a trajectory estimation submodule, an image registration submodule, an image deblurring submodule, and an average edge diagram generation submodule.

The trajectory estimation submodule estimates a camera motion trajectory and a DVS edge estimation image during exposure based on a set of DVS events. The trajectory estimation submodule may perform steps 101-102.

The image matching submodule aligns the blurred image with the DVS edge estimation image, and acquires the aligned blurred image and the currently aligned DVS edge estimation image. The image matching submodule may perform step 103.

The image deblurring submodule performs deblurring processing on the aligned blurring image using deconvolution based on the aligned DVS edge estimation image to obtain a clear image of a previous iteration step; , If the sharpness difference between the current iteration step and the previous iteration step is less than a predetermined threshold value, the clear image of the previous iteration step is output as a blurring removal result, and if the sharpness difference is greater than the threshold value, the current iteration step The sharp image is sent to the average edge diagram output module. The image deblurring submodule may perform steps 104-105.

The average edge diagram output module determines the average edge diagram based on the sharp image input to the image deblurring module. After the average edge diagram is transmitted to the image matching module, it is used as a basis for aligning the blurring image and the DVS edge estimation image. In this way, the average edge diagram of the direction of the camera motion trajectory can be determined. An average edge diagram output module may perform step 106 .

Although described with reference to the embodiments as described above, the above-described de-blurring method is not limited thereto. Various modifications, replacements and variations are possible from the above description, all of which fall within the protection scope of the claims.

Claims (14)

obtaining a blurred image to be processed and a set of DVS events recorded by a DVS sensor while the blurred image is exposed;
estimating a DVS edge estimation image based on the DVS event set; and
Performing the de-blurring process based on the DVS edge estimation image
including,
The DVS edge estimation image is determined by aligning and overlaying images formed in all time slices, and calculating a skeleton diagram of the overlapped image as the DVS edge estimation image,
How to remove image blurring.
delete According to claim 1,
A method of performing de-blurring processing based on the DVS edge estimation image,
a method of aligning the blurred image to the DVS edge estimation image and performing deblurring processing based on a result of the alignment;
The blurring image is aligned with the DVS edge estimation image, deblurring processing is performed based on the alignment result, the deblurring processing result is less than a predetermined deblurring processing value, and the number of deblurring processing is maximum. If it is less than the number of iterations, an average edge diagram is determined based on the deblurring process result, and the deblurring process result is predetermined by using the average edge diagram. rearranging the blurring image to the DVS edge estimation image until a value is satisfied or the number of deblurring processing reaches a maximum iteration number, and outputting the deblurring processing result as a deblurring result;
The blurring image is aligned with the DVS edge estimation image, deblurring processing is performed based on the alignment result, and when the number of deblurring processing is less than the maximum iteration number, an average is calculated based on the deblurring processing result. Determining an edge diagram, rearranging the blurred image to the DVS edge estimation image until the maximum number of repetitions is reached based on the average edge diagram, and outputting the blurring processing result as a deblurring result method; and
The blurring image is aligned with the DVS edge estimation image, de-blurring processing is performed based on the alignment result, and the de-blurring processing result is less than a predetermined de-blurring processing value and the number of de-blurring processing is maximum iterations. until the number is reached, an average edge diagram is determined based on the deblurring process result, and based on the average edge diagram, the deblurring process result satisfies a predetermined deblurring process value or the deblurring is removed. At least one of a method of rearranging the blurred image to the DVS edge estimation image until the number of processing reaches the maximum number of iterations, and outputting the blurring processing result as a deblurring result,
How to remove image blurring.
According to claim 1,
Performing de-blurring processing on the blurred image based on the DVS event set comprises:
estimating a camera motion trajectory and a DVS edge estimation image while the blurred image is exposed, based on the DVS event set; and
performing the de-blurring process based on the camera motion trajectory and the DVS edge estimation image;
Image blurring removal method comprising a.
According to claim 4,
The method of performing de-blurring processing based on the camera motion trajectory and the DVS edge estimation image,
The blurring image is aligned with the DVS edge estimation image, deblurring processing is performed based on the alignment result, and when the deblurring processing result is smaller than a predetermined deblurring processing value, the deblurring processing result Determines an average edge diagram in the direction of the camera motion trajectory based on the DVS edge estimation of the blurred image based on the average edge diagram until a deblurring processing result satisfies a predetermined deblurring processing value. a method of rearranging the image and outputting the blurring processing result as a deblurring result;
The blurring image is aligned with the DVS edge estimation image, deblurring processing is performed based on the alignment result, and if the set maximum number of iterations is not achieved, the camera motion trajectory direction is determined based on the deblurring processing result. An average edge diagram of the image is determined, and the blurred image is re-aligned with the DVS edge estimation image until the set maximum number of iterations is reached using the average edge diagram, and the blurring processing result is a deblurring result. way to output; and
The blurring image is aligned with the DVS edge estimation image, de-blurring processing is performed based on the alignment result, and the de-blurring processing result is less than a predetermined de-blurring processing value and the number of de-blurring processing is maximum iterations. if less than the number of deblurring processing results, determine an average edge diagram along the camera movement trajectory direction based on the deblurring processing result, and based on the average edge diagram, the deblurring processing result satisfies a preset deblurring processing value. Alternatively, the blurring image is rearranged with the DVS edge estimation image until the number of deblurring processing reaches the maximum number of iterations, and the blurring processing result is output as a deblurring result.
Image blurring removal method comprising at least one of.
According to claim 3 or 5,
The step of performing de-blurring processing based on the alignment result,
Based on the currently aligned DVS edge estimation image, deblurring processing is performed on the currently aligned blurring image using a deconvolution transform, and the sharp image obtained in the current iteration step is deblurred. the resulting step; and
If the difference between the sharpness of the images in the current iteration step and the previous iteration step is smaller than a predetermined threshold value, it is determined that a predetermined de-blurring process value is satisfied, and if the sharpness difference is greater than the threshold value, it is determined that the predetermined de-blurring processing value is satisfied. Determining that the deblurring processing value is smaller than
containing
How to remove image blurring.
According to claim 4,
Estimating a camera motion trajectory based on the DVS event set comprises:
The exposure time is divided into N time slices according to the time sequence, the DVS event located in the same time slice among the DVS event set is formed as an independent image, and the image is exposed based on the image formed within the N time slices. Including the step of determining the camera motion trajectory of
How to remove image blurring.
According to claim 7,
The method for determining the camera motion trajectory,
Among the images formed every two consecutive time slices, based on the positional relationship of the DVS event, determine the displacement of the camera in the two consecutive time slices, and the displacement of the camera every two consecutive time slices during the exposure. All of them are connected in time sequence, and the camera movement trajectory is formed during the exposure.
How to remove image blurring.
According to claim 1 or 4,
Estimating the DVS edge estimation image based on the DVS event set includes:
Dividing the exposure time into N time slices in time order, forming an image incorporating DVS events in the same time slice among the DVS event sets, aligning and overlaying images formed in all time slices, , Comprising the step of calculating a skeleton diagram of superimposed images as the DVS edge estimation image,
How to remove image blurring.
According to claim 9,
The method of aligning images of two arbitrary time slices is,
For images A and B of any two time slices,

calculate,
Where (x,y) is the 2D coordinate of the pixel point in the image, A (x,y) and B (x,y) are the pixel point values of images A and B, respectively;

2D displacement value required to align image B to image A,

Is

is the independent variable when is the minimum value, and the calculated

Determining the displacement of image B and / or image A based on, including alignment of images A and B,
How to remove image blurring.
According to claim 9,
The method of aligning and overlapping the images formed in the time slice,
For images formed every two consecutive time slices

A method of calculating , and aligning and superimposing an image of a late time to an image of an earlier time; or
For images formed every two consecutive time slices

A method of calculating , arranging and superimposing an image of an earlier time with an image of a later time according to time order
Image blurring removal method comprising at least one of.
According to claim 6,
In the case of performing the de-blurring process, the method of calculating the blurring kernel is,
blurring kernel

ego,
remind means that the matrix x is expressed in vector form, Means the operation of the gradient, I is the sharp image after deblurring processing, C is the currently aligned blurring image, E is the currently aligned DVS edge estimation image,

means to calculate the 2 norm of the vector,

means to calculate one norm of the vector,

and

is two predetermined weight values,

Is

means the independent variable when is minimum,
How to remove image blurring.
According to claim 6,
The way to determine the average edge diagram is,
For the sharpness of the image in the current iteration step, based on the DVS event set, an edge diagram in each segment direction of the estimated camera motion trajectory during exposure of the blurred image is calculated, and all edge diagrams are superimposed. Afterwards, including a method of calculating the result of calculating the skeleton diagram as the average edge diagram,
How to remove image blurring.
In the image blur removal device,
It includes an acquisition module and an image deblurring module,
The acquisition module acquires a blurred image to be processed and a set of DVS events recorded by a DVS sensor while the blurred image is exposed;
The de-blurring module estimates a DVS edge estimation image based on the DVS event set; and
Performing the de-blurring process based on the DVS edge estimation image;
The DVS edge estimation image is determined by aligning and overlaying images formed in all time slices, and calculating a skeleton diagram of the overlapped image as the DVS edge estimation image,
Video deblurring device.

Images