KR101341871B1 - Method for deblurring video and apparatus thereof - Google Patents

Abstract

A technique related to a method and apparatus for removing motion blur caused by vibration of a camera is disclosed. A method for deblurring video comprises the steps of: estimating a homography function between each frame of an input video and a frame adjacent thereto; calculating a luckiness value representing a degree of blurred pixels of each frame using the estimated homography function; and estimating a blur function of each frame using the estimated homography function and the calculated luckiness value. Thus, motion between the frames and motion blur of each frame can be estimated from the input video. Furthermore, the method comprises the steps of: selecting matching patches from a frame adjacent to all the patches of each frame of the input video; and blending the selected matching patches to restore a clear frame. Thus, the blur of each frame of the input video can be removed using the estimated motion and the motion blur. In result, a blur caused by camera motion can be effectively removed. [Reference numerals] (AA) Start; (BB) End; (S100) Estimate motion and motion blur; (S110) Estimate homography; (S120) Calculate luckiness; (S130) Estimate a blur function; (S200) Deblurring; (S210) Select matching patches; (S220) Restore a clear frame; (S230) Update the luckiness

Description

Video de-blurring method and apparatus therefor {METHOD FOR DEBLURRING VIDEO AND APPARATUS THEREOF}

The present invention relates to video deblur, and more particularly, to a method and apparatus for removing motion blur caused by camera shake.

As the spread of portable video recording apparatuses such as digital camcorders and camera phones is expanded, and video services such as YouTube are spreading, an environment in which anyone can shoot and use video is being created.

A general user who does not have professional shooting equipment usually takes a camera in hand and shoots a video. However, when shooting a video with the camera in hand without professional shooting equipment, motion blur may occur in the recorded video due to the camera shake.

In other words, the number of videos taken by the general user is increasing rapidly, but when the general user shoots the video without the professional shooting equipment, the content of the video is not only shaken because the user moves the hand and shoots the video. In many cases, each frame constituting the video is blurred and photographed.

Recently, a video stabilization system has been proposed, which smoothes the motion of a camera to remove motion blur, and has been widely applied to video editing programs such as AfterEffects of Adobe Systems, Inc. However, since the video stabilization system does not remove the blur of each frame constituting the video, there is a limit in obtaining high quality results.

Separately, image deblurring techniques have been proposed to remove camera shake. Such techniques include techniques such as single image deblurring or multiple image deblurring.

Therefore, we can think of the method of applying this deblurring technique as the simplest method for eliminating the blurring of each frame of the video, but this method cannot obtain high quality results. This is due to the spatial and temporal diversity of the blur kernels in video, the limitations of effective processing of blur due to object motion, the nature of deconvolution sensitive to noise, and the image deblur of individual frames. This is because there is a limit to effectively blur the video due to temporal mismatch.

An object of the present invention for solving the above problems is to provide a video deblurring method that can effectively remove the blur caused by the motion of the camera.

Another object of the present invention for solving the above problems is to provide a video deblurring apparatus that can effectively remove the blur caused by the motion of the camera.

According to an aspect of the present invention, there is provided a video deblurring method, comprising: estimating motion between frames and motion blur of a frame from input video, and using the estimated motion and motion blur Removing the blur of the frame.

Here, estimating motion between frames and motion blur of a frame from the input video includes estimating a homography function between each frame of the input video and an adjacent frame, and using the estimated homography function. Comprising the steps of: calculating a lukiness value representing the degree of blurring of pixels of each frame; and estimating a blur function of each frame using the estimated homography function and the calculated lukiness value .

Wherein removing the blur of the frame of the input video using the estimated motion and motion blur comprises selecting matching patches from adjacent frames for every patch of each frame of the input video; Blending the matching patches to restore the clear frame.

Here, the estimating of the blur function of each frame using the estimated homography function and the calculated lukiness value may be performed by interpolating the estimated homography function to approximate a camera shake. The function can be estimated.

Here, the estimating of the blur function of each frame using the estimated homography function and the calculated lukiness value may be performed by using a calculated lukiness value to determine a pair of clear and adjacent blurred frames. The exposure time of the frame can be estimated by selecting and comparing the pair of the selected frame.

Here, selecting matching patches from the adjacent frame for every patch of each frame of the input video may be performed by applying the estimated blur function of each frame to the adjacent frame corresponding to each frame, respectively. The blur based similarity calculated according to the similarity between the patches of the frame of the frame and the patches of the adjacent frame to which the blur function is applied can find the patches of the adjacent frame most similar to the patches of each frame as matching patches.

Here, the step of selecting matching patches from the adjacent frames for all the patches of each frame of the input video, the correction is calculated by applying the weight based on the calculated lukiness value to the calculated blur based similarity Based on the blur based similarity, the patches of the adjacent frame most similar to the patches of each frame can be found as matching patches.

Here, in the reconstructing the clear frame by blending the selected matching patches, the clear frame may be restored by blending the selected matching patches for the patches of each frame.

Here, in the reconstructing the clear frame by blending the selected matching patch, matching patches based on the calculated blur based similarity or the calculated modified blur based similarity may be blended into the patches of each frame.

Here, the method may further include blending the lukiness values of the selected matching patch to update the lukiness value of the reconstructed frame.

Here, the step of removing the blur of the frame of the input video by using the estimated motion and motion blur may be repeated a predetermined number of times.

Here, the method may further include determining a deblurring order for the input frame based on the lukiness value.

According to an aspect of the present invention, there is provided a video deblurring apparatus comprising: a motion and motion blur estimator for estimating motion between frames and motion blur of a frame from an input video; And a deblurring unit that removes blur of a frame of the input video using the blur.

When using the video deblurring method and apparatus according to the embodiment of the present invention as described above, it is possible to effectively remove the blur caused by the motion of the camera.

In addition, video deblurring methods and apparatuses include spatial and temporal diversity of blur kernels in video, limitations on effective processing of blur due to object motion, characteristics of deconvolution sensitive to noise, and individual frames. There is an advantage that can compensate for the limitation of the conventional deblurring due to the temporal inconsistency due to image deblurring.

1 is a diagram illustrating images of a motion blurred image.
2 is a conceptual diagram illustrating a blur model according to an embodiment of the present invention.
3 is an exemplary diagram illustrating a luckiness value for explaining the sharpness of an image according to an exemplary embodiment of the present invention.
4 is a flowchart illustrating a deblurring method according to an embodiment of the present invention.
5 is an exemplary view of an image to which the deblurring method according to an embodiment of the present invention is applied.
6 is a block diagram illustrating a video debluring apparatus according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

In general, a video is composed of a series of pictures, and the picture may be divided into predetermined areas such as blocks. In addition, the term "picture" may be used interchangeably with another term having an equivalent meaning such as an image, a frame, and the like, and a region of a frame may be expressed as a patch. Those skilled in the art to which the embodiments belong may understand.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating images of a motion blurred image.

Referring to FIG. 1, frames of a motion blurred video may include a sharp image region as well as a blurred image region.

As shown in FIG. 1A, a motion point of a camera can be tracked by setting a feature point in an area of the background of the frame. Since the background is static, the motion of the camera can be well represented. That is, the trajectory of the feature point shown in FIG. 1B may represent the movement of the camera.

When shooting a video with the camera in hand, blur may occur in the frame due to camera shake. Here, the shaking of the camera is an atypical motion in which the speed changes in a short time due to the shaking of the hand.

Since a camera with motion may be in a static state for a certain time, the image captured by the camera with motion may include sharp frames as well as blurred frames. In addition, a single frame may comprise a blurred image area and a clear image area. This is due to the spatially changing blur kernel.

For example, FIG. 1C shows a video frame captured with camera rotation. The center of rotation is around the center of the left image boundary. The image area near the center of rotation is clear (FIG. 1D), while the image area farther from the center of rotation is more blurred (FIG. 1E). Through this example, it may mean that local sharpness may be considered in video frames for video deblur.

According to an embodiment of the present invention, motion can be estimated based on parametric and homography as an approximation to actual motion, which can be much more complicated due to parallax.

In addition, it is possible to use an approximated motion for defining the luckiness of the pixel representing the sharpness or blurring of the image. In order to deblurr the video frame, it is possible to replace the less lucky pixel in the current frame with the more lucky pixel in the adjacent frame. Here, the term "lucky" may be used as a term indicating the degree of blurring, and the lucky pixel may mean a pixel that is less blurry than a non-lucky pixel and is clear.

Correspondence of pixels between frames can be obtained using the estimated homographics, which allows the motion model to find the best matched portion to compensate for inaccuracy. In addition, a patch-based texture synthesis approach may be applied to copy the lucky pixels to the blurred frame to preserve the object structure. And similar constraints can be added to corresponding patches in consecutive frames to maintain time consistency of the deblurred frame.

2 is a conceptual diagram illustrating a blur model according to an embodiment of the present invention.

를 위한 노출 시간이

이다. 시간 간격은

로 하고,

의 잠상(latent image)을

로 나타낸다. 근접한 프레임들간의 모션은 호모그래피(

)에 의해 근사화할 수 있다. 여기서

는 3×3 호모그래피 메트릭스(

)에 의해 파라메터라이즈된(parameterized) 왜곡(warping) 함수이고, 모션의 속도는 인접한 프레임들 사이에서 일정하다. 본 출원에서 i는 프레임의 번호를 의미할 수 있다. 본 발명에 있어서, 잠상(latent image), 잠상의 프레임(latent frame) 또는 잠상의 패치(latent patch)는 블러된 영상을 복원하기 위한 이미지, 프레임 또는 패치를 의미할 수 있다. Referring to FIG. 2, homography can be used to approximate blurs of video frames. Frame whose duty cycle is 2τ and the i th frame

Exposure time for

to be. Time interval

With

Latent image

Respectively. Motion between adjacent frames is homography (

Can be approximated by here

Is a 3 × 3 homography matrix (

Is a parameterized warping function, and the speed of motion is constant between adjacent frames. In the present application, i may mean a frame number. In the present invention, a latent image, a latent frame, or a latent patch may refer to an image, a frame, or a patch for restoring a blurred image.

는 다음의 수학식 1과 같이 나타낼 수 있다. therefore,

Can be expressed as Equation 1 below.

와

는 호모그래피 메트릭스인

와

에 의해 각각 파라메터라이즈된 왜곡(warping) 함수이고 다음의 수학식 2와 같이 정의될 수 있다. here,

Wow

Is a homography matrix

Wow

It is a warping function each parameterized by and can be defined as Equation 2 below.

는

의 역 행렬이다.
I is a 3 × 3 identity metric,

The

Is the inverse of.

Discretizing Equation 1 may yield Equation 3 below.

에서 샘플링율(sampling rate)이 될 수 있고, 실시예에 따라 20으로 설정될 수 있다. τ는 듀티 사이클로 나뉘는 샘플들의 개수가 될 수 있다. 또한, 프레임 i를 위한 블러 함수를

라 할 수 있다. Where T is

The sampling rate may be set to 20, and may be set to 20 according to an embodiment. τ may be the number of samples divided by the duty cycle. Also, blur function for frame i

.

The blur model can be similar to that developed in the multi-image deblurring method for panorama generation and can use homography as the underlying motion model. However, while the existing method treats the blur model as an accurate model, the blur model can be treated as an approximation in order to deal with a more complicated video than panoramas according to an embodiment of the present invention.

In addition, according to an embodiment of the present invention, an additional local search may be performed for side-by-side image regions of different frames in order to clearly handle modeling errors.

3 is an exemplary diagram illustrating a luckiness value for explaining the sharpness of an image according to an exemplary embodiment of the present invention.

Referring to FIG. 3, the luckiness value of two frames with motion blur is shown. 3A and 3B are two frames that are motion blurred, and FIGS. 3C and 3D are visualizations of the lukiness values of the respective pixels of FIGS. 3A and 3B, respectively. Light color means that the lukiness value is close to 1, and dark means that the lukiness value is close to zero. The sharper the pixels of FIGS. 3A and 3B are, the higher the luckiness values of FIGS. 3C and 3D are. In other words, the sharper the frame, the higher the lukiness value. Thus, the lukiness value may represent the sharpness or blur of the frame.

에서 x픽셀의 러키니스값은 수학식 4와 같이 정의할 수 있다. According to an exemplary embodiment of the present invention, the lukiness value of the pixels constituting the video frame may be calculated using absolute displacement of pixels between adjacent frames. frame

The lukiness value of x pixels in can be defined as shown in Equation 4.

는 호모그래피 H에 따른 하나의 픽셀을 다른 픽셀 위치에 매핑(mapping)하는 함수이다.

는 상수로서 20 픽셀들로 설정할 수 있다. 수학식 4는

를 통하여

프레임에서

프레임으로 카메라가 움직일 때 픽셀 x의 변위를 계산할 수 있다. here,

Is a function of mapping one pixel according to homography H to another pixel position.

Can be set to 20 pixels as a constant. Equation (4)

through

In the frame

We can calculate the displacement of pixel x when the camera moves in a frame.

와

는 I에 가깝고

는 1에 가까울 수 있으며, 픽셀x에 중심을 두는 이미지 패치는 거의 선명하다는 것을 나타낼 수 있다. 반대로,

가 작으면, 해당 패치가 큰 모션 블러를 포함하는 것을 나타낼 수 있다. 또한, 전체 프레임(

)의 러키니스값(

)은

안의 픽셀들을 위한 모든

의 평균값으로써 나타낼 수 있다. 여기서, 러키니스값은 0부터 1 사이의 값일 수 있다.
For example, when the frame-to-frame motion of pixel x is small,

Wow

Is close to I

Can be close to 1, indicating that the image patch centered on pixel x is nearly sharp. Contrary,

If is small, it may indicate that the patch includes large motion blur. Also, the full frame (

Lukiness value of)

)silver

All for the pixels inside

It can be represented as an average value of. Here, the lukiness value may be a value between 0 and 1.

4 is a flowchart for describing a deblurring method according to an embodiment of the present invention.

Referring to FIG. 4, a video deblurring method according to an embodiment of the present invention includes a motion and motion blur estimation step S100 and a deblur step S200.

Motion and motion blur estimation step S100 is performed for estimating a homography function between each frame and an adjacent frame, for each frame of the input video at step S110, using the estimated homography pixels of each frame. Comprising a step (S120) of calculating the lukiness value indicating the degree of blurring, and estimating the blur function of each frame using the estimated homography and the calculated lukiness value (S130).

The deblurring step (S200) may include selecting matching patches from neighboring frames for all patches of each frame of the input blur video (S210), blending the selected matching patches, and restoring a clear frame (S220). And blending the lukiness value of the selected matching patch to update the lukiness value of the reconstructed frame (S230).

The estimating of the homography function (S110) may include an image matching method based on a feature point to estimate homography based on a mapping between each frame of the input video and pixels constituting adjacent frames of the frame. registration method) or pixel value-based image matching method. In the case of a feature point-based method, a feature point is extracted from each frame, a corresponding feature point is found between each frame, a homography for mapping a position between the feature points is estimated, and in the case of a pixel value-based image matching method, Estimate homography that minimizes the difference in pixel values between two frames. A typical example is a method such as the Lucas-Kanade image registration method.

In operation S120, the pixel of the input frame is blurred by using a homography function based on a mapping between each frame of the input video and pixels constituting the adjacent frame. It is possible to calculate the luckiness value indicating the degree of blurredness.

Estimating the blur function of each frame by using the estimated homography and the calculated lukiness value (S130) may estimate the motion blur function by approximating the camera shake by interpolating the homography function between adjacent frames. can do. In addition, by using the calculated lukiness value, a pair of sharp and adjacent blurred frames may be selected, and the exposure time of the frame, that is, the duty cycle may be estimated by comparing the selected frames.

Selecting a matching patch from neighboring frames for all patches of each frame of the input blur video (S210) applies the blur function of each estimated frame to its adjacent frames, and then applies each patch of each frame. The patch of the adjacent frame most similar to each patch of the input frame may be found as a matching patch by using the blur based similarity calculated based on the similarity between the patches of the adjacent frames to which the blur function is applied. In addition, in order to find a clearer matching patch, a patch of an adjacent frame that maximizes the modified blur based similarity, which further considers the blur based similarity and a weight corresponding to the luckiness value, may be found as the matching patch.

In step S220 of restoring the clear frame by blending the selected matching patches, the clear frame may be restored by blending the selected matching patches for each patch of each frame. The blending may calculate the new weight for each matching patch using the calculated blur based similarity or the calculated corrected blur based similarity, and then blend the matching patches using the new weight.

In the updating of the lukiness value of the reconstructed frame (S230), the updated lukiness value may be calculated by blending the lukiness values of the selected matching patch using the weights calculated in the blending step.

In addition, the deblurring step (S200) according to an embodiment of the present invention may be performed repeatedly for a predetermined number of times for all frames of the input video to improve the deblurring effect.

In addition, since the blur degree for each input frame is different, the order of deblurring for the input frame may be determined based on the lukiness value. For example, the order of deblurring for each input frame may be determined based on an average of the lukiness values for the pixels constituting each of the at least one input frame.

A concept for describing a video debluring method according to an embodiment of the present invention is as follows.

를 산출하기 위하여, 호모그래피(

)와 듀티 사이클(τ)의 추정이 필요하다. Blur function in equation (3)

In order to calculate the homography (

) And the estimation of the duty cycle τ are required.

)를 추정하기 위하여, 표준 KLT 기법[Shi and Tomasi 1994]을 사용하는 움직임 추정(feature tracking)을 적용할 수 있다. 또한, 초기 호모그래피를 계산하기 위하여 트래킹된 피쳐 포인트(feature point)를 사용할 수 있으며, Lucad-kanade registration[Baker and Matthew 2004]을 사용하여 프레임간의 호모그래피를 개선할 수 있다. First, homography (

), We can apply feature tracking using the standard KLT technique [Shi and Tomasi 1994]. In addition, tracked feature points can be used to calculate initial homography, and Lucad-kanade registration [Baker and Matthew 2004] can be used to improve homography between frames.

)에서 어떤 두 개의 프레임간의 호모그래픽스를 추정할 수 있다. 여기서, M은 5 프레임들로 세팅할 수 있다. 프레임 i에서 프레임 j로의 호모그래피는

로 정의될 수 있고,

일 수 있다. 또한, 도 2를 참조하면

이며, 비-인접 프레임들 사이의 호모그래픽스는

와 동일한 방법으로 초기화되고 개선될 수 있다. According to an embodiment of the present invention, a local temporal window as well as between adjacent frames (

), We can estimate the homographics between any two frames. Here, M can be set to 5 frames. The homography from frame i to frame j is

Can be defined as

Lt; / RTI > Also, referring to FIG. 2

Homographics between non-adjacent frames

It can be initialized and improved in the same way as.

The video deblurring method according to an embodiment of the present invention may not update homography because it treats homography as an approximated motion model. In addition, small errors in homography estimation can be handled by local search of the matching patch in the deblurring process.

로 하면,

인 프레임들의 K 쌍이 있을 수 있다. 여기서, 프레임 러키니스 차이인

는 임계값(예:

)보다 클 수 있다. Also, a set of frame pairs can be selected to calculate τ. Here, each pair has a big difference in the lukiness measurement, so the accuracy of the blur function can be tested effectively.

,

There may be K pairs of frames. Where the frame lurkiness difference

Is the threshold value

May be greater than).

Therefore, an optimal τ that minimizes Equation 5 can be obtained.

)를 이용하여

을 가지고

를 조정할 수 있고,

의 블러 함수(

)를 사용하여 조정된 선명한 프레임(

)을 블러할 수 있다. 또한, 합성적으로 블러된 프레임과 관찰된

사이의 제곱된 차이점의 합을 계산할 수 있다. Homography (

)

To have

Can be adjusted,

'S blur function (

) Sharp frame adjusted with

) Can be blurred. Also, synthetically blurred frames and observed

You can calculate the sum of squared differences between them.

)에 의존할 수 있다. 즉 함수(

)가 정확하다면, 수학식 5의 값은 작고 그렇지 않으면 반대가 될 것이다. τ의 주어진 값을 위하여

는 수학식 3에 의해 결정되므로, 수학식 5는 τ의 에너지 함수가 될 수 있다. 또한, 본 발명의 실시예에 따르면, τ는 1에서

으로 정수값의 제한된 세트를 가짐에 따라 brute-force search 를 사용하여 수학식 5를 최소화할 수 있다.
The value of Equation 5 is the blur function for all k

Can be relied upon. Function

If) is correct, the value of Equation 5 is small, otherwise it will be reversed. for a given value of τ

Since is determined by Equation 3, Equation 5 may be an energy function of τ. Further, according to the embodiment of the present invention, τ is at 1

As we have a limited set of integer values, we can minimize Equation 5 using brute-force search.

의 복원을 위하여 인접 프레임에서의 패치를 이용할 수 있다. The video debluring method using deconvolution does not give satisfactory results. Accordingly, the video debluring method according to an embodiment of the present invention

Patches in adjacent frames may be used for the recovery.

는 프레임

에서 픽셀 x에 중심을 둔 n×n 이미지 패치로 정의될 수 있다. 예컨대, n=11일 수 있다. 본 발명의 실시예에 따르면, 시간 윈도우

에서 인접 프레임들(

)로부터 선명한 패치들의 가중된 평균을 계산함에 의해

를 디블러링할 수 있다.

Frame

Can be defined as an n × n image patch centered on pixel x. For example, n = 11. According to an embodiment of the invention, the time window

Adjacent frames in

By calculating the weighted average of the clear patches

Can be deblurred.

는

의 디블러링된 패치이고,

는 픽셀 y에 중심을 둔 왜곡(warped) 프레임(

)의 패치이다.
here,

The

Deblurred patch for,

Is a warped frame centered at pixel y (

) Patch.

는 입력 프레임의 패치와 인접한 프레임의 패치 사이의 블러 기반 유사도로, 수학식 7에 의해 구할 수 있다. weight

Is a blur based similarity between a patch of an input frame and a patch of an adjacent frame, which can be obtained by Equation 7.

는 블러된 왜곡 프레임(

)에서 y에 중심을 둔 패치이고, 현재 프레임(

)의 블러 함수(

)를

에 적용함에 의해 얻을 수 있다.

는 상수로서, 0.1이 될 수 있다.

는 왜곡 인접 프레임들(

)로부터 샘플링된

을 위한 매칭 패치(matching patch)들이다. Z는 정규화 팩터로,

이다. here,

Is a blurred distortion frame (

) Is the patch centered on y and the current frame (

Blur function of

)

It can be obtained by applying to.

Is a constant, and may be 0.1.

Is the distortion adjacent frames (

Sampled from)

Matching patches for. Z is the normalization factor,

to be.

)는 종래의 비디오 디블러링 방법의 데이터 피팅 항(fitting term)과 유사할 수 있고, 입력된 블러된 이미지로부터 잠상의 차이로 측정될 수 있다. Blur-based similarity (

) May be similar to the data fitting term of the conventional video deblurring method, and may be measured as the latent image difference from the input blurred image.

)가 되는 것의 적합성을 위하여 인접 프레임(

)으로부터 왜곡 패치(

)를 테스트할 수 있다. 블러 기반 유사도(

)는 블러된 패치(

)가

와 매칭됨에 따라 높게 나타날 수 있다. 결과적으로, 왜곡 패치(

)는 그들이 잠상의 패치(

)와 유사할 때 수학식 6에서 가중된 평균에 더 많은 기여를 할 수 있다. According to an embodiment of the present invention, a latent patch (latent patch)

Adjoining frames (

From the distortion patch (

) Can be tested. Blur-based similarity (

) Shows a blurred patch (

)end

It may appear higher as it matches. As a result, the distortion patch (

) They are patches of latent images (

) Can contribute more to the weighted average in equation (6).

)를 결정하기 위하여 왜곡 인접 프레임(

)들로부터 N 최적 매칭 패치들을 발견할 수 있다. 여기서,

이다. 또한, 매칭의 정도에 따른

를 사용할 수 있다. Patch set (

To determine the distortion adjacent frame (

N optimal matching patches can be found from here,

to be. Also, depending on the degree of matching

Can be used.

Patches may be selected by Equation 8 below.

에서 매칭 패치(

)를 발견하기 위하여 y의 범위를 탐색하기 위하여, 픽셀 x에 중심을 둔 m×m 윈도우를 사용할 수 있다. 만약

가

로부터

로의 움직임을 정확하게 추정할 수 있다면,

에서 x에 중심을 둔 패치를 사용하여 범위 m을 세팅할 수 있다.

Matching patch (

We can use the m × m window centered on pixel x to search the range of y to find if

end

from

If you can accurately estimate the movement of the furnace,

You can set the range m using a patch centered on x.

However, due to parallax and the movement of objects, the actual movement between frames is generally more complicated than single homography. Thus, using more than one value for the search range m can compensate for errors in the motion model using homography. For example, m = 21 may be used in an embodiment of the present invention.

)를 위하여 M=5로 세팅할 수 있다. 즉, 인접 프레임들과

가 패치 탐색에 포함될 수 있다.
In addition, in an embodiment of the present invention, due to misalignment between N best-matching patches, using a large value of N can lead to over-smoothed deblurred results. Accordingly, according to an embodiment of the present invention, N = 1 may be used for optimal sharpness, and Equation 6 may use an optimally-matched patch to restore a patch of latent image. Also, the time window (

Can be set to M = 5. That is, adjacent frames

May be included in the patch search.

)을 입력 프레임(

)로부터 복원하기 위하여,

안의 각각의 픽셀x에 수학식 6을 사용하여 패치를 디블러링할 수 있으나,

에서 객체 구조들의 조정 불일치를 야기할 수 있다. 따라서, 우리는

에서 디블러링된 패치들을 오버랩핑하는 효과를 병합하기 위하여 패치 기반 텍스트 합성 기법(patch-based texture synthesis approach)을 적용할 수 있다.Latent frame (

) Input frame (

To restore from

You can use the equation (6) for each pixel x inside to deblur the patch,

Can cause mismatching of the object structures. Therefore, we

A patch-based texture synthesis approach can be applied to merge the effects of overlapping the deblurred patches in.

가 픽셀 x에서

의 값이라고 하면,

는 수학식 9로 결정될 수 있다.

Is at pixel x

If the value of

May be determined by Equation 9.

는 x에 중심을 둔 공간 윈도우(nㅧn)에서

픽셀의 세트이고, 디블러링된 패치(

)는 수학식 6으로부터 산출할 수 있다. 또한,

는 x에서의

의 픽셀 값이다. Z는 정규화 팩터이고,

이며,

이다.

는 왜곡 패치(

)에서 x의 픽셀값이다.

에서 모든 픽셀을 위해 디블러링된 패치를 계산한다면, 이미지 경계를 제외한

안의 픽셀 x에 대한

에

픽셀들이 있을 것이다. 따라서, 픽셀 x는 디블러링된

개의 패치에 의해 커버될 수 있다. 또한, 비디오 디블러링을 위하여, 모든 픽셀에 대해서가 아니라 지역적으로 디블러링을 수행할 수 있다.
here,

In the space window centered on x (n ㅧ n)

Is a set of pixels, and a deblurred patch (

) Can be calculated from equation (6). Also,

Is at x

The pixel value of. Z is the normalization factor,

Lt;

to be.

Is a distortion patch (

) Is the pixel value of x.

Computes a deblurred patch for every pixel in the

For pixel x inside

on

There will be pixels. Thus, pixel x is deblured

Can be covered by two patches. In addition, for video deblurring, it is possible to perform deblurring locally instead of for every pixel.

의 패치(

)는 움직이는 객체를 포함할 수 있고, 객체의 모션은 블러 함수(

)와 다른 블러를 초래할 수 있다. 즉, 객체의 모션이 크다면, 객체의 모션이

를 위한 진정 블러 함수를 지배할 수 있다. 이러한 경우, 블러 함수 차이점으로 인하여, 지역 패치 탐색을 통하여 수학식 8에서 정의된 피팅 에러(fitting error)를 가진 매칭 패치를 다른 프레임 (

)에서 발견할 수 없다. The video deblurring method according to an embodiment of the present invention can effectively remove blur due to a moving object through local search of matching patches. However, deblurring may not be performed on objects having a large motion.

Patch of

) Can contain a moving object, and the motion of the object is a blur function (

) And other blurs. In other words, if the motion of the object is large, the motion of the object

Can dominate the true blur function. In this case, due to the blur function difference, a matching patch having a fitting error defined in Equation 8 is applied to another frame through local patch search.

Can not be found in).

는 객체 모션에 의해 이미 블러링되고,

는 동일한 형상을 가진

의 스무드된 버전이기 때문에,

와

사이의 피팅 에러는 상대적으로 작을 수 있다. 따라서,

는 그것 자체를 위하여 최적 매칭 패치가 될 수 있고, 수학식 6에 의해 계산된 잠상의 패치(

)는

와 유사하게 남을 수 있다.
Meanwhile,

Is already blurred by object motion,

Have the same shape

Since it's a smooth version of

Wow

The fitting error in between may be relatively small. therefore,

Can be an optimal matching patch for itself, and the latent image patch (

)

May remain similar to

)를 적용할 수 있다. The video deblurring method according to an embodiment of the present invention uses a new weight value in order to further improve the sharpness of the deblurred frames.

) Can be applied.

The new weight can be obtained by equation (10).

는 러키니스 맵(

)에서 픽셀 y에 중심을 둔 n×n 패치이고,

는 상수이다. 즉, 러키니스값이 작아지면 새로운 가중치도 작아질 수 있다. 러키니스 맵(

)은 왜곡 프레임(

)의 러키니스값들에 의해 구성될 수 있다. 또한, 최적 매칭 패치를 선택하기 위한 새로운 가중치(

)를 적용함에 의해 수학식 8은 수학식 11과 같이 될 수 있다. here,

Is the Luckyness map (

) Is an n × n patch centered on pixel y,

Is a constant. In other words, when the lukiness value decreases, the new weight may also decrease. Luckyness map (

) Is the distortion frame (

It can be configured by the lukiness values of). In addition, new weights for selecting the optimal matching patch (

Equation 8 may be expressed by Equation 11 by applying ().

이다. 예컨대,

를 위하여 작은 값인 0.01을 사용할 수 있다.
here,

to be. for example,

For this reason, a small value of 0.01 can be used.

In addition, the video deblurring method according to an embodiment of the present invention may use the lukiness value to determine the order of frames for performing deblur. First, all frames may be classified based on the lukiness value of the frame. In the case of a lucky frame, most of the pixels may not change after deblur.

For example, as the lukiness value of the frames is lowered, more pixels can be updated by sharper pixels from already processed frames, and the same method as the method of updating the RGB values of the pixels can be used.

As a result, the sharp pixels can be propagated from more lucky to less lucky.

In patch-based texture synthesis and image retargeting, synthesis may be performed iteratively to obtain a more consistent texture and object structure.

That is, the video deblurring method according to the embodiment of the present invention may perform a similar iterative process to improve spatial matching of deblurred frames. Thus, pixel sharpness and object structure in deblurred frames can be enhanced through iteration.

Equation 6 may be modified as shown in Equation 12 to improve deblurring through repetition.

는 p번째 반복을 통하여 복원된 i번째 잠상의 프레임으로

이다.here,

Is the i-th latent frame restored through the p-th iteration

to be.

In addition, Equation 10 may be modified as in Equation 13.

는 블러링된 왜곡 잠상의 프레임(

)에서의 패치이고,

는 p번째 반복으로 업데이트된 러키니스 맵(

)에서의 패치이다. 반복을 통한 개선에 있어서 반복의 횟수는 선명한 프레임들이 서로 얼마나 멀리 분배되어 있는지와 관련된다. 예컨대, 반복의 횟수는 시간 윈도우의 사이즈(2M+1)에 기반하여 결정될 수 있고, 선명한 프레임이 한번의 반복에서 얼마나 멀리 확장될 수 있는지 정의할 수 있다. here,

Is the frame of the blurred distortion latent image (

Is a patch from),

Is the rookieness map updated with the p iteration (

). In improvement through iteration, the number of iterations is related to how far the clear frames are distributed to each other. For example, the number of repetitions can be determined based on the size of the time window (2M + 1) and can define how far a clear frame can be expanded in one repetition.

로 정의할 수 있다. 따라서, 반복의 수는

로 계산될 수 있다.
That is, the maximum distance between two sharp adjacent frames can be found and

. Thus, the number of iterations

Lt; / RTI >

Equation 13 does not include a term related to time matching, but the video generated by repeated deblur matches almost in time. This is because the patch search was limited within a small spatial window after homography registration. Thus, for the same blurred patch in successive frames, the same clear patch can be selected in the local search of matching patches. The iterative process can also improve time consistency by allowing clear patches to propagate through multiple frames.

로서 시간적 윈도우(

)를 제한하고, 수학식 12의 계산을 위하여 세 개의 패치들(

)을 사용할 수 있다. 여기서,

과

은 이전 프레임과 다음 프레임에서 픽셀x의 매칭된 픽셀 위치이다. 예컨대, 에너지 함수에 기반한 수학식 14의 마코브 랜덤 필드(Markov random field)를 최적화함에 의해 주어진 비디오 프레임들(

)로부터 시간적으로 매끄러운 비디오 프레임들(

)을 생성할 수 있다. In addition, in order to improve temporal consistency, after the improvement through the repetition of the deblurred frames, an additional iteration of Equation 12 may be performed. In other words,

As a temporal window (

) And three patches (

) Can be used. here,

and

Is the matched pixel position of pixel x in the previous frame and the next frame. For example, given video frames, by optimizing the Markov random field of equation (14) based on an energy function,

Temporally smooth video frames (

Can be generated.

을 주어진 프레임(

)과 유사하도록 강화하고, 두 번째와 세 번째 항은 업데이트된 프레임들(

)을 그들의 이웃한 프레임들에 가까워지도록 할 수 있다. 결과적으로, 결과 프레임들은 시간 일치성과 마찬가지로 주어진 디블러링된 프레임들(

)에 가까워진다.
The patch correspondences and weights use a gradient descent method that is updated at each gradient descent step. In Equation 14, the first term relates to data accuracy

Given a frame (

), And the second and third terms are updated frames (

) Can be brought close to their neighboring frames. As a result, the resulting frames are given the deblured frames (

Close to)

5 is an exemplary view of an image to which the deblurring method according to an embodiment of the present invention is applied.

5A illustrates input frames, and FIG. 5B illustrates an image to which the video deblurring method according to an embodiment of the present invention is applied. Referring to FIG. 5, although the input frame includes motion blur, it can be seen that a significant portion of the blur has been removed through deblur. In particular, referring to FIG. 5B, it can be seen that the blur generated in the background region is effectively removed. This shows that the video deblurring method according to the embodiment of the present invention can effectively remove the blur of the image due to the motion of the camera.

6 is a block diagram illustrating a video debluring apparatus according to an embodiment of the present invention.

Referring to FIG. 6, the video debluring apparatus 10 according to the embodiment of the present invention includes a motion and motion blur estimator 100 and a deblur 200. Also, the video debluring apparatus according to the embodiment of the present invention may be an apparatus for performing the above-described video deblurring method.

The motion and motion blur estimator 100 estimates a homography function between adjacent frames with respect to each frame of the blur video, and a lukiness indicating the degree of blurring of pixels of each frame. A value calculator 120 and a blur function estimator 130 estimating a motion blur function of each frame based on the estimated homography and lukiness value.

The deblurring unit 200 includes a matching patch selection unit 210 for selecting matching patches for each patch of the blur frame from adjacent frames, and a sharp frame restoration for blending the selected patches to restore the blur frame. The unit 220 and a lukiness update unit 230 for updating the lukiness value of the blur frame according to the reconstructed frame.

The homography estimator 110 may estimate homography based on a mapping between each frame of the input video and pixels constituting adjacent frames of the frame.

The lukiness calculation unit 120 blurs the pixels of the input frame by using a homography function based on a mapping between each frame of the input video and the pixels constituting the adjacent frame. The luckiness value indicating the degree can be calculated.

The blur function estimator 130 may estimate the motion blur function by approximating the shake of the camera by interpolating a homography function between adjacent frames. In addition, by using the calculated lukiness value, a pair of sharp and adjacent blurred frames may be selected, and the exposure time of the frame, that is, the duty cycle may be estimated by comparing the selected frames.

The matching patch selector 210 applies the estimated blur function of each frame to the adjacent frames, and then calculates the matching patch according to the similarity between each patch of each frame and the patch of the adjacent frame to which the blur function is applied. The blur based similarity may be used to find a patch of an adjacent frame most similar to each patch of the input frame as a matching patch. In addition, in order to find a clearer matching patch, a patch of an adjacent frame that maximizes the modified blur based similarity, which further considers the blur based similarity and a weight corresponding to the luckiness value, may be found as the matching patch.

The clear frame restoration unit 220 may restore the clear frame by blending the selected matching patches with respect to each patch of each frame. The blending may calculate the new weight for each matching patch using the calculated blur based similarity or the calculated corrected blur based similarity, and then blend the matching patches using the new weight.

The lukiness updater 230 may calculate an updated lukiness value by blending the lukiness values of the selected matching patch using the weights calculated in the blending step.

In addition, the deblurring unit 200 according to an embodiment of the present invention may improve the deblurring effect by repeatedly performing a predetermined number of times for every frame of the input video.

In addition, since the blurring unit 200 according to the embodiment of the present invention has a difference in the degree of blurring for each input frame, the deblur order for the input frame may be determined based on the lukiness value. For example, the order of deblurring for each input frame may be determined based on an average of the lukiness values for the pixels constituting each of the at least one input frame.

Motion and motion blur estimator 100, deblur 200, homography estimator 110, luckiness value calculator 120, blur function estimator 130, matching patch in the present invention described above Although the selector 210, the clear frame reconstructor 220, and the lukiness updater 230 are started independently of each other, the motion and motion blur estimator 100, the deblur 200, and the homography estimator ( 110, the luckiness value calculator 120, the blur function estimator 130, the matching patch selector 210, the clear frame reconstructor 220, and the luckiness updater 230 are one single type, It can be implemented as one physical device. In addition to this, the motion and motion blur estimator 100, the deblur 200, the homography estimator 110, the lukiness value calculator 120, the blur function estimator 130, the matching patch selector The 210 and the clear frame reconstructor 220 and the Luckyness updater 230 may be implemented as a plurality of physical devices or groups instead of one physical device or group.

The video deblurring method according to the embodiment of the present invention can effectively remove the blur caused by the motion of the camera. In other words, blur along moving objects is not a notable artifact since it generally depends on object motion. In contrast, blur that appears in a slowly changing background can be sensitive to human perception. Therefore, the subjective quality of the video can be improved by effectively removing the blur caused by the motion of the camera using the deblurring method and apparatus according to the embodiment of the present invention.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

10: video deblur device 100: motion and motion blur estimator
200: deblur 110: homography estimation unit
120: lukiness value calculator 130: blur function estimation unit
210: matching patch selection unit 220: clear frame restoration unit
230: Luckyness update

Claims (22)

Estimating motion between frames and motion blur of the frames from the input video through comparison of each frame of the input video with adjacent frames; And
Removing blur of a frame of the input video using the estimated motion and motion blur,
Removing the blur of the frame of the input video,
Selecting matching patches from the adjacent frame for every patch of each frame of the input video; And
Blending the selected matching patches to reconstruct a clear frame. The method of claim 1, wherein estimating motion between frames and motion blur of the frame from the input video comprises:
Estimating a homography function between each frame of the input video and the adjacent frame;
Calculating a lukiness value indicating a degree to which pixels of each frame are blurred using the estimated homography function; And
Estimating a blur function of each frame using the estimated homography function and the calculated lukiness value. delete The method of claim 2, wherein the estimating the blur function of each frame using the estimated homography function and the calculated lukiness value comprises:
And a method for estimating the blur function by approximating a camera shake by interpolating the estimated homography function. The method of claim 2, wherein the estimating the blur function of each frame using the estimated homography function and the calculated lukiness value comprises:
And selecting a pair of clear frames and adjacent blurred frames by using the calculated lukiness value, and comparing the selected pairs of frames to estimate the exposure time of the frames. The method of claim 2, wherein selecting matching patches from the adjacent frame for every patch of each frame of the input video comprises:
After applying the estimated blur function of each frame to adjacent frames corresponding to the respective frames, the blur calculated based on the similarity between the patches of each frame and the patches of the adjacent frames to which the blur function is applied. Video deblurring method using based similarity to find patches of adjacent frame most similar to the patches of each frame as matching patches. The method of claim 6, wherein selecting matching patches from the adjacent frame for every patch of each frame of the input video comprises:
Patches of adjacent frames most similar to patches of each frame are matched to matching patches using a modified blur based similarity calculated by applying the calculated blur based similarity to a weight based on the calculated lukiness value. Find video deblur method. The method of claim 2, wherein blending the selected matching patches to restore a clear frame includes:
And for the patches of each frame, reconstruct the clear frame by blending the selected matching patches. The method of claim 7, wherein blending the selected matching patch to restore a clear frame,
And a matching patch based on the calculated blur based similarity or the calculated modified blur based similarity to the patches of each frame. The method according to claim 2,
And blending the lukiness values of the selected matching patches to update the lukiness value of the reconstructed frame. The method according to claim 2,
And removing the blur of the frame of the input video by using the estimated motion and motion blur a predetermined number of times. The method according to claim 2,
And determining the order of deblurring for the input frame based on the lukiness value. A motion and motion blur estimator for estimating motion between frames and motion blur of the frames from the input video by comparing each frame of the input video with an adjacent frame; And
A deblurring unit configured to remove a blur of a frame of the input video by using the estimated motion and motion blur,
The deblurring unit,
A matching patch selector that selects matching patches from the adjacent frame for every patch of each frame of the input video; And
And a clear frame reconstructing unit which reconstructs a clear frame by blending the selected matching patches. The method of claim 13, wherein the motion and motion blur estimator,
A homography estimator for estimating a homography function between each frame of the input video and the adjacent frame;
A lukiness value calculator configured to calculate a luckiness value indicating a degree of blurring of pixels of each frame by using the estimated homography function; And
And a blur function estimator for estimating a blur function of each frame using the estimated homography function and the calculated lukiness value. delete The method of claim 14, wherein the blur function estimator,
And a video deblur device which estimates the blur function by approximating a camera shake by interpolating the estimated homography function. The method of claim 14, wherein the blur function estimator,
And selecting a pair of clear frames and an adjacent blurred frame by using the calculated lukiness value, and comparing the selected pairs of frames to estimate an exposure time of the frame. The method according to claim 14, The matching patch selector,
After applying the estimated blur function of each frame to adjacent frames corresponding to the respective frames, the blur calculated based on the similarity between the patches of each frame and the patches of the adjacent frames to which the blur function is applied. Video deblurring apparatus using based similarity to find patches of adjacent frame most similar to the patches of each frame as matching patches. The method according to claim 18, The matching patch selector,
Patches of adjacent frames most similar to patches of each frame are matched to matching patches using a modified blur based similarity calculated by applying the calculated blur based similarity to a weight based on the calculated lukiness value. Finding video deblur device. The method of claim 14, wherein the clear frame recovery unit,
And for the patches of each frame, reconstruct the clear frame by blending the selected matching patches. The method of claim 19, wherein the clear frame recovery unit,
And a matching patch based on the calculated blur based similarity or the calculated modified blur based similarity to the patches of each frame. The method according to claim 14,
And a lukiness updater configured to update the lukiness value of the reconstructed frame by blending the lukiness values of the selected matching patch.

Images