KR20150090454A - Method for searching bidirectional motion using multiple frame and image apparatus with the same technique - Google Patents

Abstract

Disclosed are a method for detecting a bidirectional motion by using multiple frames and an image apparatus embedded with the bidirectional motion detection function. The method for detecting a bidirectional motion by using multiple frames according to one embodiment of the present invention comprises the steps of: inputting a first to third frames; calculating a first motion vector through a forward and backward motion detection process from the inputted second to third frames; calculating the first motion vector and the first frame into a second motion vector through the bidirectional motion detection process based on the multi frames; calculating the second motion vector to a third motion vector through a planarization process; and generating a final interpolation frame by applying an overlapped block motion compensation (OBMC) process to the third motion vector.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a bidirectional motion search method using multiple frames and a video device equipped with such a bidirectional motion search function.

One embodiment of the present invention relates to a bidirectional motion search method using multiple frames and a video apparatus equipped with such bidirectional motion search function.

In recent years, much research has been carried out in the field of video coding having a low bandwidth to realize a high picture quality in a throbbing screen such as sports. In a device using a low bandwidth, since a limited amount of data is transmitted, no frame is transmitted in the encoding process, and a lost frame is restored in the decoding process. Therefore, the frame rate up-conversion (FRUC) technique for recovering lost frames is very much interested in various consumer fields such as a multimedia system from PDA to HDTV, which is a CIF class display device.

The frame rate increase conversion scheme can be roughly divided into a pixel-based algorithm and a block-based algorithm. First, pixel-based algorithms include frame repetition, frame averaging, and linear frame interpolation. Since the above techniques do not consider a motion vector, the computational complexity is low and the interpolation is easy. However, if the motion of the object is large, the two objects overlap and blur the motion blurriness.

In order to solve this problem, a motion compensation FRUC (MC-FRUC) technique considering a motion vector, which is a block-based algorithm, has been proposed. Motion-compensated frame interpolation (MCFI), which is a basic algorithm of MC-FRUC, is a method in which the intermediate value of a motion vector obtained through motion search between a current frame and a previous frame is applied to a frame to be interpolated and interpolated.

However, using this motion vector without postprocessing results in holes and overlapped phenomena. Also, since the frame is interpolated in units of blocks without considering the boundary between the block and the block, the blocking phenomenon occurs in the interpolated image.

In order to solve these problems, a bidirectional search method and overlapped block motion compensation (OBMC) have been devised.

First, in the bi-directional search method, a motion search is performed by applying an intermediate value of an initial motion vector obtained in a single direction to a bi-directional frame. OBMC is a method of performing interpolation by changing motion vector values along the boundary of neighboring blocks in order to prevent blocking phenomenon occurring in MCFI, and obtains soft effect between blocks and blocks. However, if motion search is performed using bidirectional symmetry, if a wrong motion vector is found in a motion image having a large motion, the image quality deteriorates.

The present invention provides a bidirectional motion search method using multiple frames capable of finding a more accurate motion vector using a block-based algorithm, and an image device equipped with such a bidirectional motion search function.

An embodiment of the present invention includes: inputting first through third frames; calculating a first motion vector through forward and backward motion search processes from input second and third frames; Calculating a first motion vector as a second motion vector through a multi-frame based bidirectional motion search process, calculating the second motion vector as a third motion vector through a smoothing process, And generating a final interpolated frame by applying an overlapped block motion compensation (OBMC) process.

In this case, the first to third frames may be selected from multiple frames including first to n-th frames.

In addition, the first through third frames may be in a time-sequential order, as they progress to the first frame, and may be sequentially as they progress to the third frame.

The calculating of the first motion vector through the forward and backward motion search processes from the input second and third frames may include calculating sum of absolute differences (SAD, sum) of a block matching algorithm (BMA) of absolute difference,

The step of calculating the first motion vector and the first frame as a second motion vector through a bidirectional motion search process based on a multi-frame based on Equation (1) may include calculating a sum of bidirectional absolute differences in the forward direction and the backward direction , sum of bidirectional absolute difference).

---- (1)

---- (One)

Where Bi, Bj are the positions of the blocks, dx and dy are the initial motion vectors, P ₁ and P ₂ are the values obtained through the SBAD using forward motion vectors, P ₃ and P ₄ are obtained through SBAD using the motion vector in the reverse direction.)

The step of calculating the first motion vector and the first frame as a second motion vector through a multi-frame based bidirectional motion search process may include calculating a sum of bidirectional absolute difference (SBAD) And calculating a second block having a smaller value as the second motion vector.

The step of calculating the second motion vector as a third motion vector through the smoothing process may include calculating a motion vector of a neighboring block having a smaller sum of bidirectional absolute differences (SBAD) based on the block calculated with the second motion vector, (DFD) (displaced frame difference) as shown in the following equation (2) May be included.

---- (2)

---- (2)

의 x축과 y축 성분을 나타내며, B는 중심 블록을 나타내고, f(n-1)과 f(n)은 제2 프레임과 제3 프레임의 밝기 값인 휘도 성분을 나타냄.)(Where v _x, v _y are the second motion vectors obtained through bi-

And f (n-1) and f (n) represent brightness components, which are the brightness values of the second frame and the third frame, respectively.

In addition, in the step of calculating the third motion vector as a third motion vector through the smoothing process, the SBAD of neighboring blocks of the block calculated with the second motion vector, based on the SBAD of the block calculated with the second motion vector, Is greater than the threshold value, the second motion vector may be the third motion vector.

According to another embodiment of the present invention, there is provided a video device equipped with a bi-directional motion search function using the above-described multiple frames.

According to an embodiment of the present invention, a bi-directional motion search method using multiple frames capable of finding a more accurate motion vector using a block-based algorithm and an image device equipped with such bidirectional motion search function can be provided .

1 is a flowchart illustrating a bi-directional motion search method using multiple frames according to an embodiment of the present invention.
FIGS. 2A and 2B are diagrams illustrating motion search for forward and backward directions in the bidirectional motion search method using multiple frames according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating a bidirectional search using a triple frame in a bi-directional motion search method using multiple frames according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating motion vector planarization using a threshold in a bidirectional motion search method using multiple frames according to an embodiment of the present invention. Referring to FIG.
5 is a PSNR for the direction motion prediction result for the Ice image in Table 1. [
FIG. 6 is a PSNR for the direction motion prediction result for the Foreman image in Table 1. FIG.

Hereinafter, embodiments of the present invention will be described in detail. However, the present invention is not limited thereto, and the present invention is only defined by the scope of the following claims.

First, an embodiment of the present invention includes a step of inputting first through third frames (S100), a step of calculating a first motion vector through forward and backward motion search processes from the input second and third frames S200), the first motion vector and the first frame are calculated as a second motion vector through a multi-frame based bidirectional motion search process (S300), the second motion vector is subjected to a flattening process to obtain a third motion vector (S500) of generating a final interpolated frame by applying an overlapped block motion compensation (OBMC) process to the third motion vector (S500) Provides a search method.

1 is a flowchart illustrating a bi-directional motion search method using multiple frames according to an embodiment of the present invention.

는 각각 차례대로 현재 프레임, 보간 될 프레임, 현재 프레임의 이전 프레임(n-1번째 프레임), n-1번째 프레임의 이전 프레임(n-2번째 프레임)을 말하며, MMV(modified motion vector)는 제2 움직임 벡터, 즉, 수정된 움직임 벡터를 뜻하고, FMV(final motion vector)는 제3 움직임 벡터, 즉, 최종 움직임 벡터를 뜻한다. Referring to Figure 1,

(N-2) th frame of the (n-1) th frame, and the MMV (modified motion vector) of the previous frame 2 motion vector, i.e., a modified motion vector, and a final motion vector (FMV) refers to a third motion vector, i.e., a final motion vector.

First, an image frame composed of an RGB color model is first converted into a YCbCr color model, and a motion search is performed using Y, which is a brightness value. Then, a first motion vector, i.e., an initial motion vector, is obtained by performing forward and backward motion search using a previous frame (n-1 < th > frame) of the current frame and two current frames (S200).

The obtained first motion vectors are combined with a previous frame (n-2) th frame of the (n-1) th frame to generate a second motion vector, that is, a modified motion vector (S300).

In this case, vectors determined to be erroneous in the second motion vector are derived as a third motion vector, i.e., a final motion vector, through a planarization process described later (S400).

Finally, OBMC is applied to the third motion vector to generate a final interpolated frame (S500).

Hereinafter, a process of performing a bidirectional motion search method using multiple frames according to an embodiment of the present invention will be described in detail with n = 3 as an example.

First, when the first through third frames are input (S100), a first motion vector is calculated through forward and backward motion search processes from the input second and third frames (S200).

One-way motion search can be used to apply motion-compensated frame interpolation (MCFI) through bi-directional motion search. However, if the unidirectional motion search is performed in only one direction, it is difficult to obtain an accurate motion vector for a motion large image.

Accordingly, in one embodiment of the present invention, a motion search is performed for forward and backward directions, and more accurate motion vectors are selected by referring to various motion vectors.

FIGS. 2A and 2B are diagrams illustrating motion search for forward and backward directions in the bidirectional motion search method using multiple frames according to an embodiment of the present invention.

Referring to FIGS. 2A and 2B, a first motion vector (initial motion vector) in each direction is calculated using a sum of absolute difference (SAD) of a block matching algorithm (BMA) .

The forward direction in which the most reliable block is found in the previous frame based on the current frame and the reverse direction in which the most reliable block is found in the current frame based on the previous frame.

Here, the first motion vector is determined as a median value of the motion vector since it is a result of motion search based on the current frame or the previous frame with respect to each direction.

When the first motion vector is calculated, the first motion vector is calculated as a second motion vector through the multiframe-based bidirectional motion search process together with the first frame (S300).

More specifically, since the first motion vector of each block is determined based on the minimum value of the absolute difference sum calculation (SAD) described above, it is difficult to obtain all the accurate motion vectors according to the moving object and the background in the image.

Therefore, it is necessary to correct for a wrong motion vector. As a correction method, for example, bi-directional motion search using symmetry is available.

However, the bidirectional motion search may generate an erroneous motion vector due to the similar characteristics of an image having a large motion or an image existing within one frame.

In order to solve such a problem, the present invention uses bidirectional motion search through multiple frames.

In one embodiment of the present invention, bi-directional search using triple frames using three frames of multiple frames is presented.

FIG. 3 is a diagram illustrating a bidirectional search using a triple frame in a bi-directional motion search method using multiple frames according to an embodiment of the present invention.

Referring to FIG. 3, the bi-directional search through the triple frame additionally includes the previous frame (n-2) frame of the (n-1) th frame.

If the initial motion vector of the forward motion search based on the current frame is + v, the n-2th frame is + 3v, the previous frame is + v, and the current frame is moved by -v around the frame to be interpolated .

On the other hand, if the initial motion vector of the backward motion search based on the previous frame is + v, the block shifted by -3 v, -v of the previous frame, + v of the current frame, do.

Directional motion search is performed by setting the search range again based on the blocks moved by the initial vector. The sum of bidirectional absolute difference (SBAD) is used as the value for utilizing the n-2 th frame and the matching between the blocks as shown in the following equation (1).

---- (1)

---- (One)

Here, Sx and Sy are positions of the bi-directional search range, Bi and Bj represent the positions of blocks, and dx and dy represent initial motion vectors.

P ₁ and P ₂ are values obtained through SBAD using forward motion vectors, and P ₃ and P ₄ are values obtained through SBAD using a reverse motion vector. After calculating the SBAD between blocks, normalization is performed and the minimum SBAD of the block from the forward direction and the reverse direction is compared as shown in Equation (2) below, and a block having a small value is compared with the modified motion vector (MMV, modified motion vector, i.e., a second motion vector.

----- (2)

----- (2)

When the second motion vector is calculated, the second motion vector is calculated as a third motion vector through a smoothing process (S400).

The second motion vectors derived in S300 may include a motion vector that is erroneously estimated by a moving object or a background existing in the image, and this second motion vector causes a phenomenon that is not natural between spatially neighboring blocks .

In order to remove such blocks, a flattening process is applied in the present invention.

More specifically, in one embodiment of the present invention, a frame difference value, which is a displaced frame difference (DFD) as shown in Equation (3) below, is used.

----- (3)

----- (3)

의

축과 y축 성분을 나타내며, B는 중심 블록을 나타내고, f(n-1)과 f(n)은 이전 프레임과 현재 프레임의 밝기 값인 휘도 성분을 나타낸다.Where v _x, v _y are the modified motion vectors obtained through bi-

of

(N-1) and f (n) represent brightness components, which are the brightness values of the previous frame and the current frame, respectively.

The final motion vector (FMV), that is, the third motion vector, performs DFD with the center block and neighboring blocks, and selects the smallest value as the final motion vector. Here, the DFD performs an operation similar to the BMA described above, and derives the FMV by obtaining the DFD in all eight neighboring blocks based on the center block.

However, since this method is also dependent on the DFD, there is still a problem that an erroneous motion vector can be selected. In such a case, a blocking phenomenon or a deterioration phenomenon occurs.

Accordingly, in the present invention, a motion vector that is likely to be distorted by using a threshold value is not referred to.

FIG. 4 is a diagram illustrating motion vector planarization using a threshold in a bidirectional motion search method using multiple frames according to an embodiment of the present invention. Referring to FIG.

Referring to FIG. 4, first, DFD is performed to find neighboring blocks whose SBAD is smaller than a threshold value based on a center block. If the SBAD is greater than the threshold value in all nine blocks, the motion vector of the center block is used as it is.

Here, SBAD is the SBAD of each block that was performed in the bi-directional search. Thus, the final motion vector is determined by excluding the motion vector of the high block from the reference object of the smoothing process and performing it.

According to another embodiment of the present invention, there is provided a video apparatus equipped with a bidirectional motion search function using the above-mentioned multiple frames.

Hereinafter, an experimental example of the present invention will be described. However, the following experimental examples are merely experimental examples of the present invention, and the present invention is not limited to the following experimental examples.

Experimental Example

Way

For example, News, Foreman, Ice, Mobile, City, Football, and the like have been tested in the embodiment of the present invention, and they have CIF (352288) resolution. The block size of the motion search is 16 × 16 and the search range is 16 pixels. The range used in the bi-directional search was the area of ㅁ 2.

In order to evaluate the bidirectional motion search method using multiple frames according to an embodiment of the present invention, the original frame and the interpolated frame are compared and analyzed using a peak signal to noise ratio (PSNR).

result

The average PSNR of interpolated frames for each image was obtained and compared. The experimental algorithm is as follows.

(a) bi-directional motion search + overlapped block motion compensation (OBMC)

(b) Motion search through triple frames + OBMC

(c) bi-directional motion search of the present invention + OBMC

The images used in the experiments were frames 2 to 100. For the experiment, the frame in which the odd-numbered frames are removed from the image sequence was generated, and the same experiment was performed for the even-numbered frames. The PSNR of each image obtained by the experiment was compared with the original image.

PSNR (a) (b) (C) Crew 30.51 30.54 30.81 Foreman 31.53 31.57 31.76 Ice 29.84 29.89 30.40 Mobile 26.54 27.59 27,72 City 30.67 31.07 31.14 Football 22.49 22.49 22.66

(B) is an algorithm for performing motion search through the previous frame and the current frame based on the frame to be interpolated, Frame, and (c) is an algorithm proposed by the present invention.

5 is a PSNR for the direction motion prediction result for the Ice image in Table 1. [ FIG. 6 is a PSNR for the direction motion prediction result for the Foreman image in Table 1. FIG.

Referring to Table 1, FIG. 5, and FIG. 6, according to the bidirectional motion search method using multiple frames according to the present invention, a high PSNR value And it can be confirmed that excellent performance is obtained.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. As will be understood by those skilled in the art. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

Claims (9)

Inputting first to third frames;
Calculating a first motion vector through forward and backward motion search processes from input second and third frames;
Calculating a first motion vector and a first frame as a second motion vector through a multi-frame based bidirectional motion search process;
Calculating the second motion vector as a third motion vector through a smoothing process; And
And generating a final interpolation frame by applying an overlapped block motion compensation (OBMC) process to the third motion vector. The method according to claim 1,
Wherein the first through third frames are selected from multiple frames including first through n-th frames. The method of claim 2,
The method of claim 1, wherein the first through third frames are in a time-sequential order. The first through third frames are sequentially forwarded to the first frame, and then forwarded to the third frame. The method according to claim 1,
Wherein the calculating the first motion vector through the forward and backward motion search processes from the input second and third frames comprises:
And performing a sum of absolute difference (SAD) of a block matching algorithm (BMA). The method according to claim 1,
The step of calculating the first motion vector and the first frame as a second motion vector through a multi-frame based bidirectional motion search includes:
And calculating a sum of bidirectional absolute difference (SBAD) between forward and backward directions through the following equation (1).

----- (One)
Where Bi, Bj are the positions of the blocks, dx and dy are the initial motion vectors, P ₁ and P ₂ are the values obtained through the SBAD using forward motion vectors, P ₃ and P ₄ are values obtained through SBAD using a motion vector in the opposite direction) The method of claim 5,
The step of calculating the first motion vector and the first frame as a second motion vector through a multi-frame based bidirectional motion search includes:
Further comprising calculating a second block having a smaller value by comparing the sum of bidirectional absolute difference (SBAD) of the forward and backward directions with the second motion vector. Search method. The method of claim 6,
The step of calculating the third motion vector as a third motion vector through the smoothing process may include:
Calculating a displaced frame difference (DFD) according to the following equation (2) for neighboring blocks whose sum of bidirectional absolute differences (SBAD) is smaller than a threshold based on the block calculated with the second motion vector Wherein the motion estimation method comprises the steps of:

----- (2)
(Where v _x, v _y are the second motion vectors obtained through bi-

And f (n-1) and f (n) represent brightness components, which are the brightness values of the second frame and the third frame, respectively. The method of claim 6,
In the step of calculating the third motion vector as a third motion vector through the smoothing process,
And the second motion vector is a third motion vector when the SBAD of the neighboring blocks of the block calculated with the second motion vector is greater than the threshold value based on the SBAD of the block calculated with the second motion vector. Way motion search using multiple frames. A video apparatus equipped with a bidirectional motion search function using multiple frames according to any one of claims 1 to 8.

Images