KR100961456B1 - Method of motion estimation by using adaptive expanded block based on frame-difference - Google Patents

Abstract

Disclosed is a motion estimation method using an adaptive extension block based on frame difference. A motion estimation block determination step of determining a block requiring motion estimation using a difference value between successive frames; among blocks around blocks requiring motion estimation, neighboring blocks having a high correlation with a block requiring motion estimation need motion estimation. Adaptive Extended Block Motion Estimation based on Frame Difference including adaptive extended block determining step of determining adaptive extended block combined with block and motion vector calculating step of calculating motion vector between successive frames using adaptive extended block Configure the method. Therefore, compared to the conventional methods, it is possible to calculate a motion vector more accurately while having a smaller amount of computation.

Frame interpolation, motion estimation, adaptive, frame difference

Description

METHOD OF MOTION ESTIMATION BY USING ADAPTIVE EXPANDED BLOCK BASED ON FRAME-DIFFERENCE}

The present invention relates to a motion estimation method, and more particularly, to a motion estimation method that can be applied to a frame interpolation technique, which enables calculation of a more accurate motion vector while having a smaller amount of computation than a conventional method.

In today's display market, liquid crystal display (LCD) devices are attracting much attention as technology advances. However, in a hold-type display device in which pixel values are maintained in units of frames, such as a liquid crystal display device, motion blur occurs in a moving image. This phenomenon occurs because pixel values are maintained on a frame-by-frame basis, which causes deterioration of image quality in a moving area of a video. As a technique for reducing such a phenomenon, there is a frame interpolation technique for interpolating and inserting a new frame between existing frames. Frame interpolation reduces the time required to maintain pixel values per frame, thereby reducing motion blur.

1A and 1B are graphs illustrating a change in pixel value of a frame to explain a reduction in screen drag caused by frame interpolation.

Referring to FIG. 1A, a change in a pixel value of a frame unit when frame interpolation is not applied is illustrated. For example, when the moving picture is implemented from the nth frame 101 to the n + 3rd 104 frame, the pixel value is maintained in units of frames. 110 becomes large. That is, when frame interpolation is not applied, an unnatural screen drag phenomenon increases in the case of an image having a large motion.

Referring to FIG. 1B, a change in a pixel value of a frame in a case where frame interpolation is applied is illustrated. For example, an n frame 101-1 interpolated using the n th frame 101 and the n + 1 th frame 102 is interposed between the n th frame 101 and the n + 1 th frame 104. Similarly, the (n + 1) frame 102-1 intervenes between the n + 1 th frame 102 and the n + 2 th frame 103.

As such, if the frame refresh rate has a frequency of 60 Hz in the display apparatus when the frame interpolation illustrated in FIG. 1A is not applied, the display apparatus when the frame interpolation illustrated in FIG. 1B is applied The update rate will have a frequency of 120 Hz. In other words, the frame width up conversion (FRC: frame rate up conversion) is reduced by the frame interpolation (FRC: frame rate up conversion), so that the screen drag phenomenon is eliminated and the motion change is large. In this case, it is possible to output images that are natural and have less eye fatigue.

In this case, there are various methods as the frame interpolation method, but typical methods include a method using an average frame of frames repetitive and continuous frames and a method using motion estimation.

In particular, one of the methods that can reduce the screen drag as much as possible is to interpolate a frame using motion estimation.

Frame interpolation through motion estimation consists of two processes. The first process is to extract motion vectors by performing motion estimation, and the second process is to generate interpolation frames based on the extracted motion vectors. In particular, the most important thing in performing motion estimation for frame interpolation is to find an accurate motion vector. Motion estimation is a process of extracting a motion vector that indicates the block with the lowest error energy in the reference frame for blocks in the current frame in two consecutive frames, which requires a large amount of computation since the error energy is obtained for every position in the search region. do. In addition, when the wrong motion vector is used in the process of generating a new frame, an image break occurs and accurate motion estimation is required.

Existing widely used motion estimation techniques include general block-based global search motion estimation and overlapping block-based motion estimation.

The general block-based full-search motion estimation method is a block-based motion estimation method for all positions in a search range of a previous frame. 2,4, H.261, H.263, H.264, etc. are used in the video standards. That is, the error energy with respect to the candidate blocks in the search region of the previous frame is calculated for the blocks composed of MⅹN pixels of the current frame, and the displacement up to the block with the minimum error energy is obtained as the motion vector. The error energy is generally calculated using a sum of an absolute difference of pixel by pixel (SAD).

On the other hand, the general block-based global search method is easy to implement and is the most basic method, but incorrect motion estimation may be caused for noise or light changes in the image, zoom-in, zoom-out, and panning of the camera. There is a problem.

Next, the overlapped block motion estimation method is a method of performing motion estimation using a block extended than a size of a block to find a motion vector. There is an advantage in that false motion estimation caused by the aforementioned noise or camera panning can be reduced by using a high point, but it is not suitable for real-time implementation because it requires a large amount of computation.

SUMMARY OF THE INVENTION An object of the present invention for solving the above problems is to provide a motion estimation method that is capable of calculating more accurate motion vectors while being suitable for real-time implementation by reducing the amount of computation as compared to conventional motion estimation methods.

According to an aspect of the present invention, there is provided a method of performing motion estimation between consecutive frames for frame interpolation, comprising: a motion estimation block determination step of determining a block requiring motion estimation using a difference value between successive frames; Adaptive expansion block determination step and determining the adaptive expansion block to determine the adaptive expansion block that combines the neighboring blocks having a high correlation with the block requiring the motion estimation among the blocks around the block requiring the motion estimation with the block requiring the motion estimation Provided is a frame difference-based adaptive extended block motion estimation method including a motion vector calculating step of calculating a motion vector between consecutive frames using a type extension block.

Here, in the motion estimation block determination step, determining the block requiring motion estimation may include determining whether the motion estimation is performed using the one block when the sum of the difference values between the frames of the pixels included in the one block is greater than a predetermined threshold. This can be done by judging the necessary blocks.

Here, the adaptive extended block determination step may be performed by combining the blocks requiring the motion estimation with blocks in which the sum of the difference values of the frames included in the block is greater than a predetermined threshold among the blocks around the block requiring the motion estimation. It may be configured to determine the adaptive extension block.

In this case, blocks around the block requiring motion estimation may include up, down, left, and right blocks adjacent to the block requiring motion estimation.

When using the motion estimation method according to the present invention as described above, by reducing the number of blocks that are the object of motion estimation by using the frame difference significantly reduced the amount of computation is suitable for real-time implementation, while using the extended block It is possible to calculate an accurate motion vector even for a moving image.

In particular, the fixed block is not simply used, but the configuration of the extended block is adaptively based on the frame difference of the blocks around the block that is the object of motion estimation. It can bring the effect.

As the amount of computation is reduced, motion estimation suitable for real-time implementation is possible, and thus, a frame rate up conversion display apparatus using a frame interpolation technique can be implemented with a low cost and a relatively simple configuration.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals are used for like elements in describing each drawing.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

2 is a flowchart illustrating a motion estimation method according to the present invention.

Referring to FIG. 2, the motion estimation method according to the present invention may include a motion estimation block determination step S110, an adaptive extended block determination step S120, and a motion vector calculation step S130. Meanwhile, after the steps, the step S140 is performed to construct an interpolated frame using the calculated motion vector.

Hereinafter, each step of configuring the motion estimation method according to the present invention will be described in detail step by step.

1) Motion estimation block determination step (S110)

This step is a step of determining a block requiring motion estimation using the difference between successive frames.

The motion estimation block determination step (S110) may further include, in detail, obtaining a difference value of consecutive frames (S111) and determining a block to perform motion estimation based on the frame difference value (S112). .

As mentioned in the related art, the motion estimation is a process of extracting a motion vector indicating a block having the smallest error energy in a reference frame with respect to blocks in the current frame in two consecutive frames, so that a search area for each block to be a motion estimation target is provided. It requires a lot of computation because the error energy is found for every position in the (search range). Accordingly, the motion estimation method according to the present invention calculates a difference value between successive frames and determines a block to perform motion estimation based on the calculated difference value. That is, since a region in a frame of a video can be classified into a region without a motion and a region with a motion, only a block of a region having a motion is determined as a block to perform motion estimation.

First, in operation S111 of obtaining a difference value between successive frames, a frame difference value is generated by using a difference between pixel values of a current frame and a previous frame to distinguish an area. Equation 1 is an equation that may be used to obtain a difference value between consecutive frames in the motion estimation method according to the present invention.

[Equation 1]

Here, f (x, t) refers to the x-th pixel value of the current (t th) frame, and, f (x, t-1 ) is the mean x-th pixel value of the immediately preceding (t-1 th) frame , FD (x, t) means the difference between the t-1 th frame and the t th frame of the x th pixel.

In the above-described frame difference value, the pixel value of the region without motion has a value of '0' because it is not different from the pixel value of the previous frame. On the contrary, since the pixel of the moving area is different from the pixel value of the previous frame, it has a value of '1' or more (when an absolute value is taken). Based on this, the block having the sum of the frame difference values of the pixels constituting the block is greater than or equal to a predetermined threshold value may be determined as a block in which the motion has occurred, or it may be determined as a block in which the motion does not occur.

Therefore, in the determining of a block to perform motion estimation based on the frame difference value (S112), the block to perform motion estimation may be determined by comparing the above-described block unit total of the frame difference values with a predetermined threshold value. .

Equation 2 is an equation for determining a block to perform motion estimation based on a frame difference value, and a threshold value is used to prevent a block from which motion estimation is incorrectly determined due to noise that may occur in an image. The predetermined value obtained by the experiment can be selected and used.

[Equation 2]

는 소정의 임계값을 의미하고, i는 해당 블록을 지정하는 첫번째 픽셀의 번호를 의미하며, FD(x,t)는 i+x번째 픽셀의 t번째 프레임과 t-1 번째 프레임 간 차이값을 의미하며, FDB(i,t)는 i블록에 포함된 픽셀들의 t번째 프레임과 t-1번째 프레임간 차이값의 총합을 의미한다.From here,

Is a difference value refers to a predetermined threshold value and, i indicates the number of the first pixel that specifies the corresponding block and, FD (x, t) is between i + x-th pixel in the t-th frame and the t-1 th frame FDB (i, t) means the sum of difference values between the t th frame and the t-1 th frame of the pixels included in the i block.

3A and 3B are conceptual views illustrating the processing of the motion estimation block determination step in the motion estimation method according to the present invention.

Referring to FIG. 3A, the movement of an object in a continuous frame is illustrated by overlapping a previous frame with a current frame. That is, the object 310 in the immediately preceding frame illustrates that the position of the object 320 is moved to the right side 330 in the current frame.

Referring to FIG. 3B, the white region 340 is a portion in which a frame difference value of pixels is generated, and the gray blocks 350 are determined to be blocks having motion, and thus are determined to be blocks requiring motion estimation. That is, the frame difference value between the previous frame and the current frame is calculated according to Equation 1 above, and the motion estimation is performed on blocks in which the sum of the frame difference values of the pixels in blocks is larger than the predetermined threshold value according to Equation 2 above. When the blocks to be performed are selected, the blocks requiring the motion estimation may be determined using the gray blocks 350 illustrated in FIG. 3B.

On the other hand, blocks that are not determined to be a motion estimation block are not subject to motion estimation, and thus, for example, the corresponding block may use an immediately preceding frame or a corresponding block of the current frame as it is (S125) to construct an interpolated image. It may be directly input to (S140).

1) Adaptive expansion block determination step (S120)

In this step, among the blocks around the block requiring the motion estimation determined in the motion estimation block determining step (S110), neighboring blocks having a high correlation with the block requiring the motion estimation may be determined to correlate with the block requiring the motion estimation. It is a step of determining an adaptive expansion block in which high surrounding blocks are combined.

Basically, when performing motion estimation, it can be assumed that the object with motion has inertia and the moving object is larger than the size of the block. Based on this fact, it can be seen that the movement of neighboring blocks has a high correlation with the movement of the current block. However, it should not be used when performing motion estimation because the correlation between neighboring blocks is low in blocks with moving objects and backgrounds. In the present invention, it is determined whether a neighboring block is a block having a high correlation based on the frame difference, and if it is an available block, it is combined with a block requiring motion estimation to form an extended block. The adaptive extension block thus determined is used to obtain blocks of the previous frame and sum of an absolute difference of pixel by pixel (SAD), and a process of determining a neighboring block to be combined with a block requiring motion estimation is as follows.

First, the sum of the frame difference values of the pixels constituting the blocks with respect to the neighboring blocks of the block on which the current motion estimation is to be performed is compared with a predetermined threshold. Here, the neighboring blocks of the block on which the current motion estimation is to be performed may include blocks located above, below, left, and right of the block.

Second, among the neighboring blocks, a block having a sum total of the frame difference values of pixels constituting the block larger than a predetermined threshold value may be determined as a block having a high correlation with a block to perform current motion estimation. Include it.

4A and 4B are conceptual views illustrating a configuration example of an adaptive extension block in the motion estimation method according to the present invention.

4A and 4B are referenced in parallel with FIG. 3B described above.

When block 1 360 illustrated in FIG. 3B becomes a motion estimation target block, the left block 360-1, the right block 360-2, and the upper block 360-3 of the block 1 360 are blocks. It is determined that the block has a high correlation with one 360 to configure the adaptive extension block 410 as illustrated in FIG. 4A.

Similarly, when block 2 370 illustrated in FIG. 3B becomes a motion estimation target block, the left block 370-1 (that is, block 1 of FIG. 3B) and the upper block 370-3 of block 2 370 are used. ) Is determined to be a block having a high correlation with block 2 370 to form an adaptive extension block 420 as illustrated in FIG. 4B.

2) motion vector calculating step (S130)

This step is a step of calculating a motion vector between consecutive frames using the adaptive extension block determined in the adaptive extension block determination step (S120).

That is, the position having the smallest SAD with respect to the search area is determined using the adaptive expansion block determined in the previous step, and a motion vector is calculated from the displacement.

Subsequently, a step (S140) of generating the interpolated image by rearranging the blocks by using the motion vector calculated in the motion vector calculation step S130 is performed, and a detailed description thereof will be omitted.

Simulation result

For each of the two experimental images, the general global search method described in the prior art, the overlap block-based motion estimation method described in the prior art, and the motion estimation method according to the present invention were applied. It was verified by.

Qualitative determination of whether accurate motion estimation was made was verified by comparing the frame interpolated images through the motion estimation performed by the above-described methods.

In addition, verification of the amount of calculation is possible by comparing the number of blocks and the total number of blocks that are not subject to the calculation when the present invention is applied.

On the other hand, the quantitative evaluation of the results was performed by comparing the Peak to Signal Noise Ratio (PSNR) value applied to each result image. The PSNR value means the sum of the differences between the pixels of the input image and all the pixels after the calculation. However, since PSNR is a result of the error contained in the whole image, the result of the visual judgment and the PSNR value may be different, so the result of PSNR is not an absolute value. The higher the PSNR value, the closer the result to the original image can be determined, but generally, the result is judged to be a reliable image for 30dB or more. The PSNR equation used is shown in Equation 3 below.

&Quot; (3) "

Where FS is the size of the image, f _n (x) is the pixel value of the original image, and f _i (x) is the pixel value of the interpolated image.

1) Experiment Video 1

5A through 5D are image examples for explaining a simulation result for the experimental image 1. Referring to FIG.

FIG. 5A illustrates a current frame as an original image of experimental image 1 used in a simulation. 5B to 5D illustrate interpolation images calculated by using the current frame and the previous frame in the case of using the conventional global search method, the overlap block-based motion estimation method, and the motion estimation method according to the present invention, respectively.

Experimental image 1 has a resolution of 640 × 352, basic block size of 8 × 8, and search area of 16 × 16.

First, in the part of qualitative verification, in the conventional simple global search method (FIG. 5B) and the overlapping block-based motion estimation method (FIG. 5C), the boundary parts 512 and 513 of the human motion and the human hair part 511 and the like are used. In contrast, an incorrect motion vector is calculated and artifacts are generated, whereas in the motion estimation method (FIG. 5D) according to the present invention, an accurate motion vector is calculated and the artifact is not present.

Table 1 summarizes the average PSNR and the calculation amount calculated for each of three frames constituting the images produced by the three methods.

TABLE 1

PSNR (dB) Calculation 1 ^st frame 2 ^nd frame 3 ^rd frame Average PSNR (dB) Simple Global Search Method 41.477 42.003 41.628 42.597 100 (ref) Nested Block Base 39.113 40.118 39.646 40.656 100 Invention 41.046 41.752 41.286 42.278 46.64

Referring to Table 1, the average PSNR according to the present invention is 42.278 dB, which is about 2 dB higher than the overlap block-based motion estimation, but 0.3 dB lower than the simple global search method. The image using the method according to the present invention shows an average PSNR of about 2 dB higher than the overlapping block-based motion estimation, but it can be confirmed that the interpolated image can be sufficiently reliable with a value higher than the reference 30 dB.

On the other hand, while the conventional two methods perform motion estimation for all the blocks in terms of the amount of computation, in the present invention, about 46.64% of the blocks are compared to the conventional methods by reducing the number of motion estimation target blocks based on the frame difference value. We can estimate the amount of computation reduction by about 53% by performing the motion estimation only for these fields.

2) Experiment video 2

6A to 6D are examples of images for explaining a simulation result for experimental image 2. FIG.

FIG. 6A illustrates the current frame as the original image of Experimental Image 2 used for the simulation. 6B to 6D illustrate interpolation images calculated by using the current frame and the previous frame when the conventional global detection method, the superposition block-based motion estimation method, and the motion estimation method according to the present invention are used, respectively.

Experimental image 2 has a resolution of 720 × 304, a basic block size of 8 × 8, and a search area of 16 × 16. Experimental image 2, unlike the previous experimental image 1, is characterized in that the movement of the camera panning from the right to the left (panning) and at the same time a person moves.

Table 2 summarizes the average PSNR and the calculation amount calculated for each of three frames constituting the images produced by the three methods.

TABLE 2

PSNR (dB) Calculation 1 ^st frame 2 ^nd frame 3 ^rd frame Average PSNR (dB) Simple Global Search Method 35.167 35.141 33.718 34.675 100 (ref) Nested Block Base 31.637 31.821 30.663 31.373 100 Invention 33.488 33.514 32.368 33.123 84.05

 The experimental results of Experiment 2 show that the average PSNR of the proposed method is 33.123dB, the overlapping block motion estimation is 31.373dB, and the global search method has a PSNR of 34.675dB. Since the PSNR of the image obtained by the proposed method is higher than the reference PSNR of 30dB, it can be determined as a reliable image. Also, visual comparisons of FIGS. 6B to 6D show that there are fewer artifacts than conventional methods.

In the case of the moving image of the camera as shown in the experimental image 2, since the difference value of the frame increases on average, the number of blocks that are not subject to the motion estimation becomes smaller than the experimental image 1. However, the complexity of the proposed method is 84.05%, which is about 16% less than the existing methods, and it can be seen that the computational advantage is maintained over the existing methods.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

1A and 1B are graphs illustrating a change in pixel value of a frame to explain a reduction in screen drag caused by frame interpolation.

2 is a flowchart illustrating a motion estimation method according to the present invention.

3A and 3B are conceptual views illustrating the processing of the motion estimation block determination step in the motion estimation method according to the present invention.

4A and 4B are conceptual views illustrating a configuration example of an adaptive extension block in the motion estimation method according to the present invention.

5A through 5D are image examples for explaining a simulation result for the experimental image 1. Referring to FIG.

6A to 6D are examples of images for explaining a simulation result for experimental image 2. FIG.

Claims (6)

A method for performing motion estimation between consecutive frames for frame interpolation, A motion estimation block determination step of determining a block requiring motion estimation using a difference value between successive frames; Adaptive expansion block that determines an adaptive expansion block by combining blocks having a motion threshold that is greater than a predetermined threshold sum of interframe difference values of pixels included in the block around the block requiring motion estimation with the block requiring motion estimation. Determining step; And And a motion vector calculating step of calculating a motion vector between successive frames by using the adaptive extension block. The method of claim 1, In the motion estimation block determination step, The frame difference-based adaptive extended block motion estimation method of calculating the difference value between the consecutive frames is calculated using the following equation.

Where f (x, t) is the pixel value of the current frame, f (x, t-1) is the pixel value of the previous frame, and FD (x, t) is the t of the xth pixel. The difference between the 1 st frame and the t-1 th frame) The method of claim 1, In the step of determining the motion estimation block, the determining of the block requiring motion estimation may include converting the one block into a block requiring motion estimation when the total difference value between the frames of the pixels included in the one block is greater than a predetermined threshold. The frame difference based adaptive extended block motion estimation method, characterized in that the determination.  The method of claim 3, wherein The frame difference-based adaptive extended block motion estimation method of determining a block requiring motion estimation is performed by using the following equation.

(From here,

Denotes a predetermined threshold, i denotes the number of the first pixel designating the block, and FD (x, t) denotes the difference between the t-th frame and the t-1th frame of the i + x-th pixel. FDB (i, t) means the sum of the difference values between the t th frame and the t-1 th frame of the pixels included in the i block.) delete The method of claim 1, And the neighboring blocks include up, down, left, and right blocks adjacent to the block requiring the motion estimation.

Images