KR101709894B1 - Method for frame rate up-conversion considering the direction and magnitude of motion vectors and apparatus using the same - Google Patents

Abstract

A frame rate increasing method considering the direction and magnitude of motion vectors and an apparatus using the same are disclosed. The method of increasing a frame rate may include: determining whether a previous frame or a current frame is a dynamic frame or a static frame using an average motion vector magnitude in at least one direction of a current frame and a previous frame after bidirectional motion prediction; Directional motion prediction for a previous frame or a current frame, the direction of motion vectors detected in each of the sub-pixels in the mask of a certain size among the sub-pixels of the block generated in the extended bidirectional motion prediction, Determining whether the sizes of the motion vectors are the same within a predetermined error range if the directions are the same, and if the sizes are the same, generating between the previous frame and the current frame To be a block of intermediate frames And omitting motion vector smoothing for the motion vector.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method of increasing a frame rate considering a direction and a size of motion vectors, and a device using the same.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a frame rate increase conversion technique for a high quality image, and more particularly, to a frame rate increase method considering a direction and a size of motion vectors and an apparatus using the same.

Recently UHD TV (UltraHigh Definition TV) has been popularized and the price has become cheaper, and as the penetration rate has increased, the use of high quality video is increasing. However, as the resolution of the UHD TV increases and the size increases, the frequency becomes lower and the screen flickers.

The Extended Bilateral Motion Estimation (EBME) algorithm performs a BME (Bilateral Motion Estimation) once, then moves a half of the size of the block in the horizontal axis and the vertical axis of the image in the grid of the next block to form a new block, PSNR is higher than BME because it performs bidirectional motion prediction and finds a more accurate motion vector in the overlapped region.

However, the Extended Bilateral Motion Estimation (EBME) algorithm requires a high computation amount because it performs BME twice and repeats until all the ideal vectors are corrected in the subsequent MVS (Motion Vector Smoothing) step.

SUMMARY OF THE INVENTION It is an object of the present invention to solve the above problems and to provide a method and apparatus for determining whether or not to perform Extended Bilateral Motion Estimation (EBME), to reduce a computation amount of a system, A method of increasing the rate and an apparatus using the same.

According to an aspect of the present invention, there is provided an apparatus for increasing a frame rate in consideration of a direction and a size of motion vectors in an Extended Bilateral Motion Estimation (EBME) algorithm.

In this embodiment, dynamic frames and static frames can be classified using the average motion vector magnitudes in the x and y directions in the frame to reduce the amount of computation of EBME. In the case of a dynamic frame, EBME for finding an accurate vector is used because of a large change in a motion vector. In the case of a static frame, a BME having a low computation amount due to a small change in motion vector can be used.

In order to reduce the amount of computation in the motion vector smoothing step, the same direction and size of the motion vectors may be determined, and the same motion vectors may be passed through the motion vector smoothing step. To this end, the frame rate increasing method determines the directionality of the motion vectors of the original block or neighboring blocks of the current block, and when the neighboring motion vectors have the same direction and the neighboring motion vectors have a size difference of less than a certain size, It is possible to determine whether the motion vectors are the same or not.

According to another aspect of the present invention, there is provided a motion estimation method for estimating a motion of a block in a current frame and a current frame using a mean motion vector magnitude in at least one direction in a current frame and a previous frame, Determining whether the frame is a dynamic frame or a static frame; Performing extended bidirectional motion prediction for a previous frame or a current frame if it is determined to be a dynamic frame; Determining whether directions of motion vectors searched in each of a plurality of sub-pixels in a mask of a certain size among sub-pixels of a block generated in extended bidirectional motion prediction are the same in any one of predetermined direction ranges; Determining whether the sizes of the motion vectors are the same within a predetermined error range if the directions are the same; And omitting motion vector smoothing for a block of an intermediate frame to be generated between a previous frame and a current frame if the sizes are equal.

Here, the frame rate increasing method may further include modifying the motion vectors of the prediction units of the previous frame or the current frame using a filter if the determination result in the step of determining whether the same direction is not the same direction can do.

Here, the frame rate increasing method may further include modifying the motion vectors of the prediction units of the previous frame or the current frame using a filter if the determination result in the step of determining whether the same size is not the same size can do.

Here, the frame rate increasing method may be performed after the modifying step, or when the determination result in the step of determining whether the same size is the same size, And performing superposed block motion compensation.

Here, the frame rate increasing method may include performing motion vector smoothing on the motion vectors of the prediction units of the previous frame or the current frame if the determination result in the determination of the dynamic frame or the static frame is a static frame As shown in FIG.

Here, the frame rate increasing method may further include a step of performing a motion vector smoothing after performing the overlapping block motion compensation, or after performing the motion vector smoothing, And assigning a high weight to a frame having a large number of regions.

Here, the size of the mask may be 3x3 as a pixel unit.

According to still another aspect of the present invention, there is provided a method for decoding a current frame or a current frame by using an average motion vector magnitude of blocks of a previous frame and a current frame in at least one direction of a current frame and a previous frame, Determining whether the frame is a dynamic frame or a static frame; And performing a motion vector smoothing on the motion vectors of the prediction units of the previous frame or the current frame if the result in the determining step is a static frame.

Here, the frame rate increasing method may further include performing extended bidirectional motion prediction for prediction units of a previous frame or a current frame if the determination result in the step of determining whether the frame is a dynamic frame or a static frame is a static frame .

Here, the frame rate increasing method may further include, after performing the extended bidirectional motion prediction, determining a direction of motion vectors that are searched in each of a plurality of regions in a mask of a predetermined size among sub-prediction units generated in the extended bidirectional motion prediction, Determining which one of the direction ranges is in the same direction; Determining whether the magnitudes of the motion vectors are the same within a predetermined error range if the determination result is in the same direction; And omitting the motion vector smoothing for the previous frame or the current frame if the determination result is the same size.

Here, the frame rate increasing method may further include modifying the motion vectors of the prediction units of the previous frame or the current frame using a filter if the determination result in the step of determining whether the same direction is not the same direction can do.

Here, the frame rate increasing method may further include modifying the motion vectors of the prediction units of the previous frame or the current frame using a filter if the determination result in the step of determining whether the same size is not the same size can do.

Here, if the determination result in the step of determining whether the size is equal to or after the modifying step is equal to the same size, the method of increasing the frame rate may be such that the superimposed block motion compensation for the previous frame or the intra- The method comprising the steps of:

Here, the step of determining whether the size is the same may determine whether the size of the motion vector is smaller than 0.3.

Here, the step of determining whether the frame is a dynamic frame or a static frame may determine whether an average motion vector magnitude is greater than 0.6.

According to another aspect of the present invention, there is provided an image encoding apparatus for encoding an input image through temporal prediction, spatial prediction, transcoding, quantization, and entropy coding, Memory to store code; And a processor connected to the memory and executing the program.

Here, the processor may determine, by a program for frame rate conversion, whether a previous frame or a current frame, using the average motion vector magnitudes of the current frame and the blocks of the previous frame in at least one direction of the current frame, Directional motion prediction is performed for prediction units of a previous frame or a current frame, and if the determination result is a dynamic frame, it is determined whether or not the dynamic prediction frame is a dynamic frame or a static frame. Size of the motion vectors are determined to be the same direction in the predetermined direction ranges, and if the determination result is the same direction, it is determined that the magnitudes of the motion vectors are within a predetermined error range Same size And if the determination result is the same size, the motion vector smoothing for the current frame can be omitted.

Here, if the determination result of the same direction is not the same direction, or if the determination result of the same magnitude is not the same magnitude, the processor may use motion vectors of prediction units of the previous frame or the current frame Can be modified.

Here, the processor may perform the superimposed block motion compensation on the previous frame or blocks in the current frame before generating the intermediate frame, if the determination result of the same size or after the modification using the filter is found to be the same size have.

Here, the processor may perform motion vector smoothing on the motion vectors of the prediction units of the previous frame or the current frame if the determination result of the dynamic frame or the static frame is output as a static frame.

Here, the processor performs a superimposed block motion compensation or motion vector smoothing, and then, before generating the intermediate frame, the processor uses a motion compensation to give a high weight to a frame in which a region of the previous frame matches the current frame, can do.

When the frame rate increasing method and the apparatus using the frame rate increasing method considering the direction and size of the motion vectors according to the embodiment of the present invention as described above are used, the amount of computation of the system for image encoding and / or decoding is reduced, Can be improved.

That is, according to the present invention, it is possible to reduce the amount of computation of the system by determining whether to perform Extended Bilateral Motion Estimation (EBME) and reducing the number of execution of EBME. In addition, the amount of computation of the system can be reduced by determining whether to perform the MVS (Motion Vector Smoothing) step based on the direction and size of the motion vectors.

In addition, according to the present invention, performance and efficiency of the system can be improved by comprehensively determining whether to perform EBME and MVS. That is, the execution time is shortened and the peak signal to noise ratio (PSNR) can be improved as compared with the system using the existing EBME algorithm.

1 is an exemplary diagram for explaining a bidirectional motion prediction process for an original block that can be employed in the frame rate increasing method according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining an overlapping lattice motion vector selection process in Extended Bilateral Motion Estimation (EBME) that can be employed in the frame rate increasing method according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating an MVS (Motion Vector Smoothing) process that can be employed in the frame rate increasing method according to an embodiment of the present invention.
4 is an exemplary diagram for explaining the progress of the SOAD _CC and the SOAD _CP that can be employed in the frame rate increasing method according to an embodiment of the present invention.
5 is a flowchart illustrating a frame rate increasing method considering the direction and size of motion vectors according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining a direction determination process of motion vectors in the frame rate increasing method of FIG.
FIG. 7 is a diagram for explaining a process of determining the same direction in the motion vector determination process of FIG.
8 is a block diagram of an image encoding apparatus according to another embodiment of the present invention.
9 is a schematic block diagram of an image encoding apparatus according to another embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like are used to specify that there is a stated feature, number, step, operation, element, part or combination thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless there is a misunderstanding in the present specification, when a subscript of a character is included, subscripts can be expressed in the same size or form as ordinary characters for convenience of display.

Unless otherwise defined herein, all terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be construed in a manner consistent with the meaning in the context of the relevant art and are not to be construed as ideal or overly formal, unless explicitly defined herein.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram for explaining a bidirectional motion prediction process for an original block that can be employed in the frame rate increasing method according to an embodiment of the present invention.

Referring to FIG. 1, the bilateral motion estimation (BME) employed in the frame rate increasing method according to the present embodiment includes an arrangement of a previous frame 11 and a current frame 12 The motion vector is searched using the symmetry from the current frame 12 and the previous frame 13 with reference to the block 15 of the interpolated frame 11. Here, the interpolation frame 11 may be a frame to be interpolated by bidirectional motion prediction.

The original blocks R1, R2 are respectively searched on the search range D in the current frame 12 and the previous frame 13 corresponding to the specific block 15 of the interpolation frame 11 . The original block R1 and the new block 16 of the current frame 12 can be searched in the BME and the original block R2 and the new block 17 of the previous frame 13 can be searched. At this time, an overlap area may occur between the original block and the new block. That is, the new block may be a block that does not have an overlap region with the original block, or an overlapped block (B2) having an overlap region. The original block or the new block may be represented or referred to as a forward motion vector 18 and a backward motion vector 19 based on the block 15 of the interpolation frame 11.

The frame rate increasing method of the present embodiment can predict the motion of the original block R2 using Equations (1) and (2). Further, in order to obtain an accurate motion vector, the overlap block B2 is further defined as a position (for example, block 17) at which the SBAD (Sum of Bilateral Absolute Differential) value becomes minimum using Equations (1) and .

는 움직임 벡터 후보를 나타내며,

은 이전 프레임,

은 현재 프레임을 나타낸다.In the above equations (1) and (2)

Denotes a motion vector candidate,

Lt; / RTI >

Represents the current frame.

Next, a motion vector can be compared with an overlapped block (B2) including an area overlapping the original block (R2).

According to this embodiment, unlike the unidirectional prediction in which a hole and an overlapped region are generated in the interpolation frame generation process, a motion vector is sequentially obtained on a block-by-block basis with respect to an interpolation frame so that holes and overlapping regions may not occur, So that the motion vector can be precisely searched.

FIG. 2 is a diagram for explaining an overlapping lattice motion vector selection process in Extended Bilateral Motion Estimation (EBME) that can be employed in the frame rate increasing method according to an embodiment of the present invention.

Referring to FIG. 2, a plurality of blocks (sub-blocks) in the left block VF1 may have a 16x16 size unit. The frame rate search method can search a motion vector by performing BME on each of the sub-blocks. The nine motion vectors searched are indicated by arrows in each sub-block.

The intermediate block VF1 forms a superimposed area with a new block VF2 obtained by performing BME once and then moving the grid of the next block to the horizontal axis and the vertical axis of the image by half the length of one side of the subblock. That is, the frame rate increasing method may generate a block VF2 that partially overlaps a previous block having a vertical boundary G1 between subblocks and another vertical boundary G2. Moving vectors found in the four sub-blocks of the block VF2 are indicated by arrows in each sub-block.

If extended bidirectional motion prediction is performed on a block-by-block basis over the entire search area of the interpolation frame, that is, subdividing each sub-block into four secondary sub-blocks and subdividing the right block of FIG. 2 The motion vector of each of the secondary subblocks can be searched. The secondary subblock may have an 8x8 size unit.

According to the present embodiment, Extended Bilateral Motion Estimation (EBME) searches motion vectors of the current block through BME, and then divides the grid of the next block into half The motion vector of each sub-block is searched for and the bidirectional motion prediction can be performed again for each sub-block (secondary sub-block) of the new block VF3 through the repetition of this process, The motion vector can be precisely searched.

That is, according to the present embodiment, the motion vectors of the original blocks are obtained by performing the bidirectional motion prediction once, and then an overlap block, which is a new block shifted by half in the horizontal axis and the vertical axis direction, is formed in the original block grid. Prediction can be performed to obtain overlapping motion vectors. Then, the motion vectors can be compared with respect to overlapping areas of the original block and the overlap block. Each block can be modified to a minimum motion vector by dividing the motion vector obtained by the 16 × 16 size unit into four 8 × 8 unit blocks and comparing the SBAD with the overlapping portion.

Meanwhile, the frame rate increasing method according to the present embodiment may selectively perform or omit MVS (motion vector smoothing) in addition to BME and EBME. Hereinafter, MVS will be briefly described.

FIG. 3 is a diagram illustrating an MVS (Motion Vector Smoothing) process that can be employed in the frame rate increasing method according to an embodiment of the present invention.

인 경우에 해당할 수 있다.The MVS is intended to correct motion vectors that are erroneously judged in the motion vector finally obtained through the above BME or EBME. That is, there are motion vectors that are judged to be wrong in the finally obtained motion vector. Therefore, the frame rate increasing method of this embodiment can determine the abnormal vector. The ideal vector is calculated using Equation 3

And the like.

)는 정확한 움직임 벡터(correct motion vector)(V₁ 내지 V₈)에 미디언(median) 필터를 적용한 결과를 토대로 수정될 수 있다. 이상 벡터(

)가 한 번의 MVS 수행을 거쳐도 남아있으면, 도 3의 윈도우들(W1→W2→W3)과 같이 다시 동일한 MVS 과정을 반복 수행할 수 있다. 그리고, 각각의 움직임 벡터에 플래그(flag)를 표시하여 flag=1일 때, 이상 벡터로 판단하고, flag=0일 때 정확한 움직임 벡터로 판단할 수 있다.Specifically, as shown in Fig. 3, the ideal vector (

May be modified based on the result of applying a median filter to the correct motion vectors V ₁ through V ₈ . The ideal vector (

) Remains after one MVS operation, the same MVS process can be repeated again as in the windows (W1? W2? W3) of FIG. Then, a flag is displayed in each motion vector, and when flag = 1, it is determined as an ideal vector, and when flag = 0, it can be determined as an accurate motion vector.

은 평균 움직임 벡터를 나타낸다.In Equation 3,

Represents an average motion vector.

Meanwhile, the frame rate increasing method according to the present embodiment may selectively perform MCI (Motion Compensated Interpolation) in addition to performing or omitting optional BME, EBME, and MVS. The MCI finds a frame with a large number of matching regions between the previous frame and the current frame using a weight, and assigns a high weight to the frame. MCI can use SOAD (Sum of Overlapped Area Absolute Difference).

4 is an exemplary diagram for explaining the progress of the SOAD _CC and the SOAD _CP that can be employed in the frame rate increasing method according to an embodiment of the present invention.

)의 오차 크기를 비교할 수 있다.First, as shown in FIG. 4, the frame rate increasing method uses an extended sum of overlapped area absolute difference (SOAD) of Equation (4)

) Can be compared.

는 현재 프레임(B4)과 현재 가상 보간프레임(F2)의 확장된 블록을 나타내고,

는 이전 가상 보간프레임(F4)의 확장된 블록을 나타낸다. 그리고

와

는 확장된 블록(

또는

)의 영역에 대한

와

의 위치를 나타낸다.In Equation (4)

Represents an extended block of the current frame B4 and the current virtual interpolation frame F2,

Represents an extended block of the previous virtual interpolation frame F4. And

Wow

Is an extended block (

or

) For the region of

Wow

.

는 현재 프레임(B4)에서 확장된 블록(

)과 이전 가상 보상 프레임(F4)에서 확장된 블록(

)의 차에 대한 절대값이고,

는 현재 프레임(B4)에서 확장된 블록(

)과 현재 가상 보간프레임(F2)에서 확장된 블록의 차에 대한 절대값이다. 현재 가상 보간프레임(F2)에서 확장된 블록은 현재 프레임(B4)에서 확장된 블록(

)과 동일하거나 미리 정해진 스케일링의 위치나 크기를 가질 수 있다.Referring to Figure 4,

Lt; RTI ID = 0.0 > (B4) < / RTI &

) And the previous block (F4) in the previous virtual compensation frame

), ≪ / RTI >

Lt; RTI ID = 0.0 > (B4) < / RTI &

) And the difference between the expanded blocks in the current virtual interpolation frame F2. The block expanded in the current virtual interpolation frame F2 is expanded in the current frame B4

) Or the position or size of the predetermined scaling.

와

는 현재 프레임(B4)에서 겹쳐지는 영역을 판단하는 값 또는 정보일 수 있다.In the MCI of this embodiment,

Wow

May be a value or information for determining an overlapping area in the current frame B4.

The weighting value? May be changed according to the overlapping area of the current block and the extended block of the previous frame. This is expressed by Equation (5).

는 이전 프레임에서 확장된 블록과 이전 가상 보간프레임(F4)에서 확장된 블록의 차에 대한 절대값이고,

는 이전 프레임에서 확장된 블록과 현재 가상 보간프레임(F2)에서 확장된 블록의 차에 대한 절대값이다.In Equation (5)

Is the absolute value of the difference between the block extended in the previous frame and the block expanded in the previous virtual interpolation frame F4,

Is the absolute value of the difference between the expanded block in the previous frame and the expanded block in the current virtual interpolation frame F2.

The final expression of the interpolation frame to which the weight is applied by the above-described MCI is shown in Equation (6).

는 OBMC(Overlapped Block Motion Compensation)를 위한 함수를 나타낸다. 그리고

은 확장 블록의 이전 프레임을 나타내고,

은 확장 블록의 현재 프레임을 나타낸다.In Equation (6) above,

Represents a function for OBMC (Overlapped Block Motion Compensation). And

Represents the previous frame of the expansion block,

Represents the current frame of the extension block.

On the other hand, as the popularity of UHD (ultrahigh definition) TVs and the demand for high-quality video images increase, the frequency of the device operating in the device becomes relatively low as the resolution of the display device increases and the size increases. A frame rate up-conversion (FRUC) technique may be used to increase the frequency to solve the problem. In this case, by applying the frame rate increasing method according to the present embodiment, the performance and efficiency of image processing can be remarkably improved. Hereinafter, the frame rate increasing conversion algorithm proposed in the present embodiment will be described in detail.

5 is a flowchart illustrating a frame rate increasing method considering the direction and magnitude of motion vectors according to an exemplary embodiment of the present invention.

In the frame rate increasing method according to the present embodiment, it is possible to determine the dynamic frame and the static frame using the average motion vector magnitudes in the x-direction and the y-direction, respectively, and determine whether to perform the EBME. In addition, it is possible to determine the same motion vector in a specific direction by comparing the direction and magnitude of the motion vectors, and determine whether to perform the MVS.

=0.6)를 비교하고, 평균 움직임 벡터 크기가 임계치(

)보다 크다면, 동적 프레임으로 판단하고 임계치보다 작다면 정적 프레임으로 판단할 수 있다.For example, to determine the degree of motion in a frame, the average motion vector magnitude in each of the x and y directions in the frame and the threshold (e.g.,

= 0.6), and when the average motion vector magnitude exceeds the threshold value

), It is determined as a dynamic frame, and if it is smaller than the threshold value, it can be determined as a static frame.

In dynamic frames, EBME is used to find accurate motion vectors because there are many changes in motion vectors, and BME is used but EBME is omitted in static frames because there is little change in motion vectors. Since the motion blur phenomenon of the static frame is small, interpolation (MCI) using motion compensation can be performed without going through the overlapping block motion compensation (OBMC).

The frame rate increasing method of this embodiment can perform the following series of steps to generate an interpolated frame, a compensated frame, or an intermediate frame using the previous frame and the current frame, as shown in FIG. 5 (S50). The frame rate increasing method can be performed in a frame rate converting apparatus, a video encoding apparatus, or a computing apparatus having an image processing function, which implement this method. A frame rate increasing device, and the image encoding device or the computing device may include a frame rate conversion device.

First, the frame rate conversion apparatus can perform bi-directional motion estimation (BME) using a previous frame and a current frame in an image encoding process or the like (S51).

Next, the frame rate conversion apparatus determines whether or not a specific block of the interpolation frame has the average motion vector magnitude ( _xmag ) in the x-direction of the previous frame or the current frame is larger than the first threshold value (T1) It is possible to determine whether the average motion vector magnitude (y _mag ) in the direction is greater than the first threshold value T1 (S52).

In the present embodiment, the first threshold value T1 is a measure of the identity of a motion vector direction, and is preferably set to 0.6 as a result of experiments under various environments and conditions. Here, T1 may have a value of not less than 0 and not more than 1.0. When T1 = 1.0, the directions of the motion vectors are all included in one quadrant of one of the quadrants, and the maximum angle that can be formed by the directions of the motion vectors when T1 = 0.5 includes a right angle, T1 = 0 The directions of the motion vectors may all include the same direction or one direction.

As a result of the determination in step S52, if yes, extended bidirectional motion estimation (EBME) may be performed (S53).

After performing the EBME, the frame rate conversion apparatus can determine whether the motion vectors searched in the neighboring blocks around the specific block have the same motion direction (S54). As a result of the determination in step S54, if YES in step S54, it can be determined whether the average size of the same motion vectors is smaller than a second threshold T2 in step S55.

In the present embodiment, the second threshold value T2 is a measure of the identity of the motion vector magnitude and is preferably set to 0.3 as a result of experiments under various environments and conditions. In this case, T2 has a value of 0 or more and 1 or less, T2 = 1 is the sum or standard deviation of the deviations of the motion vector magnitudes is the largest, T2 = 0 is the smallest sum or standard deviation of the deviations of the motion vector magnitudes .

On the other hand, if the determination in steps S54 and S55 is NO, motion vector smoothing (MVS) may be performed (S56).

On the other hand, if it is determined to be YES in step S55, motion vector smoothing (MVS) may be omitted and overlapped block motion compensation (OBMC) may be performed (S57).

On the other hand, if it is determined as NO in step S52, the frame rate conversion apparatus can directly execute motion vector smoothing (MVS) without performing extended bidirectional motion estimation (EBME) S58).

After the above step S57 or S58, the frame rate conversion apparatus can perform motion compensated interpolation (MCI) (S59). Intermediate frames can be generated by performing interpolation (MCI) using motion compensation (S60).

On the other hand, in the comparison between the average motion vector magnitude and the first threshold value in the above-described embodiment, the case where the average motion vector magnitude is larger than the first threshold value in both the x-direction and the y- The present invention is not limited to such a configuration and can be implemented so as to use only one of x-direction and y-direction depending on the type, shape, encoding mode, and the like of a block or picture. Also, depending on the implementation, at least one of the various radiation directions between the x-direction and the y-direction at an intersection of the x-direction and the y-direction other than the x-direction and the y- The average motion vector magnitude can be obtained.

FIG. 6 is a diagram for explaining a direction determination process of motion vectors in the frame rate increasing method of FIG.

In the frame rate increasing method according to the present embodiment, the frame is classified into a dynamic frame and a static frame according to the degree of motion of the original block M1 in the previous frame and / or the current frame Fx on the basis of a specific block of the interpolation frame . This is expressed by Equations (7) and (8).

In the above equations (7) and (8), N is the total number of blocks in the frame, and xmag and ymag are the average motion vector magnitudes in the x and y directions within the frame, respectively. DF denotes a dynamic frame, and SF denotes a static frame.

As a result of the determination using Equations (7) and (8), when the interpolation frame or the previous frame corresponding thereto and the current frame corresponding to the current frame are dynamic frames DF, using EBME from the frame generating an interpolation frame in order, and the interpolated frame is a static frame (SF) if, as judged by the small change in the motion vectors (V ₁ to V ₈₎ in the frame a high signal processing computation amount EBME And generate the interpolation frame using only the BME having a low computation amount.

On the other hand, in the above-described embodiment, the direction and size of motion vectors of at least one original block (s) or neighboring blocks of the original block in the search area in the previous frame or the current frame based on one specific block of the interpolation frame However, the present invention is not limited to such a structure. For example, when the direction and size of the motion vectors are used, the motion vector is not a specific block of the interpolation frame The interpolation frame may be divided into a dynamic frame or a static frame in consideration of the direction and size of the motion vectors of each of the plurality of specific blocks in the at least one search area.

FIG. 7 is a diagram for explaining a process of determining whether the directions of motion vectors are the same in the motion vector determination process of FIG.

Referring to FIG. 7, after determining the dynamic frame and the static frame, the frame rate increasing method according to the present embodiment calculates a motion vector having substantially the same direction and size as a 3x3 mask (or window) The motion vectors included in the 3x3 mask may be determined as an accurate motion vector, and the MVS step may be omitted.

=0.3)을 비교할 수 있다. 그리고 M이 제2 임계치보다 작다면, 3×3 마스크에 포함된 움직임 벡터들은 동일한 움직임 벡터들로 판단되어 움직임 벡터 스무딩(MVS) 단계를 생략할 수 있다.In the process of determining the direction and magnitude of the motion vectors, first, each of the motion vectors included in the 3x3 mask is divided into four directions ((+, +), (-, - It can be determined which one of the directions belongs. When the motion vectors included in the 3x3 mask are all included in the same direction among the four directions, 3x3 of the motion vectors included in the 3x3 mask are multiplied by 3x3 It is possible to obtain the sum of the absolute values of the neighboring vector magnitude differences included in the mask. The x and y values thus obtained are represented by one value M through the Pythagorean theorem, and M and the second threshold value (

= 0.3) can be compared. If M is smaller than the second threshold value, the motion vectors included in the 3x3 mask may be determined to be the same motion vectors, and the motion vector smoothing (MVS) step may be omitted.

That is, as shown in FIG. 7, in the process of determining the directions and sizes of the same motion vectors, first, the motion vectors included in the 3x3 mask are determined from four directions, and the motion vectors included in the 3x3 mask are 4 The MVS step is performed in the case where all of the directions are included in the same direction, and the size can be obtained through Equation (9) when they are included in the same direction.

는 움직임 벡터들의 크기 차이의 합을 나타내며

과

는 x, y방향 각각의 움직임 벡터들의 크기 차이의 합을 나타낸다. In Equation (9)

Represents the sum of magnitude differences of the motion vectors

and

Represents the sum of the magnitude differences of the motion vectors in the x and y directions, respectively.

)보다 작으면(flag=0), 3×3 마스크에 포함된 움직임 벡터들을 정확한 움직임 벡터로 판단하고, MVS 단계를 수행하지 않을 수 있다.In Equation (10), M is the second threshold value

(Flag = 0), the motion vectors included in the 3x3 mask may be determined as an accurate motion vector, and the MVS step may not be performed.

8 is a block diagram of an image encoding apparatus according to another embodiment of the present invention.

8, the image encoding apparatus 20 according to the present embodiment includes a predictor 200, a subtractor 205, a transformer 210, a quantizer 215, an inverse quantizer 220, An adder 230, a filter unit 235, a decoded picture buffer (DPB) 240, and an entropy encoding unit 245.

The prediction unit 200 may include an adaptive image frame rate conversion (AIFC) module 202 or a component that performs a function corresponding to the module. In addition, the prediction unit 200 may include an intra prediction unit and an inter prediction unit.

In addition, the image encoding apparatus 20 may further include a segmentation unit 190. The dividing unit 190 may include a picture dividing unit and / or a block dividing unit. At least a part of the decomposition unit 190 includes a predictor 200, a transform unit 210, a quantization unit 215, an inverse quantization unit 220, an inverse transform unit 225, a filter unit 235, and an entropy encoding unit 245 may be included singly or in duplicate.

The subtractor 205, the transformer 210, the quantizer 215, the inverse quantizer 220, the inverse transformer 225, the adder 230, and the inverse transformer 230 of the image encoding apparatus 20, The structure, connection relationship, and operation principle of the filter unit 235, the decoded picture buffer (DPB) 240, and the entropy encoding unit 245 are well known in the art, do.

That is, the image encoding apparatus 20 according to the present embodiment may be configured such that the AIFC module 202 coupled to the prediction unit 200 generates a program for implementing the frame rate increasing method described above with reference to FIGS. 1 to 7, , And may be substantially the same as a conventional image encoding apparatus that receives video data and outputs a bit stream, except that it is implemented by a program code or the like.

9 is a schematic block diagram of an image encoding apparatus according to another embodiment of the present invention.

Referring to FIG. 9, the image encoding apparatus 10 according to the present embodiment may include a processor 102 and a memory 104. The memory 104 may store programs, program codes, and the like that implement the frame rate increasing method described above with reference to Figs. 1 to 7 (particularly Fig. 5). The processor 102 may be coupled to the memory 104 to implement a frame rate increasing method by a program.

The above-described processor 102 may include one or more cores or one or more cores and a cache memory. The processor 102 may have a multicore or single core architecture. A multi-core can refer to the integration of two or more independent cores into a single package of a single integrated circuit. And the single core architecture may refer to a central processing unit (CPU). The central processing unit may be implemented as a system on chip (SOC) in which micro control units (MCUs) and peripheral devices (integrated circuits for external expansion devices, etc.) are arranged together.

The core includes a register for storing an instruction to be processed, an arithmetic logical unit (ALU) for performing comparison, determination, and operation, a control unit for internally controlling the CPU for interpreting and executing the instruction, a control unit, an internal bus, and the like. In addition, the processor 102 may have a peripheral interface and a memory interface. The peripheral device interface connects the processor 102 to an input / output system, a communication unit or various other peripheral devices, and the memory interface can connect the processor 102 and the memory 104.

In addition, the processor 102 may include one or more data processors, image processors, or codecs (CODECs). The data processor, image processor, or codec may be comprised of one chip or may be comprised of separate chips.

The above-described processor 102 may execute various software programs to perform data input, data processing, and data output to perform a frame rate increasing method. In addition, the processor 102 may execute a specific software module (instruction set) stored in the memory 104 to perform various specific functions corresponding to the module. That is, the processor 102 may perform a frame rate increasing method implemented by software modules stored in the memory 104 in a user terminal such as a mobile device or a computing device, thereby improving the performance and efficiency of image encoding or decoding.

The above-described computing device may be a device implementing the frame rate increasing method or the image encoding method including the method, or a functional block performing the function of the device. In this case, the components of the frame rate increasing method may include a discriminator for discriminating the degree of motion in the frame, a determiner for determining the direction and size of the motion vectors, and the like. May be stored in a computer-readable medium (recording medium) in the form of software for implementation, or may be transmitted at a remote location in the form of a carrier to operate in a computing device.

The computer-readable medium may include a medium of a remote server device or a cloud system connected through a network, and may store a program or source code for performing a frame rate increasing method. The computer-readable medium may be embodied in the form of program instructions, data files, data structures, and the like, alone or in combination. Programs recorded on a computer-readable medium may include those specifically designed and constructed for the present invention or those known and available to those skilled in the computer software arts.

The computer-readable medium can also include a hardware device specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Program instructions may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. The hardware device may be configured to operate with at least one software module to perform the frame rate increasing method of the present embodiment, and vice versa.

Memory 104 may more particularly include high speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or a flash memory. In addition, the memory 104 may store software, a set of instructions, an operating system, or a combination thereof.

Here, the components of the software may include an operating system module, a communication module, a graphics module, a user interface module, a moving picture experts group (MPEG) module, a camera module, one or more application modules, and the like. A module is a set of instructions that can be represented as an instruction set or program.

The operating system includes built-in operating systems such as MS WINDOWS, LINUX, Darwin, RTXC, UNIX, OS X, iOS, Mac OS, VxWorks, Google OS, Android, And may include various components for controlling the system operation of a user terminal including a mobile device and the like. The above-described operating system may also include, but is not limited to, a function of performing communication between various hardware devices and software components (modules).

In the case where a communication function is provided, the image encoding apparatus or the computing apparatus according to the present embodiment may include a communication unit or a communication interface. The communication interface may support one or more communication protocols so that the device may be connected to the server device, the file server, or other devices on the network via the network. Such a communication interface may include one or more wireless communication subsystems. The wireless communication subsystem may include a radio frequency receiver and a transceiver and / or an optical (e.g., infrared) receiver or transceiver.

The network to which the communication interface is connected includes, for example, Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), W- (LTE-Advanced), OFDMA (Orthogonal Frequency Division Multiple Access), WiMax, WiBro, Wi-Fi (Wireless Fidelity) have.

The experimental results on the frame rate increasing method described above are as follows.

In the experiment, 8 images (Bus, Coastguard, Container, Flower, Foreman, Mobile, Mother_Daughter, Table) of CIF (352 × 288) size were used and a total of 50 even frames were generated. In this case, an interpolation frame can be generated between two adjacent even frames.

The experimental conditions were 16 × 16 for the basic block size and the search range was ± 8, and T1 = 0.6 and T2 = 0.3 were set. The quality comparison method was compared with the maximum signal-to-noise ratio (PSNR).

As shown in the following Table 1, the algorithm of the present embodiment (Proposed (dB)) shows an increase of PSNR of 0.008 dB on average compared to the EBME algorithm (EBME (dB)) of the comparative example. As shown in Table 2, it can be seen that the execution speed of the algorithm of this embodiment is shorter than that of the EBME algorithm of the comparative example.

According to the above-described embodiment, the EBME algorithm of the comparative example requires a high computation amount because the BME process is performed twice, the error vector is determined in the MVS step, and all the ideal vectors are repeated until the error is corrected. Therefore, the algorithm of the present embodiment (frame rate increasing method) determines the dynamic frame and the static frame using the respective average motion vector magnitudes in the x and y directions in the frame. In the case of a dynamic frame, it is determined that there is a large change in motion vector, and an accurate motion vector is found using EBME. In the case of a static frame, BME having a small amount of computation can be used.

Also, after determining the dynamic frame and the static frame, the algorithm of the present embodiment can further reduce the amount of computation by comparing the direction and size of several identical motion vectors and passing through the MVS step when there are several identical motion vectors in the frame .

As shown in the above experimental results, the frame rate increasing method according to the present embodiment has a PSNR improvement of 0.008 dB on average compared with the EBME algorithm of the comparative example, and the average gain is 1.186, which is comparable to the EBME algorithm of the comparative example. There is an advantage that the execution time is reduced. In addition, Coastguard and Mother_Daughter have the advantage of reducing the execution time even more by including a lot of dynamic frames.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

Claims (20)

Determining whether the previous frame or the current frame is a dynamic frame or a static frame by comparing an average value of motion vector magnitudes in an x or y direction with a threshold value in a current frame and a previous frame after bidirectional motion prediction;
Performing extended bidirectional motion prediction on the previous frame or the current frame if it is determined to be a dynamic frame;
Determining whether a direction of a motion vector detected in each of a plurality of sub-pixels in a mask of a predetermined size among sub-pixels of a block generated in the extended bidirectional motion prediction belongs to one of predetermined direction ranges;
Determining whether the magnitudes of the motion vectors are the same within a predetermined error range; And
And skipping motion vector smoothing for the block of the intermediate frame to be generated between the previous frame and the current frame if the sizes are the same. The method according to claim 1,
Further comprising modifying, using a filter, the motion vectors of the prediction units of the previous frame or the current frame if the determination result in the step of determining belongs to any one of the frames, Increase method. The method of claim 2,
Further comprising the step of modifying the motion vectors of the prediction units of the previous frame or the current frame using a filter if the determination result in the step of determining whether the sizes of the motion vectors are equal is not the same size, How to increase rate. The method of claim 3,
If the result of the determination in the step of determining whether or not the sizes of the motion vectors are the same is found to be the same size after the modifying step or before the generation of the intermediate frame, ≪ / RTI > wherein the step of performing the compensation further comprises performing compensation. The method of claim 4,
Further comprising performing motion vector smoothing on the motion vectors of the prediction units of the previous frame or the current frame if the determination result in the step of determining whether the dynamic frame or the static frame is a static frame, Increase method. The method of claim 5,
After performing the superimposed block motion compensation or after performing the motion vector smoothing, an area matching between the previous frame and the current frame using a motion compensation is generated in many frames before generating the intermediate frame Further comprising the step of providing a high weight. The method according to claim 1,
Wherein the size of the mask is 3x3. Determining whether the previous frame or the current frame is a dynamic frame or a static frame by comparing an average value of motion vector magnitudes in an x or y direction with a threshold value in a current frame and a previous frame after bidirectional motion prediction; And
And performing a motion vector smoothing on the motion vectors of the prediction units of the previous frame or the current frame if the result in the determining step is a static frame. The method of claim 8,
Further comprising performing extended bidirectional motion prediction on the prediction units of the previous frame or the current frame if the determination result in the determination of the dynamic frame or the static frame is a dynamic frame. . The method of claim 9,
After performing the extended bidirectional motion prediction,
Determining whether a direction of motion vectors detected in each of a plurality of regions in a mask of a predetermined size among sub-prediction units generated in the extended bidirectional motion prediction belongs to one of predetermined directional ranges;
Determining whether the magnitudes of the motion vectors are the same within a predetermined error range, And
If it is the same size, omitting motion vector smoothing for the previous frame or the current frame. The method of claim 10,
Further comprising modifying, using a filter, the motion vectors of the prediction units of the previous frame or the current frame if the determination result in the step of determining belongs to any one of the frames, Increase method. The method of claim 11,
Further comprising the step of modifying the motion vectors of the prediction units of the previous frame or the current frame using a filter if the determination result in the step of determining the same size is not equal, . The method of claim 12,
Performing the superimposed block motion compensation on the previous frame or the blocks in the current frame if the determination result in the determining step after the modifying step or the determining step of the same size is found to be the same size , A method for increasing the frame rate. The method of claim 10,
Wherein determining whether the motion vector magnitude is the same magnitude determines whether the motion vector magnitude is less than 0.3. The method of claim 8,
Wherein the determining whether the dynamic frame or the static frame determines whether the average value of the motion vector magnitude is greater than 0.6. 1. An image encoding apparatus for encoding an input image through temporal prediction, spatial prediction, transcoding, quantization, and entropy coding,
A memory for storing a program or a program code for encoding and frame rate conversion; And
And a processor coupled to the memory to execute the program,
Wherein the processor, by the program for the frame rate conversion,
Determining whether the previous frame or the current frame is a dynamic frame or a static frame by comparing an average value of a motion vector magnitude in the x or y direction with a threshold value in a current frame and a previous frame after bidirectional motion prediction,
If the determination result is a dynamic frame, performs an extended bidirectional motion prediction on the prediction units of the previous frame or the current frame,
Determining whether directions of motion vectors searched in each of a plurality of regions in a mask of a predetermined size among sub-prediction units generated in the extended bidirectional motion prediction belong to one of predetermined directional ranges,
Determining whether the size of the motion vectors is the same within a predetermined error range,
And omits motion vector smoothing for the current frame if the determination result is the same size. 18. The method of claim 16,
The processor determines that the motion of the prediction unit of the previous frame or the current frame is not the same as the motion of the previous frame or the current frame if the determination result of belonging to any one of them does not belong to any one, Wherein the vectors are modified using a filter. 18. The method of claim 17,
The processor may perform a superimposed block motion compensation on the previous frame or blocks in the current frame before generating the intermediate frame after the modification using the filter or when the determination result of the same size is the same size Lt; / RTI > 19. The method of claim 18,
Wherein the processor performs motion vector smoothing on the motion vectors of the prediction units of the previous frame or the current frame when the determination result of the dynamic frame or the static frame is a static frame. The method of claim 19,
Wherein the processor is configured to perform motion compensation on the current frame after performing the superimposed block motion compensation or the motion vector smoothing, To the image encoding apparatus.

Images