TWI478589B - Method of block-based motion estimation and frame rate up conversion - Google Patents

Description

Block mode action estimation method and frame rate improvement method

The present invention relates to motion estimation, and more particularly to a block-based motion estimation method that is applicable to frame rate up conversion (FRUC).

Frame rate increase (FRUC) technology is often used in digital image displays, such as digital televisions, to create one or more inner frame between two adjacent frames, thereby increasing the frame rate, for example, from 60 Hz to 120Hz or even 240Hz. The generation of the inner frame is generally an interpolation technique using motion compensation. The first figure shows an example of generating an inner frame based on the previous frame and the current frame. First, it determines the movement of the macroblock (MB) of the current frame relative to the corresponding macroblock of the previous frame. According to this action, the calculation is performed to obtain the inner frame. However, the generation of the interpolated blocks in the inner frame is random and irregular. Since the display of the frame is generally from left to right and from top to bottom, the aforementioned random situation causes difficulty or complexity in displaying the inner frame. Furthermore, the interpolated block is generally not aligned with the segmentation of the macroblock of the video frame. Therefore, the memory data access performed to process the interpolated block will also become random and pixel-based, which will cause a delay in time. In addition, the interleaved blocks overlap each other. This overlapping situation makes the display of the inner frame more complicated. Pixel-style motion compensation also affects the speed and accuracy of other processing, such as occlusion processing, so that the processing must also be pixel-based.

In view of the fact that the above traditional frame rate increase (FRUC) is delayed, complicated and inaccurate, it is necessary to propose a novel mechanism to accelerate and simplify the motion compensation when generating the inner frame (for example, increasing the frame rate). .

In view of the above, one of the objects of the embodiments of the present invention is to provide a block-based motion estimation method, so that the motion compensation of the inner frame is a block mode process, thereby causing the memory device data to be stored. The sequential access of the block mode that becomes regular is taken, thereby greatly increasing the processing speed.

According to an embodiment of the invention, a reference frame and a current frame are first provided. The current frame is divided into complex macroblocks (MB) that do not overlap each other. Then, relative to the reference frame, an action vector (MV) of each macroblock of the current frame is obtained to obtain an action vector map (MV map). According to the motion vector diagram, the motion vector of each macroblock in the illustration frame is determined within the inner frame between the reference frame and the current frame. In this embodiment, determining the action vector of each macroblock in the inset frame includes the following steps. According to the motion vector diagram, the macroblock of the current frame is inversely mapped to the inner frame, and the complex reverse mapping macroblock is generated in the inner frame. Next, the distance between the complex reverse mapping macroblock and the to-be-processed macroblock in the inner frame is determined. Finally, according to the distance, an action vector is automatically selected as a vector map to determine an action vector of the macro block to be processed.

21-24‧‧‧Steps

231-234‧‧‧Steps

40‧‧‧Search range

40A, 40B, 40C‧‧‧ macro blocks

42‧‧‧ pending macroblocks

42A, 42B, 42C‧‧‧ Reverse Mapping Macro Blocks

The first figure shows an example of generating an inner frame based on the previous frame and the current frame.

The second flowchart shows a block-based motion estimation method according to an embodiment of the present invention.

The third figure shows a detailed flow chart of step 23 of the second figure.

The fourth figure illustrates a related schematic diagram of the third graph process.

The second flowchart shows a block-based motion estimation method according to an embodiment of the present invention, which is applicable to frame rate improvement (FRUC).

In step 21, the first frame (generally referred to as a reference frame) and the current frame are first provided, by interpolation technique A new frame is created between the previous frame and the current frame. In general, the frame of time N and the frame of time N+2 are provided, and the inner frame is generated at time N+1. In this embodiment, each frame is divided into rectangular areas that do not overlap each other, and is called a macroblock (MB). For example, each macroblock can be 4x4 or 16x16 pixels in size.

Next, in step 22, an action vector map (MV map) is obtained. The motion vector map is also divided into macroblocks that do not overlap each other as described above. Each macroblock in the motion vector diagram contains an action vector that represents the movement or displacement of the macroblock of the current frame relative to the corresponding macroblock of the previous frame. In detail, block matching is performed by each block between the current frame and the previous frame to generate an action vector of the macro block of the motion vector diagram. For each macroblock in the current frame, the block matching action estimate finds the best matching block from the previous frame (eg, when it reaches the sum of absolute difference (SAD)), Thus, the motion vector of the macroblock of the current frame is obtained. The action vector diagram contains the motion vectors for each macroblock of the current frame. Therefore, the motion vector diagram of this embodiment is generally also referred to as a forward motion vector diagram.

Next, in step 23, according to the forward motion vector map obtained in step 22, the motion vector of each macroblock in the inner frame is determined. In this embodiment, the decision of the action vector of the inner frame is performed one block by one block or block-based. For example, macroblocks are processed from left to right and from top to bottom. The third figure shows a detailed flow chart of step 23, and the fourth figure illustrates a related schematic diagram of the third figure flow.

In step 231, a search range 40 (fourth map) is first determined in the current frame, which is related to the to-be-processed macro block 42 in the inset frame. Next, in step 232, the macroblocks (ie, 40A, 40B, and 40C) within the search range are inversely mapped to the action vector map (eg, the motion vectors of the macroblocks 40A, 40B, and 40C). The inner inset frame (as indicated by the solid arrows) thus produces reverse mapped macroblocks 42A, 42B, and 42C, respectively. Next, in step 233, the distances of the reverse mapping macroblocks 42A/42B/42C and the macroblocks 42 to be processed are determined, respectively. In this embodiment, the distance between the macroblocks is defined as the upper left corner of each macroblock. The distance (as indicated by the dashed arrow). Finally, in step 234, because the reverse mapping macroblock 42C corresponding to the macroblock 40C is closest to the macroblock 42 to be processed, the motion vector of the macroblock 40C is selected as the macroblock to be processed. The motion vector of block 42. In detail, the motion vector of the macroblock 42 to be processed is half of the motion vector of the macroblock 40C. For convenience of explanation, the macroblocks 40A, 40B, and 40C represent all the macroblocks in the search range of this embodiment.

Next, in step 24 (second diagram), according to the motion vector selected in step 234, a block is generated in a block by block. According to steps 23 and 24, all the macroblocks in the inner illustration frame are processed and displayed in sequence.

According to the above embodiment, since the motion compensation of the inner frame is in a block mode, the data access of the memory device can be a block-based regular access, instead of the pixel method in the conventional method ( Pixel-based random access, which can increase the processing speed in a large amount. Furthermore, the block mode action compensation allows other processes, such as masking, to be processed in a block manner.

The above embodiment is based on a forward motion vector diagram to perform reverse mapping. However, in other embodiments, a reverse motion vector map can also be used. In other words, the forward and reverse motion vector maps are used simultaneously to obtain two or more candidate motion vectors for the macroblock to be processed within the inner frame. The reverse motion vector map is also divided into macroblocks that do not overlap each other as described above. Each macroblock of the inverse action vector diagram contains an action vector representing a frame of time N (eg, the previous frame) of the macroblock relative to time N+2 (eg, the current frame) The movement or displacement of the corresponding macroblock. In other words, the direction of motion estimation of the inverse motion vector diagram is opposite to the direction of motion estimation of the forward motion vector diagram. Similar to step 23 of the second figure, the inverse action vector map is used to determine the inverse motion vector for each macroblock in the inner frame, the details of which are similar to steps 231-234 of the third figure. Compared to using only the forward motion vector diagram, if the forward and reverse motion vector diagrams are used simultaneously for the block mode motion compensation, better performance can be obtained, but more resources and time delays must be used.

The above description is only the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention; all other equivalent changes or modifications which are not departing from the spirit of the invention should be included in the following Within the scope of the patent application.

21-24‧‧‧Steps

Claims (10)

A block mode action estimation method includes: providing a reference frame and a current frame, the current frame being divided into complex macroblocks (MB) that do not overlap each other; and relative to the reference frame, An action vector (MV) of each of the macroblocks of the current frame is used to obtain an action vector map (MV map); and according to the action vector map, between the reference frame and the current frame An inset frame, determining an action vector of each macroblock in the inset frame; wherein the step of determining an action vector of each macroblock in the inset frame comprises: according to the motion vector diagram, The macroblock of the current frame is inversely mapped to the inner frame to generate a complex reverse mapping macroblock in the inner frame; respectively determining the complex reverse mapping macro in the inner frame And a distance between the block and the macro block to be processed; and selecting an action vector from the action vector map to determine an action vector of the macro block to be processed according to the distance. The method for estimating the operation of the block mode according to the first aspect of the patent application, wherein the reference frame is the previous frame. The method for estimating a block mode according to the first aspect of the patent application, wherein the macro block of the inner frame is processed from left to right and top to bottom to determine the motion vector. The method for estimating a block mode according to claim 1, wherein each action vector of the action vector map is generated by block matching between the current frame and the reference frame (block) Matching), wherein the motion vector represents the movement of the macroblock of the current frame relative to the corresponding macroblock of the reference frame. The method for estimating the block mode according to the first aspect of the patent application further includes a step of determining a search range in the current frame, which is related to the to-be-processed macro block, and is used in the search range. The macroblock is inversely mapped to the inner frame. The method for estimating a block mode according to the first aspect of the patent application, wherein the reverse mapping macro block The distance from the to-be-processed macroblock is the distance between the corner of the reverse mapping macroblock and the corresponding corner of the to-be-processed macroblock. A frame rate increasing method includes: providing a reference frame and a current frame, the current frame being divided into complex macroblocks (MB) that do not overlap each other; and the current frame is obtained relative to the reference frame An action vector (MV) of each of the macroblocks of the frame is used to obtain an action vector map (MV map); according to the motion vector map, between the reference frame and the current frame An illustration frame determining an action vector of each macro block in the inner frame; and generating an inner frame according to an action vector of each macro block in the inner frame; wherein the determining the inner frame The step of the motion vector of each macroblock includes: inversely mapping the macroblock of the current frame to the inner frame according to the motion vector map, to generate a complex inverse in the inner frame Mapping the macroblock; respectively determining a distance between the complex reverse mapping macroblock and a to-be-processed macroblock in the inner frame; and selecting an action vector from the motion vector map according to the distance Used to determine the motion vector of the macro block to be processed. The frame rate increasing method according to claim 7, wherein the macro block of the inner frame is processed from left to right and from top to bottom to determine the motion vector. The method for improving the frame rate according to claim 7 further includes a step of determining a search range in the current frame, which is related to the to-be-processed macro block, and is used in the search range. , the macroblock is inversely mapped to the inner inset frame. The frame rate increasing method of claim 7, wherein the distance between the reverse mapping macroblock and the to-be-processed macroblock is a corner of the reverse mapping macroblock and the waiting Handling the distance between the corresponding corners of the macroblock.