TW201346839A - Motion de-blurring method based on motion compensation - Google Patents

Abstract

A motion de-blurring method based on motion compensation is disclosed, which comprises the following steps: (A) calculating pixel differences at the same positions between a current frame and a reference frame, so as to obtain a plurality of pixel difference values; (B) calculating the summation of absolute values of the pixel difference values, so as to output a total error value; (C) according to the total error value, a first threshold value, a second threshold value and a third threshold value, selecting one of a first processing unit, a second processing unit and a third processing unit to perform interpolation processing to the current and reference frames; (D) if the total error value is less than the first threshold value and greater than the second threshold value, the first processing unit executing a one-way compensation operation mode; and (E) if the total error value is less than or equal to the second threshold value and greater than or equal to the third threshold value, the second processing unit executing a bi-directional compensation operation mode.

Description

Method of moving blur based on dynamic compensation

The invention relates to a method for moving motion blur, in particular to a method for moving motion blur based on dynamic compensation.

With the development of technology, display technology is more and more developed. In order to reduce the size of the screen, reduce weight and solve power consumption, liquid crystal display (LCD) has almost replaced the traditional cathode ray tube (Cathode Ray Tube). , CRT) screen. In addition, as the size of the liquid crystal screen continues to increase, in order to improve the quality of the video, the resolution is gradually improved. However, due to the "long response time" and "hold-type" driving method of the liquid crystal display, and the "human eye persistence" characteristic, the phenomenon of movement blur is likely to occur.

In order to effectively solve the motion blur caused by high-speed image movement and increase the smoothness of the image, many scholars have proposed many methods to improve the motion blur, such as: accelerating the action of liquid crystal molecules, reducing the response time of the liquid crystal screen, and inserting Black technology, as well as supplemental illustration technology.

The so-called interpolation technique is to insert a supplementary illustration surface or insert a black screen by using motion compensation technology between two pictures, thereby increasing the frequency of the picture. Since the high-quality TV picture frequency may be 120 Hz, 240 Hz or higher, Frame Rate Up-Conversion (FRUC) is currently the most frequently discussed subject.

However, in the process of supplementing the illustration surface, because the current algorithm can not achieve 100% accurate compensation, it is easy to cause counter-effects, thus compensating for the wrong picture. At this time, only the action of the supplementary illustration surface can be turned off to avoid the error of the illustration, but the function of eliminating the motion blur is also lost. Therefore, the problem of the movement blur caused by the liquid crystal display cannot be completely solved, so it is necessary to seek A solution.

Accordingly, it is an object of the present invention to provide a method for moving motion blur based on dynamic compensation.

Therefore, the method for moving motion blur based on dynamic compensation is applicable to an image processing apparatus for processing a current picture and a reference picture to output an inner picture plane, the image processing apparatus including a difference calculation unit, A difference statistics unit, an operation mode selection unit, a first processing unit, a second processing unit and a third processing unit, the method comprising the following steps.

First, the difference calculation unit calculates a pixel difference between each first pixel in the current picture and each second pixel in the reference picture corresponding to each first pixel position to obtain a plurality of pixel difference values.

Then, the difference statistic unit calculates the sum of the absolute values of the pixel difference values to output an error total value.

Then, the operation mode selection unit selects the first processing unit, the second processing unit, and the third according to the relationship between the total error value and a first threshold value, a second threshold value, and a third threshold value. One of the processing units performs an interpolation process between the current picture and the reference picture, wherein the first threshold value is greater than the second threshold value, and the second threshold value is greater than the third threshold value.

Then, when the total error value is less than the first threshold value and greater than the second threshold value, the operation mode selection unit selects the first processing unit, and then the first processing unit is in a one-way compensation operation mode pair. Interpolation processing is performed between the current picture and the reference picture to generate the inner picture plane.

When the total error value is less than or equal to the second threshold value and greater than or equal to the third threshold value, the operation mode selection unit selects the second processing unit, and then the second processing unit adopts a bidirectional compensation operation mode. Interpolating processing between the current picture and the reference picture to generate the inner picture plane.

The effect of the present invention is that the total value of the error between the current picture and the two pictures of the reference picture is compared with the first, second, and third threshold values by the operation mode selection unit, and the first, second, and third thresholds are processed. One of the units selects the compensation operation mode that is most suitable for interpolation, and thus solves the problem that a compensation operation mode alone cannot balance the types of various images.

The above and other technical contents, features and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments.

Referring to FIG. 1 and FIG. 2, a preferred embodiment of the method for removing motion blur based on dynamic compensation according to the present invention is implemented by an image processing apparatus 2.

The image processing device 2 is configured to process a current frame (Current Frame) 11 and a reference frame (Reference Frame) 12, and output an Interpolated Frame 13. The image processing device 2 includes a difference calculation unit 21, a difference statistics unit 22, an operation mode selection unit 23, a first processing unit 24, a second processing unit 25, and a third processing unit 26.

The difference calculation unit 21 is configured to receive the current picture 11 and the reference picture 12 of the color space of the luminance-chrominance (YUV Color Space), wherein the current picture 11 and the reference picture 12 have a resolution of 252. × 288 is a frame, and it includes a plurality of pixels (Pixel), respectively. Each of the pixels has a corresponding pixel value, and the pixel value has a luminance value (Luma) or a chroma value (Chroma), and the difference calculation unit 21 is further configured to respond to each of the first pixels in the current picture 11 The pixel difference 211 between the two pictures is calculated for each pixel in the reference picture 12 at the same position of each first pixel, and then the pixel difference 211 is output to the difference value unit 22.

The difference statistic unit 22 is configured to calculate the sum of the absolute values of the pixel difference 211 values, and transmit the error total value 221 obtained through the calculation to the operation mode selecting unit 23.

The operation mode selection unit 23 selects the first, second, and third processing units (24, 25) according to the relationship between the error total value 221 and a first threshold value, a second threshold value, and a third threshold value. And one of the 26) interpolating the current picture 11 and the reference picture 12, wherein the first threshold value is greater than the second threshold value, and the second threshold value is greater than the third threshold value.

In the preferred embodiment, the frame having a resolution of 252×288 is taken as an example. The first threshold is 90000, the second threshold is 60000, and the third threshold is 10000. The value of the threshold is proportional to the resolution of the frame.

Therefore, the operation mode selection unit 23 selects the first processing unit 24 and further the first processing unit by determining that the error total value 221 is smaller than the first threshold value and greater than the second threshold value. The current picture 11 and the reference picture 12 are processed in a one-way compensation operation mode.

Referring to FIG. 3 and FIG. 4 , the unidirectional compensation operation mode first divides the current picture 11 into a plurality of macro blocks (Macs, MBs) of the same size and not overlapping each other, and passes through one of the current pictures 11 . The macro block MB _Cur and the reference picture 12 corresponding to the coordinate position of the _macro block MB _{Cur are used} in a predefined search range to utilize the block comparison (Block) via the Motion Estimation method (Block) The matching method searches for the most similar macroblock MB _Ref in the reference picture 12.

In the preferred embodiment, the search range is a block search for each pixel coordinate position in the search range by means of a full search, and the operation of the mean absolute difference (Mean Absolute Difference) is used. The method is based on the selection of similar giant blocks, and the average absolute value difference is as shown in the following equation (1):

among them,

f _K : current picture 11;

f _KT : reference picture 12; and

MN: A giant block of size M×N.

For example, as shown in FIG. 3, after finding the most similar macroblock MB _Ref in the reference picture 12 corresponding to the _macro block MB _Cur in the current picture 11, the _macro area can be directly used. The relative coordinate position difference between the block MB _Cur and the macro block MB _Ref is used to calculate the motion vector MV (Motion Vector, MV), assuming that the _macro block MB _Cur coordinate position in the current picture 11 is (x+i, y+j And the most similar macroblock MB _Ref coordinate position in the reference picture 12 is (x, y), then the motion vector MV is (-i, -j).

Referring to FIG. 4, after the motion vector MV is obtained through the motion estimation, the current picture 11 and the reference picture 12 are obtained by the motion compensation (MC) algorithm in the unidirectional compensation operation mode. An intermediate state, wherein the macro block MB _{Int of} the inner illustration surface 13 is a macro block corresponding to the _macro block MB _{Cur of} the current picture 11 and the _macro block MB _Ref of the reference picture 12, and the inner picture surface 13 The giant block MB _Int is calculated by dividing its motion vector MV by two.

In the preferred embodiment, in order to solve the hole phenomenon which is easy to occur in the one-way compensation operation mode, the inner illustration surface 13 is such that the macro block MB _Int is transmitted through a motion vector map (Motion Vector Map). Rearranged and reconstructed, the inner illustration surface 13 can also be expressed as:

I : inside illustration face 13;

x : the horizontal coordinate position of the current screen 11;

y : the vertical coordinate position of the current screen 11;

M V _x : horizontal movement vector; and

MV _y : Vertical motion vector.

In addition, referring to FIG. 1 and FIG. 5, the operation mode selecting unit 23 selects the second by determining that the error total value 221 is less than or equal to the second threshold value and is greater than or equal to the third threshold value. The processing unit 25, and further the second processing unit 25, processes the current picture 11 and the reference picture 12 in a bidirectional compensation operation mode.

The bidirectional compensation operation mode includes a forward compensation algorithm and a backward compensation algorithm, and the details of the program executed by the backward compensation algorithm are similar to the one-way compensation operation mode, and the backward compensation algorithm is also the same The global search method is also used to perform block alignment through the motion estimation algorithm, and then the motion vector MV is obtained, and the inner graph surface 13 based on backward compensation is reconstructed via motion compensation.

However, the forward compensation operation mode is also first to divide the reference picture 12 into a plurality of macro blocks of the same size and not overlapping each other, and the current picture is displayed via the macro block MB _Ref in the reference picture 12 . global search range of 11, the embodiment using the motion estimation algorithm to the block matching, to find out the current picture and the reference picture macroblock MB _Ref. 12 the most similar macroblock of 11 MB _Cur Then, the motion vector MV is derived, and the macroblock MB _{Int of} the inner illustration plane 13 is also obtained through the motion compensation algorithm, and the inner graph surface 13 based on the forward compensation can be reconstructed.

Therefore, the inner illustration surface 13 in the two-way compensation operation mode can be obtained from the calculation of the inner surface 13 of the forward compensation and the backward compensation, as shown in the following equation (2):

among them,

I : inside illustration face 13;

x : the horizontal coordinate position of the inner illustration surface 13;

y : the vertical coordinate position of the inner illustration surface 13;

M V _x : horizontal movement vector; and

MV _y : vertical motion vector

I ( x + , y + , t -1): based on the forward estimate within the illustration plane 13

I ( x - , y - , t +1): based on the backward estimate within the illustration plane 13

In the preferred embodiment, in order to reduce the block effect that is easy to occur during the bidirectional compensation operation mode, when the forward and backward compensation algorithms are used for block comparison, an overlap based block is used. Overlapped Block-Based Motion Estimation (OBME), and the ratio of the extent of the central pixel in the macroblock does not exceed the range of the original giant block.

In addition, referring to FIG. 1, the operation mode selecting unit 23 selects the first condition by determining that the error total value 221 is greater than or equal to the first threshold value, or the error total value 221 is less than the third threshold value. The third processing unit 26, and further the third processing unit 26 interpolates between the current picture 11 and the reference picture 12 in a picture repeat operation mode, the picture repeat operation mode is to copy the reference picture 12, and the copy is performed by the copy The reference picture 12 is used as the inner picture plane 13.

Referring to FIG. 1 and FIG. 2, a preferred embodiment of the method for moving motion blur based on dynamic compensation is applied to the image processing apparatus 2 for processing the current picture 11 and the reference picture 12 to output the inner image. Illustration face 13. The method consists of the following steps:

First, as shown in step S31, the image processing device 2 reads the luminance-chrominance (YUV) value of the video sequence (not shown).

Then, as shown in step S32, the current picture 11 and the reference picture 12 are input to the image processing device 2, wherein the current picture 11 is the kth frame in the image sequence, and the reference picture 12 is the image sequence. The k-1th frame in the middle.

Then, as shown in step S33, the difference calculating unit 21 calculates a pixel difference 211 between each first pixel in the current picture 11 and each second pixel in the reference picture 12 corresponding to each first pixel position, to The pixel difference 211 value is obtained.

Next, as shown in step S34, the difference statistic unit 22 calculates the sum of the absolute values of the values of the pixel differences 211 to output a quantized result indicating the change of the picture between the current picture 11 and the reference picture 12. The total error is 221.

Then, as shown in step S35, the operation mode selecting unit 23 selects the first, second, and third processing units according to the numerical relationship between the error total value 221 and the first, second, and third threshold threshold values. One of (24, 25, 26) interpolating between the current picture 11 and the reference picture 12, wherein the first threshold is greater than a second threshold, and the second threshold is greater than the first Three thresholds.

Then, as shown in step S36, when the operation mode selection unit 23 determines that the error total value 221 is smaller than the first threshold and greater than the second threshold, it indicates that the current picture 11 and the reference picture 12 are The picture changes slightly, or the light changes are relatively strong, so the algorithm with lower complexity is used, the operation mode selecting unit 23 selects the first processing unit 24, and then the first processing unit 24 uses the one-way compensation. The operation mode processes the current picture 11 and the reference picture 12 to generate the inner picture plane 13.

Alternatively, as shown in step S37, when the operation mode selection unit 23 determines that the error total value 221 is less than or equal to the second threshold value and is greater than or equal to the third threshold value, it indicates that the current picture 11 is a gentle Moving the picture, the operation mode selecting unit 23 selects the second processing unit 25, and then the second processing unit 25 combines the bidirectional compensation operation mode and the overlap block based motion estimation algorithm to the current picture 11 and The reference picture 12 is processed to generate the inner picture plane 13.

Alternatively, as shown in step S38, when the operation mode selecting unit 23 determines that the error total value 221 is greater than or equal to the first threshold value, or the error total value 221 is less than the third threshold value, the third processing The unit 26 processes the current picture 11 and the reference picture 12 in the picture repeat operation mode, and copies the reference picture 12 through the picture repeat operation mode to form the inner picture plane 13.

In the preferred embodiment, when the error total value 221 is greater than or equal to the first threshold value, the current picture 11 is a transition picture; and when the error total value 221 is less than the third threshold value, Indicates that the current picture 11 is a static picture.

Then, as shown in step S39, the internal difference picture generated by one of the steps S36 to S38 is output, and then inserted into two consecutive pictures of the current picture 11 and the reference picture 12.

As shown in Table 1, the PSNR value statistics of the algorithm proposed in the non-patent academic literature; wherein the algorithm proposed in the literature [1] is a global search and a bidirectional compensation operation method, the average PSNR value is about Is 29.136. However, the literature [2] added a modified motion vector to make the PSNR value slightly improved, but the effect is still limited. In the literature [3] to [6], the mobile estimation algorithm based on overlapping blocks and the modified motion vector are gradually added. Although the PSNR value is gradually improved, the complexity of the algorithm is also improved. The disadvantage of using symmetry dynamic estimation is that when the moving object is close to the background color, it may be easy to compensate for the wrong picture during the interpolation because the correct motion vector cannot be found. In the literature [7], the method of forward compensation operation is adopted, and the PSNR value is about 29.083 on average.

Therefore, the architecture proposed by the preferred embodiment is to use the first, second, and third thresholds as the basis for determining the variation of the global search collocation image sequence to make the most accurate adaptive compensation. As shown in Table 1, after comparison, it is found that the method based on dynamic compensation of the moving motion blur can effectively improve the PSNR value in various test films, and most of them are better than the algorithms proposed in each document.

Non-patent academic literature mentioned in Table 1:

[1] SH Lee, YC Shin, S. Yang, HH Moon, and RH Park, "Adaptive motion-compensated interpolation for frame rate up-conversion," Consumer Electronics, IEEE Transactions on, vol. 48, no. 3, pp 444-450, 2002.

[2] J. Zhang, L. Sun, S. Yang, and Y. Zhong, “Position prediction motion-compensated interpolation for frame rate up-conversion using temporal modeling,” in Image Processing, 2005. ICIP 2005. IEEE International Conference On,2005,vol. 1,p. I-53.

[3] T. S. Chong, O. C. Au, W. S. Chau, and T. W. Chan, “Multiple objective frame rate up conversion,” in Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on, 2005, pp. 253-256.

[4] K. Hilman, HW Park, and Y. Kim, "Using motion-compensated frame-rate conversion for the correction of 3: 2 pulldown artifacts in video sequences," Circuits and Systems for Video Technology, IEEE Transactions on, vol 10, no. 6, pp. 869-877, 2000.

[5] SH Lee, O. Kwon, and RH Park, "Weighted-adaptive motion-compensated frame rate up-conversion," Consumer Electronics, IEEE Transactions on, vol. 49, no. 3, pp. 485-492, 2003 .

[6] YT Yang, YS Tung, and JL Wu, "Quality enhancement of frame rate up-converted video by adaptive frame skip and reliable motion extraction," Circuits and Systems for Video Technology, IEEE Transactions on, vol. 17, no. 12, pp. 1700-1713, 2007.

[7] BW Jeon, GI Lee, SH Lee, and RH Park, "Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure," Consumer Electronics, IEEE Transactions on, vol. 49, no. 3, Pp. 499-508, 2003.

In summary, the preferred embodiment of the present invention determines the amount of picture change between the current picture 11 and the reference picture 12 in the video sequence, and refers to the first, second, and third thresholds, and timely complements the interpolation technique. Automatic and correct compensation, so that the problem caused by the conventional image surface 13 formed by varying degrees of picture change can be avoided, and then the movement blur easily generated on the liquid crystal display and the input delay can be reduced, so that it can be achieved. The object of the invention.

The above is only the preferred embodiment of the present invention, and the scope of the invention is not limited thereto, that is, the simple equivalent changes and modifications made by the scope of the invention and the description of the invention are All remain within the scope of the invention patent.

11. . . Current picture

12. . . Reference picture

13. . . Inside illustration

2. . . Image processing device

twenty one. . . Difference calculation unit

211. . . Pixel difference

twenty two. . . Difference statistics unit

221. . . Total error

twenty three. . . Operation mode selection unit

twenty four. . . First processing unit

25. . . Second processing unit

26. . . Third processing unit

S31~S39. . . step

MV. . . Moving vector

MB _Cur . . . Current screen giant block

MB _Ref . . . Reference screen giant block

MB _Int . . . Inside illustration giant block

1 is a block diagram showing an apparatus for implementing a preferred embodiment of the method for moving motion blur based on dynamic compensation of the present invention;

2 is a flow chart showing a preferred embodiment of the method for removing motion blur based on dynamic compensation according to the present invention;

3 is a schematic diagram showing that a macroblock of a current picture in the preferred embodiment compares a most similar macroblock in a search range in a reference picture;

4 is a schematic diagram showing a one-way compensation operation mode in the preferred embodiment;

Figure 5 is a schematic diagram showing a bidirectional compensation operation mode in the preferred embodiment.

S31~S39. . . step

Claims (10)

A method for removing motion blur based on dynamic compensation is applicable to an image processing apparatus for processing a current picture and a reference picture to output an internal illustration surface, the image processing apparatus comprising a difference calculation unit and a difference a statistical unit, an operation mode selection unit, a first processing unit, a second processing unit, and a third processing unit, the method comprising the following steps: (A) the difference calculation unit calculates each first in the current picture a pixel is different from a pixel of each second pixel in the reference picture corresponding to each first pixel position to obtain a plurality of pixel difference values; (B) the difference statistical unit calculates a sum of absolute values of the pixel difference values And outputting an error total value; (C) the operation mode selecting unit selects the first processing according to a relationship between the total error value and a first threshold value, a second threshold value, and a third threshold value Interpolating the current picture and the reference picture by one of the unit, the second processing unit, and the third processing unit, wherein the first threshold is greater than the second threshold, and the second gate The value is greater than the third threshold value; (D) when the total error value is less than the first threshold value and greater than the second threshold value, then the operation mode selection unit selects the first processing unit, and then the first processing The unit interpolates the current picture and the reference picture in a one-way compensation operation mode to generate the inner picture plane; and (E) when the total error value is less than or equal to the second threshold value and greater than or equal to the When the third threshold is used, the operation mode selection unit selects the second processing unit, and then the second processing unit interpolates the current picture and the reference picture in a bidirectional compensation operation mode to generate the inner Illustration. The method for moving motion blur based on dynamic compensation according to claim 1, wherein in the step (D), the first processing unit combines a global search method in performing the one-way compensation operation mode Interpolation processing is performed between the current picture and the reference picture. The method for moving motion blur based on dynamic compensation according to claim 1, wherein in the step (E), the second processing unit combines a global search method pair in performing the two-way compensation operation mode Interpolation processing is performed between the current picture and the reference picture. The method for moving motion blur based on dynamic compensation according to claim 1, wherein in the step (E), the second processing unit combines the bidirectional compensation operation mode and a motion estimation based on overlapping blocks The algorithm performs interpolation processing between the current picture and the reference picture. The method for removing motion blur based on dynamic compensation according to claim 1 of the patent application, further comprising the step (F), when the total value of the error is greater than or equal to the first threshold, or the total value of the error is smaller than the third When the threshold is set, the third processing unit performs interpolation processing between the current picture and the reference picture in a picture repeat operation mode, and the picture repeat operation mode copies the reference picture to generate the inner picture plane. The method for moving motion blur based on dynamic compensation according to claim 5, wherein the operation mode selection unit indicates that the current picture is one by determining that the total error value is greater than or equal to the first threshold value. Transition screen. The method for moving motion blur based on dynamic compensation according to claim 5, wherein the operation mode selection unit determines that the error total value is less than or equal to the second threshold value and greater than or equal to the third threshold When the value is displayed, it indicates that the current picture is a smooth moving picture. The method for moving motion blur based on dynamic compensation according to claim 5, wherein the operation mode selection unit indicates that the current picture is a static picture by determining that the total error value is smaller than the third threshold value. . The method for moving motion blur based on dynamic compensation according to claim 5, wherein the image data of the current picture and the reference picture is image data of a color space of luminance-chroma. The method for moving motion blur based on dynamic compensation according to claim 1, wherein the image data of the current picture and the reference picture is image data of a color space of luminance-chrominance.