JP2008245135A - Interpolation frame generating method and interpolation frame generating device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an interpolation frame generating technique for appropriately coping with a defect of an image at the end part of a screen. <P>SOLUTION: The interpolation frame generating method for generating a new frame image positioned between input frame images by using two or more inputted frame images has steps of: detecting a motion vector of an object within the frame image by block matching processing between the input frame images, and generating an interpolation image using the detected motion vector. In the step of generating the interpolation image, when the corresponding pixel of the detected motion vector direction is positioned outside the screen on one side of a source frame, the interpolation image is generated using only the pixels of the frame on the side having no pixel positioned outside the screen. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for creating and inserting an interpolation frame between frame images constituting a moving image and displaying the motion of an object as a smooth and natural motion.

  A motion vector is obtained by performing block matching between two or more input frame images before and after, and a new interpolation frame image to be interpolated between the input frame images is created using this to perform frame number conversion. Thus, many methods for improving the motion blur of the hold display type display have been proposed.

  For example, when displaying a moving image on a liquid crystal display (LCD), the LCD displays a frame image (hereinafter simply referred to as a frame) at a rate of 60 frames / second, for example. This frame is a progressive scanning signal obtained by processing an interlace of 60 fields / second, for example. That is, the LCD continues to display one frame for 1/60 second.

  When viewing such a video displayed on the LCD, the image of the previous frame remains as an afterimage in human eyes. For this reason, a moving object in the image may appear blurred or the movement of the object may appear unnatural. Such a phenomenon becomes more prominent as the screen becomes larger.

  In order to prevent such blurring of moving images, a method of displaying a moving image by inserting an interpolation frame between two consecutive frames is known (Patent Document 1). In this method, the motion vector (the direction and distance the object moved) of each block is detected by matching the image blocks that make up the frame between two or more input frames of the previous frame and the subsequent frame. Is done. Using the motion vector of each block, a new interpolated frame located between the input frames is created. By inserting an interpolated frame between two input frames, the number of frames is increased to display a moving image.

  The block matching is a method for detecting which image block in a certain frame matches an image block of a predetermined size. The difference between the pixels corresponding to each other in the image block in the previous frame and any of the image blocks in the subsequent frame is calculated, and the subsequent frame in which the accumulated value (SAD: Sum of Absolute Difference) is minimized The middle image block is detected as the image block most similar to the previous frame middle image block. The difference in the position of the most similar image block between the previous frame and the subsequent frame is detected as a motion vector.

By the way, when an interpolation frame is generated using a motion vector estimated by block matching, a failure may occur, and the countermeasure in Patent Document 2 is to identify a motion vector at an interpolation pixel position and adjacent based on a vertical component of the vector. This is a pixel interpolation processing device that selects and corrects a pixel to be corrected. That is, there is a case where an uncovered area where pixels are not written may occur, and a patch-like countermeasure method is described in which correction of different selected pixels is performed on the area depending on motion vector conditions. However, there is a need for a countermeasure against a potential global image failure at the edge of the screen, but a technique that can appropriately cope with it is not disclosed.
JP 2005-6275 A JP 2003-299035 A

  An interpolation frame creation technique capable of appropriately dealing with an image failure at the edge of the screen is provided.

  According to one aspect of the present invention for achieving the above object, an interpolation frame creating method for creating a new frame image located between the input frame images using two or more input frame images. A step of detecting a motion vector of an object in a frame image by a block matching process between the input frame images, and a step of creating an interpolation image using the detected motion vector, If the corresponding pixel position in the detected motion vector direction exists outside the screen on one side of the original frame, the creating step is interpolated using only the pixels of the frame on the side having no pixels located outside the screen. An interpolation frame creation method characterized by creating an image is provided.

  According to the present invention, it is possible to provide an interpolation frame creation technique that can appropriately cope with the failure of the image at the edge of the screen.

  FIG. 1 is a block diagram showing an embodiment of an interpolation frame creation device (frame number conversion device) according to the present invention. The interpolated frame creation apparatus 10 in FIG. 1 includes a frame memory unit 10, a motion vector detection unit 12, an interpolated image creation unit 13, and a determination unit 14. The motion vector detection unit 12 detects a motion vector by block matching processing from, for example, two consecutive frames in the input image signal. That is, the motion vector detection unit 12 calculates a difference absolute value between pixel values corresponding to each other in image blocks in two input frame images, and a motion vector corresponding to a minimum value of the cumulative addition value (SAD) of the difference absolute value. Are selected as motion vector candidates. The frame rate of the input image signal is, for example, 60 frames / second.

The interpolated image creating unit 13 creates an interpolated frame based on the detection result of the motion vector detecting unit 12, and inserts it between the two frames. The frame rate of the output image signal into which the interpolation frame is inserted is, for example, 120 frames / second.
The determination unit 14 determines whether the reference pixel of the motion vector detected by the motion vector detection unit 12 is within the effective screen or outside the effective screen. That is, the determination unit 14 grasps the position of the reference pixel of the motion vector detected by the motion vector detection unit 12 with a counter (not shown) inside the determination unit 14, and based on the result, whether the reference pixel is within the effective screen or the effective screen Judge whether it is outside.

  That is, the determination unit 14 determines for each pixel whether the position of the reference pixel of the detected motion vector is within the effective screen or outside the effective screen. Then, it is determined for each pixel whether an interpolation frame is generated using the detected motion vector or an interpolation frame is generated using a part of the motion vector as described later. Then, the result is passed to the interpolation image creation unit 13. The interpolation image creation unit 13 creates an interpolation image based on the determination result of the determination unit 14.

  The interpolation image creation unit 13 creates an interpolation frame using the motion vector detected by the motion vector detection unit 12. At this time, an interpolation frame is created using the pixel based on the motion vector determined as the motion vector to be used by the determination unit 14 as an effective pixel. Alternatively, the interpolation image creation unit 13 creates an image created based on the motion vector determined as the motion vector to be used by the determination unit 14, and an image created based on the motion vector determined as another motion vector to be used. Are combined (blended) to create an interpolation frame. For example, the average pixel value is taken.

  The motion vector detection unit 12, the interpolation image creation unit 13, and the determination unit 14 can be configured as hardware using individual electronic circuits or software executed by a CPU (not shown).

  FIG. 2 is a diagram for explaining an example of the block matching process. That is, as a motion vector detection method using block matching, a pixel of a corresponding position is obtained by translating a block of a predetermined shape with respect to a point object on two frames before and after the insertion position of the interpolated frame image. A method of calculating a difference value between pixel values for all the pixels in the block, obtaining a cumulative value (SAD: Sum of Absolute Difference), and setting the direction in which the SAD value is minimum as the motion vector of the block, etc. It has been known.

As shown in FIG. 2, there is a method for obtaining a motion vector through block matching processing between image blocks located at point-symmetric positions. That is, in the method of FIG. 2, image blocks that are point-symmetrical on the front frame 20 and the rear frame 22 sandwiching the insertion position of the interpolation image block 41 in the interpolation frame 21 are compared for each pixel. To calculate SAD. A vector connecting the image blocks that are most similar (having the smallest SAD) is determined as a motion vector. This comparison is performed in a predetermined search range 40 in the previous frame 20 and a corresponding search range 42 in the rear frame 22.

  When the combination of image blocks most similar to each other is, for example, the image block 43 and the image block 44, a vector from the image block 43 to the image block 44 is determined as the motion vector of the interpolated image block 41. Based on this motion vector and image data of image blocks 43 and 44 that are most similar to each other, an interpolation image block 41 in the interpolation frame 21 is created. In addition, although this motion vector (including magnitude and direction) is shown in FIG. 2 as three-dimensional for convenience of explanation, in actual processing, it is a vector indicated in two dimensions on the frame.

  In the method shown in FIG. 2, a block having a specified shape is translated in a point-symmetric manner on two frames before and after the insertion position of the interpolated frame image as a center. Then, the difference value of the pixel values of the pixels at the corresponding positions is calculated for all the pixels in the block, an accumulated value (SAD) is obtained, and the direction in which the SAD value is minimum is determined as the motion vector of the block. To do.

Next, the image block size and motion vector search range will be described.
FIG. 3 shows a specific example of the macroblock size and its search range. These blocks and search ranges can be applied to the block matching process shown in FIG. Here, in order to simplify the description, the block matching process in the horizontal direction will be mainly described.

  In FIG. 3, MB is a macroblock, and MSR is a search range of the macroblock MB. The size of the macroblock MB is, for example, 64 pixels × 4 pixels as shown by the inner solid line in FIG. This macro block MB is shifted on the previous frame 20 by −16 to +16 pixels in the horizontal direction and −2 to +2 pixels in the vertical direction. At the same time, the macro block MB is shifted on the rear frame 22 by +16 to −16 pixels in the horizontal direction and +2 to −2 pixels in the vertical direction.

  That is, when the interpolated image block 41 (in this case, a block having a size matching the macroblock MB) inserted on the interpolated frame 21 is used as the center of the point object, the macroblock MB is shifted, for example, by +12 pixels on the previous frame 20 Then, the macro block MB is shifted by -12 pixels on the rear frame 22. SAD is calculated by comparing pixel values of corresponding image blocks in both frames 20 and 22 in units of corresponding pixels. Therefore, in the case of block matching using the macroblock MB, the search range MSR on the previous frame 20 and the subsequent frame 22 are both 96 pixels × 8 pixels here.

  The optimum block size for accurately performing motion vector detection by block matching varies depending on the resolution of the input frame and how the object included in the frame moves.

Hereinafter, the block matching processing operation by the motion vector detection unit 12 will be described.
FIG. 4 is a graph showing the relationship (SAD characteristics) between the shift amount of the image block and the SAD in the block matching process. This block matching process is performed by the motion vector detection unit 12. In order to simplify the description, only the block matching processing for horizontal object movement will be described.

  In this case, when the target image block 43 in the previous frame including the object is shifted in units of one pixel from the position of the central image block 44 in the search range in the rear frame 22, it is shifted by 10 pixels as shown in FIG. SAD is minimal (here, minimum). A motion vector is detected based on the shift amount S0 and the direction of the minimum point PS0. Therefore, here, the motion vector of the target image block 41 is detected as 10 pixels in the horizontal direction. As a result, for example, an image block obtained by shifting the target image block 43 by five pixels in the horizontal direction from the corresponding corresponding position in the interpolation frame is created as an intra-interpolation image block. The shift amount S0 and its direction indicate the position within the search range of the image block 41.

Hereinafter, the processing operation of the screen edge by the motion vector will be described.
For the sake of simplicity, an example in which an image that is horizontally scrolled on the whole screen is input to the screen as shown in FIG. In the normal interpolation pixel generation process, if both the pixels in the preceding and following frames are in the screen indicated by the broken line as shown in FIG. Interpolated pixels are generated. The average value of the base pixels of V1 and V2, and the average value of the base pixels of V3 and V4 are used.

  FIG. 6 shows an example of a case where an image that is horizontally scrolled horizontally is input to the screen as shown in FIG. 5, but as shown in FIG. This is a case of being located outside the screen of the original frame image.

  In this case, an interpolated image is generated only from the pixels of the frame on the side that does not have a corresponding pixel outside the screen. That is, the value of the base pixel of V2 and the value of the base pixel of V3 are used. As a result, it is possible to generate an interpolated image with little failure even at the screen edge.

(Operation details of interpolation frame creation device)
Next, the operation content of the interpolation frame creation device 10 will be described. The interpolation frame creation device 10 performs an interpolation frame creation process according to the flowchart shown in FIG. 7, and creates an interpolation frame.

  When the interpolation frame creation process is started, the interpolation frame creation device 10 proceeds to step 1 to perform motion vector detection, and then proceeds to step 2 to create an interpolation frame.

In motion vector detection, the motion vector detection unit 12 performs block matching of two image frames to detect a motion vector.
FIG. 8 is a flowchart showing a processing procedure of the interpolation frame creating apparatus 10 in which the processing of FIG. In FIG. 8, the motion vector detection unit 12 to the determination unit 14 assume that the detected motion vector of the determination target P [X, Y] is input as (2Xa, 2Ya), for example (step S21 in FIG. 8). Then, the determination unit 14 determines the motion vector range (whether a correct image can be obtained) of the interpolated image P [X, Y] based on the motion vector as [X−Xa, Y−Ya] and [X + Xa, Y + Ya]. ] (Step S22).

  If both [X-Xa, Y-Ya] and [X + Xa, Y + Ya] are in the correct image range, the determination unit 14 determines that all the pixels based on the detected motion vector are correct, and uses this motion vector. An interpolation frame is created (step S23). If one of the two coordinates is not in the correct image range, the determination unit 14 notifies the interpolation image creation unit 13 to that effect. In response to this, the interpolated image creation unit 13 creates an interpolated frame from pixels based on the correct motion vector of the image (step S24).

  For example, in the case of the search range in FIG. 3, if the X coordinate is 0 and Xa is 16, for example, 16 pixels in the horizontal direction of the rear frame are not averaged with the pixels of the previous frame and are used to create an interpolation frame. It will be. If Xa is 16, for example, at the opposite end, 16 pixels in the horizontal direction of the previous frame are used for creating an interpolated frame without being averaged with the pixels of the subsequent frame.

  In motion vector detection, pixel data at the edge of the screen continues outside the screen, or is detected to some extent by using a method in which pixel values outside the screen are processed as specific fixed values such as gray luminance. I can do it. However, if the same method is used at the time of generating the interpolation pixel, pixel generation is performed using data that does not exist originally, so that the generated image is broken.

  In motion vector detection, a method in which cumulative addition processing is not performed on pixels located outside the screen can be used. In this case, since SAD becomes small and advantageous, adjustment such as normalization by multiplying SAD by the reciprocal of the pixel ratio for which cumulative addition has been performed is suitable.

  Usually, an interpolated image is generated from the corresponding pixels on the previous and subsequent frames in the motion vector direction obtained as described above. There are cases where there are pixels that are effective only on one side. For this reason, there is a problem that an interpolation image cannot be correctly generated at the edge of the screen in the method of taking the average value of corresponding pixels in the two previous and subsequent frames to obtain an interpolation pixel.

  This embodiment is made to solve the above problem. When an interpolation image is generated, the pixel corresponding to the motion vector direction is located outside the screen of the original frame image on one side at the edge of the screen. Generates an interpolated image from only the pixels of the frame on the side having no corresponding pixel outside the screen. As an effect, it is possible to generate an interpolated image with less failure at the screen edge.

The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention.
Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements according to different embodiments may be appropriately combined.

The block block diagram which shows one Embodiment of the interpolation frame production apparatus by this invention. The figure for demonstrating an example of a block matching process. The figure which shows the specific example of the size of a block, and its search range. The graph which shows the relationship between the shift amount of the image block in a block matching process, and SAD. FIG. 1 represents a motion vector used for creating an interpolation image in a specific area in the screen. FIG. 2 represents a motion vector used for creating an interpolated image in a specific area in the screen. 5 is a flowchart showing an operation procedure of processing in the interpolation frame creation device 10; 5 is a flowchart showing an operation procedure of interpolation frame creation processing in the interpolation frame creation device 10.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Interpolation frame production apparatus, 11 ... Frame memory, 12 ... Motion vector detection part, 13 ... Interpolation image creation part, 14 ... Determination part

Claims (8)

An interpolation frame creation method for creating a new frame image located between the input frame images using two or more input frame images,
Detecting a motion vector of an object in a frame image by block matching processing between the input frame images;
Creating an interpolated image using the detected motion vector,
In the step of creating the interpolation image, when the corresponding pixel position in the detected motion vector direction exists outside the screen on one side of the original frame, only the pixel in the frame on the side not having the pixel located outside the screen An interpolated frame creating method characterized by creating an interpolated image using an image. 2. The interpolation frame creation method according to claim 1, wherein in the step of detecting the motion vector, a pixel positioned outside the screen is regarded as a pixel positioned at an end of the screen extending. 2. The interpolation frame creation method according to claim 1, wherein in the step of detecting the motion vector, a pixel located outside the screen is regarded as having a specific fixed value. 2. The interpolation frame creation method according to claim 1, wherein the step of detecting the motion vector performs a cumulative addition process of pixel differences in addition to pixels located outside the screen. An interpolation frame creation device that creates a new frame image located between the input frame images using two or more input frame images,
Detecting means for detecting a motion vector of an object in a frame image by block matching processing between the input frame images;
Creating means for creating an interpolated image using the detected motion vector,
If the corresponding pixel position in the detected motion vector direction exists outside the screen on one side of the original frame, the creating means for creating the interpolated image is a pixel of the frame on the side not having the pixel located outside the screen. An interpolated frame creating apparatus that creates an interpolated image using only 6. The interpolated frame creation apparatus according to claim 5, wherein the detection means for detecting the motion vector regards a pixel located outside the screen as a pixel located at an end of the screen extending. 6. The interpolated frame creation apparatus according to claim 5, wherein the detection means for detecting the motion vector regards a pixel located outside the screen as having a specific fixed value. 6. The interpolated frame creation apparatus according to claim 5, wherein the detection means for detecting the motion vector performs cumulative addition processing of pixel differences in addition to the pixels located outside the screen.

Images