JP2006270823A - Method and device for image rate conversion - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and device by which properly-interpolated motion pictures can be produced even in the case of motion pictures containing a picture having a low degree of correlation with the previous and next pictures due to a scene change and strobe light emitting. <P>SOLUTION: In the picture having the low degree of correlation between blocks of motion vectors and many matching errors between pictures as obtained by an estimated result of motions, repeating pictures for requiring no motion compensation are defined as an interpolation picture or interpolated pictures are produced by fixing a interpolation mode to one direction. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image rate conversion method for converting an image rate that is the number of frames / field per second by signal processing in order to efficiently transmit, store, and display a moving image signal such as a television image with high image quality. And an image rate conversion apparatus.

  The moving image signal is generally interlaced scanning at 60 fields per second. A video captured by a normal video camera has this image rate, but when film is used as a material, the original image is 24 to 30 frames per second. Similarly, in the case of animation and computer graphics, the image rate is lower than 60. Broadcasting for mobiles has 15 frames per second regardless of the original image rate in order to lower the bit rate. Since an image with such a low image rate is not sufficiently smooth, it is desirable to display it at a higher image rate.

  On the other hand, the PAL system and SECAM system used in Europe, China, and the like, and the digital moving image system with 625 scanning lines compatible therewith, have 50 fields per second. When these images are broadcast in Japan or the United States, it is necessary to convert them to 60 fields per second.

  The video of 50 fields per second becomes more prominent when the room is bright, so it is desired to display the video at a higher image rate such as 100 fields per second.

  Unlike a cathode ray tube (CRT), a liquid crystal display (LCD) holds an image for one field. Therefore, a moving image continues to emit light at the same position with respect to the movement of the viewpoint, and there is a phenomenon that time integration appears to be blurred. In order to solve this, the light emission time may be shortened, but the luminance of the image is reduced accordingly. In order to reduce blur without causing a decrease in luminance, the image rate is increased and the time for one field is shortened.

  For these many reasons, image rate conversion processing is required. In order to convert the image rate, it is necessary to form a new image by interpolation at a time position different from that of the original image signal. As an image interpolation method, there are a case where processing in the time direction is performed spatially at the same image position without using motion compensation, and a case where an image is formed by moving the image spatially using motion compensation. .

  In the case of conversion for matching signal forms that do not take image quality problems into consideration or flicker reduction, the object can be achieved without necessarily performing motion compensation. However, in the case of simple image repetition without motion compensation, the smoothness of motion and motion blur on the LCD are not improved. In the case of weighted addition (linear interpolation) from before and after the motion compensation is not performed, the motion is not improved, and a double image is generated and the image becomes difficult to see.

  In order to improve the smoothness of motion and motion blur on the LCD, it is necessary to form an interpolated image using motion compensation. However, unlike image coding, etc., there is no image to be interpolated in the first place, so motion estimation for motion compensation is performed using surrounding (front and back) images, and appropriate motion estimation is not easy. Absent.

  Here, the image rate includes a field rate for interlace scanning, a frame rate for sequential scanning, the number of frames per second of the film, and the like. The unit of the image rate is indicated by fps, but there can be both field per second and frame per second.

FIG. 3 shows a configuration of a conventional example of image rate conversion. The main parts of this image rate conversion are those shown in Patent Documents 1 and 2, and three types of forward interpolation from the past frame, backward interpolation from the future frame, and bidirectional linear interpolation from both frames. The interpolation method is adaptively switched in units of blocks according to the motion of the image.
A moving image coming from the moving image input terminal 1 is temporarily stored in the image buffer 2. This image is supplied to three types of interpolators, a forward interpolator 3, a bidirectional interpolator 4, and a reverse interpolator 5, and an interpolated image formed by each interpolator is supplied to an adaptive interpolation selector 6. The adaptive interpolation selector 6 determines the quality of each interpolation image and selects an appropriate interpolation image in units of blocks. The interpolated image thus obtained is given to the image order controller 8. The image order controller 8 outputs the input moving image given from the image buffer 2 and the interpolated image given from the adaptive interpolation selector 6 in an appropriate order. A moving image signal whose image rate is converted in this way is output from the moving image output 9.
JP 2004-320278 A JP 2004-320279 A

  The conventional moving image code stream multiplexing apparatus performs adaptive interpolation, so images such as covered (the object is covered with the background and the background disappears) and uncovered (the background where the object moved and hidden) appear. It responded to the partial change of. However, when the brightness of the entire screen differs greatly only for one image due to scene changes or strobe light emission, it is not possible to properly perform motion estimation for other images with extremely low correlation between images. When the images before and after the interpolation target are completely different as described above, if the interpolation is performed from the front and back adaptively, the different images are switched for each part, and the entire image tends to be inappropriate.

  The present invention has been made paying attention to the above points, and as a result of motion estimation, the interpolation mode is fixed to one direction in an image having a very low correlation between blocks of a motion vector or an image having a very low correlation between images, It is an object of the present invention to provide an image rate conversion method and an image rate conversion apparatus that can obtain an image without failure by repeating without performing motion compensation.

  In the present invention, when converting the image rate of a moving image by inter-image interpolation using motion compensation, a motion vector in units of blocks is obtained between the images of the incoming image, and the inter-block correlation of the motion vector is calculated within one image. At least one of the motion vector activity detected in step 1 and the motion vector matching error detected in one image is obtained as an activity, and the activity is compared with a predetermined value. When the value is lower than a predetermined value, the normal interpolation mode is used to form an interpolated image by interpolation between motion compensated images. An image rate conversion method and an image rate conversion apparatus for arranging and outputting an interpolated image and an incoming image in a predetermined time relationship.

  Also, when converting the image rate of a moving image by inter-image interpolation using motion compensation, a motion vector in units of blocks is obtained between the images of the incoming image, and the inter-block correlation of the motion vector is detected within one image. At least one of the detected motion vector activity and the matching error between the motion vectors detected in one image is obtained as an activity, and the activity is compared with a predetermined value. If the value is lower than the value, an interpolation image is formed by interpolating motion compensated images from the previous and subsequent images as the normal interpolation mode. An image rate conversion method and an image rate conversion apparatus for forming an image and arranging and outputting the interpolated image and the incoming image in a predetermined time relationship.

  In the present invention, when performing adaptive interpolation from the preceding and following images, as a result of motion estimation, the correlation between the motion vectors is low and the interpolation mode is fixed in one direction in an image with many matching errors between images. If the brightness of the entire screen is significantly different due to scene changes or strobe light emission, the interpolated image from different directions is not switched for each part, such as other images with extremely low correlation between images. Interpolated images can be formed.

  In addition, as a result of motion estimation, in an image with a low motion vector inter-block correlation and a large inter-image matching error, it is possible to repeat the motion vector without performing motion compensation. An appropriate interpolated image can be formed as one image without unstable change.

<Image rate conversion in the first embodiment>
An image rate conversion method and an image rate conversion apparatus according to a first embodiment of the present invention will be described. FIG. 1 shows the configuration, and the same components as those in the conventional example of FIG. In FIG. 1, a motion estimator 10, an activity detector 11, a mode determiner 12, and an image switch 7 are added as compared to FIG.

  In the embodiment, the difference from the conventional example is a method of forming an interpolated image in a scene change or low correlation image, a method of forming an interpolated image in a normal image, and a moving image having a different image rate after the interpolated image is formed. The forming method is basically the same as the conventional example.

  Here, both the input image and the output image are sequentially scanned images, the conversion is only the image rate, and the conversion of the scanning line structure and the number of pixels in the image are not performed.

<Adaptive motion compensation interpolation>
A moving image signal coming from the moving image input terminal 1 is given to the image buffer 2. The image buffer 2 holds images for several frames, and the held images are output in a timely manner according to the processing. An image signal for forming an interpolated image is supplied to the forward interpolator 3, the bidirectional interpolator 4, and the backward interpolator 5. Each interpolator performs motion estimation between predetermined frames, determines a motion vector, and motion-compensates the reference image according to the motion vector to form an interpolated image. This is shown in FIG.

  In the forward interpolator 3, an image input in the past from the interpolated image is used as a reference image, and the reference image is spatially moved to form an interpolation image. In the bi-directional interpolator 4, both an image input in the past and an image input after the interpolated image are used as reference images, both reference images are moved in a spatially opposite relationship, and both reference images are moved. Is added to form an interpolated image. In the backward interpolator 5, an image input after the interpolated image is used as a reference image, and the reference image is spatially moved to form an interpolated image.

  These three types of interpolation images are selected by the adaptive interpolation selector 6. Which interpolation image is selected is described in detail in Japanese Patent Application Laid-Open No. 2004-320279, and is determined in units of blocks according to the matching state between images including the images before and after. The adaptive interpolation image obtained in this way is given to the image switch 7.

<Forced one-way interpolation>
The image switch 7 switches the forward interpolation image, the backward interpolation image, and the adaptive interpolation image in units of images according to the interpolation mode information given from the mode determination unit 12. The selected final interpolation image is given to the image order controller 8. In order to clarify the processing function, the adaptive interpolation selector 6 and the image switch 7 are shown separately. However, since the switching process for each image can be realized by the switching process for each block, the operation of the adaptive interpolation selector 6 is described. Can also be realized as a common block in the image.

  The image order controller 8 replaces the final interpolated image and the input image given from the image buffer 2 in accordance with the time relationship, and outputs them from the moving image output 9 as a moving image signal of the target image rate. When the conversion is 1: 2 such as 30 fps to 60 fps, the ratio of the interpolation image and the original image is the same, and the interpolation image and the input image are alternately inserted. At this time, since the interpolated image is delayed due to the inter-image processing, it is necessary to delay the input image.

<Determination of interpolation mode>
The image temporarily stored in the image buffer 2 is given to the motion estimator 10. The motion estimator 10 performs motion estimation between images before and after the interpolation target image. Since this motion estimation is similar to the motion estimation performed by the bidirectional interpolator 4, it can also serve as processing. The motion estimator 10 gives the obtained motion vector (MV) and the minimum block matching error amount of each block calculated by motion estimation to the activity detector 11.

The activity detector 11 calculates the MV inter-block correlation and the block matching error amount for each image. For the MV correlation between blocks, an accurate autocorrelation value may be calculated, but here, it is assumed that the absolute value of the difference for each component between adjacent blocks is added for one image (D _V ). FIG. 5 shows the state of MV. In the case of a normal image, MV has a considerably high correlation as shown on the left side. On the other hand, if the images for which motion estimation is performed are completely different, the MV is random as shown on the right side, and the correlation between blocks is low.

Block-matching error amount D _M is an index of the inter-image correlation, a block sum of squared errors or absolute errors, adds one image. Only one of the MV inter-block correlation and the matching error amount may be used. These parameters are given to the mode determiner 12 when one image is completed.

The mode determiner 12 compares each parameter with a predetermined threshold determined in advance according to the image format, and sets the one-way mode when the parameter is larger than the threshold. When there is only one type of parameter, the determination is made by comparing with the threshold values (T _V1 , T _M1 ). However, when there are two types of parameters, another threshold value (T _V2 , T _V2 , T _M2 ) is provided.

To summarize, the forced unidirectional interpolation mode is set when any of the conditions D _V > T _V1 , D _M > T _M1 , (D _V > T _V2 and D _M > T _M2 ) is satisfied, and otherwise Is the normal interpolation mode. Note that T _V1 > T _V2 and T _M1 > T _M2 . T _V is T _V1 = 10 pixels, T _V2 = 5 pixels, etc. as one component (vertical or horizontal) of one vector. T _M is a value per pixel (number divided by the number of pixels in one image), such as T _M1 = 40, T _M2 = 20, etc., when the input signal has an 8-bit accuracy and an absolute value error.

  In the case of forced one-way interpolation, whether forward interpolation or reverse interpolation is used is interpolated from images that are close in time depending on the time positions of the interpolated image and the preceding and succeeding images. This state is shown in FIG. 7, which is a 1: 5 conversion from 24 fps to 120 fps. In this case, four images are interpolated between the input images, but the front two images are in the forward direction and the rear two images are in the reverse direction. In the case of 1: 2 such as conversion from 30 fps to 60 fps, since the interpolated image is exactly in the center, the forward direction and the reverse direction are equivalent, and the direction is determined in advance.

<Image Rate Conversion in Second Embodiment>
An image rate conversion method and an image rate conversion apparatus according to the second embodiment of the present invention will be described. FIG. 2 shows the configuration, and the same components as those in the first embodiment of FIG. 2 includes a motion compensation interpolator 22 instead of the forward interpolator 3, the bidirectional interpolator 4, the reverse interpolator 5, and the adaptive interpolation selector 6 as compared with FIG. The operation of the switch 23 is different. In the following, only the parts that are different from the first embodiment will be described.

  The motion compensation interpolator 22 is arbitrary about the interpolation processing method, and may be the same as that of the first embodiment, but may be simpler one-way interpolation. However, if there is no motion compensation, there is no effect in improving the smoothness of motion and improving the motion blur of the LCD, so motion compensation is performed. The interpolation image formed by the motion compensation interpolator 22 is given to the switch 23. The image switch 23 switches the interpolation image and the input image supplied from the image buffer in units of images according to the interpolation mode information. The final interpolated image selected by the image switch is given to the image order controller 8. The operation of the image order controller 8 and the subsequent operations are the same as those in the first embodiment. Here, the input image supplied from the image buffer 21 is an image as it is without motion compensation, and the interpolation image uses motion compensation.

The operations of the motion estimator 10 and the activity detector 11 are the same as those in the first embodiment. However, the determination threshold of the mode determiner 24 is slightly different. In the second embodiment, since motion compensation is stopped, determination is made with emphasis on motion vector activity rather than inter-image matching. Specifically, T _V1 and T _V2 are left as they are, and T _M1 and T _M2 are slightly increased. As a result, the probability of entering the non-normal mode is slightly lower than in the first embodiment, but it is reasonable because motion compensation is performed in the first embodiment.

<Image Rate Conversion in Third Embodiment>
An image rate conversion method and an image rate conversion apparatus according to a third embodiment of the present invention will be described. This is a combination of the first embodiment of FIG. 1 and the second embodiment of FIG. Although the configuration is not shown, it is almost the same as in FIG. 1 except for the operations of the image switch 7 and the mode determiner 12.
As an option of the image switch 7, the output of the image buffer 2 is also added, and the non-normal interpolation mode has two stages of unidirectional interpolation and unidirectional interpolation without motion compensation. In accordance with this, the mode determination is also performed in two stages.

It is a figure which shows the structural example of the image rate conversion method thru | or image rate conversion method of 1st Example of this invention. It is a figure which shows the structural example of the image rate conversion method thru | or image rate conversion method of 2nd Example of this invention. It is a figure which shows the structural example of the conventional image rate conversion apparatus. It is a figure which shows the mode of image rate conversion. It is a figure which shows the mode of a motion vector. It is a figure which shows the mode of the interpolation by an interpolation direction. It is a figure which shows an image time position and an interpolation direction.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Moving image input terminal 2, 2, 21 ... Image buffer, 3 ... Forward direction interpolator, 4 ... Bidirectional interpolator, 5 ... Reverse direction interpolator, 6 ... Adaptation Interpolation selector, 7, 23 ... Image switch, 8 ... Image order controller, 9 ... Moving image output terminal, 10 ... Motion estimator, 11 ... Activity detector, 12, 24: Interpolation mode determiner, 21: Image buffer, 22: Motion compensation interpolator

Claims (4)

An image rate conversion method for converting an image rate of a moving image by inter-image interpolation using motion compensation,
Motion estimation (10) for obtaining a motion vector in units of blocks between incoming images;
Activity level for obtaining at least one of the motion vector activity level in which the inter-block correlation of the motion vector is detected in one image and the inter-image activity level in which the motion vector matching error is detected in one image and outputting it as the activity level Detection (11);
Interpolation that outputs interpolation mode information by comparing the degree of activity with a predetermined value and in normal interpolation mode when the degree of activity is lower than the predetermined value and as one-way interpolation mode when the degree of activity is higher than the predetermined value Mode determination (12);
When the interpolation mode is the normal interpolation mode, an interpolation image is formed from the previous and subsequent images by interpolation between motion compensated images. When the interpolation mode is the unidirectional interpolation mode, an interpolation process (3 4, 5, 6, 7)
An image rate conversion method comprising: performing interpolation conversion (8) for arranging and outputting an interpolation image formed by the interpolation means and an incoming image in a predetermined time relationship. An image rate conversion method for converting an image rate of a moving image by inter-image interpolation using motion compensation,
Motion estimation (10) for obtaining a motion vector in units of blocks between incoming images;
Activity level for obtaining at least one of the motion vector activity level in which the inter-block correlation of the motion vector is detected in one image and the inter-image activity level in which the motion vector matching error is detected in one image and outputting it as the activity level Detection (11);
Interpolation mode determination that outputs interpolation mode information by comparing the degree of activity with a predetermined value, and when the degree of activity is lower than a predetermined value, a normal interpolation mode, and when the degree of activity is higher than a predetermined value, a repetition mode (24) and
When the interpolation mode is a normal interpolation mode, an interpolation image is formed by interpolation between motion compensated images, and when the interpolation mode is a repetition mode, an interpolation process (22, 23) for forming an interpolation image identical to an adjacent frame;
An image rate conversion method comprising: performing interpolation conversion (8) for arranging and outputting an interpolation image formed by the interpolation means and an incoming image in a predetermined time relationship. An image rate conversion device that converts an image rate of a moving image by inter-image interpolation using motion compensation,
Motion estimation means (10) for obtaining a motion vector in units of blocks between images of incoming images;
Activity level for obtaining at least one of the motion vector activity level in which the inter-block correlation of the motion vector is detected in one image and the inter-image activity level in which the motion vector matching error is detected in one image and outputting it as the activity level Detection means (11);
Interpolation that outputs interpolation mode information by comparing the degree of activity with a predetermined value and in normal interpolation mode when the degree of activity is lower than the predetermined value and as one-way interpolation mode when the degree of activity is higher than the predetermined value Mode determination means (12);
Interpolation means for forming an interpolated image from the previous and subsequent images by interpolation between motion compensated images when the interpolation mode is the normal interpolation mode, and forming an interpolated image from only the unidirectional image in the unidirectional interpolation mode ( 3, 4, 5, 6, 7),
An image rate conversion apparatus comprising: an interpolation image formed by the interpolation means; and an arrangement conversion means (8) for arranging and outputting an incoming image in a predetermined time relationship. An image rate conversion device that converts an image rate of a moving image by inter-image interpolation using motion compensation,
Motion estimation means (10) for obtaining a motion vector in units of blocks between images of incoming images;
Activity level for obtaining at least one of the motion vector activity level in which the inter-block correlation of the motion vector is detected in one image and the inter-image activity level in which the motion vector matching error is detected in one image and outputting it as the activity level Detecting means (11);
Interpolation mode determination that outputs interpolation mode information by comparing the degree of activity with a predetermined value, and when the degree of activity is lower than a predetermined value, a normal interpolation mode, and when the degree of activity is higher than a predetermined value, a repetition mode Means (24);
Interpolating means (22, 23) for forming an interpolated image by interpolation between motion compensated images when the interpolation mode is the normal interpolation mode, and forming the interpolated image as the same as the adjacent frame in the repetitive mode;
An image rate conversion apparatus comprising: an interpolation image formed by the interpolation means; and an arrangement conversion means (8) for arranging and outputting an incoming image in a predetermined time relationship.

Images