JP2011087228A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and an image processing method capable of performing frame rate conversion processing where deterioration in image quality in a telop region is suppressed. <P>SOLUTION: This image processing apparatus has a telop detecting means for detecting a region of a telop and the motion of the telop from an input video image; a motion vector detecting means for dividing the input video image into a plurality of blocks and detecting a motion vector by a block; and an interpolation frame generating means for generating an interpolation frame using the motion vector. When the region of the telop is detected by the telop detecting means, the interpolation frame generating means alternately perform first interpolation frame generation processing and second interpolation frame generation processing. In the first interpolation frame generation processing, a motion vector based on the motion of the telop is used as a motion vector of the block corresponding to the region of the telop. In the second interpolation frame generation processing, a motion vector detected by the motion vector detecting means is used as the motion vector of the block corresponding to the region of the telop. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method.

  A technique for increasing the frame rate of an input video by detecting motion information (motion vector) between two consecutive frames of the input video and generating an interpolated frame for interpolating between the frames. Yes (frame rate conversion process). When generating an interpolation frame, the input video is divided into a plurality of blocks, and a motion vector for each block is detected. Then, an interpolation frame is generated using the detected motion vector. If such a technique is used, the motion in the video can be expressed smoothly.

  When the video includes a telop, the viewer is likely to focus on the telop. Therefore, a great effect can be expected by generating an interpolation frame and displaying the telop smoothly. Patent Document 1 discloses a technique for detecting a telop area from a video and generating an interpolation frame (interpolation frame generation process) using a motion vector based on the motion of the telop as a motion vector of a block corresponding to the telop area. Is disclosed. According to the technique disclosed in Patent Document 1, telops can be displayed smoothly.

  However, a block corresponding to a telop area often includes a telop and a non-telop video (hereinafter referred to as background), and the telop and background often have different motions. Therefore, when a motion vector based on the motion of a telop is used as the motion vector of such a block, the background is dragged by the motion of the telop, and the background is disturbed in the generated interpolated frame image. In addition, if a motion vector based on the background motion is used, the telop is disturbed. That is, the image quality deteriorates regardless of whether the motion vector based on the background motion or the motion vector based on the telop motion is used as the motion vector of the block corresponding to the telop region.

JP 2008-107753 A

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method capable of performing a frame rate conversion process in which deterioration of image quality in a telop area is suppressed.

  An image processing apparatus according to the present invention is an image processing apparatus that converts a frame rate by inserting an interpolated frame between frames of an input video, and detects a telop area and a motion of the telop from the input video. A telop detection unit; a motion vector detection unit that divides an input video into a plurality of blocks and detects a motion vector for each block; and an interpolation frame generation unit that generates an interpolation frame using the motion vector. The interpolation frame generation means uses the motion vector based on the motion of the telop as the motion vector of the block corresponding to the telop area when the telop area is detected by the telop detection means. And the motion vector detected by the motion vector detecting means as the motion vector of the block corresponding to the telop area. And performing a second interpolation frame generation process using the torque alternately.

  The image processing method of the present invention is an image processing method for converting a frame rate by inserting an interpolated frame between frames of an input video, and detects a telop area and a motion of the telop from the input video. A telop detection step, a motion vector detection step that divides the input video into a plurality of blocks and detects a motion vector for each block, and an interpolation frame generation step that generates an interpolation frame using the motion vector. In the interpolation frame generation step, when a telop area is detected in the telop detection step, a first interpolation frame generation process using a motion vector based on the motion of the telop as a motion vector of a block corresponding to the telop area And motion vector detection as the motion vector of the block corresponding to the telop area And performing a second interpolation frame generation process using the motion vector detected by the step alternately.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of performing a frame rate conversion process in which deterioration of image quality in a telop area is suppressed.

FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing apparatus according to the present embodiment. The figure which shows an example of the block corresponding to the area | region of a telop. The figure which shows an example of the interpolation frame image produced | generated. The figure which shows an example of the interpolation frame image produced | generated. The figure which shows an example of the interpolation frame image produced | generated. The figure which shows an example of the interpolation frame image produced | generated.

Exemplary embodiments of the present invention will be described below with reference to the drawings. The image processing apparatus according to the present embodiment converts a frame rate by inserting an interpolation frame between frames of an input video (input video) (frame rate conversion processing).
<Example 1>
In this embodiment, one interpolated frame for interpolating between frames of an input video is generated, and a video with a frame rate of 60 fps (input video) is converted into a video with a frame rate of 120 fps (output video).

The configuration of the image processing apparatus according to this embodiment will be described below. FIG. 1 is a block diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment.
In FIG. 1, the frame delay unit 101 delays the input frame by one frame period and outputs the delayed frame to the motion vector detection unit 102.
The motion vector detection unit 102 divides the input video into a plurality of blocks, and detects a motion vector for each block. Specifically, the input frame image (frame image) and the frame image one frame before the frame acquired from the frame delay unit 101 are divided into matrix blocks. Then, a motion vector between these frames is detected for each block using an existing technique such as a block matching method. The detected motion vector is output to the telop detection unit 104, the motion vector correction unit 105, and the motion vector switching unit 106.
The interpolation frame generation unit 103 generates an interpolation frame using the motion vector acquired from the motion vector switching unit 106 (interpolation frame generation processing), and outputs it.

The telop detection unit 104 detects the telop area and the movement of the telop from the input video. Specifically, the telop area and its motion are detected by statistically analyzing the motion vector information acquired from the motion vector detection unit 102. The motion vector detection unit 102 detects a block in which a motion vector based on the motion of the telop is detected as a motion vector, and outputs the detection result to the motion vector correction unit 105.
In this embodiment, the telop area and its motion are detected by statistically analyzing the information of the motion vector, but the telop area and its motion detection method are not limited to this. For example, the detection may be performed using pixel data of an input frame image, feature amounts regarding difference data between frames, or both of them.

  The motion vector correction unit 105 corrects the motion vector acquired from the motion vector detection unit 102 based on the telop detection result acquired from the telop detection unit 104, and outputs the correction to the motion vector switching unit 106. For example, the motion vector correction unit 105 acquires information indicating a block in which a telop motion is detected from the telop detection unit 104 as a motion vector, and regards the block and its surrounding blocks as blocks corresponding to the telop area. Then, the motion vectors of these blocks (the motion vectors acquired from the motion vector detection unit 102) are replaced with motion vectors based on the motion of the telop detected by the telop detection unit 104. The motion vectors of blocks other than the block corresponding to the telop area are output as they are.

The motion vector switching unit 106 outputs the motion vector of each block to the interpolation frame generation unit 103. Specifically, the motion vector V1 acquired from the motion vector detection unit 102 or the motion vector V2 acquired from the motion vector correction unit 105 is output. Note that which one of the motion vectors V1 and V2 is output is controlled by the switching timing control unit 107.
The switching timing control unit 107 controls the motion vector switching unit 106. Specifically, the motion vector switching unit 106 is controlled so that the motion vector V1 and the motion vector V2 are alternately switched and output every time an interpolation frame is generated.

  With the above configuration, in the present embodiment, when the telop detection unit 104 detects the telop area in the interpolation frame generation unit 103, the first interpolation frame generation processing and the second interpolation frame generation processing are alternately performed. To be done. The first interpolation frame generation process is an interpolation frame generation process using a motion vector based on the motion of the telop as a motion vector of a block corresponding to the telop area. The second interpolation frame generation process is an interpolation frame generation process using a motion vector detected by the motion vector detection unit 102 as a motion vector of a block corresponding to a telop area. Accordingly, it is possible to perform a frame rate conversion process in which deterioration of image quality in the telop area is suppressed.

Hereafter, the said effect is demonstrated in detail.
FIG. 2 is a diagram showing nine blocks corresponding to a telop area. For convenience of explanation, alphabets (identifiers) A to I are attached to the respective blocks. 3 to 5 show examples of generated interpolation frame images (interpolated frame images). 3 to 5 are examples of images in a telop area in which a telop (character “R”) moves from right to left on a background including a triangular image. Also, the nine blocks shown in FIGS. 3 to 5 correspond to the blocks A to I in FIG. 2, respectively.

First, the example of FIG. 3 will be described.
FIG. 3 shows an example in which an interpolation frame generation process using a motion vector always detected by the motion vector detection unit 102 is performed. The blocks E and F in the interpolated frame image 1 inserted between the frames of the input video (original frame images 1 and 2) have a large proportion (number of pixels) occupied by the telop. A based motion vector (telop motion vector) is detected. On the other hand, in the other blocks (blocks A to D, G to I), since the background occupies a large proportion, a motion vector (background motion vector;
In the example of FIG. 3, “still” is detected. Similarly, in the interpolated frame image 2, a telop motion vector is detected as a motion vector only for the blocks D and E.
In this way, when there are a plurality of objects in the same block, which object's motion vector is detected by the motion vector detecting unit 102 depends on the number of pixels occupied in each object's block and the image feature amount. It depends on complex conditions such as. Therefore, a detection result that is preferable in terms of image quality is not always obtained. As a result, with such a method, the image quality of the interpolated frame image may be greatly degraded.
In the example of FIG. 3, in the telop area of all interpolated frame images, the number of blocks in which the background motion vector is detected is larger than the block in which the telop motion vector is detected. The telop collapses.

Next, the example of FIG. 4 will be described.
FIG. 4 shows an example in which a telop area and its motion are detected, and an interpolation frame generation process is always performed using the telop motion vector as the motion vector of the block corresponding to the telop area. Specifically, a block in which a telop motion vector is detected as a motion vector is detected, and the block and its surrounding blocks are regarded as blocks corresponding to the telop area. Then, the motion vectors of those blocks are replaced with the detected telop motion vectors. As a result, when generating the interpolation frame, the telop motion vector is used as the motion vector of the block corresponding to the telop area (blocks A to I in the example of FIG. 4), and the telop disturbance is suppressed. Interpolated frames can be generated. However, since the motion is often different between the telop and the background, there is a high possibility that the background is disturbed by such a method (in the example of FIG. 4, the background is stationary and is disturbed).
Further, when the motion vector detected by the motion vector detection unit 102 is used, the telop area is not necessarily greatly disturbed in the interpolated frame image. Despite the fact that no significant disturbance occurs in the telop area, when the interpolated frame generation process using the motion vector of the telop as the motion vector of the block corresponding to the telop area is performed, the telop disturbance is not significantly suppressed, The background will be greatly disturbed.
Thus, there is a trade-off relationship between the image quality of the telop and the image quality of the background in the interpolated frame image.

Therefore, in this embodiment, when the telop detection unit 104 detects a telop area in the interpolation frame generation unit 103, the first interpolation frame generation process and the second interpolation frame generation process are alternately performed.
An example of the interpolated frame image generated in this embodiment is shown in FIG. In the example of FIG. 5, as in FIGS. 3 and 4, the motion vector detection unit 102 performs the motion vector of the telop as the motion vectors of the blocks E and F of the interpolated frame image 1 and the blocks D and E of the interpolated frame image 2. Shall be detected. In addition, a background motion vector is detected as a motion vector of another block.

  The telop detection unit 104 detects that the motion vectors of the blocks E and F of the interpolation frame image 1 and the blocks D and E of the interpolation frame image 2 are telop motion vectors. Then, the motion vector correction unit 105 replaces the motion vectors of the blocks E and F of the interpolation frame image 1 and the blocks around the blocks D and E of the interpolation frame image 2 with the motion vector of the telop. Therefore, the motion vectors of the blocks A to I output from the motion vector correction unit 105 when generating the interpolated frame images 1 and 2 are telop motion vectors.

In the present embodiment, when generating the interpolation frame image 1, the motion vector acquired from the motion vector correction unit 105 is used (first interpolation frame generation processing). Specifically, the switching timing control unit 107 controls the motion vector switching unit 106 so that such a motion vector is output to the interpolation frame generation unit 103. In this case, since the motion vector of the telop is used as the motion vector of the blocks A to I, the interpolated frame image 1 is generated as shown in the example of FIG. Will be.

  On the other hand, when generating the interpolation frame image 2, the motion vector acquired from the motion vector detection unit 102 is used (second interpolation frame generation processing). That is, the telop motion vector is used as the motion vector of the blocks D and E, and the background motion vector is used as the motion vector of the other blocks. Therefore, as the interpolated frame image 2, as shown in the example of FIG. 3, the background disturbance is suppressed, but an interpolated frame image in which the telop is disturbed is generated.

  Thus, in this embodiment, the first interpolation frame generation process and the second interpolation frame generation process are alternately performed, and the frame rate of the input video is converted to a higher frame rate. Specifically, an interpolation frame in which the telop disturbance is suppressed (interpolation frame in which the background is distorted) and an interpolation frame in which the background disturbance is suppressed (an interpolation frame in which the telop is distorted) are alternately arranged between the frames of the input video. Inserted. As a result, the frequency of telop disturbance can be reduced as compared with the case where the interpolation frame generation process using the motion vector always detected by the motion vector detection unit 102 is performed as shown in FIG. Further, as shown in FIG. 4, the frequency of background disturbance can be reduced as compared with the case of performing an interpolation frame generation process that always uses a telop motion vector as a motion vector of a block corresponding to a telop area. Therefore, the trade-off problem between the image quality of the telop and the image quality of the background can be alleviated. In addition, since these interpolated frame images are displayed at a high frame rate, the telop and background disturbance of the interpolated frame image are averaged, and large image quality deterioration can be prevented.

<Example 2>
In this embodiment, an input video having a frame rate of 60 fps is converted into an output video having a frame rate of 180 fps by inserting two interpolation frames between the frames of the video.
Note that the functional configuration of the present embodiment is the same as that of the first embodiment (FIG. 1), and thus the description thereof will be omitted. Hereinafter, an example of the generated interpolated frame image will be described in detail.

An example of the interpolation frame image generated in this embodiment is shown in FIG. In the example of FIG. 6, it is assumed that the motion vector detection unit 102 detects the motion vector of the telop as the motion vector of the blocks E and F of the interpolated frame images 1 and 2. In addition, a background motion vector is detected as a motion vector of another block.
The telop detection unit 104 detects that the motion vectors of the blocks E and F of the interpolated frame images 1 and 2 are telop motion vectors. Then, the motion vector correction unit 105 replaces the motion vectors of the blocks around the blocks E and F of the interpolated frame images 1 and 2 with the motion vector of the telop. Therefore, the motion vectors of the blocks A to I output from the motion vector correction unit 105 when generating the interpolated frame images 1 and 2 are telop motion vectors.

  In the present embodiment, when generating the interpolation frame image 1, the motion vector acquired from the motion vector correction unit 105 is used (first interpolation frame generation processing). In this case, since the motion vector of the telop is used as the motion vector of the blocks A to I, the interpolated frame image 1 is generated as shown in the example of FIG. Will be.

On the other hand, when generating the interpolation frame image 2, the motion vector acquired from the motion vector detection unit 102 is used (second interpolation frame generation processing). That is, the telop motion vector is used as the motion vector of the blocks E and F, and the background motion vector is used as the motion vector of the other blocks. Therefore, as the interpolated frame image 2, as shown in the example of FIG. 3, the background disturbance is suppressed, but an interpolated frame image in which the telop is disturbed is generated.

  Thus, in this embodiment, the first interpolation frame generation process and the second interpolation frame generation process are alternately performed, and the frame rate of the input video is converted to a higher frame rate. As a result, similar to the first embodiment, the above-described problem of trade-off between the image quality of the telop and the image quality of the background can be alleviated. Deterioration can be prevented.

<Example 3>
In the present embodiment, as in the first embodiment, an input video having a frame rate of 60 fps is converted into an output video having a frame rate of 120 fps. Further, the interpolation frame generation method is changed according to the movement of the telop. Hereinafter, the configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. Note that description of the functional configuration similar to that of the first embodiment is omitted.

The telop detection unit 104 outputs the detected telop region and information on the motion (for example, information on the motion vector of the telop and the block in which the vector is detected) to the motion vector correction unit 105 and the switching timing control unit 107. .
The switching timing control unit 107 controls the motion vector switching unit 106 according to the motion of the telop. Specifically, when the telop detection unit 104 detects a telop region that moves in the vertical direction, the motion vector switching unit 106 is controlled so that the motion vector V1 and the motion vector V2 are alternately output. In other cases, the motion vector switching unit 106 is controlled to output the motion vector V1 or the motion vector V2. Accordingly, when the telop detection unit 104 detects a telop region that moves in the vertical direction, the interpolation frame generation unit 103 alternately performs the first interpolation frame generation process and the second interpolation frame generation process. It becomes. In other cases, the first interpolation frame generation process or the second interpolation frame generation process is performed. As a result, telop and background disturbance can be suppressed in the telop area moving in the vertical direction as compared with the conventional method. Whether to perform the first interpolation frame generation process or the second interpolation frame generation process for a telop area that moves in a direction other than the vertical direction depends on the purpose (whether the disturbance of the telop or the background is reduced). You may choose according to it.

  As described above, according to the image processing apparatus and the image processing method according to the present embodiment, when a telop area is detected, the first interpolation frame generation process and the second interpolation frame generation process are alternately performed. To be done. Accordingly, it is possible to perform a frame rate conversion process in which deterioration of image quality in the telop area is suppressed.

  In the above-described first to third embodiments, the case where the frame rate of the input video is converted to a frame rate that is two to three times has been described. It suffices if the frame rate of the input video is converted by inserting an interpolation frame. For example, the frame rate of the input video may be converted to a frame rate of 5 or 8 times.

In the present embodiment, the motion vector V1 and the motion vector V2 are alternately switched and output every time an interpolation frame is generated. That is, the first interpolation frame generation process and the second interpolation frame generation process are alternately switched every time an interpolation frame is generated. However, the configuration of the present invention is not limited to this. For example, the motion vector V1 or the motion vector V2 may be continuously output a plurality of times. That is, a plurality of interpolation frames may be generated by the first interpolation frame generation process or the second interpolation frame generation process. If the first interpolation frame generation process and the second interpolation frame generation process are alternately performed when a telop area is detected, the above-described effect can be obtained.

  102: Motion vector detection unit, 103: Interpolated frame generation unit, 104: Telop detection unit

Claims (3)

An image processing apparatus for converting a frame rate by inserting an interpolation frame between frames of an input video,
Telop detection means for detecting the telop area and the movement of the telop from the input video;
A motion vector detection means for dividing the input video into a plurality of blocks and detecting a motion vector for each block;
Interpolation frame generation means for generating the interpolation frame using the motion vector;
Have
When the telop detection unit detects a telop region, the interpolation frame generation unit uses a motion vector based on the motion of the telop as a motion vector of a block corresponding to the telop region. An image processing apparatus alternately performing a generation process and a second interpolation frame generation process using a motion vector detected by the motion vector detection means as a motion vector of a block corresponding to the telop area . The interpolation frame generation means alternately performs the first interpolation frame generation process and the second interpolation frame generation process when a telop area moving in the vertical direction is detected by the telop detection means, 2. The image processing apparatus according to claim 1, wherein the first interpolation frame generation process or the second interpolation frame generation process is performed in a case other than the above. An image processing method for converting a frame rate by inserting an interpolation frame between frames of an input video,
A telop detection step of detecting a telop area and a movement of the telop from the input video;
A motion vector detection step of dividing the input video into a plurality of blocks and detecting a motion vector for each block;
An interpolation frame generation step for generating the interpolation frame using the motion vector,
In the interpolation frame generation step, when a telop area is detected in the telop detection step, a first interpolation frame using a motion vector based on the motion of the telop as a motion vector of a block corresponding to the telop area An image processing method comprising alternately performing a generation process and a second interpolation frame generation process using the motion vector detected in the motion vector detection step as a motion vector of a block corresponding to the telop area .

Images