JP2011244330A - Frame rate conversion method and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a frame rate conversion processing in which an image quality is improved without increasing a processing load excessively by generating an output image with use of a motion vector based on multiple input images having different times.SOLUTION: A motion vector is detected from an input image A and an input image B which are temporally continuous. Then, for the image A, a motion compensated image of an intermediate time between the image A and the image B is generated using the motion vector detected from the image A and the image B. On the other hand, a motion compensated image of an intermediate time between the image A and the image B is generated using the image B among the input images and the detected motion vector. The generated these two motion compensated images of the intermediate time are then blended, and a generated image (image C) is output as the image of the intermediate time between the image A and the image B so as to be subjected to a frame rate conversion.

Description

  The present invention relates to a frame rate conversion method and apparatus, and more specifically, to a technique for improving the quality of an input image by generating an output image using image components at a plurality of times by motion compensation. It is related.

First, as an example of a frame rate conversion apparatus having a motion detection unit and a motion compensation unit, when a moving image with a frame rate of 60 Hz is input, a process for converting the frame rate of the moving image to double is shown in FIG. Show.
The images 901, 902, and 903 in (1) of FIG. 9 are images that are continuously input in time. From the position of the objects 904, 905, and 906 on the screen, the object moves in the lower right direction within the screen at a constant speed.

  (2) in FIG. 9 shows an output moving image obtained by converting the frame rate of the input moving image in (1) by a factor of two. The output images 907, 909, and 911 are the same images as the input images 901, 902, and 903, and the output images 908 and 910 are newly generated images. The objects 912 and 913 in the newly generated output images 908 and 910 detect the motion direction of the object in the input image, calculate the position of the object at the intermediate time of the input image, and compensate the motion of the input image. Is generated by

An example of a block diagram for realizing the processing of FIG. 9 is shown in FIG. The frame memory 1001 accumulates an input image and supplies the image to the motion vector detection unit 1002 and the frame generation unit (motion compensation unit) 1003 at an optimal timing by control from a timing controller (not shown).
The motion vector detection unit 1002 reads the temporally continuous images A and B from the frame memory 1001 and detects the motion of the object between the images.
The frame generation unit 1003 generates and outputs a motion compensated image from the image C supplied from the frame memory 1001 and the motion vector output from the motion vector detection unit 1002.
Further, the frame generation unit 1003 outputs the image C read from the frame memory 1001 as it is when the input image is an output image as in the output images 907, 909, 911 of FIG.

The images A, B, and C will be described more specifically.
As shown in FIG. 11, images A and B are temporally continuous images in the input image. A motion vector detection unit 1002 detects a motion vector from the images A and B. Here, the image A is used as the image C. In the frame generation unit 1003, an image at an intermediate time between the image A and the image B is used for the image C (= image A) read from the frame memory 1001 using the motion vector detected by the motion vector detection unit 1002. Generate and output. When the images A and B are used as output images as they are, the image C read from the frame memory 1001 is output as it is.

  Further, as shown in FIG. 12, an image B may be used as the image C. In this case, the frame generation unit 1003 uses the motion vector detected by the motion vector detection unit 1002 with respect to the image C (= image B) read from the frame memory 1001, so that it is between the images A and B. Generate and output an intermediate time image. When the images A and B are used as output images as they are, the image C read from the frame memory 1001 is output as it is.

  An example of processing of the motion vector detection unit 1002 in FIG. 10 is shown in FIG. The motion vector detection unit 1002 detects the motion of the object between the motion source image 1301 and the motion destination image 1302. In order to correspond to the motion of a plurality of objects, the motion original image 1301 is divided into a plurality of blocks in a grid pattern, and the motion vector of each block is detected. For example, the motion vector for the block 1305 of the motion source image is detected from the region of the motion search range 1307 centering on the block 1306 at the same position of the motion destination image. In the example of FIG. 13, a block 1308 is detected as a destination block of the block 1305, and a vector from the block 1305 to the block 1308 becomes a motion vector.

  Next, FIG. 14 shows an example of an image quality improving apparatus using a plurality of images having different times. The images 1401, 1402, and 1403 in (1) of FIG. 14 are images that are continuously input in time. From the positions of the objects 1404, 1405, and 1406 on the screen, the object moves in the screen at a constant speed in the lower right direction. (2) in FIG. 14 shows an output image of the high image quality processing. The output image 1407 is generated by blending (mixing) the input image 1401 with the input image 1401 that has been motion compensated by the motion vector 1.

An example of a block diagram for realizing the processing of FIG. 14 is shown in FIG. The frame memory 1501 accumulates input images, and supplies the images to the motion vector detection unit 1502, the frame generation unit (motion compensation unit) 1503, and the blending unit 1504 at an optimal timing by control from a timing controller (not shown).
The motion vector detection unit 1502 reads the temporally continuous images A and B from the frame memory 1501 and detects the motion of the object between the images. The frame generation unit 1503 generates and outputs a motion compensated image from the image C supplied from the frame memory 1501 and the motion vector output from the motion vector detection unit 1502. The blend unit 1504 blends and outputs the motion compensated image output from the frame generation unit 1503 and the image D supplied from the frame memory 1501.

The images A, B, C, and D in FIG. 15 will be described more specifically.
As shown in FIG. 16, images A and B are temporally continuous images in the input image. A motion vector detection unit 1502 detects a motion vector from the images A and B. Here, the image A is used as the image C. The frame generation unit 1503 uses the motion vector detected by the motion vector detection unit 1502 for the image C (= image A) read from the frame memory 1501 to obtain an intermediate time image between the image A and the image B. Generate and output. The blend unit 1504 uses the image D (= image B) read from the frame memory 1501 and blends the image D with the intermediate time image generated by the frame generation unit 1503. Thereby, output images for the images A, B, C, and D are obtained. When the images A and B are used as output images as they are, the images C and D read from the frame memory 1501 are output as they are.

The blending process will be described more specifically.
The blending process mixes the signal components of two images. For example, if the result of blending image A and image B is image C,
Image C = Image A × Blend ratio + Image B × (1-Blend ratio)
It becomes.
Generally, the blend ratio is obtained from the similarity between the image A and the image B for each pixel position in the image. Specifically, when the similarity is high (when the difference between the image A and the image B is small), the blend ratio is increased.

With reference to FIG. 17, the processing for improving the image quality by blending will be described in more detail.
For example, noises n1, n2, and n3 are mixed in input images having different shooting times during shooting, analog transmission, compression encoding using MPEG, and the like. These noises are not exactly the same, and are different for each input image. Therefore, by blending images with different shooting times, random noise is neutralized, and the noise feeling of the obtained image can be reduced.

  As shown in FIG. 18, when an interpolated image is generated by IP conversion for each field image for an interlaced image and scaled by a scaler to be converted into a panel size image, the shape f1 of the object of the adjacent image , F2, and f3 change slightly. Further, since the moving distance of the object is irrelevant to the captured image pitch, adjacent moving images do not completely match. Even in such a case, by blending images having different shooting times, the information amount of the object can be increased and the image quality can be improved.

  In the example of FIG. 15, the motion compensation image generated by the frame generation unit 1503 and the input image D are blended by the blending unit 1504. However, if these images are the same image, the image changes even after blending. do not do. However, normally, the images to be blended do not become the same image due to noise and scaling processing as described above. Therefore, image quality can be improved by blending images.

  Next, another example of an image quality improving apparatus using a plurality of images having different times is shown in FIGS. As shown in FIGS. 17 and 18 above, images with different shooting times are generally different due to noise or the like. Therefore, not only two-frame images with different shooting times but also three or more frames are blended, or past output images are used for blending. The image quality improvement effect by increasing the amount of information can be expected.

FIG. 19 shows a process of generating an output image using an input image of three frames or more (acyclic type), and FIG. 20 shows a process of generating an output image using an input image and a past output image. (Cyclic type).
In the case of the process shown in FIG. 19, a motion vector is detected for a plurality of temporally continuous input images. Then, using the detected motion vector, a motion compensated image at a time corresponding to a specific input image is generated, and these motion compensated images are blended with the specific input image to improve image quality.

  In the case of FIG. 20, motion vectors are detected for a plurality of temporally continuous input images. The detected motion vector is applied to the output image to generate a motion compensated image at a time corresponding to the specific input image, and the generated motion compensated image is blended with the specific input image to improve the image quality. Yes.

  For example, Non-Patent Document 1 shows an example in which super-resolution processing is performed between a plurality of frames with respect to a technique for improving image quality using a plurality of frames of an input image.

Nikkei Electronics, 2009.2.9, "Non-standard television that challenges beauty", Nikkei Inc.

  The frame rate conversion process and the image quality improvement process require a motion detection process and a motion compensation process. A large memory and arithmetic processing are required for the motion detection processing and motion compensation processing between images. For this reason, if the frame rate conversion process and the image quality improvement process are performed independently, the amount of memory and the calculation amount of the entire process increase. Even in the processing described in Non-Patent Document 1, there is a problem that the amount of processing during image processing is large.

  The present invention has been made in view of the above circumstances, and generates high-quality images without excessively increasing the processing load by generating output images based on a plurality of input images at different times using motion vectors. It is an object of the present invention to provide a frame rate conversion method and apparatus that enable a converted frame rate conversion process.

  In order to solve the above problems, a first technical means of the present invention is a frame rate conversion method executed by an image processing apparatus that converts a frame rate of an input image and outputs the frame rate, and is input to the video processing apparatus. A motion vector detection step for detecting a motion between the input images and obtaining a motion vector, a motion compensation step for generating a motion compensated image using the motion vector detected in the motion vector detection step, and the motion compensation A mixing step of mixing a plurality of images including motion-compensated images generated in the step, wherein the mixing step mixes components of the input image at different times.

  According to a second technical means, in the first technical means, the motion compensation step generates a plurality of motion compensation images generated based on a plurality of input images at different times, and the mixing step includes the motion By mixing a plurality of motion compensation images generated in the compensation step, an image between the input images is generated and the frame rate is converted.

  In a third technical means, in the first technical means, the motion compensation step generates a motion compensated image based on an output image corresponding to a part of the input images among a plurality of input images at different times, A motion compensation image is generated based on another input image, and the mixing step mixes the motion compensation image generated in the motion compensation step to generate an image between the input images, thereby generating the frame. It is characterized by converting the rate.

  According to a fourth technical means, in the first technical means, the motion compensation step generates an image between the input images using a motion compensated image generated based on an output image corresponding to the input image, and a time A motion compensation image corresponding to the time of another input image is generated based on an output image corresponding to a part of the input images, and the mixing step is generated in the motion compensation step. The motion compensated image corresponding to the time of the other input image and the other input image are mixed.

  A fifth technical means is a frame rate conversion device that converts a frame rate of an input image and outputs the same, and detects a motion vector by detecting a motion between input images input to the frame rate conversion device. A motion compensation unit that generates a motion compensated image using the motion vector detected by the motion vector detection unit, and a mixture that includes a plurality of images including the motion compensated image generated by the motion compensation unit The mixing unit mixes components of input images at different times.

  According to a sixth technical means, in the fifth technical means, the motion compensation unit generates a plurality of motion compensated images generated based on a plurality of input images at different times, and the mixing unit By mixing a plurality of motion compensation images generated by the compensation unit, an image between the input images is generated and the frame rate is converted.

  According to a seventh technical means, in the fifth technical means, the motion compensation unit generates a motion compensated image generated based on an output image corresponding to a part of the input images among a plurality of input images at different times. And generating a motion compensation image generated based on another input image, and the mixing unit mixes the motion compensation image generated by the motion compensation unit to generate an image between the input images. Thus, the frame rate is converted.

  According to an eighth technical means, in the fifth technical means, the motion compensation unit generates an image between the input images based on a motion compensated image generated based on an output image corresponding to the input image, and a time Among the plurality of different input images, a motion compensated image corresponding to the time of another input image is generated based on an output image corresponding to a part of the input images, and the mixing unit is generated in the motion compensation step The motion compensated image corresponding to the time of the other input image and the other input image are mixed.

  According to the present invention, by generating an output image based on a plurality of input images at different times using motion vectors, it is possible to perform a frame rate conversion process with high image quality without excessively increasing the processing load. Become. In particular, according to the present invention, it is possible to share the frame rate conversion process and the motion detection process necessary for the image quality enhancement process, thereby greatly reducing the processing amount.

It is a figure explaining the frame rate conversion process by the 1st Embodiment of this invention. It is a figure explaining the 1st Embodiment of this invention concretely. It is a block diagram for demonstrating the structural example of the frame rate conversion apparatus which concerns on this invention. It is a figure explaining the frame rate conversion process by the 2nd Embodiment of this invention. It is a block diagram for demonstrating the other structural example of the frame rate conversion apparatus which concerns on this invention. It is a figure explaining the frame rate conversion process by the 3rd Embodiment of this invention. It is a block diagram for demonstrating the further another structural example of the frame rate conversion apparatus which concerns on this invention. It is a figure explaining the example of a process when the invention is applied to 4 times frame rate conversion. It is a figure for demonstrating the process which converts the frame rate of the conventional moving image into 2 times. It is a figure which shows an example of the block diagram for implement | achieving the conversion processing of the frame rate of FIG. FIG. 10 is a diagram for further explaining the frame rate conversion processing of FIG. 9. FIG. 10 is another diagram for further explaining the frame rate conversion processing of FIG. 9. It is a figure explaining an example of a process of the motion vector detection part of FIG. It is a figure explaining an example of the conventional image quality improvement apparatus using the several image from which time differs. It is a block diagram for implement | achieving the process of FIG. It is a figure which demonstrates the process of FIG. 14 more concretely. It is a figure explaining the process which improves the image quality by blending. It is another figure explaining the process which improves the image quality by blending. It is a figure explaining the other example of the conventional image quality improvement apparatus using the several image from which time differs. It is a figure explaining the further another example of the conventional image quality improvement apparatus using the several image from which time differs.

  In the embodiment according to the present invention, when performing the process of converting the frame rate of the input image, the motion between the input images is detected to obtain a motion vector, and the frame rate of the input image is converted using the motion vector. Generate an output image. Here, the frame rate is converted by generating an image between the input images by using the motion compensated images generated based on the input images having different times and mixing the components of the input images having different times. In the following, embodiments of the present invention will be described in more detail.

(Embodiment 1)
An input image and an output image obtained by the processing of the first embodiment of the present invention are shown in FIG. The images 101, 102, and 103 in (1) of FIG. 1 are images that are continuously input in time. From the positions of the objects 104, 105, and 106 on the screen, the object is moving in the screen in the lower right direction at a constant speed.

  (2) of FIG. 1 shows an output moving image obtained by converting the frame rate of the input moving image of (1) by a factor of two. The output images 107, 109, and 111 are images corresponding to the same time as the input images 101, 102, and 103, respectively, and the output images 108 and 110 are images at an intermediate time of the input images. The output image (2) in FIG. 1 is generated by blending a plurality of input images. The process of blending images corresponds to the image mixing process of the present invention. The output images 108 and 109 are generated from the input images 101 and 102, and the output images 110 and 111 are generated from the input images 102 and 103.

  FIG. 2 is a diagram for explaining the above processing more specifically. The images 101, 102, and 103 are images that are continuously input in time. Here, attention is paid to the image 101 and the image 102. These temporally continuous input images are referred to as image A and image B. Here, first, a motion vector is detected from the images A and B. As a motion vector detection method, a conventionally known method can be appropriately used.

Then, a motion compensated image at an intermediate time between the image A and the image B is generated for the image A in the input image by using the motion vector detected by the image A and the image B. Since this motion compensated image is an image at an intermediate time between the image A and the image B, information of ½ of the detected motion vector is used. On the other hand, a motion compensated image at an intermediate time between the image A and the image B is generated using the image B in the input image and the detected motion vector.
Then, the generated motion compensated images at the intermediate time are blended, and the obtained generated image (referred to as image C) is output as an image at an intermediate time between images A and B.

  When an image D whose time coincides with the image B is generated as an output image, the following processing is performed although not shown. Here, a motion compensated image at the same time as the image B is generated from the image A in the input image and the detected motion vector. In this case, the position information of the detected motion vector is used to generate an image at the same time as the image B. Then, the motion compensated image generated from the image A and the motion vector and the image B are blended, and the obtained generated image (image D) is output as an image at the same time as the image B. By these processes, components of the input image at different times are mixed in the output image.

FIG. 3 is a block diagram for explaining a configuration example of the frame rate conversion apparatus according to the present invention, and shows a configuration example for executing the processing of the first embodiment.
In FIG. 3, the input image is written in the frame memory 201. Input images (image A and image B) having different times are read from the frame memory 201 and supplied to the motion vector detection unit 202. The motion vector detection unit 202 detects a motion between the supplied two frames of images and generates a motion vector.

  The first frame generation unit (motion compensation unit) 203 uses the image A in the input image and the motion vector detected by the motion vector detection unit 202 to use a motion compensated image at an intermediate time between the image A and the image B. Is generated. On the other hand, the second frame generation unit (motion compensation unit) 204 uses the image B in the input image and the motion vector detected by the motion vector detection unit 202 to move between the images A and B at an intermediate time. A compensation image is generated.

  The blending unit (mixing unit) 205 blends the motion compensation images at the intermediate times generated by the first and second frame generation units 203 and 204, and uses the generated image as the intermediate time between the images A and B. Output as an image.

  On the other hand, when an image whose time coincides with the image B is generated as the output image, the first frame generation unit 203 uses the image A in the input image and the motion vector detected by the motion vector detection unit 202 to generate the image B. A motion compensated image at the same time is generated. On the other hand, the second frame generation unit 204 outputs the image B in the input image as a motion compensated image. The blend unit 205 blends the images at the same time as the image B output from the first and second frame generation units 203 and 204, and outputs the generated image as the image at the same time as the image B. .

  As described above, in the first embodiment of the present invention, the motion detection unit 202 that detects a motion between input images and obtains a motion vector, and the first and second that generate an output image using the motion vector. In a frame rate conversion apparatus including frame generation units (motion compensation units) 203 and 204, a plurality of motion compensation images generated based on each of a plurality of input images at different times are mixed using detected motion vectors. By generating an image between input images, random noise can be reduced, and image quality can be improved by increasing the amount of information of a moving object. At this time, the frame rate conversion process and the motion vector detection process necessary for the image quality enhancement process can be shared, and the processing amount can be greatly reduced.

(Embodiment 2)
An input image and an output image obtained by the processing of the second embodiment of the present invention are shown in FIG. The images 401, 402, and 403 in (1) of FIG. 4 are images that are input continuously in time. From the positions of the objects 404, 405, and 406 on the screen, the object moves in the screen at a constant speed in the lower right direction.

  (2) in FIG. 4 shows an output moving image obtained by converting the frame rate of the input moving image in (1) by a factor of two. The output images 407, 409, and 411 are images at the same time as the input images 401, 402, and 403, respectively, and the output images 408 and 410 are images at an intermediate time of the input images. Also, the output image of FIG. 4B includes a plurality of input image components. The output image 409 is generated from the past output image 407 and the input image 402, and the output image 408 is generated from the output image 409 and the input image 402.

  Here, attention is paid to input images 401 and 402 that are temporally continuous, and these input images are referred to as image A and image B, respectively. An output image (referred to as image E) whose time coincides with image B is generated from input image B and an output image (referred to as image C) at the same time as input image A.

  An output image (referred to as image D) at an intermediate time between image A and image B is generated by the output image (image C) immediately before image D and input image B. Here, a motion vector is detected from the temporally continuous input image A and image B, and the motion vector is applied to the output image C to generate a motion compensated image at an intermediate time between the images C and E. On the other hand, the motion vector is applied to the image B to generate motion compensation images of the images A and B and intermediate times. These two motion compensated images are blended to generate and output an image D at an intermediate time between the images A and B.

On the other hand, when generating the image E, the motion vector detected from the image A and the image B is applied to the image C to generate a motion compensated image at the same time as the image B. Then, the motion compensation image is blended with the image B to generate and output an image E.
The image C is generated in the same manner as the image E, and is a blended image of the motion compensated image based on the past output image and the input image (image A) at that time. By these processes, components of the input image at different times are mixed in the output image.

  As described above, among a plurality of input images having different times, a motion compensation image generated based on an output image corresponding to a part of the input images and a motion compensation image generated based on another input image are mixed. By generating an image between input images, it is possible to expect a random noise reduction effect and an image quality improvement effect due to an increase in the information amount of the mobile station object.

FIG. 5 is a block diagram for explaining another configuration example of the frame rate conversion apparatus according to the present invention, and shows a configuration example for executing the processing of the second embodiment.
In FIG. 5, the input image is written into the first frame memory 501. Input images (image A and image B) having different times are read from the first frame memory 501 and supplied to the motion vector detection unit 502. The motion vector detection unit 502 detects a motion between the supplied two frames of images and generates a motion vector.

The first frame generation unit (motion compensation unit) 503 applies the motion vector detected by the motion vector detection unit 502 to the image (image C) read from the second frame memory 504, and the image A motion compensated image at an intermediate time between A and B is generated and supplied to the blending unit (mixing unit) 505. On the other hand, the second frame generation unit (motion compensation unit) 506 applies the motion vector detected by the motion vector detection unit 502 to the image B of the input image, and moves between the image A and the image B at the intermediate time. A compensation image is generated and supplied to the blending unit 505.
The blending unit 505 blends the motion compensated two-frame images to generate an output image.

  On the other hand, when an output image whose time coincides with the image B is generated as the output image, the first frame generation unit 503 moves the image (image C) read from the second frame memory 504. The motion vector detected by the vector detection unit 502 is applied, a motion compensated image at the same time as the image B is generated, and supplied to the blending unit 505. On the other hand, the second frame generation unit 506 outputs the image B in the input image as a motion compensated image. The blending unit 505 blends the images at the same time as the image B output from the first and second frame generation units 503 and 506, and sets the obtained generated image as an image at the same time corresponding to the image B. Output. An output image at the same time as the input image is written in the second frame memory 504.

(Embodiment 3)
FIG. 6 shows an input image and an output image obtained by the processing according to the third embodiment of the present invention. In the second embodiment, two motion compensated images are blended to generate an output image at an intermediate time of the input image. However, an output image at an intermediate time is generated using only one frame of the motion compensated image. You may make it do.

The images 601, 602, and 603 in FIG. 6 (1) are images that are continuously input in time. From the positions of the objects 604, 605, and 606 on the screen, the object moves in the lower right direction on the screen at a constant speed.
(2) in FIG. 6 shows an output moving image obtained by converting the frame rate of the input moving image in (1) by a factor of two. The output images 607, 609, and 611 are images at the same time as the input images 601, 602, and 603, respectively, and the output images 608 and 610 are images at intermediate times of the input images. In addition, the output image of FIG. 6B includes a plurality of input image components. The output image 609 is generated from the past output image 607 and the input image 602, and the output image 608 is generated from the output image 607.

Here, attention is paid to input images 601 and 602 that are temporally continuous, and these input images are referred to as image A and image B, respectively. An output image (referred to as image E) whose time coincides with image B is generated from input image B and an output image (referred to as image C) at the same time as input image A.
An output image (image D) at an intermediate time between the images A and B is generated by an output image (image C) immediately before the image D. Here, a motion vector is detected from temporally continuous input image A and image B, and the motion vector is applied to output image C to generate a motion compensated image at an intermediate time between image C and image E. Output as D.

On the other hand, when generating the image E, the motion vector detected from the image A and the image B is applied to the image C to generate a motion compensated image at the same time as the image B. Then, the motion compensation image is blended with the image B to generate and output an image E.
The image C is generated in the same manner as the image E, and is a blended image of the motion compensated image based on the past output image and the input image (image A) at that time. By these processes, components of the input image at different times are mixed in the output image.

  In this way, an image between the input images is generated by the motion compensated image generated based on the output image corresponding to the input image, and at this time, the image corresponds to the input image used for generating the image between the input images. By generating an output image by mixing a motion compensated image generated based on an output image corresponding to a part of the input images among a plurality of input images at different times and other input images, a random state Noise reduction effect and an image quality improvement effect due to an increase in the amount of information of the mobile station object can be expected.

FIG. 7 is a block diagram for explaining a configuration example of the frame rate conversion apparatus according to the present invention, and shows a configuration example for executing the processing of the third embodiment.
In FIG. 7, the input image is written in the first frame memory 701. Input images (image A and image B) having different times are read from the first frame memory 701 and supplied to the motion vector detection unit 702. The motion vector detection unit 702 detects a motion between the supplied two frames of images and generates a motion vector.

The frame generation unit (motion compensation unit) 703 applies the motion vector detected by the motion vector detection unit 702 to the image (image C) read from the second frame memory 705, and performs image A and image A motion compensated image at an intermediate time with B is generated and supplied to the blending unit (mixing unit) 704. The image at the intermediate time is output as an image D without being blended in the blending unit 704.
On the other hand, when generating the image E at the same time as the image B, the frame generation unit 703 detects the image (image C) read from the second frame memory 705 by the motion vector detection unit 702. The motion vector is applied, a motion compensated image at the same time as the image B is generated, and supplied to the blend unit 704. The blending unit 704 blends this motion compensated image with the image B and outputs it as an image E. An output image at the same time as the input image is written in the second frame memory 705.

(Embodiment 4)
In each of the above embodiments, an example of processing for converting the frame rate to twice is shown. However, the frame rate conversion processing according to the present invention is not limited to the double frame rate conversion. For example, FIG. It can also be applied to the four times frame rate conversion as shown in FIG.
A motion vector is detected from temporally continuous input images 801 and 802, and an image 804 having a time of ¼ between the input images 801 and 802 is generated using the detected motion vector. Here, the motion vector is applied to the output image 803 to generate a motion compensated image at ¼ time, and the motion vector is applied to the input image 802 to generate a motion compensated image at ¼ time. By blending these two motion compensation images, an output image 804 is generated and output.

By blending the motion-compensated image based on the output image and the motion-compensated image based on the input image as described above, the output image 805 is also obtained for the half time and the third time between the images 801 and 802. 806 can be generated respectively. Further, an output image 807 at the same time as the image 802 is generated by blending a motion compensated image obtained by applying a motion vector to the output image 803 and the input image 802.
Thus, in the embodiment according to the present invention, the frame rate after conversion is not limited and can be appropriately applied to any frame rate such as four times.

101, 102, 103 ... input image, 104, 105, 106 ... object, 107, 108, 109, 110, 111 ... output image, 201 ... frame memory, 202 ... vector detection unit, 203 ... first frame generation unit, 204 ... second frame generation unit, 205 ... blending unit, 401, 402, 403 ... input image, 407 to 411 ... output image, 501 ... first frame memory, 502 ... motion vector detection unit, 503 ... first Frame generation unit, 504 ... second frame memory, 505 ... blending unit, 506 ... second frame generation unit, 601, 602, 603 ... input image, 604, 605, 606 ... object, 607 to 611 ... output image, 701: First frame memory, 702: Motion vector detection unit, 703: Frame generation unit, 704: Blending unit, 70 ... second frame memory, 801,802 ... input image, 803-807 ... output image, 901,902,903 ... input image, 904,905,906 ... object, 907-911 ... output image, 912,913 ... object , 1001 ... Frame memory, 1002 ... Vector detection unit, 1003 ... Frame generation unit, 1301 ... Motion source image, 1302 ... Motion destination image, 1303, 1304 ... Object, 1305, 1306 ... Block, 1307 ... Search range, 1308 ... Block , 1401-1403 ... input image, 1404, 1405, 1406 ... object, 1407, 1408 ... output image, 1501 ... frame memory, 1502 ... vector detection unit, 1503 ... frame generation unit, 1504 ... blending unit.

Claims (8)

A frame rate conversion method executed by an image processing apparatus that converts a frame rate of an input image and outputs the frame rate,
A motion vector detection step for detecting a motion between input images input to the video processing device to obtain a motion vector;
A motion compensation step for generating a motion compensated image using the motion vector detected in the motion vector detection step;
Mixing a plurality of images including the motion compensated image generated in the motion compensation step,
The frame rate conversion method characterized in that the mixing step mixes components of input images at different times. The frame rate conversion method according to claim 1,
The motion compensation step generates a plurality of motion compensation images generated based on each of a plurality of input images at different times,
In the frame rate conversion method, in the mixing step, a plurality of motion compensated images generated in the motion compensation step are mixed to generate an image between the input images to convert the frame rate. The frame rate conversion method according to claim 1,
The motion compensation step generates a motion compensation image based on an output image corresponding to a part of the input images among a plurality of input images at different times, and generates a motion compensation image based on another input image. And
In the frame rate conversion method, the mixing step converts the frame rate by mixing the motion compensated images generated in the motion compensation step and generating an image between the input images. The frame rate conversion method according to claim 1,
The motion compensation step generates an image between the input images based on a motion compensated image generated based on an output image corresponding to the input image, and a part of the input images among a plurality of input images at different times Generating a motion compensated image corresponding to the time of another input image based on the output image corresponding to
The frame rate conversion method characterized in that the mixing step mixes the motion compensation image corresponding to the time of the other input image generated in the motion compensation step and the other input image. A frame rate conversion device for converting a frame rate of an input image and outputting the converted image,
A motion vector detection unit that detects a motion vector by detecting a motion between input images input to the frame rate conversion device;
A motion compensation unit that generates a motion compensated image using the motion vector detected by the motion vector detection unit;
A mixing unit that mixes a plurality of images including the motion compensated image generated by the motion compensation unit,
The mixing unit mixes components of an input image at different times. In the frame rate conversion device according to claim 5,
The motion compensation unit generates a plurality of motion compensation images generated based on each of a plurality of input images at different times,
The frame rate conversion device, wherein the mixing unit mixes a plurality of motion compensated images generated by the motion compensation unit to generate an image between the input images and convert the frame rate. In the frame rate conversion device according to claim 5,
The motion compensation unit generates a motion compensation image based on an output image corresponding to a part of the input images among a plurality of input images at different times, and generates a motion compensation image based on another input image. And
The frame rate conversion apparatus, wherein the mixing unit converts the frame rate by mixing the motion compensated images generated by the motion compensation unit and generating an image between the input images. In the frame rate conversion device according to claim 5,
The motion compensation unit generates an image between the input images based on a motion compensated image generated based on an output image corresponding to the input image, and a part of the input images among a plurality of input images having different times Generating a motion compensated image corresponding to the time of another input image based on the output image corresponding to
The frame rate conversion device, wherein the mixing unit mixes a motion compensated image corresponding to a time of the other input image generated in the motion compensation step and the other input image.

Images