JP2010124369A - Frame interpolation device, image encoder, and image decoder - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a moving image with a high image quality by obtaining an accurate motion vector by simple processing and configuration and generating an interpolation frame using the motion vector. <P>SOLUTION: A frame interpolation device includes: a camera 101 for taking a photograph of an image; an encoder 102 for coding the image photographed by the camera 101; a motion vector coding unit 107 for coding the motion vector of the image photographed by the camera 101; a decoder 103 for decoding and outputting the image coded by the encoder 102; a motion vector decoding unit 108 for decoding the motion vector coded by the motion vector coding unit 107, and a frame interpolation unit 105 for generating an interpolation frame from the decoded frame outputted from the decoder 103 and the motion vector outputted from the motion vector decoding unit 108. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a frame interpolation device, an image encoding device, and an image decoding device, and more specifically, by interpolating a non-encoded frame in addition to an encoded frame when displaying a moving image. The present invention relates to a frame interpolation device, an image encoding device, and an image decoding device that can increase the number of frames that can be displayed per unit time.

  When recording a moving image taken by a mobile phone or a digital video camera (DVC) on a recording medium, inter prediction coding is performed using high correlation in the time direction in the moving image. The amount of data is reduced and recorded on a recording medium. The encoded data recorded on the recording medium is not only reproduced / displayed by the terminal device that recorded the moving image, but also inserted in the player to display the reproduced image on the television, or A moving image reproduced on the terminal is transferred to a television via various interfaces, and the reproduced image is displayed on the television. At this time, particularly when shooting with a mobile phone, the compression rate is high, so that all frames may be skipped and encoded without being encoded.

  In recent years, liquid crystal televisions, which account for the majority of TV shipments, have been implemented with a technique for improving the image quality by driving the liquid crystal panel at 120 Hz and increasing the number of display frames by FRC (Frame Rate Conversion) or the like. . For this reason, when a captured moving image is reproduced, not only the encoded frame but also an interpolation frame is generated from a decoded frame obtained by reproducing the encoded frame, thereby securing a frame necessary for display.

  With regard to such a frame interpolation technique, for example, Patent Document 1 discloses a frame interpolation technique for realizing accurate frame interpolation with a simple interpolation method and apparatus even when a plurality of subjects having different motions exist. Is disclosed. Here, first, the area dividing unit compares the motion compensation interframe difference absolute value of the small block to be decoded with a threshold value for each pixel, and divides it into a first area of pixels smaller than the threshold value and a second area of larger pixels. . The interpolation value calculation unit averages the pixel values in the first area and the corresponding pixel values in the reference block to form an interpolation frame. The interpolation value calculation unit determines occlusion / appearance in the decoding target frame in the second area, sets one of the decoding target block or the corresponding area of the reference block as a reference area, and re-searches reference frames that do not belong to the reference area And Next, the pixel value of the re-search reference frame determined from ½ of the second motion vector obtained between the reference region and the re-search reference frame is copied to the second region of the interpolation frame to form an interpolation frame.

  5 and 6 are block diagrams respectively showing configuration examples of a conventional image encoding device and image decoding / display device that realize frame interpolation. 5 and 6, 30 is an image encoding device, 40 is an image decoding / display device, 301 is a camera that captures moving images, 302 is an encoder that encodes moving images, and 303 is a code output from the encoder 302. A decoder that decodes the encoded data; 305, a frame interpolation unit that interpolates a frame from the decoded frame and the motion vector decoded by the decoder; 306, a motion vector detection unit that detects a motion vector used for frame interpolation; It is a display unit that displays the output decoded frame and the interpolation frame generated by the frame interpolation unit 305.

  First, in the configuration example of FIG. 5, the image decoding / display apparatus 40 outputs the motion vector decoded by the decoder 303 to the frame interpolation unit 305, and the frame interpolation unit 305 uses the motion vector included in the encoded data. Interpolated frames are generated. On the other hand, in the configuration example of FIG. 6, the image decoding / display apparatus 40 detects a new motion vector using the decoded frame output from the decoder 303 by the motion vector detection unit 306 and performs frame interpolation on the detected motion vector. The data is output to the unit 305. The frame interpolation unit 305 generates an interpolation frame using the motion vector.

  Here, there are various methods such as MPEG (Moving Picture Experts Group) -1, MPEG-2, MPEG-4 AVC (Advanced Video Coding), VC1, and VP8 as a method of moving image encoder and decoder. Any method may be used for the encoder 302 of the image encoding device 30 and the decoder 303 of the image decoding / display device 40.

FIG. 7 is a diagram for explaining a conventional general generation method of an interpolation frame. Here, an example is shown in which an interpolated frame that is located in the middle of two decoded frames in time is generated. The block on the interpolation frame to be interpolated is located at a half of the motion vector connecting each block of the decoded frame (2) and the corresponding block of the decoded frame (1). This is assuming constant velocity motion. This block is generated by the average value of each pixel of the corresponding decoded frame (1) and decoded frame (2) block. Alternatively, the pixel value of the block of either the decoded frame (1) or the decoded frame (2) is generated. As described above, the motion vector used here may be a motion vector used for inter prediction included in the encoded data, or the motion vector detection unit 306 uses the decoded frame (1) and the decoded frame (2). A newly detected motion vector may be used.
JP 2000-224593 A

In the frame interpolation process as described above, if a frame is interpolated using a motion vector different from the actual motion, the image quality is deteriorated. Therefore, obtaining an accurate motion vector is a very big problem in improving image quality. However, conventionally, there is a problem that it is difficult to obtain an accurate motion vector due to the following factors.
(1) The motion vector used in the moving image encoding process is determined so that the rate distortion is optimal, and does not necessarily reflect the motion of the object accurately.
(2) When a motion vector is detected on the decoding side, it is difficult to obtain an accurate motion vector using two decoded frames if deterioration due to encoding is large.
(3) Even if an accurate motion vector between two decoded frames can be detected, the interpolation frame is interpolated on the assumption that the object is moving at a constant speed, so that the object is moving at a constant speed. If not, a correct interpolation position on the interpolation frame cannot be obtained.

On the other hand, it has become possible to acquire a high frame rate moving image by improving the performance of the imaging element of the camera.
The present invention has been made in view of the above-described circumstances, and obtains an accurate motion vector by a simple process and configuration, and generates an interpolated frame using the motion vector to generate a high-quality moving image. An object of the present invention is to provide a frame interpolation device, an image encoding device, and an image decoding device which can be used.

  In order to solve the above problems, a first technical means of the present invention encodes a camera that captures an image, an image encoding unit that encodes an image captured by the camera, and a motion vector of the image captured by the camera. A motion vector encoding unit, an image decoding unit that decodes and outputs an image encoded by the image encoding unit, a motion vector decoding unit that decodes a motion vector encoded by the motion vector encoding unit, It has a frame interpolation unit that generates an interpolation frame from the decoded frame output from the image decoding unit and the motion vector output from the motion vector decoding unit.

  According to a second technical means, in the first technical means, the camera has a motion vector detecting unit that detects a motion vector of a frame of the captured image, and the motion vector encoding unit is detected by the motion vector detecting unit. It is characterized in that a motion vector is encoded and output.

  According to a third technical means, in the second technical means, the camera outputs an output frame to the image encoding unit to encode the captured image, and detects a motion vector without outputting to the image encoding unit. A switching unit that switches between non-output frames to be input to the unit, and the motion vector detection unit inputs the non-output frame and the output frame, detects a motion vector related to the non-output frame, and It is characterized by outputting.

  The fourth technical means includes a camera that captures an image, an image encoding unit that encodes an image captured by the camera, and a motion vector encoding unit that encodes a motion vector of the image captured by the camera, The image and motion vector encoded by the image encoding unit and the motion vector encoding unit are output separately, respectively.

  Means for inputting an encoded image and an encoded motion vector of an image captured by a camera, an image decoding unit for decoding and outputting the encoded image, and a motion for decoding the encoded motion vector The image processing apparatus includes a vector decoding unit, and a frame interpolation unit that generates an interpolation frame from the decoded frame output from the image decoding unit and the motion vector output from the motion vector decoding unit.

  According to the present invention, a frame interpolation device and an image decoding device that can generate an accurate motion vector by a simple process and configuration and generate an interpolated frame using the motion vector to generate a high-quality moving image. An apparatus and an image encoding apparatus can be provided. In particular, since a motion vector necessary for generating an interpolation frame is detected on the encoding side using a high frame rate moving image, accurate frame interpolation using an accurate motion vector becomes possible.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a block diagram showing a configuration of an image encoding device and an image decoding / display device for explaining an embodiment of a frame interpolation device according to the present invention. The embodiment of the frame interpolation apparatus according to the present invention includes an image encoding device 10 that encodes a moving image and an image decoding / display device 20. The image encoding device 10 includes a camera 101, an encoder 102, a motion vector detection unit 106, and a motion vector encoding unit 107. The image encoding device 10 corresponds to the embodiment of the image encoding device of the present invention, and the image decoding / display device 20 corresponds to the embodiment of the image decoding device of the present invention.

  The encoder 102 corresponds to the image encoding unit of the present invention, encodes an image output from the camera 101, and outputs the encoded image to the decoder 103 of the image decoding / display device 20. In the present embodiment, the image output to the encoder 102 is output after being switched by a switch 202 of the camera 101 described later, and is selected as an output frame from images captured by the camera 101. . Since the encoder 102 has the same configuration as that of the conventional technology, the description thereof is omitted.

  The camera 101 captures a moving image at a high frame rate and outputs a moving image and a motion vector. The camera 101 includes a motion vector detection unit 106 in order to output a motion vector. The motion vector encoding unit 107 encodes the motion vector output from the camera 101 and outputs the encoded motion vector to the image decoding / display device. At this time, the motion vector is encoded separately from the moving image captured by the camera 101. As a motion vector encoding method, a method generally used for encoding moving image data can be applied. For example, the difference value between the motion vector and the prediction vector may be variable-length encoded, or other methods may be used as appropriate.

  As shown in FIG. 1, the encoded motion vector data output from the motion vector encoding unit 107 is sent to the image decoding / display device 20 separately from the encoded video data. Also, the encoded data of the moving image and the encoded data of the motion vector may be recorded on a recording medium (not shown). The encoded data of the moving image and the encoded data of the motion vector stored in the recording medium are read by the image decoding / display device 20, so that these encoded data are directly received from the image encoding device 10. It is possible to execute the same processing as when output to.

  As described above, there are a plurality of moving image encoding methods, and they are not unified. However, as in the embodiment according to the present invention, by encoding a motion vector separately from the encoding of the moving image, even if the encoding method of the moving image is determined by the product or application standard, the motion vector Is not affected by this encoding.

  FIG. 2 is a diagram for explaining a more detailed configuration example of the camera 101. In the camera 101, an image sensor 201 capable of high-speed data output is provided. The image sensor 201 is assumed to be capable of shooting at a frame rate twice that of a normal moving image, for example. In recent years, high-speed cameras have been commercialized for consumer use as well, and many image sensors for high frame rates are on the market. In the following description, it is assumed that the frame rate of the moving image to be encoded is 30 fps (Frames Per Second), and the image sensor 201 captures the image at 60 fps, which is twice that.

FIG. 3 is a diagram for explaining an output frame of an image sensor of the camera, an output frame and a motion vector output from the camera. FIG. 3A shows a moving image captured by the image sensor 201 of the camera 101 at 60 fps in the time direction for each frame.
FIG. 3B is a diagram illustrating an output frame and a motion vector output from the camera. A 60 fps moving image captured by the camera 101 includes an output frame (shown by a solid line) output from the camera 101, and the camera 101. Is divided into non-output frames (indicated by dotted lines) that are not output. In order to represent a non-output frame that is not output from the camera 101, the camera 101 outputs a motion vector instead.

  Here, first, in the camera 101, a moving image captured by the image sensor 201 is divided into an output frame (30 fps) and a non-output frame (30 fps) by a switch (switching unit) 202, and each is input to the motion vector detection unit 106. . The motion vector detection unit 106 searches the motion of each block divided into blocks using block matching or a gradient method, and obtains a motion vector of a non-output frame.

FIG. 4 is a diagram for explaining a method of obtaining a motion vector between the output frame and the non-output frame. As a method for obtaining a motion vector between an output frame and a non-output frame, two methods shown in FIGS. 4A and 4B can be applied.
In the method of FIG. 4A, a non-output frame is divided into blocks, and the position of the output frame corresponding to each block is obtained as a motion vector. Also. In the method of FIG. 4B, the output frame is divided into blocks, and the position of the non-output frame corresponding to each block is obtained as a motion vector.
In the above example, the motion vectors of the non-output frame and the two output frames before and after the non-output frame are respectively obtained, but the motion vector of the non-output frame only before or after the non-output frame is obtained. It may be.

  When the motion vector is used for interpolation in the image decoding / display device 20, an interpolation method corresponding to the method (FIG. 4A or FIG. 4B) in which motion is detected in advance by the image encoding device 10 is used. There must be. Therefore, a corresponding motion detection and interpolation method is set in advance between the image encoding device 10 and the image decoding / display device 20, or a motion vector is encoded by the motion vector encoding unit 107 of the image encoding device 10. Before conversion, a mechanism for sending a flag for notifying which of the motions is detected in FIG. 4 (A) or FIG. 4 (B) is required.

  As described above, the camera 101 captures a moving image at 60 fps by the image sensor 201 and outputs the captured moving image at a rate of 1 frame per 2 frames (30 fps). The data for one frame thinned out in the time direction is output not as a moving image but as a motion vector of a non-output frame obtained by the motion vector detection unit 106.

  In the above example, a moving image of 60 fps is captured by the image sensor 201 and a 30 fps moving image and a motion vector of 30 fps are output. However, in the present invention, for example, the image sensor 201 It is possible to shoot at 120 fps and output a moving image of 60 fps, or shoot at 90 fps with the image sensor 201b and output a moving image of 45 fps, which can support various frame rates. is there.

  The ratio of output frames to non-output frames is 1: 1 in the above example, but various frame rates such as 1: 2 and 1: 3 can be supported. For example, when shooting at 90 fps with the image sensor 201, if the output frame is 30 fps and the non-output frame is 60 fps, the ratio of the output frame to the non-output frame is 1: 2. Alternatively, for example, when the image sensor 201 captures images at 120 fps, if the output frame is 30 fps and the non-output frame is 90 fps, the ratio of the output frame to the non-output frame is 1: 3.

As described above, in a non-output frame, a motion vector is output from the camera 101 instead of the moving image, thereby encoding a 30 fps moving image and a 30 fps moving vector rather than encoding a 60 fps moving image. The code amount can be reduced by doing so.
Compared to a configuration in which only a moving image of 30 fps is encoded and a motion vector is obtained by the image decoding / display device 20, a motion vector is obtained by using an actual frame of a moving image photographed by the image coding device 10. An accurate motion vector can be obtained.

Therefore, in this embodiment, the generated interpolated frame is generated by a technique such as FRC by generating the interpolated frame by the image decoding / display apparatus 20 using the motion vector encoded by the image encoding apparatus 10. The image quality is higher when compared with the generated conventional interpolation frame.
In addition, when detecting a motion vector, the smaller the frame interval, the fewer false motion vector detection. For example, if a 30 fps moving image and a 210 fps motion vector are output for a moving image shot at a high frame rate such as 240 fps, the frame interval for detecting the motion vector is reduced. Thereby, the accuracy of the detected motion vector is also increased, and the image quality of the frame interpolated using the detected motion vector can be improved.

  In the above example, the motion vector is obtained for the moving image output from the image sensor 201 of the camera 101. However, color correction, color conversion, and gamma correction are not performed inside the image sensor 201. By detecting a motion vector using raw data, it is possible to output a motion vector for a high frame rate moving image without being restricted by the amount of data that can be output from the image sensor 201.

Next, an embodiment of the image decoding / display device 20 will be described. The image decoding / display apparatus 20 illustrated in FIG. 1 includes a decoder 103, a display unit 104, a motion vector decoding unit 108, and a frame interpolation unit 105.
The decoder 103 corresponds to the image decoding unit of the present invention, decodes the image output from the encoder 102 of the image encoding device 10, and outputs the decoded image to the frame interpolation unit 105. Since the decoder 103 and the display unit 104 have the same configuration as that of the conventional technology, the description thereof is omitted.

The motion vector decoding unit 108 decodes the received motion vector encoded data separately from the moving image encoded data, and outputs the decoded motion vector to the frame interpolation unit 105.
The frame interpolation unit 105 generates an interpolation frame using the decoded frame decoded by the decoder 103 and the motion vector decoded by the motion vector decoding unit 108. Here, the frame interpolation unit 105 calculates the pixel value on the interpolation frame according to the method in which the motion vector detection unit 106 of the image encoding device 10 detects the motion vector. For example, when the motion vector detection unit 106 detects a motion vector by the method of FIG. 4A, the corresponding position (motion) on the decoded frame temporally before or after each block obtained by dividing the interpolation frame. ) Is required. Therefore, the frame interpolation unit 105 of the image decoding device / display device 20 calculates the pixel value of the interpolation frame using the pixel value of the block at the position of the decoded frame before or after corresponding to each block of the interpolation frame, and Generate interpolated frames.

  In addition, when the motion vector detection unit 106 detects a motion vector by the method of FIG. 4B, the motion vector indicates the position on the interpolation frame corresponding to each block of the decoded frame. First, the position on the interpolation frame corresponding to the decoded frame is calculated, the pixel value at the corresponding position on the interpolation frame is calculated based on the pixel value of each block of the decoded frame, and an interpolation frame is generated.

  In the above example, the image decoding / display device 20 receives the encoded data and the motion vector of the moving image from the image encoding device 10, and the frame interpolation unit 105 generates an interpolation frame. The encoded image data and motion vector of the moving image are recorded on a recording medium by the image encoding device 10, and the encoded data and motion vector of the moving image are read from the recording medium by the image decoding / display device 20. An interpolation frame can also be generated by processing.

As described above, in the above-described embodiment according to the present invention, the image decoding / display device 20 includes the encoded data of the moving image received directly from the image encoding device 10 or via the transmission path or the recording medium. By performing frame interpolation using the motion vector of the interpolation frame (non-output frame), it is possible to display a moving image at a frame rate higher than the frame rate of the decoded frame.
Further, the image encoding device 10 detects a motion vector for an interpolation frame using an actual moving image frame photographed by the camera 101, and delivers the encoded data to the image decoding / display device 20. Therefore, an accurate motion vector can be obtained from the conventional method for obtaining a motion vector for interpolating an interpolation frame from two decoded frames, and accurate frame interpolation is possible. Then, the frame-interpolated moving image can be improved in image quality.

FIG. 1 is a block diagram showing a configuration of an image encoding device and an image decoding / display device for explaining an embodiment of a frame interpolation device according to the present invention. It is a figure for demonstrating operation | movement of the camera of FIG. 1, and a motion vector detection part. FIG. 6 is another diagram for explaining the operation of the camera and the motion vector detection unit in FIG. 1. It is a figure for demonstrating the method of calculating | requiring the motion vector between an output frame and a non-output frame. It is a block diagram which shows an example of the conventional image coding and image decoding and display apparatus which implement | achieve frame interpolation. It is a block diagram which shows the other example of the conventional image coding and image decoding and display apparatus which implement | achieve frame interpolation. It is a figure for demonstrating the conventional general production | generation method of an interpolation frame.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Image coding apparatus, 20 ... Image decoding / display apparatus, 30 ... Image coding apparatus, 40 ... Image decoding / display apparatus, 101 ... Camera, 102 ... Encoder, 103 ... Decoder, 104 ... Display part, 105 ... Frame Interpolation unit 106 ... Vector detection unit 107 107 Vector encoding unit 108 108 Vector decoding unit 201 Image pickup element 201 b Image pickup element 202 Switch 202 302 Encoder 303 Decoder 305 Frame interpolation unit 306 ... A motion vector detection unit.

Claims (5)

A camera for taking images,
An image encoding unit that encodes an image captured by the camera;
A motion vector encoding unit that encodes a motion vector of an image captured by the camera;
An image decoding unit for decoding and outputting the image encoded by the image encoding unit;
A motion vector decoding unit for decoding the motion vector encoded by the motion vector encoding unit;
A frame interpolation apparatus comprising: a frame interpolation unit that generates an interpolation frame from the decoded frame output from the image decoding unit and the motion vector output from the motion vector decoding unit. The frame interpolating device according to claim 1, wherein
The camera has a motion vector detection unit that detects a motion vector of a frame of a captured image,
The frame interpolation device, wherein the motion vector encoding unit encodes and outputs the motion vector detected by the motion vector detection unit. The frame interpolation device according to claim 2, wherein
The camera switches between an output frame output to the image encoding unit for encoding a captured image and a non-output frame input to the motion vector detection unit without outputting to the image encoding unit. Having a switching part,
The frame interpolation device, wherein the motion vector detection unit inputs the non-output frame and the output frame, detects a motion vector related to the non-output frame, and outputs the motion vector to the motion vector encoding unit. A camera for taking images,
An image encoding unit that encodes an image captured by the camera;
A motion vector encoding unit that encodes a motion vector of an image captured by the camera,
An image encoding apparatus, wherein the image and the motion vector encoded by the image encoding unit and the motion vector encoding unit are output separately. Means for inputting an encoded image and an encoded motion vector of an image taken by a camera;
An image decoding unit for decoding and outputting the encoded image;
A motion vector decoding unit for decoding the encoded motion vector;
An image decoding apparatus comprising: a frame interpolation unit that generates an interpolation frame from the decoded frame output from the image decoding unit and the motion vector output from the motion vector decoding unit.

Images