JP2011035747A - Method, device and program for decoding moving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a moving image at high frame rate with a smooth motion using encoded data of a moving image at low frame rate as input. <P>SOLUTION: A motion vector for every block is calculated from decoding results by a motion vector calculation unit 12, and a frame rate according to size of the motion vector is calculated for every block by a frame rate decision unit 13. The motion vector is converted based on a value of the frame rate for every block by a motion vector division processing unit 14, and an interpolation frame to be inserted between decoded original frames is generated using the converted motion vector by an interpolation frame generation unit 15. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for decoding and reproducing a moving image, and in particular, to increase the number of frames by interpolating frames constituting a moving image with a small amount of calculation to increase the frame rate of a low frame rate at high speed. About.

  The moving image data is expressed as a series of still images (frames) corresponding to a certain interval of time. As the time interval (frame interval) decreases, that is, as the number of still images per unit time (frame rate) increases, the motion image becomes smoother and the quality is improved.

  Although the display device has a displayable frame rate that is improved with the performance improvement and can display 200 frames or more per second, a moving image that has already been encoded at a lower frame rate such as 30 frames or 60 frames per second. The display performance at the high frame rate of the display device cannot be fully utilized for the display. In order to display such a moving image at a high frame rate, it is necessary to interpolate frames that are insufficient to improve the frame rate.

  On the other hand, it is not appropriate in terms of cost and processing performance to provide a moving image playback device such as a home video playback device with a complicated image processing function, and therefore it is necessary to perform frame interpolation by a simple method.

  The above technical background will be described in more detail as follows. As a method for improving the frame rate by interpolating the frames, for example, as described in Non-Patent Document 1, using an optical flow, or as described in Non-Patent Document 2, morphing is performed. Many methods have been proposed, such as those used to generate an interpolated image and those using autocorrelation as described in Non-Patent Document 3. While these can generate high-quality interpolated images, they require a large amount of calculation for generating the interpolated images, and are not suitable for processing with devices with limited processing capabilities such as home video devices.

  A method using encoding information has been proposed as a method of performing frame interpolation with a device with limited processing capability (see Non-Patent Documents 4 to 6). These are The motion of the image from the reference frame to the frame to be interpolated is obtained using information such as the motion vector employed in the encoding such as H.264, and the pixel value to be interpolated is estimated from the obtained vector to generate the image. That's it. In this method, it is not necessary to estimate the motion of the image again by optical flow estimation or the like with a large processing amount, and the calculation amount of pixel generation is smaller than the methods of Non-Patent Documents 1 to 3 described above.

  In addition, as shown in Non-Patent Document 7 and Non-Patent Document 8, when the quality of the moving image is constant, the number of frames per unit time (frame rate) and the code amount per frame, that is, the number of bits, are a trade-off. Therefore, increasing one increases the other. In other words, if the amount of code required to encode two different areas on the image with the same quality in units of frames is the same, the apparent movement is smaller if the apparent movement speeds of the two areas are different. However, the quality as a moving image is high, and the quality varies depending on the speed of the apparent movement. For this reason, in order to bring the quality of the moving images in the two areas close to each other, the frame rate may be changed according to the moving speed of the moving images.

KRISHNAMURTHY R, WOODS JW, MOULIN P: “Frame Interpolation and Bidirectional Prediction of Video Using Compactly Encoded Optical-Flow Fields and Label Fields”, IEEE Transactions on Circuits and Systems for Video Technology Vol.9, No.5, Page.713- 726 (1999.08). Takashi Yamamoto, Miki Haseyama: “A Study on Frame Interpolation Based on Morphing High Accuracy of Region Segmentation in Adaptive Corresponding Point Estimation”, IEICE Technical Report Vol.107, No.487 (ITS2007-65), Page.19-24 (2008.02.12). Yasutoshi Takeyama, Nao Mishima, Go Ito: “High-Precision Frame Interpolation Technique for 120Hz LCDs—Autocorrelation Block Matching”, Proceedings of the 2007 Annual Conference of the Institute of Image Information and Television Engineers (CD-ROM), Vol. .ROMBUNNO.3-7 (2007.08.01). Ryo Oshikiri, Tatsuro Fujisawa, Yoshihiro Kikuchi: “One-Seg Frame Interpolation Using Coding Information”, 2008 Video Media Processing Symposium, Vol.13th, Page.161-162 (2008.10.29). Yuki Tanaka, Kenji Mito: “One-Seg Frame Interpolation Technology Using H.264 Syntax Elements”, IEICE Technical Report Vol.107, No.224 (MoMuC2007 47), Page.23-28 (2007.09. 13). Yuichi Dehara, Shunichi Sekiguchi, Kazuo Sugimoto, Kotaro Asai: “A Study on Improvement of Temporal Resolution of Low Rate Video Using Coding Information”, Proceedings of the IEICE Conference, Vol.2005, Information and Systems 2, D-11-4, Page.4 (2005.03.07). Junichi Yamagata, Shinichi Tanaka, Toshiyuki Yoshida: “About the flattening of the quality of the encoded video by buffer control”, 2007 Image Coding Symposium (PCSJ2007), P-2.11, Page.29-30 (2007.10.31-11.2) . Shinichi Tanaka, Daigoro Miyagawa, Toshiyuki Yoshida, Takefumi Tsukuba, Toshio Nomura: “Image quality control and bit rate reduction effect based on MOS”, 2008 Image Coding Symposium (PCSJ2008), P-2.11, Page.29- 30 (2008.10.29-31)

  In order to improve the frame rate of the moving image, a necessary number of frames may be interpolated between the frames of the original moving image, but the amount of calculation increases according to the number of frames to be interpolated. In order to improve from a frame rate of an image of about 30 fps (frames / second) to a frame rate exceeding 200 fps, it is necessary to interpolate and insert a plurality of frames between two adjacent frames. It is necessary to reduce.

  When displaying at the same frame rate, the slow-moving part is smoother than the fast-moving part. Therefore, to maintain a constant quality of the entire moving image, the frame rate of the slow-moving part is changed. It is possible to make it lower than the frame rate of the fast part. If the frame rate can be changed in accordance with the speed of motion for each image area, the amount of calculation for frame interpolation can be further reduced while maintaining the smoothness of the moving image.

  The present invention pays attention to this point, and an object thereof is to provide a technique capable of reducing the amount of calculation for frame interpolation more while maintaining the smoothness of a moving image.

  In order to solve the above problems, the present invention determines the number of frames to be interpolated for each region on the moving image according to the size of the motion vector, and based on the motion vector when generating the interpolated image. The frame rate of a moving image can be improved with a small amount of calculation by calculating from the motion vector of the encoded data of, and calculating the size by reducing the size to a value corresponding to the size of the frame interval. Makes it possible to generate

  That is, the present invention provides a decoder that outputs encoded information in encoded data for calculating a motion vector of a moving image block, means for determining a block motion vector from the encoded information, and a size of the motion vector. Means for determining a frame rate after frame interpolation, means for generating a motion vector corresponding to the interpolation frame based on the frame rate, and means for generating an interpolated image from the image of the reference frame and the motion vector.

  The operation of the present invention is as follows. In order to interpolate a frame, a pixel to be interpolated is predicted from the pixels of the original moving image data. Therefore, the amount of calculation increases as the number of frames to be interpolated increases. Therefore, the smaller the number of interpolation frames, the smaller the amount of calculation and the easier the processing.

  Based on the magnitude of the motion vector obtained from the decoder, the magnitude of the motion of the image at a time interval of one frame can be estimated for each block. If the number of frames to be interpolated is determined so that the magnitude of image motion per frame falls within a certain range between the minimum and maximum values, the number of frames to be interpolated can be reduced while maintaining smoothness of motion. be able to. By performing substantial frame interpolation only on the frames thus determined, a high frame rate moving image with smooth motion can be generated from a low frame rate moving image with a small amount of processing.

  According to the present invention, it is possible to provide a moving picture decoding apparatus that generates a high frame rate moving picture with smooth motion from a low frame rate moving picture with a small amount of processing.

It is a block diagram showing the apparatus structural example of this invention. It is the flowchart which showed the whole process of embodiment of this invention. It is a flowchart showing the processing content of a motion vector division | segmentation process part.

  FIG. 1 shows an example of the apparatus configuration of the present invention. The moving picture decoding apparatus 10 shown in FIG. 1 includes MPEG-2 (ISO / IEC 13818-2), MPEG-4 (ISO / IEC 14496), H.264, and H.264. H.264 (see Reference 1) or the like is input as data encoded by a moving image encoding method using interframe compression based on motion compensation.

[Reference 1]: ITU-T H.264 SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS, Infrastructure of audiovisual services. Coding of moving video. Advanced video coding for generic audiovisual Services.
In these encoding methods, a frame of moving image data is divided into blocks, and the amount of data is reduced by using motion compensation that predicts motion from other frames. The motion prediction is expressed by a motion vector, and the difference between the predicted value and the actual block pixel is further encoded as an error. Actually, motion vectors are not represented by absolute values, but the amount of data is reduced by predicting from the motion vector values of the blocks around the block and expressing the error by encoding it. (See Reference 2).

[Reference 2]: Supervised by Satoshi Okubo, Junya Kakuno, Yoshihiro Kikuchi, Teruhiko Suzuki, “Revised H.264 / AVC textbook”, published by Impress R & D, 2006.
As described above, since the code data itself includes motion vector information, it is possible to obtain a motion vector for each block at high speed by using the code data. Further, a motion vector per one frame interval of a block can be obtained from a difference in frame number with a reference frame serving as a motion vector reference by a method as shown in Non-Patent Document 4.

The decoder 11 of the moving picture decoding apparatus 10 decodes the input encoded data to generate image data of the frame, and at the same time, information on the motion vector for each block of the frame used at the time of generating the image data and the reference frame number Is output. Here, the motion vector v _d output from the decoder for each block is
v _d = (v _dx , v _dy )
And

  In these encodings, since block prediction is possible in both directions of the previous frame and the subsequent frame in time, the decoder output corresponds to the prediction from the frame of the previous time in advance. (Direction).

  The motion vector calculation unit 12 calculates the motion vector v per frame of the original image of the block from the motion vector for each block of the decoder output and the reference frame number as follows. n is the difference between the reference frame number and the current frame number.

v = v _d / n = (v _dx / n, v _dy / n)
H. In coding such as H.264, bi-prediction and bi-prediction are possible in which motion prediction is performed from two frames and the average of two prediction values is used as pixel prediction. When a plurality of motion vectors are set in one block by bi-prediction or the like, one motion vector is generated from the two vectors. The simplest method is to select one of the motion vectors. Another method is to take an average value of two motion vectors per frame.

  In these encodings, a frame (I frame) for which inter-frame prediction is not performed is inserted. Since no motion vector information can be obtained for a frame for which inter-frame prediction is not performed, the motion vector information of the immediately preceding frame is used as it is.

  The frame rate determination unit 13 determines the frame rate of each block based on the motion vector data as follows, and transmits it to the motion vector division processing unit 14.

  The frame rate determination unit 13 first determines a block for which frame interpolation is not performed based on the image of the block. When a block is evaluated for the likelihood of noise generation by frame interpolation and the motion vector may not represent the motion of the image, that is, when the reliability of the motion vector is lower than a certain amount, the block It is assumed that frame interpolation between adjacent original image frames is not performed. When no inter-frame prediction is performed and no motion vector is set for a block, frame interpolation based on the image of the block is not performed. The reliability of the motion vector can be obtained by, for example, a method disclosed in Non-Patent Document 5.

Once the block for frame interpolation is determined, the frame rate for each block is determined. Let r be the ratio of the frame rate when the frame is interpolated to the frame rate of the original moving image. That is, if the frame rate r ₀ of the original image and the frame rate after frame interpolation are r ₁ , r is determined as follows.

r = r ₁ / r ₀
If the motion vector per frame interval at the original frame rate is v = (v _x , v _y ), the motion vector v ′ per frame interval at the frame rate increased by r times is ,
v ′ = (v _x / r, v _y / r)
| V ′ | = | v | / r
r = | v | / | v ′ |
It is.

Here, the frame rate determination unit 13, the length of the motion vector of the target | v _t | and the maximum value max of the _{r 1 (r 1) = m} × r 0 (m is an integer) holds.

r is a value obtained by rounding | v | / | v _t | to an integer (for example, an integer greater than or equal to | v | / | v _t |), and r ₁ (r ₁ = r × r ₀ ) There divisor of max (r ₁₎ below and max (r _1), i.e. r is set to be a divisor of m. In this way, the frame rate r ₁ after block interpolation is obtained. For blocks that do not perform interpolation, r ₁ = r ₀ .

  The motion vector division processing unit 14 divides the motion vector data for each block received from the motion vector calculation unit 12 based on the frame rate for each block received from the frame rate determination unit 13, includes an interpolation frame, Set a new motion vector for a block of frames.

The vector division in the motion vector division processing unit 14 is performed as follows. When the motion vector is (v _x , v _y ) and the frame rate is r ₁ (r = r ₁ / r ₀ ), the motion vector v ′ of the block is
v ′ = (v _x / r, v _y / r)
It is.

When increasing the frame rate from r ₀ to max (r ₁ ), max (r ₁ ) / r ₀ −1 immediately before the original frame
A frame is inserted. Here, the number of the frame to be inserted is set in ascending order of time.
1, 2,..., Max (r ₁ ) / r ₀ −1
It is attached.

  In order from the interpolation frame number 1, the motion vector of each block is set as follows (see the flowchart of FIG. 3).

Let n be the number of the interpolation frame.
(1) When r ₁ = r ₀ , the motion vector is a zero vector.
(2) When r ₁ = r ₀ is not true (max (r ₁ ) / r ₁ is an integer),
When n / (max (r ₁ ) / r ₁ ) is an integer, the motion vector is set to v ′ obtained above. Otherwise, the motion vector is the zero vector.

  Based on the motion vector for each block of the frame and the reference frame data in the reference frame buffer 16, the interpolation frame generation unit 15 generates an image of the interpolation frame as follows and sends it to the frame buffer 17.

  The interpolation frame generation unit 15 determines a block of the interpolation frame at a position shifted by v ′ pixels from the position of the same block of the image data in the reference frame buffer 16 based on the motion vector v ′ received from the motion vector division processing unit 14. An interpolation image is generated using the block information as an image of the interpolation frame image. Further, the generated interpolated image is stored in the reference frame buffer 16 in order to generate the next interpolated frame.

  In the reference frame buffer 16, frame image data is input and stored from the decoder 11 and the interpolation frame generation unit 15 to generate an interpolation frame.

  The frame buffer 17 inserts the frame generated by the interpolation frame generation unit 15 between the frames generated from the original encoded data, and outputs the frame.

  FIG. 2 is a flowchart showing the entire processing of the present invention. The decoder 11 decodes the frame image and sends motion vector information to the motion vector calculation unit 12 (step S10). The motion vector calculation unit 12 calculates a motion vector per frame of each block (step S11). The frame rate determination unit 13 determines the frame rate of each block based on the magnitude of the motion vector (step S12). The motion vector division processing unit 14 divides the motion vector data for each block based on the frame rate for each block, and determines the motion vector of each block of the interpolation frame (step S13).

  The interpolation frame generation unit 15 performs the following processing for each interpolation frame. First, each block image of the interpolation frame is generated based on the reference image and the motion vector (step S14). The generated interpolation frame is inserted after the original frame (step S15). The generated frame is used as a reference image, and the processing after step S14 is similarly repeated for the next interpolation frame until all the interpolation frames are completed (steps S16 to S18).

  When the processing of one interpolation frame is completed, using the original image as a reference image in the same manner, the processing from step S10 onward is repeated for the next interpolation frame until the processing of all frames is completed, and the frame to be decoded ends. If so, the process ends (steps S19 to S21).

FIG. 3 is a flowchart showing the processing contents of the motion vector division processing unit 14. First, the interpolation frame number n is initialized to 1 (step S30). Next, the first block (coordinates are (x, y)) is set as a processing target (step S31). The following processing is performed on the block of the interpolation frame number n and coordinates (x, y) (step S32). Assume that the interpolated frame rate is r ₁ and the original frame rate is r ₀ (step S33).

If r ₁ = r ₀ and n / (max (r ₁ ) / r ₁ ) is not an integer, the motion vector of the block is set to a zero vector (steps S34, S35, S36). In other cases, the motion vector of the block is set to v ′ described above (steps S34, S35, S37).

If the processing of the last block of the frame has not been completed, the next block is selected, the coordinates are set to (x, y), and the processing after step S32 is similarly repeated (steps S38 and S39). When the processing of the last block of the frame is completed, it is determined whether or not the processing of the last interpolation frame has been completed. If not, 1 is added to n, and similarly for the next interpolation frame. The processes after step S31 are repeated (steps S40 and S41). When the process of the last interpolation frame is completed, that is, when n = max (r ₁ ) / r ₀ −1, the process is terminated.

  In the above embodiment, the decoder 11 outputs the motion vector for each block of the frame obtained by decoding the encoded data and the information of the reference frame number to the motion vector calculation unit 12.

  As another embodiment, instead of writing the motion vector and reference frame number information from the decoder 11 to the motion vector calculation unit 12, the coding information included in the encoded data is written to the motion vector calculation unit 12. Is also possible. Here, the coding information is information used when a motion vector is calculated for block motion compensation in inter-frame prediction in the coding method. Specifically, it refers to information other than the information used for intra prediction encoding and DCT coefficients.

  By using the encoding information, it is possible to obtain a motion vector for each block representing the motion of the image more accurately than the motion vector used in the encoding at high speed without performing an operation on the pixel such as optical flow estimation. Is possible.

  When the encoding information is input, the motion vector calculation unit 12 uses a method as shown in Non-Patent Document 4, Non-Patent Document 5, and Non-Patent Document 6 for each frame, 8 × 8 pixels on the frame. A motion vector per one frame interval of the block is obtained and transmitted to the frame rate determination unit 13 and the motion vector division processing unit 14 as motion vector information.

  As described above, in the method of examining the motion of each block from the encoded information, not only the motion vector information in the above-described embodiment but also additional information such as block type information is used to more accurately determine the motion vector of the image. Can be calculated.

  The above-described video decoding process can be realized by a computer and a software program, and the program can be recorded on a computer-readable recording medium or provided through a network.

DESCRIPTION OF SYMBOLS 10 Moving image decoding apparatus 11 Decoder 12 Motion vector calculation part 13 Frame rate determination part 14 Motion vector division | segmentation process part 15 Interpolation frame production | generation part 16 Reference frame buffer 17 Frame buffer

Claims (4)

A method of decoding a moving image encoded by an encoding method that performs inter-frame prediction by motion compensation,
A process of calculating a motion vector for each block of the frame from the encoded data information of the decoded frame;
Depending on the size of the motion vector, a process for calculating a frame rate for each block having a high frame rate for a block with a large motion vector and a low frame rate for a block with a small motion vector;
Transforming the motion vector of the block based on the value of the frame rate for each block;
A method for decoding a moving image, comprising: generating frame image data based on a motion vector obtained by the conversion, and generating an interpolated frame to be inserted between the decoded original frames. The frame rate calculated for each block is a frame rate that is equal to or less than the maximum frame rate that is an integral multiple of the original frame rate and is a divisor of the maximum frame rate,
The original frame rate is r ₀ , the calculated frame rate is r ₁ , the maximum frame rate is r _max , and the motion vector v per frame interval at the original frame rate is v = (v _x , v _y ), When n is the number of an interpolated frame to be interpolated in order to make the original frame rate r _{0 the} calculated frame rate r ₁ , n × r ₁ / r _max is an integer when converting the motion vector of the block. In this case, the converted motion vector v ′ is set to v ′ = (v _x × r ₀ / r ₁ , v _y × r ₀ / r ₁ ). In other cases, the converted motion vector v ′ is set to zero. The moving image decoding method according to claim 1, wherein the conversion is a vector. An apparatus for decoding a moving image encoded by an encoding method that performs inter-frame prediction by motion compensation,
Means for calculating a motion vector for each block of the frame from the encoded data information of the decoded frame;
Depending on the magnitude of the motion vector, a means for calculating a frame rate for each block having a high frame rate for a block with a large motion vector and a low frame rate for a block with a small motion vector;
Means for converting the motion vector of the block based on the value of the frame rate for each block;
A moving image decoding apparatus comprising: means for generating image data of a frame based on a motion vector obtained by the conversion, and generating an interpolated frame to be inserted between the decoded original frames.   A moving picture decoding program for causing a computer to execute the moving picture decoding method according to claim 1.

Images