JP5304709B2 - Moving picture decoding apparatus, moving picture decoding method, and moving picture decoding program - Google Patents

Description

  The present invention relates to a moving image signal decoding technique.

  In recent years, services that deliver digital image and sound content via broadcast waves such as satellite and terrestrial waves and networks have been put into practical use, and content with a huge amount of information can be efficiently recorded and transmitted. In order to do so, a high-efficiency encoding technique is required. As high-efficiency encoding of moving images, the correlation between pixels that are spatially adjacent in the same frame of a moving image signal, and the correlation between temporally adjacent frames and fields, represented by MPEG4-AVC, are used. A method of compressing information using it is used.

  In MPEG4-AVC, as a compression using temporal correlation, a local decoded image of a frame that has already been encoded is used as a reference image for a target image that is an encoding target frame, and a two-dimensional block of a predetermined size is used. A motion amount (hereinafter referred to as “motion vector”) between the target image and the reference image is detected in units (hereinafter referred to as “target block”), and a predicted image based on the target block and the motion vector is detected. The generated motion compensated prediction is used.

  In MPEG4-AVC, the size of a target block in a 16 × 16 pixel two-dimensional block (hereinafter referred to as “macroblock”), which is a unit of encoding processing, is changed, and a motion vector for each target block is obtained. By using a method of predicting using, a method of storing a plurality of reference images and selecting a reference image used for prediction, and a method of generating a motion predicted image by obtaining a motion vector between two reference images and a target block, It is possible to improve the prediction accuracy of motion compensated prediction, thereby reducing the amount of information.

  Also, in motion compensated prediction, it is necessary to encode and transmit the generated motion vector, and in order to prevent an increase in the amount of information due to the motion vector, a predicted motion vector predicted from the motion vector for a decoded block around the target block By encoding using values, it is possible to use motion compensated prediction called direct mode in which no motion vector is transmitted.

  However, since the prediction of the motion vector cannot always be obtained with high accuracy, as shown in Patent Document 1, both the encoding side and the decoding side detect a motion vector between reference images, and the motion vector is A method of generating a predicted motion vector of a target block and assuming a direct mode on the assumption that it is continuous in time is also presented.

JP 2008-154015 A

  In motion compensated prediction in conventional moving picture coding represented by MPEG4-AVC, the following problems cannot be solved, and hence improvement in coding efficiency is hindered.

  The first problem is the degradation of the quality of the motion compensated predicted image due to the degradation of the quality of the decoded image used as the reference image, especially the degradation mixed in the motion compensated predicted image when high-compression encoding is performed. While the component deteriorates the prediction accuracy, it is necessary to encode information for restoring the deteriorated component as a prediction difference, and the amount of information is increasing.

  The second problem is that the motion vector prediction is not accurate enough for image signals with little temporal and spatial motion continuity, and the predicted image quality when using the direct mode is effective. It is a point not to do. This degradation occurs when adjacent blocks have different motions across the target object, and the motion vector used for prediction when the motion is large in time has moved corresponding to the motion of the original target block This degradation occurs because a block of positions is assumed. Similarly, when the motion changes with time, the prediction is not successful and deterioration occurs.

  The third problem is an increase in the amount of code required for motion vector transmission when using prediction using two reference images or motion compensated prediction in units of fine blocks. When two reference images are used, prediction deterioration is smoothed by adding the reference images, and the influence of the deterioration component can be reduced. To increase. Also, in motion compensation in fine block units, it is possible to obtain appropriate motion according to the boundary of the object, and the accuracy of the predicted image is improved, but it is necessary to transmit motion vectors in fine units. The amount of code increases.

  Patent Document 1 is a technique presented to solve the second problem described above. When a spatially uniform motion is present, a motion vector obtained between reference images is a target block. The motion vector prediction accuracy is improved because the motion passes through the position, but if the motion is not spatially uniform, it is the predicted motion vector obtained without using the target block information Therefore, the motion is different from that of the target block, and the prediction is not sufficient. In addition, in order to capture a large motion, a motion vector detection process over a wide range between reference images is required for both the encoding device and the decoding device, which causes a problem that the amount of calculation increases.

  The present invention has been made in view of such a situation, and an object of the present invention is to provide a technique for improving the efficiency of motion compensated prediction by improving the quality of a predicted image while suppressing an increase in the amount of calculation in an encoding device and a decoding device. It is to provide.

  In order to solve the above problems, a video decoding device according to an aspect of the present invention includes a motion vector decoding unit that decodes a motion vector for a decoding target block from an encoded stream, and a first reference image using the motion vector. A reference image synthesizing unit that generates a synthesized reference block obtained by synthesizing the extracted first reference block and a predetermined region of at least one other reference image, a synthesized reference block as a predicted block, a prediction block, and a decoding target A decoding unit that generates a decoded image by adding the prediction difference block decoded from the block;

In order to solve the above-described problem, a video decoding device according to an aspect of the present invention includes a motion vector decoding unit that decodes a first motion vector for a decoding target block from an encoded stream, and a first motion vector. A motion vector separation unit that generates two motion vectors, a first reference block in a specific area having a size equal to or larger than the decoding target block, extracted from the first reference image using the second motion vector, and the like A block having the same size as the decoding target block is extracted from the combined reference block using a reference image combining unit that generates a combined reference block that combines a predetermined region of at least one reference image and a first motion vector. , Adding the motion compensated prediction unit having the extracted block as a prediction block, the prediction block, and the prediction difference block decoded from the decoding target block The Rukoto, and a decoder for generating a decoded image.

  In the motion vector separation unit, the accuracy of the input first motion vector is M pixel accuracy (M is a real number), and the accuracy of the second motion vector to be generated is N pixel accuracy (N is a real number: N> M). And the second motion vector is a value obtained by converting the first motion vector to N pixel accuracy, and the specific area is based on the position of the first reference image indicated by the second motion vector. It may have an area of ± N / 2 pixels or more.

  According to this configuration, if the accuracy of the decoded motion vector is M pixel accuracy, the motion vector is converted to N pixel accuracy rougher than the M pixel, and the reference image subjected to motion compensation prediction is converted based on the converted motion vector value. By synthesizing other reference images, it is possible to perform the same synthesizing process on the decoding side as the encoding device, and by combining the difference value between the converted motion vector value and the received motion vector value, motion compensation is performed. By using it as a phase correction value of a predicted image, a motion compensated predicted image with a small prediction residual generated on the encoding device side can be obtained by a decoding device with one motion vector value.

  The reference image synthesis unit may include an inter-reference image motion vector detection unit that detects a third motion vector between the first reference block and a second reference image that is another reference image. The reference image synthesis unit calculates an average value or a weighted average value for each pixel of the second reference block extracted from the second reference image using the third motion vector and the first reference block. Thus, a synthesized reference block may be generated.

  According to this configuration, a motion vector value between another reference image is obtained with respect to a motion compensated predicted image predicted using the first reference image, and the motion compensated predicted image obtained from the other reference image is added. By taking the average, it is possible to generate a prediction image corresponding to the removal of the coding degradation component and the slight luminance change of the decoding target, and the coding efficiency can be improved.

  The inter-reference image motion vector detection unit may detect a plurality of third motion vectors between the first reference block and the second reference image in units of blocks smaller than the first reference block. . The reference image synthesis unit combines a plurality of second reference blocks in units of small blocks extracted from the second reference image using a plurality of third motion vectors, and calculates an average for each pixel with the first reference block. A composite reference block may be generated by calculating a value or a weighted average value.

  According to this configuration, with respect to the motion compensated predicted image predicted using the first reference image, a motion vector value in a smaller unit than the target motion compensated predicted image between other reference images is obtained, By performing the synthesis process with the motion compensated prediction image acquired in fine units according to the motion vector, it is possible to generate a prediction image corresponding to a minute temporal deformation of the object of the decoding target, and encoding efficiency Can be improved.

  The inter-reference-picture motion vector detection unit performs the first time difference according to the two time differences between the first time difference between the first reference picture and the decoding target block and the second time difference between the second reference picture and the decoding target block. The third motion vector may be detected by searching for a motion within a predetermined range around the motion vector value obtained by converting the motion vector 2.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the quality of a prediction image can be improved and the efficiency of motion compensation prediction can be improved, suppressing the increase in the computational complexity in an encoding apparatus and a decoding apparatus.

It is a block diagram which shows the structure of the moving image encoder of Embodiment 1 of this invention. It is a block diagram which shows the structure of the moving image decoding apparatus of Embodiment 1 of this invention. It is a conceptual diagram which shows the synthetic | combination image motion compensation prediction method in this invention. It is a block diagram which shows the structure of the multiple reference image synthetic | combination part in the moving image encoder of Embodiment 1 of this invention. It is a block diagram which shows the structure of the multiple reference image synthetic | combination part in the moving image decoding apparatus of Embodiment 1 of this invention. It is a block diagram which shows the structure of the moving image encoder of Embodiment 2 of this invention. It is a block diagram which shows the structure of the moving image decoding apparatus of Embodiment 2 of this invention. It is a conceptual diagram which shows the operation | movement of the synthetic | combination image motion compensation prediction process in Embodiment 2 of this invention. It is a block diagram which shows the structure of the multiple reference image synthetic | combination part in the moving image encoder of Embodiment 2 of this invention. It is a flowchart for demonstrating operation | movement of the multiple reference image synthetic | combination part and a synthetic | combination image motion compensation prediction part in the moving image encoder of Embodiment 2 of this invention. It is a figure which shows an example of the process order of the encoding process in Embodiment 2 of this invention, and reference image management. It is a figure which shows an example of the motion vector detection range between the reference images in Embodiment 2 of this invention. It is a figure which shows an example of the additional information to the slice header in Embodiment 2 of this invention. It is a figure which shows an example of the additional information to the motion compensation prediction mode in Embodiment 2 of this invention. It is a block diagram which shows the structure of the multiple reference image synthetic | combination part in the moving image decoding apparatus of Embodiment 2 of this invention. It is a flowchart for demonstrating operation | movement of the motion vector separation part, multiple reference image synthetic | combination part, and a synthetic | combination image motion compensation prediction part in the moving image decoding apparatus of Embodiment 2 of this invention. It is a conceptual diagram which shows the operation | movement of the synthetic | combination process of the reference image in Embodiment 3 of this invention. It is a conceptual diagram which shows the operation | movement of the composite image motion compensation prediction process in Embodiment 4 of this invention. It is a block diagram which shows the structure of the multiple reference image synthetic | combination part in the moving image encoder and moving image decoder of Embodiment 4 of this invention. It is a flowchart for demonstrating operation | movement of the synthetic | combination determination part in the moving image encoder and moving image decoder of Embodiment 4 of this invention. It is a conceptual diagram which shows the operation | movement of the synthetic | combination image motion compensation prediction process in Embodiment 5 of this invention. It is a block diagram which shows the structure of the multiple reference image synthetic | combination part in the moving image encoder and moving image decoder of Embodiment 5 of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
First, the moving picture coding apparatus according to the first embodiment will be described. FIG. 1 is a block diagram showing a configuration of the moving picture encoding apparatus according to the first embodiment.

  As shown in FIG. 1, the moving picture coding apparatus according to Embodiment 1 includes an input terminal 100, an input picture buffer 101, a block division unit 102, an intra-frame prediction unit 103, a motion vector detection unit 104, and a motion compensation prediction unit 105. , Motion vector prediction unit 106, multiple reference image synthesis unit 107, prediction mode determination unit 109, subtractor 110, orthogonal transform unit 111, quantization unit 112, inverse quantization unit 113, inverse orthogonal transform unit 114, adder 115, An intra-frame decoded image memory 116, a decoded reference image memory 117, an entropy encoding unit 118, a stream buffer 119, an output terminal 120, and a code amount control unit 121 are provided.

  The point in which the multiple reference image synthesis unit 107 is provided and the operation in the processing block and the motion compensation prediction unit 105 are the features in the first embodiment of the present invention. With regard to the other processing blocks, a moving image such as MPEG4-AVC is used. The same processing as the processing blocks constituting the encoding processing in the encoding device can be applied.

  A digital image signal input from the input terminal 100 is stored in the input image buffer 101. The digital image signal stored in the input image buffer 101 is supplied to the block dividing unit 102, and is cut out as an encoding target block in units of macroblocks composed of 16 × 16 pixels. The block division unit 102 supplies the extracted encoding target block to the intra-frame prediction unit 103, the motion vector detection unit 104, the motion compensation prediction unit 105, and the subtractor 110.

  In the intra-frame prediction unit 103, the decoding target image input from the block division unit 102 and the decoded image of the area that has been encoded with respect to the periphery of the encoding target block stored in the intra-frame decoding image memory 116. Is input, and prediction using correlation within the frame is performed. For example, the pixel value is predicted in a plurality of predetermined directions in units of 4 × 4 pixels, 8 × 8 pixels, and 16 × 16 pixels for the encoding target block, and is selected as a unit of prediction processing Prediction using the correlation of adjacent pixels in the screen is performed using a method called intra prediction that generates a predicted image together with information indicating the direction (intra prediction mode). The predicted image and the selected intra prediction mode are output from the intra-frame prediction unit 103 to the prediction mode determination unit 109.

  In the motion vector detection unit 104, the encoding target block input from the block division unit 102 and the decoded image of the frame that has been encoded on the entire screen and stored in the decoded reference image memory 117 are input as the reference image. Then, motion estimation is performed between the encoding target block and the reference image. As a general motion estimation process, a reference image at a position moved by a predetermined movement amount from the same position on the screen is cut out, and the movement amount that minimizes the prediction error when the image is used as a prediction block is determined as a motion vector. As a value, a block matching process that is obtained while changing the movement amount is used. The detected motion vector value is output to the motion compensation prediction unit 105 and the multiple reference image synthesis unit 107.

  The motion compensation prediction unit 105 receives the motion vector value obtained by the motion vector detection unit 104, generates motion compensation prediction images for a plurality of block sizes of 16 × 16 or less and a plurality of reference images, and a block division unit The prediction signal with the least difference information to be encoded is selected for the encoding target block input from 102, and the combined reference image signal input from the multiple reference image combining unit 107 is also a prediction signal candidate. As described above, the prediction signal with the least difference information to be encoded is selected. The motion compensation prediction unit 105 outputs the selected motion compensation prediction mode and the prediction signal to the prediction mode determination unit 109. The motion compensation prediction mode includes mode information indicating whether or not motion compensation is performed using the synthesized reference image.

  The motion vector prediction unit 106 calculates a predicted motion vector value using the motion vectors of the surrounding encoded blocks, and supplies the motion vector detection unit 104 and the motion compensated prediction unit 105.

  By using the predicted motion vector value, the motion vector detection unit 104 detects the optimal motion vector value by taking into account the amount of code necessary for encoding the difference between the motion vector predicted value and the motion vector value. To do. Similarly, the motion compensation prediction unit 105 takes into account the amount of code required when encoding the difference between the motion vector prediction value and the motion vector value, and the reference image used as the block unit for the optimal motion compensation prediction and Select a motion vector value.

  The multi-reference image synthesis unit 107 inputs a motion vector value for one reference image output from the motion vector detection unit 104 and a plurality of reference images stored in the decoded reference image memory 117, and uses the multi-reference image. A reference image composition process is performed. The synthesized reference image signal is output to the motion compensation prediction unit 105. The detailed operation of the multiple reference image composition unit 107 will be described later.

  The prediction mode determination unit 109 applies the most prediction to the coding target block input from the block division unit 102 based on the prediction mode and the prediction image for each prediction method input from the intra-frame prediction unit 103 and the motion compensation prediction unit 105. A prediction signal with little difference information to be encoded is selected, and a prediction image block for the selected prediction method is output to the subtractor 110 and the adder 115, and the entropy encoding unit 118 performs prediction as additional information. The mode information and information that requires encoding according to the prediction mode are output.

  The subtractor 110 calculates the difference between the encoding target block supplied from the block dividing unit 102 and the prediction image block supplied from the prediction mode determination unit 109, and supplies the result to the orthogonal transform unit 111 as a difference block. .

  The orthogonal transform unit 111 generates DCT coefficients corresponding to the orthogonally transformed frequency component signal by performing DCT transform on the difference block in units of 4 × 4 pixels or 8 × 8 pixels. Further, the orthogonal transform unit 111 collects the generated DCT coefficients in units of macroblocks and outputs them to the quantization unit 112.

  The quantization unit 112 performs quantization processing by dividing the DCT coefficient by a different value for each frequency component. The quantization unit 112 supplies the quantized DCT coefficient to the inverse quantization unit 113 and the entropy coding unit 118.

  The inverse quantization unit 113 performs inverse quantization by multiplying the quantized DCT coefficient input from the quantization unit 112 by a value divided at the time of quantization, and the result of the inverse quantization is obtained. The decoded DCT coefficient is output to the inverse orthogonal transform unit 114.

  The inverse orthogonal transform unit 114 performs inverse DCT processing to generate a decoded difference block. The inverse orthogonal transform unit 114 supplies the decoded difference block to the adder 115.

  The adder 115 adds the prediction image block supplied from the prediction mode determination unit 109 and the decoded difference block supplied from the inverse orthogonal transform unit 114 to generate a local decoding block. The local decoded block generated by the adder 115 is stored in the intra-frame decoded image memory 116 and the decoded reference image memory 117 in a form subjected to inverse block conversion. In the case of MPEG-4 AVC, adaptive filtering is applied to the block boundary where the coding distortion of each block is likely to appear as a boundary before the local decoding block is input to the decoded reference image memory 117. In some cases, processing to be performed is performed.

  The entropy encoding unit 118 converts the quantized DCT coefficient supplied from the quantization unit 112, the prediction mode information supplied from the prediction mode determination unit 109, and information that requires encoding according to the prediction mode. On the other hand, variable length encoding of each information is performed. Specifically, intra prediction mode and prediction block size information in the case of intra-frame prediction, and prediction block size, reference image designation information, and motion vector in the case of motion compensation prediction and synthesized image motion compensation prediction, The difference value from the predicted motion vector value is information that requires encoding. The information subjected to the variable length coding is output from the entropy coding unit 118 to the stream buffer 119 as a coded bit stream.

  The encoded bit stream stored in the stream buffer 119 is output to a recording medium or a transmission path via the output terminal 120. Regarding the code amount control of the encoded bit stream, the code amount control unit 121 is supplied with the code amount of the encoded bit stream stored in the stream buffer 119 and compared with the target code amount. In order to approach the target code amount, the fineness (quantization scale) of the quantization unit 112 is controlled.

  Subsequently, a moving picture decoding apparatus that decodes an encoded bitstream generated by the moving picture encoding apparatus according to Embodiment 1 will be described. FIG. 2 is a configuration diagram of the moving picture decoding apparatus according to the first embodiment.

  As shown in FIG. 2, the moving picture decoding apparatus according to Embodiment 1 includes an input terminal 200, a stream buffer 201, an entropy decoding unit 202, a prediction mode decoding unit 203, a prediction image selection unit 204, an inverse quantization unit 205, and an inverse. Orthogonal transformation unit 206, adder 207, intra-frame decoded image memory 208, decoded reference image memory 209, output terminal 210, intra-frame prediction unit 211, motion vector prediction decoding unit 212, motion compensation prediction unit 213, and multiple reference image synthesis Part 215.

  The point in which the multiple reference image synthesis unit 215 is provided and the operation in the processing block and the motion compensation prediction unit 213 are the characteristics in the first embodiment of the present invention. For other processing blocks, a moving image such as MPEG4-AVC is used. The same processing as the processing blocks constituting the decoding processing in the decoding device can be applied.

  The encoded bit stream input from the input terminal 200 is supplied to the stream buffer 201, and the stream buffer 201 absorbs the code amount variation of the encoded bit stream and is supplied to the entropy decoding unit 202 in a predetermined unit such as a frame. The The entropy decoding unit 202 performs variable-length decoding on the encoded prediction mode information, the additional information corresponding to the prediction mode, and the quantized DCT coefficient from the encoded bitstream input via the stream buffer 201. Then, the quantized DCT coefficient is output to the inverse quantization unit 205, and the prediction mode information and additional information corresponding to the prediction mode are output to the prediction mode decoding unit 203.

  Regarding the inverse quantization unit 205, the inverse orthogonal transform unit 206, the adder 207, the intra-frame decoded image memory 208, and the decoded reference image memory 209, the local decoding process of the moving image coding apparatus according to the first embodiment of the present invention. Processing similar to that of a certain inverse quantization unit 113, inverse orthogonal transform unit 114, adder 115, intra-frame decoded image memory 116, and decoded reference image memory 117 is performed. The decoded image stored in the intra-frame decoded image memory 208 is displayed as a decoded image signal on the display device via the output terminal 210.

  The prediction mode decoding unit 203 performs motion vector prediction decoding when motion compensation prediction or synthesized motion compensation prediction is selected as the prediction mode from the prediction mode information input from the entropy decoding unit 202 and additional information corresponding to the prediction mode. The motion compensation prediction mode or the composite image motion compensation prediction mode, which is information indicating the predicted block unit, and the decoded difference vector value are output to the unit 212, and the prediction mode information is output to the prediction image selection unit 204. Output. Also, the prediction mode decoding unit 203 outputs information indicating selection and additional information according to the prediction mode to the intra-frame prediction unit 211 and the motion compensation prediction unit 213 according to the decoded prediction mode information. To do.

  The predicted image selection unit 204 selects a predicted image for the decoding target block output from either the intra-frame prediction unit 211 or the motion compensated prediction unit 213 according to the prediction mode information input from the prediction mode decoding unit 203. And output to the adder 207.

  When the decoded prediction mode indicates intra-frame prediction, the intra-frame prediction unit 211 receives the intra prediction mode as additional information according to the prediction mode from the prediction mode decoding unit 203, and according to the intra prediction mode. The decoded image of the region where decoding is completed is input to the periphery of the decoding target block stored in the intra-frame decoded image memory 208, and prediction using the intra-frame correlation is performed in the same intra prediction mode as the encoding device. Done. The intra-frame prediction unit 211 outputs the intra-frame prediction image generated by the prediction to the prediction image selection unit 204.

  The motion vector predictive decoding unit 212 uses the motion vector of the neighboring decoded block for the decoded difference vector value input from the prediction mode decoding unit 203, and performs the motion prediction using the same method as that performed by the encoding device. A vector value is calculated, and a value obtained by adding the difference vector value and the predicted motion vector value is output to the motion compensated prediction unit 213 and the multiple reference image synthesis unit 215 as the motion vector value of the decoding target block. The motion vector is decoded by the number encoded according to the block unit of the prediction process indicated in the motion compensation prediction mode or the composite image motion compensation prediction mode.

  The motion compensation prediction unit 213 receives the motion vector value input from the motion vector prediction decoding unit 212, the combined reference image signal input from the multiple reference image combining unit 215, and the prediction mode input from the prediction mode decoding unit 203. A motion compensation prediction image is generated based on information indicating whether the motion compensation prediction mode and the composite image motion compensation prediction are added information as additional information according to the information, and the generated motion compensation prediction image is output to the prediction image selection unit 204. .

  In the multiple reference image synthesis unit 215, the motion vector value for one reference image indicated in the synthesized image motion compensation prediction mode output from the motion vector prediction decoding unit 212 and the multiple reference images stored in the decoded reference image memory 209. And a reference image synthesis process using a plurality of reference images is performed. The synthesized reference image signal is output to the motion compensation prediction unit 213.

  The multi-reference image synthesis unit 215 is paired with the multi-reference image synthesis unit 107 in the moving picture coding apparatus according to Embodiment 1 of the present invention. The detailed operation of this block will be described later.

  Hereinafter, a motion compensation prediction prediction image generation method using a synthesized reference image, which operates in the video encoding device and the video decoding device according to Embodiment 1, will be described with reference to FIG.

  FIG.3 (c) is a conceptual diagram which shows the synthetic | combination image motion compensation prediction method in this invention. FIGS. 3A and 3B are conceptual diagrams of motion compensation prediction using a plurality of reference images used in MPEG4-AVC.

  FIG. 3 (a) detects a motion vector between two reference pictures called bi-directional prediction and a coding target block, transmits a motion vector for each reference picture, In this method, the average value of reference blocks indicated by motion vectors is used as a predicted image. By synthesizing two reference images, it is possible to generate a prediction image composed of a reference image by averaging a small luminance change of an encoding target and a function of removing an encoding degradation component as a motion adaptive filter in the time direction. .

  FIG. 3B is a technique for performing prediction using two reference images without transmitting a motion vector called a temporal direct mode. When a block at the same position as the encoding target block of the reference image 2 is generated by motion compensated prediction from the reference image 1, assuming that the motion is temporally continuous, the encoding target A motion vector value between the block, reference image 1 and reference image 2 is generated, and bidirectional prediction is performed using the motion vector. A prediction image obtained by synthesizing two reference images can be generated without transmitting a motion vector. However, when the motion vector value between the reference image 1 and the reference image 2 is large as shown in FIG. The motion represented by the motion vector value indicates the motion at a position spatially separated from the encoding target block, and is generated implicitly limited to the motion when temporally continuous. Therefore, the temporal direct mode does not function effectively when the continuity of motion vector values is temporally small.

  In the method disclosed in Patent Document 1, for the purpose of improving the quality of the above-described temporal direct mode, both the encoding side and the decoding side detect motion in a block existing at a symmetrical position around the encoding target block between reference images. By doing this, it is a technique to generate motion vectors with temporal continuity across the encoding target block, but it can function effectively for conditions that are not spatially continuous, but there is little temporal continuity In the case, it does not function as well as the time direct mode.

  As shown in FIG. 3C, the prediction configuration of the composite motion compensated prediction of the present invention detects and encodes the motion vector in the reference image 1 and applies it to the reference block indicated by the motion vector in the reference image 1. On the other hand, the motion vector detection between the reference images for the reference image 2 is performed on both the encoding side and the decoding side, and only the motion vector value for the reference image 1 is transmitted to generate a synthesized reference image using two reference images. By doing so, it is possible to perform good bidirectional prediction for an image whose spatial and temporal continuity is not maintained, and it is possible to realize motion compensation prediction processing with less motion vector information than conventional bidirectional prediction.

  Next, FIG. 4 shows a configuration diagram of a plurality of reference image synthesis units in the moving picture coding apparatus according to Embodiment 1, and the operation of reference image synthesis processing will be described. As shown in FIG. 4, the multiple reference image synthesis unit 107 includes a standard reference image acquisition unit 400, a motion vector detection range setting unit 401, an inter-reference image motion vector detection unit 402, a synthesized reference image acquisition unit 403, and a reference image synthesis unit. 404 and a composite image memory 405.

  First, the motion vector detection unit 104 inputs the motion vector value MV1 between the first reference image and the encoding target block to the standard reference image acquisition unit 400 and the motion vector detection range setting unit 401. The reference reference image acquisition unit 400 acquires the reference block of the first reference image from the decoded reference image memory 117 using the input MV1. The reference reference image acquisition unit 400 outputs the acquired first reference block to the inter-reference image motion vector detection unit 402 and the reference image synthesis unit 404.

  Subsequently, the motion vector detection range setting unit 401 sets a range for detecting a motion vector between the second reference images for the first reference block. For the detection range of motion vectors between reference images, a method of implicitly setting the same detection range in the encoding device and the decoding device can be applied, but the detection range setting for each frame or reference image used is encoded. It is also possible to use a method of transmitting as information. In the first embodiment, the detection range is set implicitly (eg, ± 32 pixels), and the center of the motion vector detection range is set to the same position as the position of the encoding target block in the reference image. .

  The inter-reference image motion vector detection unit 402 performs the second reference image in the motion vector detection range specified by the motion vector detection range setting unit 401 with respect to the first reference block input from the standard reference image acquisition unit 400. Are obtained from the decoded reference image memory 117 via the synthesized reference image acquisition unit 403, an error value such as block matching is calculated, and a motion vector having a small value is calculated as a motion vector between reference images. Similarly, with respect to the detection accuracy of inter-reference image motion vectors, it is possible to apply a technique for implicitly detecting motion vectors with the same detection accuracy in the encoding device and the decoding device. However, the motion vector is used for each frame or each reference image used. It is also possible to use a method of transmitting the detection accuracy of the data as encoded information. Here, as an implicit setting, the detection accuracy is 1/4 pixel accuracy. The inter-reference image motion vector detection unit 402 outputs the calculated inter-reference image motion vector to the reference image synthesis unit 404.

  In the reference image synthesis unit 404, the first reference block is input from the standard reference image acquisition unit 400, and the inter-reference image motion vector is input from the inter-reference image motion vector detection unit 402, which is indicated by the inter-reference image motion vector. The second reference block is obtained by acquiring the reference block of the second reference image from the decoded reference image memory 117 via the synthesized reference image acquisition unit 403. The reference image composition unit 404 performs composition processing between the first reference block and the second reference block. The synthesis process in the first embodiment employs a method of generating a reference block that is synthesized by taking an average for each pixel of the first reference block and the second reference block, for example. The reference image synthesis unit 404 outputs the synthesized reference block to the motion compensation prediction unit 105 via the synthesized image memory 405.

  Next, the configuration of the multiple reference image synthesis unit 215 in the moving picture decoding apparatus according to Embodiment 1 is shown in FIG. As shown in FIG. 5, the multiple reference image synthesis unit 215 includes a standard reference image acquisition unit 1000, a motion vector detection range setting unit 1001, an inter-reference image motion vector detection unit 1002, a synthesized reference image acquisition unit 1003, and a reference image synthesis unit. 1004 and a composite image memory 1005, and the operations of the reference reference image acquisition unit 400, motion vector detection range setting unit 401, inter-reference image motion vector detection unit 402, and composite reference image acquisition shown in FIG. Operations similar to those of the unit 403, the reference image synthesis unit 404, and the synthesized image memory 405 are performed.

  First, the decoded motion vector value MV1 is input from the motion vector predictive decoding unit 212 to the standard reference image acquisition unit 1000 and the motion vector detection range setting unit 1001. The reference reference image acquisition unit 1000 acquires the reference block of the first reference image from the decoded reference image memory 209 using the input MV1. The reference reference image acquisition unit 1000 outputs the acquired first reference block to the inter-reference image motion vector detection unit 1002 and the reference image synthesis unit 1004.

  Subsequently, the motion vector detection range setting unit 1001 sets a range for detecting a motion vector between the second reference images for the first reference block. Regarding the detection range, in Embodiment 1, the detection accuracy of ¼ pixel accuracy is set by an implicit setting, and the center of the motion vector detection range is ± 32 pixels at the same position as the position of the block to be encoded in the reference image. The detection range. The motion vector detection range setting unit 1001 outputs information on the set motion vector detection range to the inter-reference image motion vector detection unit 1002.

  The inter-reference image motion vector detection unit 1002 performs the second reference image in the motion vector detection range specified by the motion vector detection range setting unit 1001 with respect to the first reference block input from the standard reference image acquisition unit 1000. Are obtained from the decoded reference image memory 209 via the synthesized reference image acquisition unit 1003, an error value such as block matching is calculated, and a motion vector having a small value is calculated as a motion vector between reference images. The inter-reference image motion vector detection unit 1002 outputs the calculated inter-reference image motion vector to the reference image synthesis unit 1004.

  In the reference image synthesis unit 1004, the first reference block is input from the standard reference image acquisition unit 1000, and the inter-reference image motion vector detection unit 1002 is input, which is indicated by the inter-reference image motion vector. By acquiring the reference block of the second reference image from the decoded reference image memory 209 via the synthesized reference image acquisition unit 1003, the second reference block is obtained. The reference image synthesis unit 1004 performs a synthesis process between the first reference block and the second reference block. The reference image synthesis unit 1004 outputs the synthesized reference block to the motion compensation prediction unit 213 via the synthesized image memory 1005.

  With respect to the reference image obtained by calculating the motion vector between the encoding target block and the first reference image by the moving image encoding device and the moving image decoding device according to Embodiment 1 of the present invention, By synthesizing the reference image, it is possible to realize a motion compensated predicted image with a small prediction residual with additional information that only transmits one motion vector.

  In addition, by using the value of the motion vector between the reference images and the motion vector value MV1, it is possible to generate a motion vector value between the block to be encoded and the second reference image. It can be stored in the motion vector predictive decoding unit 212 and used as a predicted motion vector value for the subsequent encoding target block. Thus, the motion vector value that can be recognized by the decoding apparatus is increased and the prediction accuracy of the motion vector is improved, so that the motion vector can be transmitted with less information.

(Embodiment 2)
Next, the second embodiment will be described. In the second embodiment, the accuracy of motion vectors used for reference image synthesis processing is roughened, and fine phase alignment is performed on the synthesized reference image. FIG. 6 is a block diagram showing a configuration of the moving picture encoding apparatus according to the second embodiment.

  As shown in FIG. 6, the moving picture coding apparatus according to Embodiment 2 includes an input terminal 100, an input picture buffer 101, a block division unit 102, an intra-frame prediction unit 103, a motion vector detection unit 104, and a motion compensation prediction unit 105. , Motion vector prediction unit 106, multiple reference image synthesis unit 107, synthesized image motion compensation prediction unit 108, prediction mode determination unit 109, subtractor 110, orthogonal transform unit 111, quantization unit 112, inverse quantization unit 113, inverse orthogonal A conversion unit 114, an adder 115, an intra-frame decoded image memory 116, a decoded reference image memory 117, an entropy encoding unit 118, a stream buffer 119, an output terminal 120, and a code amount control unit 121 are provided. Compared to the first embodiment, the function of the composite image motion compensation prediction unit 108 is added, and the operation of the multiple reference image composition unit 107 is different. Only the operation of the functional blocks related to the added composite image motion compensation prediction unit 108 will be described.

  The motion compensation prediction unit 105 receives the motion vector value obtained by the motion vector detection unit 104, and similarly to the first embodiment, motion compensation prediction images for a plurality of block sizes of 16 × 16 or less and a plurality of reference images. For the encoding target block input from the block dividing unit 102, the prediction signal with the least difference information to be encoded is selected, and the selected motion compensation prediction mode and the prediction signal are selected as the prediction mode determination unit 109. Output to.

  The motion vector prediction unit 106 calculates a predicted motion vector value using the motion vectors of the surrounding encoded blocks in the same manner as in the first embodiment, and the motion vector detection unit 104, the motion compensation prediction unit 105, and the synthesis This is supplied to the image motion compensation prediction unit 108.

  The multi-reference image synthesis unit 107 inputs a motion vector value for one reference image output from the motion vector detection unit 104 and a plurality of reference images stored in the decoded reference image memory 117, and uses the multi-reference image. A reference image composition process is performed. The combined reference image signal is output to the combined image motion compensation prediction unit 108.

  The combined image motion compensation prediction unit 108 is input from the block dividing unit 102 using the combined reference image signal input from the multiple reference image combining unit 107 and the predicted motion vector value input from the motion vector prediction unit 106. The prediction signal with the least difference information to be encoded is selected for the encoding target block, and the selected combined image motion compensation prediction mode and prediction signal are output to the prediction mode determination unit 109. Detailed operations of the multiple reference image synthesis unit 107 and the synthesized image motion compensation prediction unit 108 will be described later.

  Next, a moving picture decoding apparatus that decodes an encoded bitstream generated by the moving picture encoding apparatus according to Embodiment 2 will be described. FIG. 7 is a configuration diagram of the moving picture decoding apparatus according to the second embodiment.

  As shown in FIG. 7, the moving picture decoding apparatus according to Embodiment 2 includes an input terminal 200, a stream buffer 201, an entropy decoding unit 202, a prediction mode decoding unit 203, a prediction image selection unit 204, an inverse quantization unit 205, and an inverse. Orthogonal transformation unit 206, adder 207, intra-frame decoded image memory 208, decoded reference image memory 209, output terminal 210, intra-frame prediction unit 211, motion vector prediction decoding unit 212, motion compensation prediction unit 213, motion vector separation unit 214 A multiple reference image synthesis unit 215 and a synthesized image motion compensation prediction unit 216. The functions of the motion vector separation unit 214 and the synthesized image motion compensation prediction unit 216 are added to the first embodiment, and the operation of the multiple reference image synthesis unit 215 is different. Only the operation of functional blocks related to the added motion vector separation unit 214 and synthesized image motion compensation prediction unit 216 will be described.

  The prediction mode decoding unit 203 performs the same processing as in Embodiment 1, but according to the decoded prediction mode information, the intra-frame prediction unit 211, the motion compensation prediction unit 213, and the combined motion compensation prediction unit 216 The portion for outputting the information indicating the selection and the additional information corresponding to the prediction mode is different as the operation.

  The prediction image selection unit 204 selects the prediction image to be selected in accordance with the prediction mode information input from the prediction mode decoding unit 203, in addition to the intra-frame prediction unit 211 and the motion compensation prediction unit 213, and combines motion compensation prediction. The decoding target block output from any one including the unit 216 is input, selection processing is performed, and the result is output to the adder 207.

  The motion vector prediction decoding unit 212 calculates the motion vector value of the decoding target block by the same method as in Embodiment 1, and outputs the motion vector value to the motion compensation prediction unit 213 and the synthesized image motion compensation prediction unit 216. The motion vector is decoded by the number encoded according to the block unit of the prediction process indicated in the motion compensation prediction mode or the composite image motion compensation prediction mode.

  The motion compensated prediction unit 213 receives a motion compensated prediction image from the motion compensated prediction mode as additional information corresponding to the motion vector value input from the motion vector prediction decoding unit 212 and the prediction mode input from the prediction mode decoding unit 203. The generated motion compensated prediction image is output to the prediction image selection unit 204.

  The motion vector separation unit 214 converts a motion vector value input from the motion vector predictive decoding unit 212 into a predetermined pixel accuracy (hereinafter referred to as a reference motion vector value), and a motion vector value. Are divided into difference vector values (hereinafter referred to as correction vector values), and the reference motion vector values are output to the plurality of reference image synthesis units 215, and the correction vector values are output to the synthesized image motion compensation prediction unit 216. Output. The reference motion vector value and the correction vector value are decoded by the number encoded according to the block unit of the prediction process indicated in the composite image motion compensation prediction mode.

  In the multiple reference image synthesis unit 215, the reference motion vector value for one reference image indicated in the synthesized image motion compensated prediction mode output from the motion vector separation unit 214 and the multiple reference images stored in the decoded reference image memory 209. And a reference image synthesis process using a plurality of reference images is performed. The combined reference image signal is output to the combined image motion compensation prediction unit 216.

  The synthesized image motion compensation prediction unit 216 corrects one reference image indicated by the synthesized reference image signal input from the multiple reference image synthesis unit 215 and the synthesized image motion compensation prediction mode output from the motion vector separation unit 214. Using the vector value, a prediction block for the decoding target block is cut out from the synthesized reference image signal. The composite image motion compensation prediction unit 216 generates a composite motion compensation prediction image generated by combining the extracted prediction blocks with respect to all the blocks indicated in the composite image motion compensation prediction mode. Output to.

  The multiple reference image synthesis unit 215 and the synthesized image motion compensation prediction unit 216 are paired with the multiple reference image synthesis unit 107 and the synthesized image motion compensation prediction unit 108 in the moving image coding apparatus according to Embodiment 2 of the present invention. The detailed operation of these blocks and the motion vector separation unit 214 will be described later.

  Hereinafter, the overall mechanism of the prediction image generation method of the composite image motion compensation prediction that operates in the video encoding device and the video decoding device of Embodiment 2 will be described with reference to FIG. The operation will be described.

  FIG. 8 is a conceptual diagram showing the operation of the composite image motion compensation prediction process in Embodiment 2 of the present invention. The encoding apparatus first detects a motion vector between the encoding target frame and the first reference image using the encoding target frame and the reference image as a reference as a first reference image, and the first motion vector value MV1 is generated. In the configuration of FIG. 6, MV1 is obtained by the motion vector detection unit 104. Here, the accuracy of MV1 is assumed to be N pixels (example: one pixel). When the motion vector value detected by the motion vector detection unit 104 is finer than N pixel accuracy, MV1 is generated by rounding the detected motion vector value to N pixel accuracy.

  Next, the inter-reference image motion vector between the first reference block and the second reference image is detected with the reference block cut out from the first reference image by MV1 as the first reference block. Based on the detected motion, for the first reference block and its surroundings, a predicted image with M pixel (eg, 1/4 pixel) accuracy with M <N is generated by means such as filtering, and the second reference A predicted image with the same accuracy is generated for the second reference block cut out from the image using the inter-reference image motion vector and its surroundings, and a combined predicted image including the surroundings is generated using these.

  The motion vector detection with M pixel accuracy is performed on the prediction image including the surroundings generated last by means such as block matching with the encoding target block, and the second motion vector value MV2 detected as a result is detected. , Encoding and transmitting as a motion vector between the encoding target block and the first reference image, and subtracting the combined predicted image specified by MV2 from the encoding target block as a combined motion compensated prediction block, Encode and transmit the difference block.

  On the other hand, on the decoding device side, the first motion vector value MV1 is restored by rounding the first received motion vector value MV2 to N pixel accuracy. Next, the inter-reference image motion vector between the first reference block and the second reference image is detected with the reference block cut out from the first reference image by MV1 as the first reference block. Based on the detected motion, a predicted image with M pixel accuracy is generated for the first reference block and its surroundings by means such as filtering defined on the encoding side, and an inter-reference image motion vector is generated from the second reference image. A predicted image with the same accuracy is generated for the second reference block cut out using and the surrounding area, and a combined predicted image including the surrounding area is generated using these.

  The synthesized motion compensated prediction block that is the same as that generated on the encoding device side is obtained by cutting out the predicted image synthesized at the position specified by the second motion vector value MV2 from the predicted image including the periphery generated last. Can be generated.

  In this mechanism, the first reference block is used as a template having information close to the encoding target block, and image synthesis with motion compensation is performed with other reference images, so that MPEG-4 AVC or the like is performed. In the motion compensated prediction, the same effect as in the first embodiment can be obtained in which a prediction signal having characteristics close to the prediction (bidirectional prediction) using two reference images can be generated. At the same time, there is no need to determine a motion vector value by a synthesizing process with fine accuracy at a quarter accuracy level on the encoding device side, and a motion vector value based on a rough accuracy motion vector value at one pixel accuracy level. Correction with fine accuracy (M pixel accuracy) can be performed on the synthesized reference image, so that the phase that has moved slightly in the synthesis process with a small amount of processing or the noise component of the reference image in the synthesis process A motion vector value that takes into account the removed result is obtained, and a highly accurate predicted image block can be generated.

  In addition, in the decoding device, an image that can be generated by directly acquiring the pixel of the reference image of the one-pixel accuracy level can be used for motion vector detection between the reference images in the synthesis process. There is also an effect that the motion vector detection processing can be performed in parallel.

  Next, FIG. 9 shows the configuration of the multiple reference image synthesis unit 107 in the encoding apparatus that realizes the mechanism shown in FIG. 8, and the composite image that operates in the multiple reference image synthesis unit 107 and the synthesized image motion compensation prediction unit 108. A flowchart of the motion compensation prediction process is shown in FIG.

  As shown in FIG. 9, the multiple reference image synthesis unit 107 includes a standard reference image acquisition unit 400, a motion vector detection range setting unit 401, an inter-reference image motion vector detection unit 402, a synthesized reference image acquisition unit 403, and a reference image synthesis unit. 404 and a composite image memory 405.

  First, the motion vector detection unit 104 inputs the motion vector value MV1 between the first reference image and the encoding target block to the standard reference image acquisition unit 400 and the motion vector detection range setting unit 401. The reference reference image acquisition unit 400 acquires the reference block of the first reference image from the decoded reference image memory 117 using the input MV1. The reference block acquisition area is a reference with M pixel accuracy (M <N) of the target block ± N / 2 pixels or more based on the position of the first reference image moved by the value of MV1 with respect to the encoding target block. The area necessary for creating the image is taken. For example, when N is 1 pixel, M is 1/4 pixel, and a 6-tap filter used in MPEG-4 AVC is used as an expansion filter necessary for generating a 1/4 pixel precision image, encoding is performed. In addition to the target block size, a reference image of an area of ± 3 pixels is acquired as a first reference block. The reference reference image acquisition unit 400 outputs the acquired first reference block to the inter-reference image motion vector detection unit 402 and the reference image synthesis unit 404.

  Subsequently, the motion vector detection range setting unit 401 sets a range for detecting a motion vector between the second reference images for the first reference block. For the detection range of motion vectors between reference images, a method of implicitly setting the same detection range in the encoding device and the decoding device can be applied, but the detection range setting for each frame or reference image used is encoded. It is also possible to use a method of transmitting as information. The detection range setting algorithm in the second embodiment will be described later with reference to FIG. The motion vector detection range setting unit 401 outputs information on the set motion vector detection range to the inter-reference image motion vector detection unit 402.

  The inter-reference image motion vector detection unit 402 performs the second reference image in the motion vector detection range specified by the motion vector detection range setting unit 401 with respect to the first reference block input from the standard reference image acquisition unit 400. Are obtained from the decoded reference image memory 117 via the synthesized reference image acquisition unit 403, an error value such as block matching is calculated, and a motion vector having a small value is calculated as a motion vector between reference images. Similarly, with respect to the detection accuracy of inter-reference image motion vectors, it is possible to apply a technique for implicitly detecting motion vectors with the same detection accuracy in the encoding device and the decoding device. However, the motion vector is used for each frame or each reference image used. It is also possible to use a method of transmitting the detection accuracy of the data as encoded information. The inter-reference image motion vector detection unit 402 outputs the calculated inter-reference image motion vector to the reference image synthesis unit 404.

  In the reference image synthesis unit 404, the first reference block is input from the standard reference image acquisition unit 400, and the inter-reference image motion vector is input from the inter-reference image motion vector detection unit 402, which is indicated by the inter-reference image motion vector. The second reference block is obtained by acquiring the reference block of the second reference image from the decoded reference image memory 117 via the synthesized reference image acquisition unit 403. The reference image composition unit 404 performs composition processing between the first reference block and the second reference block. The synthesis process in Embodiment 2 employs a method of generating a reference block that is synthesized by taking the average of each pixel of the first reference block and the second reference block, for example. The reference image synthesis unit 404 outputs the synthesized reference block to the synthesized image motion compensation prediction unit 108 via the synthesized image memory 405.

  Next, the operation of the composite image motion compensation prediction process using these configurations will be described using the flowchart of FIG. FIG. 10 shows a flow of the operation of the composite image motion compensation prediction in the encoding process of one frame. As for the operation of other processing units, it is possible to use a conventional moving image encoding process such as MPEG-4 AVC.

  At the start of processing for one frame, first, a compositing target reference image for each reference image is determined (S500). It is possible to select and use a plurality of reference images for motion compensation prediction in the second embodiment. FIG. 11 shows an example of the processing order of encoding processing and reference image management in Embodiment 2, and will be described.

  A process called I-slice that performs coding without using motion compensation prediction is performed for the first frame or completely. The decoded image encoded by the I slice is stored in the decoded reference image memory 117, and becomes a reference image of a frame to be subsequently encoded.

  The P slice is a frame that enables compression by temporal correlation using motion compensated prediction using a decoded image of a previous frame in time as a reference image. In the example of the encoding processing order of the second embodiment in FIG. 11, all the decoded images of P slices are used as reference images. The added reference images are stored in the decoded reference image memory 117 and stored up to a predetermined number of reference images.

  A B slice is a frame in which motion compensation prediction can be performed by adding two reference images, and motion compensation prediction with high prediction accuracy is possible using temporally preceding and following reference images. When using an image, it is necessary to encode two motion vectors. In the example of the encoding processing order of the second embodiment in FIG. 11, the decoded image of the B slice is not used as a reference image.

  As in the example shown in FIG. 11, when four reference images can be stored in the encoding process in which the B slice is set for each frame, a new reference image is encoded after the I frame and P frame are encoded. Is stored, and when four or more images are stored, one reference image is discarded and a new decoded image is used as a reference image. In the example of reference image management of the second embodiment in FIG. 11, the oldest frame in time is selected as the reference image to be discarded.

  Thus, since a plurality of reference images can be selectively used for the encoding target frame, first, a process of determining a reference image to be synthesized for each reference image is performed. By defining an implicit rule and making the same determination in the encoding device / decoding device, correct composition processing can be performed.

  For example, when the encoding target frame is a B slice, a reference image closest to the encoding target frame that has a temporal relationship across the encoding target frame with respect to the first reference image that is a basic reference image is selected. The second reference image is a reference image used for composition. When the encoding target is a P slice, when the first reference image is the reference image closest to the encoding target frame, the second closest reference image is set as the second reference image, and otherwise. The second reference image is the reference image closest to the encoding target frame.

  When the reference images to be synthesized with respect to all the reference images are determined, the motion vector detection accuracy between the reference images is subsequently determined (S501). Here, the motion vector detection accuracy is set to ¼ pixel, which is the detection accuracy transmitted in the final combined motion compensation prediction. For example, the motion vector detection accuracy can be obtained by obtaining a motion with finer accuracy such as １ ／ pixel accuracy. It is also possible to perform the synthesis process with fine accuracy without increasing the accuracy of the motion vector to be transmitted.

  Subsequently, a motion vector detection range between reference images is determined (S502). Regarding the detection range, it is also possible to take the entire area of the second reference image as the motion vector detection range for all the first reference blocks, and by performing detection processing with the same definition as the decoding device, Although the encoding apparatus in the second embodiment functions, in order to reduce the amount of calculation in detecting a motion vector between reference images, a detection range as shown in FIG. 12 is set.

FIG. 12 is an example of a motion vector detection range between reference images in the second embodiment. If the input time of the encoding target image is Poc_Cur, the input time of the first reference image is Poc_Ref1, and the input time of the second reference image is Poc_Ref2, the motion vector MV1 from the first reference image for the encoding target block is When the search range of the second reference image is based on the position of the encoding target block, the search center position is α = MV1 × (Poc_Cur-Poc_Ref2) / (Poc_Cur-Poc_Ref1)
As shown in the above, the motion vector prediction value between the encoding target block and the second reference image when it is assumed that the motion is temporally continuous is set.

  However, since there are many situations that are not temporally continuous changes, such as camera movements and object movements, a second reference image suitable for synthesis processing can be obtained by searching for a motion vector for a specific region with the search position as the center. To get the reference block. In the example shown in FIG. 12, an area of ± 4 pixels is designated as the specific area.

  Specifically, in S502, the calculation of the search center position for obtaining the synthesized reference image for each encoding target block is performed only by performing the process of determining the definition of ± 4 pixels. Calculated every time.

  Subsequently, information that causes the decoding device to perform the same processing by transmitting the encoded bit stream within the processing definition in units of frames is transmitted in a slice header that transmits information in units of frames. FIG. 13 shows an example of additional information to the slice header in the second embodiment.

  Since the slice header of FIG. 13 is based on the slice header in MPEG-4 AVC, the described portion is only related to the added information. Since the combined motion compensation prediction is not used in the I slice which is a prediction method between frames, information added in the case other than the I slice is transmitted.

  First, 1 bit of refinement_mc_enable, which is information for controlling whether to perform combined motion compensation prediction in units of slices, is transmitted. Furthermore, when refinement_mc_enable is 1 (combined motion compensation prediction is performed), the following three pieces of information are transmitted.

  One is information indicating whether to switch adaptively with the conventional motion compensation prediction, or to replace the conventional motion compensation prediction with the combined motion compensation prediction, and transmits 1 bit as refinement_mc_adaptive.

Second, 2-bit data is transmitted as refinement_mc_matching_range_full as information indicating a motion vector detection range between reference images. As an example, 2-bit data indicates that the following detection range is defined.
00 ± 1 pixel
01 ± 2 pixels
10 ± 4 pixels
11 ± 8 pixels

Third, 2-bit data is transmitted as refinement_mc_matching_subpel as information indicating the accuracy of motion vector detection between reference images. As an example, 2-bit data indicates that the following detection accuracy is defined.
00 1 pixel accuracy (no decimal precision detection is performed)
01 1/2 pixel accuracy
10 1/4 pixel accuracy
11 1/8 pixel accuracy

  In this way, after the setting is determined in units of frames, the combined motion compensation prediction process is performed on the macroblock that is the encoding target block in the encoding target frame. For each macroblock (S504), for all reference images (S505), first motion vector detection is performed using the first selected reference image as the first reference image (S506).

  The above detection processing can be performed by the motion vector detection unit 104 used in the conventional motion compensation prediction. However, if the conventional motion compensation prediction is not used, whether the same motion vector detection processing is added to the combined motion compensation prediction. Alternatively, the predicted motion vector value output from the motion vector predicting unit 106 can always be used as the first motion vector value as the first motion vector value.

  When a predicted motion vector value is used as the first motion vector value, the difference motion vector value becomes a shift amount from the center position as a search result for a minute range after the reference image is synthesized, and the difference to be transmitted There is an advantage that motion vector information is reduced.

The first motion vector is assumed to have one pixel accuracy, and if the motion vector value input from the motion vector detection unit 104 or the motion vector prediction unit 106 is a motion vector with accuracy less than one pixel, it is rounded to one pixel accuracy. Apply actions. For example, when the input motion vector value MV1org has a 1/4 pixel accuracy, the first motion vector value MV1 is obtained by calculation as follows.
MV1 = (MV1org + 2) >> 2
Subsequently, a first reference block is acquired from the first motion vector (S507). As described in the description of FIG. 9, the first reference block has a target block ± 1/2 pixel or more based on the position of the first reference image moved by the value of MV1 with respect to the encoding target block. In addition to the encoding target block size, a reference image of an area of ± 3 pixels is acquired as an area necessary for creating a reference image with 1/4 pixel accuracy.

  Next, the detection range of the second reference image is set from the first motion vector (S508). Which reference image is used for the second reference image is determined by the definition determined in S500. Regarding the detection range, the detection range shown in FIG. 12 described in S502 is set. For the set detection range, motion vector detection between reference images is performed between the first reference block and the second reference image (S509).

  Subsequently, a second reference block is acquired using the detected inter-reference image motion vector (S510). The second reference block has the same coding block size ± 3 pixels as the first reference block on the basis of the position of the second reference image moved by the motion vector value between the reference images with respect to the first reference block. A reference image of the area is acquired.

  Next, the first reference block and the second reference block are synthesized to generate a synthesized reference image block (S511). As an algorithm to be synthesized, a synthesized reference image block is generated by calculating an average value for each pixel of the first reference block and the second reference block. Note that the synthesized reference image block can also support weighted prediction used in MPEG-4 AVC. The synthesized reference image block can be weighted, and the first reference block and the first reference block It is also possible to take the weighted addition average by making the addition ratio of the two reference blocks inversely proportional to the distance from the image to be encoded. When switching between these, information for specifying the addition method is displayed in units of frames or macros. Transmit in blocks.

  Subsequently, motion vector detection in a very small range is performed between the synthesized reference image block and the encoding target block to generate a second motion vector value (S512). Specifically, when a first motion vector value is detected with a 1-pixel accuracy and a 1 / 4-pixel accuracy motion vector, the first motion vector is indicated by the first motion vector MV1 with respect to the first reference block. A block of the same size as the encoding target block is cut out from the same position of the synthesized reference image block while moving horizontally and vertically in a unit of 1/4 pixel within the range of ± 1/2 pixel based on the position. Perform block matching with the block.

As a result of block matching, the one with the smallest error evaluation value with respect to the encoding target block is calculated as the second motion vector value MV2. If the movement amount indicating the moved range is MVdelta,
MV2 = (MV1 << 2) + MVdelta
Is output as

  Here, since MVdelta is calculated with ¼ pixel accuracy in both horizontal and vertical directions with −2 ≦ MVdelta <2, by performing processing of MV1 = (MV2 + 2) >> 2 on the decoding side for MV2. The first motion vector can be restored.

  Subsequently, based on the position indicated by the obtained second motion vector value MV2, a synthesized motion compensated prediction block is cut out from the synthesized reference image block, and an error evaluation value is calculated. The error evaluation value is not only the sum of errors due to block matching etc., but also the prediction difference block that is subtracted from the encoding target block using the amount of code required for transmission of motion vectors etc. and the obtained synthesized motion compensated prediction block It is also possible to calculate the distortion amount between the code amount and the decoded input image as a calculated value, taking into account the code amount required when encoding.

  The processing from S506 to S513 is performed on all reference images, and when the reference image is not the last reference image (S514: NO), the next reference image is selected as the first reference image (S515). ), The process returns to S506. If the reference image is the last reference image (S514: YES), the one with the smallest error evaluation value is selected from the second motion vector values obtained for all the reference images, and the selected second image is selected. Information indicating the first reference image used when calculating the second motion vector value and the second motion vector value is output to the prediction mode determination unit 109 together with the error evaluation value (S516).

  The prediction mode determination unit 109 compares the error evaluation value with other prediction modes and determines an optimal prediction mode (S517).

  The prediction difference block, which is the difference between the predicted image in the determined prediction mode and the encoding target block, and the additional information related to the prediction mode are encoded (S518), thereby completing the encoding process for one macroblock. To do.

  The motion vector value is stored in the motion vector prediction unit 106 for use in the motion vector prediction of the subsequent macroblock similarly when the conventional motion compensated prediction is selected and when the combined motion compensated prediction is selected. Is done. The second motion vector value transmitted in the synthesized motion compensated prediction is managed separately because the motion vector value of the first reference image without the synthesis process has the same correlation as in the conventional motion compensated prediction. By handling the same in the same manner, it is possible to increase the motion vector values that can be referred to in the peripheral blocks, and to maintain the same motion vector prediction accuracy as that in the prior art.

  In the flowchart of FIG. 10, the description is made assuming that the size of the encoding target block in the macroblock is one, but as in MPEG-4 AVC, 16 × 16, 16 × 8, 8 × 16, 8 × 8 , 8 × 4, 4 × 8, 4 × 4, etc., can be combined motion compensated prediction, in which case, an error evaluation value of the combined motion compensated prediction for each block size is calculated, The block size with the smallest error evaluation value is selected, and the decoding device can recognize the selection result by transmitting the prediction mode.

  When refinement_mc_adaptive = 1 is transmitted as the slice header information shown in FIG. 13, a process of adaptively switching between normal motion compensation prediction and synthesized motion compensation prediction is performed.

  FIG. 14 shows an example of additional information to the motion compensation prediction mode in the second embodiment. The switching information is a reference image unit that transmits a motion vector according to an applied mode in cases other than the direct mode (Direct) using only the predicted motion vector value (Intra) and the predicted motion vector value without using a motion vector, It is transmitted as 1-bit ON / OFF information. In FIG. 14, refmc_on_l0 [mbPartIdx] and refmc_on_l1 [mbPartIdx] are the corresponding information.

  In bi-directional prediction in B slice (additional prediction is performed using two reference images), it is also possible to select whether to use a synthesized reference image as a predicted image for each reference image, In combination with the selection of the reference image, the combined motion compensated prediction can be performed from a maximum of four reference images with two motion vectors, and the quality of the predicted image can be further improved.

  When the encoding of the macroblock is completed, if it is not the last macroblock (S519: NO), the next macroblock is designated (S520), and the process proceeds to S504. If it is the last macroblock (S519: YES), the encoding process for one frame is terminated.

  Next, FIG. 15 shows the configuration of the multiple reference image synthesis unit 215 in the decoding apparatus that implements the mechanism of FIG. 8, and the motion vector separation unit 214, the multiple reference image synthesis unit 215, and the synthesized image motion compensation prediction unit 216 operate. The flowchart of the composite image motion compensation prediction process is shown in FIG. 16, and the detailed operation will be described.

  As shown in FIG. 15, the multiple reference image synthesis unit 215 includes a standard reference image acquisition unit 1000, a motion vector detection range setting unit 1001, an inter-reference image motion vector detection unit 1002, a synthesized reference image acquisition unit 1003, and a reference image synthesis unit. 1004, and a synthesized image memory 1005, and the operations are the reference reference image acquisition unit 400, motion vector detection range setting unit 401, inter-reference image motion vector detection unit 402, and synthesized reference image acquisition shown in FIG. Operations similar to those of the unit 403, the reference image synthesis unit 404, and the synthesized image memory 405 are performed.

  First, the motion vector separation unit 214 inputs the motion vector value MV1 generated by the motion vector separation unit 214 from the decoded motion vector value MV2 to the standard reference image acquisition unit 1000 and the motion vector detection range setting unit 1001. .

  Specifically, MV1 is generated by an operation of MV1 = (MV2 + 2) >> 2, and a motion vector value between the first reference image and the encoding target block in the encoding device can be acquired. The reference reference image acquisition unit 1000 acquires the reference block of the first reference image from the decoded reference image memory 209 using the input MV1. The reference reference image acquisition unit 1000 outputs the acquired first reference block to the inter-reference image motion vector detection unit 1002 and the reference image synthesis unit 1004.

  Subsequently, the motion vector detection range setting unit 1001 sets a range for detecting a motion vector between the second reference images for the first reference block. The motion vector detection range setting unit 1001 outputs information on the set motion vector detection range to the inter-reference image motion vector detection unit 1002.

  The inter-reference image motion vector detection unit 1002 performs the second reference image in the motion vector detection range specified by the motion vector detection range setting unit 1001 with respect to the first reference block input from the standard reference image acquisition unit 1000. Are obtained from the decoded reference image memory 209 via the synthesized reference image acquisition unit 1003, an error value such as block matching is calculated, and a motion vector having a small value is calculated as a motion vector between reference images. The inter-reference image motion vector detection unit 1002 outputs the calculated inter-reference image motion vector to the reference image synthesis unit 1004.

  In the reference image synthesis unit 1004, the first reference block is input from the standard reference image acquisition unit 1000, and the inter-reference image motion vector detection unit 1002 is input, which is indicated by the inter-reference image motion vector. By acquiring the reference block of the second reference image from the decoded reference image memory 209 via the synthesized reference image acquisition unit 1003, the second reference block is obtained. The reference image synthesis unit 1004 performs a synthesis process between the first reference block and the second reference block. The reference image synthesis unit 1004 outputs the synthesized reference block to the synthesized image motion compensation prediction unit 216 via the synthesized image memory 1005.

  Next, the operation of the composite image motion compensation prediction process on the decoding device side using these configurations will be described with reference to the flowchart of FIG. Also in FIG. 16, the flow of the operation | movement of the synthetic | combination image motion compensation prediction in the decoding process of 1 frame similarly to FIG. 10 is shown. As for the operation of other processing units, it is possible to use a conventional video decoding process such as MPEG-4 AVC.

  At the start of the decoding process for one frame, the slice header is first decoded to obtain information related to the reference image (S1100). Information indicating the coding order as shown in FIG. 11 and information specifying the reference image are transmitted in the slice header, and information related to the combined motion compensated prediction as shown in FIG. 13 is also decoded.

  Next, a process of determining a reference image to be synthesized for each reference image is performed (S1101). In the second embodiment, the decoding device makes a determination similar to the processing shown in the operation of the encoding device.

  When the reference images to be synthesized are determined for all the reference images, the motion vector detection accuracy between the reference images is set using the decoded refinement_mc_matching_subpel (S1102).

  Similarly, the motion vector detection range between reference images is set using refinement_mc_matching_range_full decoded from the slice header (S1103).

  After the setting in units of frames is confirmed, when synthesized motion compensated prediction is used for a macroblock that is a decoding target block in the decoding target frame, a process for generating a synthesized motion compensated prediction block is performed. The

  For each macroblock (S1104), when the prediction mode is not the combined motion compensated prediction mode for the first time (S1105: NO), prediction processing is performed in another prediction mode, and decoding processing is performed using the generated prediction image. (S1106).

  When the prediction mode is the combined motion compensation prediction mode (S1105: YES), information indicating the first reference image is acquired (S1107). The information indicating the reference image is encoded together with the prediction mode similarly to MPEG-4 AVC, and can be acquired together with the prediction mode information of the decoded macroblock.

  Subsequently, the motion vector value MV2 decoded by the motion vector predictive decoding unit 212 is acquired (S1108). The MV2 is subjected to separation processing in the motion vector separation unit 214 to generate MV1 (S1109). Specifically, the calculation of MV1 = (MV2 + 2) >> 2 is performed as described above.

  Subsequently, the first reference block is acquired using MV1 (S1110). As described in the description of FIG. 9, the first reference block is a reference image with a 1/4 pixel accuracy of ± 1/2 pixel or more of the target block on the basis of the position of the first reference image moved by the value of MV1. In addition to the encoding target block size, a reference image of an area of ± 3 pixels is acquired as an area necessary for creating the image.

  Next, the detection range of the second reference image is set from the first motion vector (S1111). Which reference image is used for the second reference image is selected in the same way as the encoding device according to the definition determined in S1101. As for the detection range, the detection range set in S1103 is used. For the set detection range, motion vector detection between reference images is performed between the first reference block and the second reference image (S1112).

  Subsequently, a second reference block is acquired using the detected inter-reference image motion vector (S1113). The second reference block has the same coding block size ± 3 pixels as the first reference block on the basis of the position of the second reference image moved by the motion vector value between the reference images with respect to the first reference block. A reference image of the area is acquired.

  Next, the first reference block and the second reference block are synthesized to generate a synthesized reference image block (S1114).

  Subsequently, an image block of an area corresponding to the position designated by MV2 is extracted from the synthesized reference image block by moving MV2-MV1 with respect to the position designated by MV1 (S1115). The moving 1/4 pixel component is generated by the calculation of MV2− (MV1 << 2) because MV1 has 1 pixel accuracy and MV2 has 1/4 pixel accuracy. The extracted image block is output to the predicted image selection unit 204 as a synthesized motion compensated prediction block (S1116).

  Subsequently, the decoding process of difference information is performed using the synthesized motion compensated prediction block (S1117), whereby the decoding process of one macroblock is completed. If the decoded macroblock is not the last macroblock of one frame (S1118: NO), the macroblock to be decoded next is designated (S1119), and the process returns to S1105.

  When the last macroblock of one frame is decoded (S1118: YES), the processing of one frame is completed.

  In the composite image motion compensation prediction process in the decoding apparatus according to the second embodiment, since the final MV2 value is known in advance, for the first reference block, reference is made in units of one pixel of the encoding target block size. It is also possible to acquire only image blocks, perform motion vector detection between reference images, generate a reference image in 1/4 pixel units that is required when generating a combined reference image, and perform combining processing. The same synthesized reference image as that of the encoding device can be generated while reducing the increase in the amount of calculation due to the filtering process.

  With respect to the reference image obtained by performing motion compensation prediction by obtaining a motion vector between the encoding target block and the first reference image by the moving image encoding device and the moving image decoding device according to the second embodiment of the present invention, By synthesizing the reference image and performing motion vector detection (correction) in a very small range on the synthesized predicted image, the quality of the predicted image is improved, and phase changes such as the improved edge portion are added. A motion-compensated prediction image with little prediction residual with corrected motion could be generated.

  Furthermore, assuming that the accuracy of the motion vector obtained for the first reference image is N pixel accuracy, the range of motion vector detection (correction) performed on the synthesized predicted image is ± N / 2 pixels. The motion compensation predicted image is obtained from the first reference image on the decoding device side by one motion vector value, and the synthesized predicted image is transmitted by performing the correction with fine accuracy and transmitting the motion vector value of the correction result. It is possible to obtain a motion compensated predicted image in which the phase change is corrected, and to encode / decode a motion compensated predicted image with a small prediction residual without increasing additional information.

  Further, when the motion vector value for the first predicted image is predicted from the motion vector values of the decoded peripheral blocks, it is only necessary to receive only the correction value for the predicted image synthesized as the motion vector value to be decoded. The amount of vector information was reduced.

  In Embodiment 2 of the present invention, a motion vector value between a motion compensated predicted image predicted using the first reference image and another reference image is obtained, and the motion obtained from the other reference image is obtained. By taking the addition average with the compensated prediction image, it is possible to generate a prediction image corresponding to the removal of the encoding degradation component and the slight luminance change of the decoding target, and the encoding efficiency can be improved.

  As for the second motion vector, a synthesized reference image is generated using one determined result in the encoding device and synthesized motion compensation prediction is performed. However, a plurality of first motion vectors are prepared. Even when the combined motion compensated prediction is performed using the same method based on the respective motion vectors and the optimal second motion vector is encoded, the decoding apparatus increases the amount of computation by the process described in the second embodiment. Decoding is possible, and the encoding apparatus can perform optimum combined motion compensation prediction with M pixel accuracy by determination in units of N pixels, and is suitable for the first embodiment while suppressing an increase in encoding processing. Combined motion compensation prediction is possible.

(Embodiment 3)
Next, a video encoding device and a video decoding device according to Embodiment 3 will be described. In the third embodiment, the configuration of the moving image encoding device and the moving image decoding device is the same as that of the second embodiment, and only the reference image combining process in the multiple reference image combining unit is different. Specifically, only the arithmetic processing performed in the reference image synthesis units 404 and 1004 and the flowcharts S511 and S1114 in the description of the second embodiment is different.

  FIG. 17 is a conceptual diagram showing the operation of reference image synthesis processing in Embodiment 3, and the calculation processing will be described. In the second embodiment, averaging is performed by performing a uniform addition averaging process on all pixel values in the block in the synthesis process. However, in the third embodiment, the first process is performed in the synthesis process. An error value is calculated for each pixel between the reference block and the second reference block, and the weighting of the second reference block and the first reference block for each pixel is changed according to the absolute value of the error. To calculate a weighted average value.

Specifically, when the error is small, the weighting is performed equally, and the pixel of the second reference block is not added above the threshold value. Assuming that the pixel value of the first reference block is P1, and the pixel value of the second reference block is P2, the addition ratio value is obtained from the pixel error absolute value | P1-P2 | A certain α is calculated. Using the calculated α, the combined pixel value PM for each pixel is
PM = P1 × (1−α) + P2 × α
Calculated by

  These synthesizing processes can be performed implicitly by the encoding device / decoding device, but information indicating whether simple averaging is performed on the slice header or the like, or adaptive addition for each pixel is performed. It is also possible to send and select.

  When simple averaging is performed in the second embodiment, it is possible to generate a predicted image having characteristics corresponding to the average value prediction from two reference images with one motion vector. 3, the first reference block is used as a reference to remove the distortion caused by the encoding deterioration or the like while the first reference block is used as a reference, and the signal with respect to the portion where the change in the edge component or the like is large By storing the characteristics, a predicted image in which the quality of the first reference image block is improved is generated in a process common to encoding and decoding. As a result, in the video encoding device / video decoding device according to the third embodiment, in addition to the effects of the first embodiment, it is possible to remove the coding degradation component while maintaining the signal characteristics of the characteristic part such as the edge component. It was possible to generate a high-quality prediction image that was performed, and improved the encoding efficiency.

  Even when the reference image has only one image in the coding structure as shown in FIG. 11, the first reference image is designated as the second reference image, and the reference image within the first reference image is designated. By performing block matching and adaptively adding the texture components, it is possible to remove the deteriorated components, and with the configuration of the present invention, a good effect can be exhibited.

(Embodiment 4)
Next, a video encoding device and a video decoding device according to Embodiment 4 will be described. In the fourth embodiment, the multiple reference image synthesis unit uses a plurality of second reference images, and the block unit of motion vector detection from the second reference image for the first reference block is the encoding target block. It is characterized by being composed of smaller block units.

  FIG. 18 is a conceptual diagram showing the operation of the composite image motion compensation prediction process in Embodiment 4 of the present invention. Since the relationship between the encoding device side and the decoding device side is the same as that in the second embodiment shown in FIG. 8, only the conceptual diagram showing the operation on the encoding device side is shown.

  On the encoding device side, a motion vector is detected using the encoding target frame and a reference image as a reference as a first reference image, and processing for generating MV1 is performed in the same manner as in the first embodiment.

  Next, the reference block cut out from the first reference image by MV1 is divided into smaller block units than the encoding target block, and the inter-reference image motion vector between the second reference image is detected in smaller block units. As an example, when the encoding target block is 16 × 16 pixels having the same size as the macroblock, the unit for obtaining the inter-reference image motion vector is 8 × 8 pixels. Then, using the inter-reference image motion vectors between a plurality of (four in the example, two) detected second reference images, a small target to be synthesized with respect to the target first reference block region. Generate multiple reference blocks in block units.

  For the image around the target block when generating the second reference block, as shown in FIG. 18, only the area not included in the other divided blocks is acquired in units of small blocks. However, it is also possible to acquire a peripheral area of ± 3 pixels in all the small blocks and use it for the synthesis processing in the same manner as in the first embodiment, so that a synthesized image that smoothly connects the boundaries of the small blocks can be obtained. It is possible to generate. A second reference block corresponding to the first reference block is generated using the plurality of reference blocks of the small block.

  Subsequently, a motion vector between reference images is similarly detected for a third reference image different from the second reference image. Similarly, for the third reference image, the inter-reference image motion vector between the first reference block is detected in units of small blocks. A plurality of reference blocks are generated based on the detected inter-reference image motion vector, and a third reference block corresponding to the first reference block is generated using the plurality of reference blocks.

  In the combining process, the second reference block is combined for each pixel or each small block with respect to the first reference block, or the third reference block is combined, or both the second and third reference blocks are combined. Is determined using the pixel values of the first reference block and the pixel values of the second and third reference blocks.

  For the predicted image including the surroundings synthesized in this way, motion vector detection with M pixel accuracy is performed between the encoding target block and means such as block matching in the same manner as in the first embodiment. The detected second motion vector value MV2 is encoded and transmitted as a motion vector between the current block to be encoded and the first reference image, and a synthesized predicted image designated by MV2 is synthesized and motion compensated prediction. As a block, the difference block is encoded and transmitted by subtracting from the encoding target block.

  The configuration of the moving image encoding device and the moving image decoding device in the fourth embodiment is the same as that in the second embodiment, but the configuration and processing in the multiple reference image synthesis unit are different. FIG. 19 is a block diagram showing the configuration of the multi-reference image synthesis unit in the fourth embodiment, and FIG. 20 is a flowchart for explaining the operation. The fourth embodiment will be described. The configuration and operation of the multi-reference image synthesizer in the encoding device and the decoding device are the same except that the operation of the processing block connected to the multi-reference image synthesizer is different. .

  As illustrated in FIG. 19, the multiple reference image synthesis unit according to Embodiment 4 includes a standard reference image acquisition unit 1400, a motion vector detection range setting unit 1401, a second inter-reference image motion vector detection unit 1402, and a second reference image acquisition. Unit 1403, reference image synthesis unit 1404, synthesized image memory 1405, third reference image acquisition unit 1406, third inter-reference image motion vector detection unit 1407, and synthesis determination unit 1408. The operations of the third reference image acquisition unit 1406, the third inter-reference image motion vector detection unit 1407, and the synthesis determination unit 1408 are new effects in the fourth embodiment compared to the multiple reference image synthesis unit of the first embodiment. However, the third reference image acquisition unit 1406 and the third inter-reference image motion vector detection unit 1407 are integrated with the second inter-reference image motion vector detection unit 1402 and the second reference image acquisition unit 1403. As in the configuration in the second embodiment, it is also possible to operate with the configuration of the inter-reference image motion vector detection unit 402 and the synthesized reference image acquisition unit 403. In FIG. 19, in order to explain the operation, they are described as separate blocks.

  First, the motion vector detection unit 104 inputs the motion vector value MV1 between the first reference image and the encoding target block to the standard reference image acquisition unit 1400 and the motion vector detection range setting unit 1401. The reference reference image acquisition unit 1400 acquires the reference block of the first reference image from the decoded reference image memory 117 using the input MV1. The reference block acquisition area is a reference with M pixel accuracy (M <N) of the target block ± N / 2 pixels or more based on the position of the first reference image moved by the value of MV1 with respect to the encoding target block. The area necessary for creating the image is taken. The reference reference image acquisition unit 1400 outputs the acquired first reference block to the second inter-reference image motion vector detection unit 1402, the synthesis determination unit 1408, and the third inter-reference image motion vector detection unit 1407.

  Subsequently, in the motion vector detection range setting unit 1401, a range for detecting a motion vector between the second reference images for the first reference block and a range for detecting a motion vector between the third reference images are set. Set. As for the detection range setting algorithm, the same processing as the detection range setting in the first embodiment is performed for the motion vector detection between reference images with the second reference image and the motion vector between reference images with the second reference image. The detection is performed individually and the range is determined. The motion vector detection range setting unit 1401 outputs information on the set motion vector detection range to the second reference inter-image motion vector detection unit 1402 and the third reference inter-image motion vector detection unit 1407.

  The second inter-reference image motion vector detection unit 1402 performs the second reference in the motion vector detection range specified by the motion vector detection range setting unit 1401 with respect to the first reference block input from the standard reference image acquisition unit 1400. The reference block of the reference image is acquired from the decoded reference image memory 117 via the second reference image acquisition unit 1403, and the block is obtained for each 8 × 8 block size that is ¼ the size of the encoding target block. An error value such as matching is calculated, and a motion vector having a small value is calculated as a second reference inter-image motion vector. The second inter-reference image motion vector detection unit 1402 outputs the four calculated inter-second reference image motion vectors to the synthesis determination unit 1408.

  Similarly, the third inter-reference image motion vector detection unit 1407 performs the motion vector detection range specified by the motion vector detection range setting unit 1401 on the first reference block input from the standard reference image acquisition unit 1400. The reference block of the third reference image is acquired from the decoded reference image memory 117 via the third reference image acquisition unit 1406, and for each 8 × 8 block size that is ¼ the size of the encoding target block. Then, an error value such as block matching is calculated, and a motion vector having a small value is calculated as a motion vector between the third reference images. The third inter-reference image motion vector detection unit 1407 outputs the four calculated third inter-reference image motion vectors to the synthesis determination unit 1408.

  In the synthesis determination unit 1408, the first reference block input from the standard reference image acquisition unit 1400 is indicated by the four second reference image motion vectors input from the second reference image motion vector detection unit 1402. A second reference block corresponding to the first reference block is generated from the decoded reference image memory 117 via the second reference image acquisition unit 1403 using the reference blocks of the plurality of second reference images.

  Similarly, the combination determination unit 1408 uses the reference blocks of the plurality of third reference images indicated by the four third reference image motion vectors input from the third reference image motion vector detection unit 1407 to A third reference block corresponding to the first reference block is generated from the decoded reference image memory 117 via the reference image acquisition unit 1406.

  The synthesis determining unit 1408 calculates an error value between the second reference block, the third reference block, and the first reference block that are generated subsequently, and uses the error value relationship to determine the first reference block. On the other hand, the selection of the second reference block and the third reference block to be combined and the addition ratio are determined. The confirmation algorithm will be described later.

  The combination determination unit 1408 generates a combined reference block by combining the second reference block and the third reference block with the first reference block using the determined addition ratio, and combines the reference The block is output to the composite image motion compensation prediction unit 108 via the composite image memory 1405.

  Next, FIG. 20 shows a flowchart for explaining the operation of the determination process in the combination determination unit 1408, and the detailed operation will be described. First, the first reference block is input in units of encoding target blocks (S1500). The reference image block has a filter coefficient in order to perform a movement amount movement in a unit of 1/4 pixel within a range of ± 1/2 pixel with respect to the size of the encoding target block in the combined motion compensation prediction unit. In addition, an area of ± 3 pixels is acquired (when the encoding target block is 16 × 16 pixels, an area of 22 × 22 pixels is acquired).

  Subsequently, the four second reference image motion vectors calculated by the second reference image motion vector detection unit 1402 are input (S1501). Using each input motion vector, a small reference image block in units of 8 × 8 pixels is acquired from the second reference image (S1502). The acquisition area of the small reference image block is 14 × 14 pixels.

  For the adjacent portion of the small reference image block, it is determined whether or not the pixels of the overlapping portion of the adjacent reference image block are reflected (S1503). Specifically, when the difference value between adjacent acquired motion vectors is within ± 1 pixel, the adjacent blocks are overlapped and connected smoothly. If it is larger than ± 1 pixel, it is determined that it is a reference block obtained for a different object, the overlapping portion is not reflected, and the pixel of the corresponding small reference block is set as it is. According to the above determination, the small reference image blocks are superimposed to generate a second reference block composed of 22 × 22 pixels (S1504).

  Subsequently, four third reference image motion vectors calculated by the third reference image motion vector detection unit 1407 are input (S1505). As in the case of the second reference image, a small reference image block is acquired from the third reference image using each motion vector (S1506), and the processing of the adjacent boundary portion of the small reference image block is confirmed (S1507). The third reference block is generated by superimposing the small reference image blocks (S1508).

  Next, combining processing is performed on a pixel basis using the first reference block, the second reference block, and the third reference block. For each pixel in the reference block (S1509), an error absolute value | P1-P2 | between the pixel value P1 of the first reference block and the pixel value P2 of the second reference block is calculated (S1510). Similarly, an absolute error value | P1-P3 | between P1 and the pixel value P3 of the third reference block is calculated (S1511), and an absolute error value | P2-P3 | of P2 and P3 is calculated (S1512).

  In the determination of the synthesis process in the third embodiment, the addition ratio of P1, P2, and P3 is determined using the three values | P1-P2 |, | P1-P3 |, and | P3-P2 | (S1513). Is done.

  First, when | P1-P2 | and | P1-P3 | are both smaller than a threshold value β (example: 8), P1, P2, and P3 are added and averaged with the same weight. That is, the ratio of P1, P2, and P3 is 1: 1: 1.

  Next, when | P1-P2 | is smaller than the threshold β and | P1-P3 | is larger than the threshold γ (for example, 16), only P2 is added to P1. That is, the ratio of P1, P2, and P3 is 1: 1: 0. When the relationship between P2 and P3 is reversed, that is, when | P1-P3 | is smaller than the threshold β and | P1-P2 | is larger than the threshold γ (for example, 16), the ratio of P1, P2, and P3 is 1: 0: 1.

When both | P1-P2 | and | P1-P3 | are larger than the threshold value γ, the value of | P2-P3 | is investigated. If | P2-P3 | is smaller than the threshold value δ (example: 4), it is determined that an error has occurred in the pixel value of P1, and the pixel value is updated using P2 and P3. Addition process in the direction. Specifically, the ratio of P1, P2, and P3 is 1: 2: 2.
When | P2-P3 | is larger than the threshold γ, P2 and P3 are excluded from the synthesis targets, and the addition ratios of P1, P2, and P3 are 1: 0: 0.

  In cases other than the above conditions, the average value of P2 and P3 is added to P1 and averaged. That is, the ratio of P1, P2, and P3 is 2: 1: 1. The weighted average of P1, P2, and P3 is calculated according to the ratio determined in this way, and the combined reference block pixel value PM is generated (S1514).

  Similar processing is performed on all the pixels in the reference block. If it is not the last pixel in the reference block (S1515: NO), the next pixel is set (S1516), and the process returns to S1510. If it is the last pixel in the reference block (S1515: YES), the reference image synthesis process for the encoding target block is terminated.

  In the moving image encoding device and the moving image decoding device according to the fourth embodiment, the motion compensated prediction which is the target between the other reference images with respect to the motion compensated predicted image predicted using the first reference image By obtaining a motion vector value in a smaller unit than the image and performing a synthesis process with the motion compensated prediction image obtained in a fine unit according to each motion vector, the encoding target object is not increased. The prediction image corresponding to the minute temporal deformation of the object is generated to improve the encoding efficiency. In addition, the motion compensated prediction image predicted using the first reference image is added by obtaining a correlation with another reference image, selecting a plurality of reference images suitable for combining, and performing a combining process. An appropriate composite image can be generated from a plurality of reference images without sending information, and the encoding efficiency is further improved.

(Embodiment 5)
Next, a video encoding device and a video decoding device according to Embodiment 5 will be described. In the fifth embodiment, the multi-reference image synthesis unit performs super-resolution enlargement processing on the first reference image using a plurality of reference images, and synthesizes an enlarged image as a result of super-resolution enlargement. A feature is that the reference image is configured to be used for motion compensation prediction.

  First, a conceptual diagram showing the operation of the composite image motion compensation prediction process in Embodiment 5 is shown in FIG. 21 and described. In the configurations of the first to fourth embodiments, a pixel value with decimal pixel accuracy of less than one pixel is generated by filtering each reference image, and a synthesized reference image is generated using the generated reference image. However, in the fourth embodiment, a pixel value with decimal pixel accuracy of less than one pixel obtained by enlarging the first reference image is generated by pasting from another reference image, and the frequency band after pasting is adjusted. Thus, a reference image with high definition and reduced influence of coding deterioration is generated.

  As shown in FIG. 21, with respect to the first reference block acquired based on the first motion vector detected between the encoding target block and the first reference image, the second block in the specific range. The reference image and the third reference image are set, the motion detection for the first reference block is performed on the pixels within the specific range, and the pixel is pasted (registration), thereby enlarging the reference block with decimal pixel accuracy. Is generated, and a filter for adjusting to a predetermined band is applied to reflect the component.

  By repeating the above process a plurality of times, the encoding deterioration in the first reference block is removed, and a high-definition reference image with decimal pixel accuracy is generated. By using the high-definition reference signal generated in this way for the composite image motion compensation prediction, a predicted image with less high frequency components of the prediction residual is generated.

  FIG. 22 is a block diagram showing the configuration of the multiple reference image synthesizer in the video encoding device and video decoding device of Embodiment 5, and the operation will be described. Similar to the fourth embodiment, in the fifth embodiment, the configuration and operation of the multiple reference image synthesis unit in the encoding device and the decoding device are the same except that the operation of the processing block connected to the multiple reference image synthesis unit is different. In order to perform the operation, the behavior in the encoding device will be shown and described.

  As shown in FIG. 22, the multiple reference image synthesis unit according to the fifth embodiment includes a standard reference image acquisition unit 1700, a registration target range setting unit 1701, a registration unit 1702, a synthesized reference image acquisition unit 1703, and a band limiting filter unit. 1704, a composite image memory 1705, and a reconstruction end determination unit 1706.

  First, the motion vector detection unit 104 inputs the motion vector value MV1 between the first reference image and the encoding target block to the standard reference image acquisition unit 1700 and the registration target range setting unit 1701. The reference reference image acquisition unit 1700 acquires the reference block of the first reference image from the decoded reference image memory 117 using the input MV1. The reference reference image acquisition unit 1700 outputs the acquired first reference block to the registration unit 1702.

Subsequently, the registration target range setting unit 1701 sets an area to be registered from other reference images for the first reference block. In particular,
Similar to the motion vector detection range between the reference images in the first embodiment shown in FIG. 12, the position indicated by the motion amount obtained by extending or reducing MV1 in accordance with the distance from the encoding target image is ± The range of L pixels is set as a target area for registration. The value of L requires a wider range than the motion vector detection range between reference images in the first embodiment, and is set to L = 32, for example.

  The target area set in the registration target range setting unit 1701 is sent to the registration unit 1702 and subjected to registration processing. The registration unit 1702 first performs horizontal and vertical X-fold enlargement processing on the first reference block. In the present embodiment, by setting X = 4, an enlarged image that can be used for motion compensation in 1/4 pixel units is generated.

  In the enlarged image, a pixel in which a pixel value exists and a pixel generated by the enlargement process are subjected to different processes in the registration process. As shown in FIG. 21, the registration is acquired from the decoded reference image memory 117 via the first enlarged reference block in a predetermined pixel unit (eg, 4 × 4 pixels) and the synthesized reference image acquisition unit 1703. A motion vector is calculated by performing block matching at intervals of one pixel with a reference block configured in units of one pixel of another reference image. When the calculated motion vector does not indicate a position with 1 pixel accuracy, pixel pasting from another reference image is performed on a pixel that has not existed conventionally, and thus the pasted pixel value is filtered. Replaces the pixel value generated by. When the position of one-pixel accuracy is indicated, and when a pixel sticks to the same place from a plurality of reference images, the highest is based on the distribution and frequency of pixel values attached to each pixel position after the registration is completed. The value that should be at the corresponding position is calculated, and the pixel value is replaced with the value.

  The registered reference block is output from the registration unit 1702 to the band limiting filter unit 1704. The band limiting filter unit 1704 applies a band limiting filter that assumes frequency characteristics of the originally enlarged reference image to the input reference block after registration.

  For pixel position values other than one-pixel accuracy that did not stick at the time of registration, using the surrounding registered pixel values without using the values generated when the image was first enlarged, Filtered. As a result, the influence of the registration is also reflected on the pixel value at the position where it is not attached.

  The reference block as a result of applying the band limiting filter is stored in the synthesized image memory 1705 by the band limiting filter unit 1704. The composite image memory 1705 sends the stored reference block to the reconstruction end determination unit 1706.

  In the reconfiguration end determination unit 1706, the reference block sent from the band limiting filter unit 1704 and subjected to the previous band limiting filter is secured inside, and compared with the input reference block. When the change is small as a comparison result (the amount of change is smaller than the previous change) and the change this time is small, it is determined that the reconstruction processing for super-resolution is completed. The reconstruction end determination unit 1706 ends the synthesis process from the plurality of reference images for the current encoding target block.

  At the end, the stored reference block is output from the composite image memory 1705 to the composite image motion compensation prediction unit 108. If not completed, the bandwidth limit is applied to the difference between the stored reference block and the reference block to which the previous bandwidth limit filter is applied with respect to the reference block to which the previous bandwidth limit filter is applied. A high-frequency component is extracted by applying a filter having an inverse characteristic of the filter, and an updated image of the reference block reflecting the generated high-frequency component information is input again to the registration unit 1702, and the registration image from another reference image is again input. Is applied. By repeating the registration process a plurality of times, high-definition components are reconstructed in stages on the reference block, and a high-quality reference block is generated.

  Regarding super-resolution processing including specific registration and its reflection, there is a method other than the configuration of the fifth embodiment, and even when the method is applied, it is based on a synthesized reference image that has been subjected to super-resolution processing. There is an effect that the motion compensation prediction can be realized without transmitting an additional motion vector.

  According to the moving image encoding device and the moving image decoding device in the fifth embodiment, the super-resolution processing is performed on the motion compensated prediction image predicted using the first reference image using another reference image. By creating the predicted image as a predicted image, a predicted image in which the high-frequency component from which the reference image has disappeared is restored is generated, and motion vector detection is performed to finely adjust the super-resolved reference image for high frequency. Since a motion vector that takes into account the phase of the component can be transmitted, there is a new effect that the prediction residual of the high-frequency component can be significantly reduced without an increase in additional information.

  The moving picture encoding apparatus and moving picture decoding apparatus presented as the first, second, third, fourth, and fifth embodiments are physically a CPU (central processing unit), a memory, and the like. Can be realized by a computer equipped with a recording device, a display device such as a display, and a communication means to the transmission path, and the means having each presented function is realized as a program on the computer and executed. Is possible. In addition, the program can be provided by being recorded on a computer-readable recording medium, provided from a server through a wired or wireless network, or provided as data broadcasting of terrestrial or satellite digital broadcasting. is there.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  DESCRIPTION OF SYMBOLS 101 Input image buffer, 102 Block division part, 103 Intra-frame prediction part, 104 Motion vector detection part, 105 Compensation prediction part, 106 Motion vector prediction part, 107 Multiple reference image synthetic | combination part, 108 Compensation prediction part, 109 Prediction mode determination part 110 subtractor, 111 orthogonal transform unit, 112 quantization unit, 113 inverse quantization unit, 114 inverse orthogonal transform unit, 115 adder, 116 intra-frame decoded image memory, 117 decoded reference image memory, 118 entropy encoding unit, 119 Stream buffer, 121 Code amount control unit, 201 Stream buffer, 202 Entropy decoding unit, 203 Prediction mode decoding unit, 204 Predicted image selection unit, 205 Inverse quantization unit, 206 Inverse orthogonal transform unit, 207 Adder, 20 Intra-frame decoded image memory, 209 decoded reference image memory, 211 intra-frame prediction unit, 212 motion vector prediction decoding unit, 213 compensation prediction unit, 214 motion vector separation unit, 215 multiple reference image synthesis unit, 216 compensation prediction unit, 400 standard Reference image acquisition unit, 401 motion vector detection range setting unit, 402 inter-reference image motion vector detection unit, 403 synthesized reference image acquisition unit, 404 reference image synthesis unit, 405 synthesized image memory, 1000 standard reference image acquisition unit, 1001 motion vector A detection range setting unit, 1002 a motion vector detection unit, 1003 a synthesized reference image acquisition unit, 1004 a reference image synthesis unit, and 1005 a synthesized image memory.

Claims (7)

A motion vector decoding unit for decoding the first motion vector for the decoding target block from the encoded stream;
A motion vector separation unit that generates a second motion vector from the first motion vector;
A first reference block of a specific area having a size equal to or larger than the decoding target block, extracted from the first reference image using the second motion vector, and a predetermined area of at least one other reference image; A reference image synthesis unit that generates a synthesized reference block;
Using the first motion vector, a block having the same size as the decoding target block is extracted from the synthesis reference block, and the extracted block is a prediction block;
A moving picture decoding apparatus comprising: a decoding unit that generates a decoded image by adding the prediction block and a prediction difference block decoded from the decoding target block. In the motion vector separation unit, the accuracy of the input first motion vector is M pixel accuracy (M is a real number), and the accuracy of the second motion vector to be generated is N pixel accuracy (N is a real number: N > M), and the second motion vector is a value obtained by converting the first motion vector to N pixel accuracy,
The specific area, the location relative to the first reference image indicated by the second motion vector, motion picture according to claim 1, characterized in that it comprises a target block ± N / 2 pixels or more regions Image decoding device. The reference image synthesis unit includes an inter-reference image motion vector detection unit that detects a third motion vector between the first reference block and a second reference image that is another reference image.
The reference image synthesis unit includes an average value or a weighted average value for each pixel of the second reference block extracted from the second reference image using the third motion vector and the first reference block. by calculating the moving picture decoding apparatus according to claim 1 or 2, characterized in that to generate the synthetic reference block. The inter-reference image motion vector detection unit detects a plurality of third motion vectors between the first reference block and the second reference image in units of blocks smaller than the first reference block. And
The reference image synthesis unit combines a plurality of second reference blocks in small blocks extracted from the second reference image using the plurality of third motion vectors, and 4. The moving picture decoding apparatus according to claim 3 , wherein the synthesized reference block is generated by calculating an average value or a weighted average value for each pixel. The inter-reference-picture motion vector detection unit detects two time differences between a first time difference between the first reference picture and the decoding target block and a second time difference between the second reference picture and the decoding target block. around the motion vector value obtained by converting the second motion vector according, by searching a motion within a predetermined range, according to claim 3 or 4, characterized in that detecting the third motion vector Video decoding device. Decoding a first motion vector for a decoding target block from an encoded stream;
Generating a second motion vector from the first motion vector;
A first reference block of a specific area having a size equal to or larger than the decoding target block, extracted from the first reference image using the second motion vector, and a predetermined area of at least one other reference image; Generating a synthesized composite reference block;
Extracting a block having the same size as the decoding target block from the synthesis reference block using the first motion vector, and setting the extracted block as a prediction block;
A moving picture decoding method comprising: a step of adding a prediction block and a prediction difference block decoded from the decoding target block to generate a decoded image. A function of decoding a first motion vector for a decoding target block from an encoded stream;
A function of generating a second motion vector from the first motion vector;
A first reference block of a specific area having a size equal to or larger than the decoding target block, extracted from the first reference image using the second motion vector, and a predetermined area of at least one other reference image; A function to generate a synthesized reference block;
A function of extracting a block having the same size as the decoding target block from the synthesis reference block using the first motion vector, and setting the extracted block as a prediction block;
A moving picture decoding program that causes a computer to realize a function of generating a decoded image by adding the prediction block and a prediction difference block decoded from the decoding target block.