JP4645515B2 - Motion compensated image coding method and motion compensated image coding apparatus - Google Patents

Description

  The present invention is a high-efficiency encoding that encodes a moving image signal with a smaller amount of information in order to efficiently transmit, store, and display a moving image signal such as a television image with high image quality. It belongs to the motion compensated image encoding process for encoding the prediction residual by inter-frame prediction.

<Motion compensated interframe predictive coding>
As for coding of a moving image signal, motion compensation interframe predictive coding is generally used. This is also the case for international standards represented by MPEG. The incoming video signal is subtracted from the prediction signal corresponding to the incoming signal, and the prediction residual is encoded. The prediction signal is formed by motion compensation, and its block shape, size, and accuracy are determined by the standard. Note that the motion compensation accuracy refers to how finely the unit performs motion compensation, and does not indicate the accuracy of motion vectors or the overall performance of motion compensation (prediction residual amount).

In the standard system, H.261 is 1 pixel accuracy, MPEG-1, 2, 4 is 0.5 pixel accuracy, MPEG-4
Advanced Simple Profile, H.264, SMPTE VC-1 has 0.25 pixel accuracy, and new standardized ones have higher accuracy. All block shapes are square. The block size is basically 16x16 pixels up to MPEG-2, MPEG-4,
VC-1 is 8 × 8 pixels, and H.264 is 4 × 4 pixels, and even finer changes are supported. The higher the motion compensation accuracy and the smaller the block size, the smaller the prediction residual, but the information on the motion vector increases. Therefore, the block size is adaptively switched while observing the prediction residual amount.

  In these standard systems, the standard defines the decoding process, and the local decoding process also complies with the standard in the encoding. Here, it is common to use a prediction image obtained by common motion compensation for both addition and prediction subtraction in local decoding, but prediction is inherently free, and motion compensation is not necessarily limited to local decoding. They do not have to be identical. Therefore, encoding with different motion compensation accuracy has been proposed for prediction and local decoding.

<Image coding with motion compensation accuracy changed by prediction and local decoding>
FIG. 3 shows a configuration of a conventional example of image coding in which motion compensation accuracy is changed between prediction and local decoding. This technique is shown in Japanese Patent No. 2897649 (registration date: March 12, 1999) “motion compensated prediction coding apparatus”. Another similar technique in which the motion compensation resample filter is changed between prediction and local decoding is disclosed in Japanese Patent Application Laid-Open No. 11-164306 (publication date: June 18, 1999) “Motion Compensated Video Encoding Device and Its It is shown in "Method".

  The moving image signal coming from the moving image input 1 is subtracted from the high-precision prediction signal given from the minute motion compensator 32 by the subtracter 2, and the obtained prediction residual signal is given to the DCT 3. DCT3 performs discrete cosine transform (DCT) on the prediction residual signal in units of 8 × 8 pixels to obtain coefficients. The coefficient is quantized with a predetermined step width by the quantizer 4 to become a fixed-length code, variable-length coded by the variable-length coder 5 and information-compressed prediction residual.

  The coefficient of the fixed-length code is dequantized by the inverse quantizer 9 to become a local decoding coefficient, and the inverse transform of DCT3 is performed by the IDCT 15 to become a local decoding prediction residual. The local decoded prediction residual is added with a motion compensated prediction signal with a specified accuracy given from the motion compensator 8 by the adder 14 and is stored in the reference image memory 13 as a locally decoded image.

  On the other hand, the motion estimator 12 performs motion estimation from the incoming image signal and the reference image obtained from the reference image memory 13. The prescribed accuracy of motion estimation is a square with a block size of 16 × 16 pixels, and the motion compensation accuracy is 0.5 pixel. The accuracy of the obtained motion vector is the same as this. The motion vector having the specified accuracy is supplied to the minute motion estimator 31, the motion compensator 8, and the variable length encoder 16. The motion compensator 8 motion-compensates the inter-picture prediction reference image stored in the reference image memory 13 with a motion vector with a specified accuracy, and an adder 14 and a minute motion compensator for the obtained specified accuracy prediction signal. 32.

  The minute motion estimator 31 performs a re-search with 0.25 pixel accuracy on the periphery of the motion vector based on the motion vector with the specified accuracy. The obtained minute motion vector is given to the minute motion compensator 32. The minute motion compensator 32 minutely moves the predicted image with the specified accuracy according to the minute motion vector. Since this movement is a process of one pixel or less, it is realized by changing the coefficient of the resampling filter. The high-precision prediction signal obtained in this way is given to the subtracter 2.

  The variable length encoder 5 performs variable length encoding on the prediction residual that has become a fixed length code, and the variable length encoder 16 performs variable length encoding on a motion vector with a specified accuracy. Each variable-length code is code-multiplexed by the multiplexer 10 to be a code string, which is output from the code string output 11.

Such encoding is greatly different in reproduction image quality compared to normal inter-picture prediction encoding using the same prediction signal in inter-picture prediction and local decoding. Since the prediction itself is highly accurate, a minute motion compensation deviation component is deleted, and a change in the level direction or a large change becomes a prediction residual. Therefore, the prediction residual amount is reduced, and if the transfer bit rate is the same, the quantization becomes fine. Since motion compensation in decoding remains at the standard accuracy, a minute motion compensation deviation component is ignored in the original prediction residual, and a change in the level direction or a large change is improved in reproducibility. As a result, the reproduced image is slightly displaced in each frame, but becomes a reproduced image with less noise and distortion. The minute positional deviation that occurs is less than the prescribed motion compensation accuracy (0.5 pixels for MPEG2) and varies depending on the frame, so it is difficult to detect visually. Note that since motion compensation for local decoding is performed with a specified accuracy, there is no mismatch between the encoder and the decoder, and no error remains (accumulates).
Japanese Patent No. 2897649 JP-A-11-164306

  Conventional motion-compensated image encoding performs motion compensation for inter-picture prediction with higher accuracy than specified accuracy, so it was possible to reduce the prediction residual compared to the case of specified accuracy. Even if the accuracy of the conventional motion compensation based on the parallel translation of the square is improved, the change, the rhombus distortion caused by the time difference between the top and bottom of the image, the deformation of the object in the image due to the movement other than the parallel movement, etc. There was no adequate prediction, and the prediction residual remained large.

  The present invention has been made paying attention to the above points, and motion compensation for local decoding processing is performed in a specified block shape, and motion compensation for inter-picture prediction includes deformation of an unspecified block shape. Motion compensation can reduce prediction residuals even for changes due to motion other than parallel movement, and the prediction residuals can be quantized finely, resulting in motion compensated image coding that provides high-quality playback images. It is an object to provide a method and a motion compensated image coding apparatus.

  The present invention obtains a specified motion vector in a specified square block from an incoming image and a reference image in motion compensation coding in which motion compensation processing of decoding and local decoding is specified in a square block. With the motion vector, the reference image is compensated for motion as specified to obtain a specified predicted image. Subsequently, based on the prescribed motion vector, the motion estimation is performed again from the incoming image and the reference image including the deformation of the block shape other than the rectangular block to obtain a motion vector that is not prescribed, and the reference image is obtained from the motion vector. Motion compensation is performed including deformation of the block shape other than the rectangular block to obtain an unpredicted predicted image. Then, the prediction image outside the specified range is subtracted from the encoded image, and the obtained prediction residual is quantized after predetermined processing to obtain a quantized prediction residual, which is then inversely quantized and subjected to predetermined processing. The prediction residual decoded in this way is obtained, and the prescribed prediction image is added to obtain a reference image for predicting the next encoded image. Finally, there is provided a motion compensated image coding method and a motion compensated image coding apparatus that perform variable length coding on the quantized prediction residual and the prescribed motion vector to obtain a code string.

  Also, in motion compensation coding where the motion compensation processing for decoding and local decoding is defined by a square block, motion estimation is performed from the incoming image and the reference image, including deformation of the block shape other than the prescribed rectangular block. Obtaining an unspecified motion vector, deleting information on the deformation of the block shape from the motion vector, obtaining a specified motion vector corresponding only to the specified square block, and converting the reference image to the specified motion Motion compensation is performed using vectors to obtain a prescribed predicted image. Subsequently, motion compensation is performed on the reference image including deformation of the block shape other than the rectangular block using the motion vector outside the specification to obtain a prediction image outside the specification. Then, the prediction image outside the specified range is subtracted from the encoded image, and the obtained prediction residual is quantized after predetermined processing to obtain a quantized prediction residual, which is then inversely quantized and subjected to predetermined processing. The prediction residual decoded in this way is obtained, and the prescribed prediction image is added to obtain a reference image for predicting the next encoded image. Finally, there is provided a motion compensated image coding method and a motion compensated image coding apparatus that perform variable length coding on the quantized prediction residual and the prescribed motion vector to obtain a code string.

<First Embodiment Motion Compensated Image Coding>
The motion compensation image coding according to the first embodiment of the present invention will be described. FIG. 1 shows the configuration, and the same reference numerals are given to the same components as those of the conventional example of FIG. Compared to FIG. 3, FIG. 1 includes a shape deformation estimator 6 instead of the minute motion estimator 31 and a deformation motion compensator 7 instead of the minute motion compensator 32.

  The first embodiment of the present invention differs from the conventional example in the prediction signal forming method for obtaining the residual, and the encoding method and the local decoding process of the prediction residual are basically the same as those in the conventional example. is there.

<Encoding process>
The motion image signal coming from the motion image input 1 is subtracted from the motion compensated prediction signal including the deformation of the block shape given from the deformation motion compensator 7 by the subtractor 2, and the obtained prediction residual signal is subtracted. It is given to DCT3. DCT3 performs discrete cosine transform (DCT) on the prediction residual signal in units of 8 × 8 pixels to obtain coefficients. The coefficient is quantized with a predetermined step width by the quantizer 4 to become a fixed-length code, variable-length coded by the variable-length coder 5 and information-compressed prediction residual. These encoding processes are the same as those of the conventional example except for the prediction signal, but the prediction residual becomes smaller and the generated code amount decreases as the prediction signal changes. When the quantization is controlled so that the generated code amounts are equivalent, the quantization step width becomes fine and the quantization error decreases.

  The coefficient of the fixed-length code is dequantized by the inverse quantizer 9 to become a local decoding coefficient, and the inverse transform of DCT3 is performed by the IDCT 15 to become a local decoding prediction residual. The local decoded prediction residual is added to the standard-compensated motion compensated prediction signal given from the motion compensator 8 by the adder 14 and is stored in the reference image memory 13 as a locally decoded image. These local decoding processes are basically the same as in the conventional example.

  On the other hand, the motion estimator 12 performs motion estimation from the incoming image signal and the reference image obtained from the reference image memory 13. The motion estimation is performed with the block shape being a square, the block size being 16 × 16 pixels, and the motion compensation accuracy being 0.5 pixels, and the obtained motion vector meets this rule. This prescribed motion vector is given to the shape deformation estimator 6, the motion compensator 8, and the variable length encoder 16. The motion compensator 8 performs motion compensation on the reference image for inter-picture prediction stored in the reference image memory 13 with a prescribed motion vector, and provides the obtained prescribed prediction signal to the adder 14.

  The shape deformation estimator 6 performs a re-search on the shape deformation of the image based on the square block formed by the prescribed motion vector. Information about the obtained shape deformation is given to the deformed motion compensator 7 together with a prescribed motion vector. The deformed motion compensator 7 reads out a pixel necessary for forming a prediction signal from the reference image memory 13 according to a specified motion vector, and uses the information related to shape deformation (including position information below the pixel of the specified motion vector) for the pixel. And deform. The motion-compensated prediction signal including the block shape deformation obtained in this way is supplied to the subtractor 2.

  The variable length encoder 5 performs variable length encoding on the prediction residual that has become a fixed length code, and the variable length encoder 16 performs variable length encoding on the specified motion vector information. Each variable-length code is code-multiplexed by the multiplexer 10 to be a code string, which is output from the code string output 11. These variable length encoding processes are the same as in the conventional example.

<Shape deformation estimation and deformation motion compensation>
The operations of the shape deformation estimator 6 and the deformation motion compensator 7 will be described in detail. This is because the motion compensation accuracy of the conventional minute motion estimator 31 and the minute motion compensator 32 is merely improved, but the motion compensation is changed in units of pixels in the block to cope with the deformation of the image accompanying the motion. To do. Although the basic idea is similar to affine transformation, it is different from the technique called motion compensation or global motion compensation by affine transformation. It is a point not to let you. This is because when the deviation from the motion compensation of local decoding increases, the distortion of the reproduced image also increases.

  The candidate for shape deformation may be a rhombus or trapezoid as shown in FIG. 4, a reduced or enlarged whole, a pincushion shape using a curve instead of a straight line, or a tall shape. If only a straight line is generalized, various shapes can be realized by moving the positions of the four ends of the block two-dimensionally as shown in FIG. In this case, even if the amount of movement of the edge is only ± 1 pixel, there are 9 patterns of 3 × 3 in the horizontal and vertical directions, and in all combinations of 4 edges, there are 6561 patterns of 9 × 9 × 9 × 9. Furthermore, when ± 0.5 pixel is added, there are 25 patterns of 5 × 5, and 390625 patterns of 25 × 25 × 25 × 25. It is not easy to search for all of these.

  As a more rational shape estimation method, a normal motion vector search is performed with an 8 × 8 pixel block obtained by dividing a 16 × 16 pixel block into four as shown in FIG. The 8 × 8 pixel block remains a square and is not deformed. This search is performed with a search range of 0.75 pixels and a search accuracy of 0.25 pixels with reference to a prescribed motion vector. The four motion vectors determined as a result are regarded as the movement amount at the center of each 8 × 8 pixel block, and an estimated deformation pattern is determined. The search for each 8 × 8 pixel block is the same as the re-search with only the normal motion compensation accuracy improved, and it becomes 49 patterns of 7 × 7 vertically and horizontally, and can be easily realized.

  Each pixel value is formed by motion estimation and motion compensation. When the shape of a block is determined as shown in FIG. 5B, the position of each pixel is determined by dividing it into equal parts. This position is rounded to 0.25 pixel accuracy as shown in FIG. 7, and each pixel is formed from surrounding pixels by an interpolation filter. In FIG. 5, the rectangle is the pixel position, but in FIG. 7, the intersection of the broken lines is the original virtual position, and the broken circle is the pixel position on the filter processing. Although the position seems to have shifted considerably, the error generated as a pixel value is not large. In the motion estimation, the formed pixels are differentiated from the target image in the same manner as in normal block matching, and are evaluated based on the mean square error of blocks or the sum of absolute values.

  The pixel value forming filter may be a simple filter such as linear interpolation in the shape deformation estimator 6, but a higher order filter using peripheral pixels is desired in the deformation motion compensator 7. An example is shown in FIG. 8, where the 0.5 pixel position is formed by using 6 peripheral pixels, and the 0.25 pixel and 0.75 pixel positions are interpolated with the original pixel position of 0.5. It is given by the average of the pixels at the pixel position.

<Motion Compensated Image Coding of Second Embodiment>
The second motion compensated image coding according to the present invention will be described. FIG. 2 shows the configuration, and the same reference numerals are given to the same components as those of the first embodiment of FIG. 2 includes a motion deformation estimator 21 and a motion information former 22 instead of the motion estimator 12 and the shape deformation estimator 6 as compared with FIG. What is different as the processing operation is a method for forming a prescribed motion vector. In the first embodiment, a prescribed motion vector is determined first, but the first embodiment is based on shape deformation information. The prescribed motion vector is determined above. In this case, it can be said that both the motion vector including the block deformation and the specified motion vector can be optimized, and it is easy to improve the overall performance.

  The predictive code processing from the moving image input 1 to the code sequence output 11, the local decoding processing from the inverse quantization 9 to the reference image memory 13, and the processing operations of the motion compensator 8 and the modified motion compensator 7 are shown in FIG. This is the same as the first embodiment.

  The operation of the motion deformation estimator 21 is similar to performing both the motion estimator 12 and the shape deformation estimator 6 of FIG. However, it is not always necessary to use a prescribed motion vector as a basis, and both motion and shape change can be searched simultaneously to determine the optimum motion and shape change. The obtained motion vector and shape change information are given to the deformed motion compensator 7 and the motion information generator 22.

  The motion information generator 22 forms a prescribed motion vector, and obtains a motion vector with the smallest error from the determined motion vector and shape change. As shown in FIG. 6 (b), it is sufficient that the area overlapping with the shape represented by the motion vector and the shape change is maximized. The prescribed motion vector obtained in this way is slightly different from the motion vector obtained by the motion estimator 12 of FIG. 1, and is less shifted from the image obtained by the deformation motion compensation.

  In the present invention, motion compensation for local decoding processing is performed by a prescribed rectangular block, and motion compensation for inter-picture prediction is performed by compensating for motion including deformation of a block shape other than a non-regulated rectangular block. It is possible to reduce the prediction residual for image changes due to zooming, etc., rhombus distortion caused by time lag between the top and bottom of the image, and deformation due to movement other than parallel movement such as rotation of objects in the image. Since the quantization becomes finer, a high-quality reproduced image can be obtained. Since the deformation amount of the predicted image is limited to about one pixel at the maximum with respect to motion compensation used in local decoding, a component that is not encoded as a prediction residual is a minute shift, and a large deformation or level change is a prediction residual. Therefore, it is encoded and reproduced.

  In addition, motion estimation including motion vector and block shape deformations other than the specified square block is performed, and information regarding block shape deformation is deleted from the obtained non-specified motion vector, and only the rectangular block is specified. By obtaining the motion vector, the prescribed motion vector can be set to an advantageous value, and it is possible to form an optimal motion compensation comprehensively.

The figure which shows the structural example of the motion compensation image coding of the 1st Embodiment of this invention. The figure which shows the structural example of the example motion compensation image coding of the 2nd Embodiment of this invention. The figure which shows the structural example of the motion compensation image coding of a prior art example. The figure which shows embodiment of block deformation | transformation. The figure which shows two-dimensional block deformation | transformation and embodiment of a pixel. The figure which shows embodiment of the shape estimation from a small block, and formation of a regular motion vector. The figure which shows embodiment of rounding of a pixel position and pixel movement. The figure which shows embodiment of 0.25 pixel precision motion compensation.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Moving image input, 2 ... Subtractor, 3 ... DCT, 4 ... Quantizer, 5, 16 ... Variable length encoder, 6 ... Shape deformation | transformation estimator, 7 ... Deformation motion compensator, 8 ... Motion compensator , 9 ... Inverse quantizer, 10 ... Multiplexer, 11 ... Code stream output, 12 ... Motion estimator, 13 ... Reference image memory, 14 ... Adder, 15 ... IDCT, 21 ... Motion deformation estimator, 22 ... Motion information generator, 31 ... minute motion estimator, 32 ... minute motion compensator

Claims (4)

A motion compensation coding method in which motion compensation processing of decoding and local decoding is defined by a square block,
A motion estimation step for obtaining a prescribed motion vector in a prescribed rectangular block from an incoming image and a reference image;
A first motion compensation step for performing motion compensation on a reference image as prescribed by the prescribed motion vector to obtain a prescribed predicted image;
Based on the specified motion vector, the motion estimation is performed again including the deformation of the block shape other than the square block from the incoming image and the reference image to obtain an unspecified motion vector, and the unspecified motion vector is used. Then, a second motion compensation step for performing motion compensation on the reference image including deformation of the block shape other than the rectangular block to obtain an unpredicted predicted image;
A prediction residual quantization step of subtracting the unpredicted prediction image from the encoded image, quantizing the obtained prediction residual after predetermined processing, and obtaining a quantized prediction residual;
A reference image for inversely quantizing the quantized prediction residual, obtaining a decoded prediction residual by performing a predetermined process, and adding the prescribed prediction image to predict a next encoded image Obtaining a local decoding step;
A motion compensation image coding method comprising: variable length coding the quantized prediction residual and the prescribed motion vector to obtain a code string. A motion compensation coding method in which motion compensation processing of decoding and local decoding is defined by a square block,
A motion estimation step of estimating a motion including a deformation of a block shape other than a prescribed square block from an incoming image and a reference image, and obtaining a motion vector outside the specification;
The information on the deformation of the block shape is deleted from the non-prescribed motion vector to obtain a prescribed motion vector corresponding only to the prescribed rectangular block, the reference image is motion-compensated with the prescribed motion vector, A first motion compensation step for obtaining a predicted image;
A second motion compensation step of performing motion compensation on a reference image including deformation of a block shape other than a rectangular block by using the motion vector outside the specification to obtain a prediction image outside the specification;
A prediction residual quantization step for subtracting the unpredicted prediction image from the encoded image, quantizing the obtained prediction residual after predetermined processing, and obtaining a quantized prediction residual;
A reference image for dequantizing the quantized prediction residual, obtaining a decoded prediction residual by performing predetermined processing, and adding the prescribed prediction image to predict a next encoded image Obtaining a local decoding step;
A motion compensation image coding method comprising: variable length coding the quantized prediction residual and the prescribed motion vector to obtain a code string. A motion compensation encoding apparatus in which motion compensation processing for decoding and local decoding is defined by a square block,
A motion estimation means for obtaining a prescribed motion vector in a prescribed rectangular block from an incoming image and a reference image;
First motion compensation means for performing motion compensation on a reference image as prescribed by the prescribed motion vector to obtain a prescribed predicted image;
Based on the specified motion vector, the motion estimation is performed again from the incoming image and the reference image including the deformation of the block shape other than the rectangular block to obtain an unspecified motion vector, and the reference image is determined by the unspecified motion vector. A second motion compensation unit that performs motion compensation including deformation of a block shape other than a rectangular block and obtains an unpredicted predicted image;
A prediction residual quantization means for subtracting the unpredicted prediction image from the encoded image, quantizing the obtained prediction residual after predetermined processing, and obtaining a quantized prediction residual;
A reference image for inversely quantizing the quantized prediction residual, obtaining a decoded prediction residual by performing a predetermined process, and adding the prescribed prediction image to predict a next encoded image Local decoding means to obtain
A motion-compensated image coding apparatus comprising: variable-length coding means for variable-length coding the quantized prediction residual and the prescribed motion vector to obtain a code string. A motion compensation encoding apparatus in which motion compensation processing for decoding and local decoding is defined by a square block,
A motion estimation means for estimating a motion including a deformation of a block shape other than a prescribed square block from an incoming image and a reference image, and obtaining a motion vector outside the regulation;
Information relating to the deformation of the block shape is deleted from the non-prescribed motion vector to obtain a prescribed motion vector corresponding only to the prescribed rectangular block, the reference image is motion compensated with the prescribed motion vector, and prescribed First motion compensation means for obtaining a predicted image of
Second motion compensation means for performing motion compensation on a reference image including deformation of a block shape other than a square block by using the motion vector outside the specification, and obtaining a prediction image outside the specification;
A prediction residual quantization means for subtracting the unpredicted prediction image from the encoded image, quantizing the obtained prediction residual after predetermined processing, and obtaining a quantized prediction residual;
A reference image for inversely quantizing the quantized prediction residual, obtaining a decoded prediction residual by performing a predetermined process, and adding the prescribed prediction image to predict a next encoded image Local decoding means to obtain
A motion-compensated image coding apparatus comprising: variable-length coding means for variable-length coding the quantized prediction residual and the prescribed motion vector to obtain a code string.

Images