JP3218874B2 - Image encoding apparatus and image encoding method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing system in which all pixels in the same patch are not restricted to having a common motion vector, and the horizontal and vertical components of the motion vector of the pixel are values other than an integer multiple of the distance between adjacent pixels. The present invention relates to an image encoding device that performs motion compensation that can take the following formula, and an image decoding device that receives and decodes a code output by the encoding device.

[0002]

2. Description of the Related Art It is known that, in high-efficiency coding of moving images, motion compensation utilizing similarity between temporally adjacent frames has a great effect on information compression. The motion compensation process is expressed as follows using mathematical expressions.
The predicted image of the frame to be coded (current frame) is denoted by P (x, y), and the reference image (decoded image of a frame that is temporally close to P and has already been coded) is denoted by R ( x, y). Also, assuming that x and y are integers,
In P and R, it is assumed that a pixel exists at a point where the coordinate value is an integer. At this time, the relationship between P and R is

[0003]

(Equation 1)

[0004] Here, assuming that the image is divided into n small areas (patches), Pi represents the pixels included in the i-th patch of the image. Also, the conversion function f
i (x, y) and gi (x, y) represent the spatial correspondence between the image of the current frame and the reference image. Here, the motion vector of the pixel (x, y) in Pi is (x-fi (x, y),
y-gi (x, y)). The international standard H.264 of the current video coding system is used. 261, MPEG1, MPE
In G2 and the like, a method called block matching in which fi (x, y) and gi (x, y) are constants irrelevant to x and y is adopted. However, in order to achieve a higher information compression rate than these standard encoding methods, it is necessary to employ a more advanced motion compensation method. As such a new motion compensation scheme, a motion compensation scheme that allows pixels in the same patch to have different motion vectors, where fi (x, y) and gi (x, y) are not constants, has recently been proposed. I have.
The transformation function in these methods is affine transformation

[0005]

(Equation 2)

[Nakaya et al., "Basic Study on Motion Compensation Based on Triangular Patches", IEICE Technical Report, IE90-106, Hei 2-03), bilinear transformation

[0007]

(Equation 3)

[0008] Example using GJ Sullivan and RL
Baker, "Motion compensation for video compression
using control grid interpolation ", Proc. ICASSP ''
91, M9.1, pp.2713-2716, 1991-05), perspective transformation

[0009]

(Equation 4)

(V. Seferdis and M. Ghanbari,
"General approach to block-matching motion estim
ation '', Optical Engineering, vol. 32, no.7, pp.
1464-1474, 1993-07) have already been reported. Here, aij, bij, and cij are motion parameters estimated for each patch. If the value of the conversion function is not an integer,
It is necessary to find a luminance value at a point where the coordinate value is not an integer and no pixel actually exists in the reference image. In this case, bilinear interpolation using four neighboring pixels is often used as the processing. If this interpolation method is described by a mathematical expression, R (x + p, y + q) is expressed as 0 ≦ p, q <1.

[0011]

(Equation 5)

## EQU1 ##

When transmitting the motion information, the image coding apparatus only needs to transmit to the receiving side information that can specify the motion parameter of the transform function in some form. For example, it is assumed that an affine transformation is used as a transformation function and the shape of the patch is a triangle. In this case, whether the six motion parameters are directly transmitted or the motion vectors of the three vertices of the patch are transmitted,
On the receiving side, six motion parameters ai1 to ai6 can be reproduced.

A feature of the motion compensation system in which the conversion function is not a constant is that a mathematical operation is required when obtaining a motion vector for each pixel. In the calculation of the motion vector (calculation of the conversion function), if there is a difference in the calculation accuracy between the transmitting side and the receiving side, the predicted images obtained by the encoding device and the decoding device will be different (mismatches will occur). May occur). Since the mismatch in the predicted image has a property of being accumulated on the receiving side, even a small error per frame may seriously affect the image quality of the reproduced image. This problem does not exist in block matching in which all pixels in a block follow the same motion vector, and that motion vector is encoded and transmitted as motion information as it is.

A description will be given of a countermeasure against this problem by taking as an example a case where an affine transformation (Equation 2) is used as a transformation function and bilinear interpolation (Equation 5) is used as an interpolation method. As one method of solving the problem, a method is conceivable in which the error of the operation result of Expression 5 is sufficiently smaller than the quantization step size of the luminance value by sufficiently increasing the operation accuracy of Expression 2 and Expression 5. Can be Consider the case where this measure is adopted. First, it is assumed that the calculation accuracy of Equation 5 is sufficiently high. At this time, for example, R (0, 0) = R (0, 1) = 0, R (1,
0) = R (1,1) = 255, and 0 <x <1;
If 0 <y <1, (hereinafter referred to as “worst condition”), an error with a magnitude of 1/255 or more in fi (x, y) always leads to an error in the quantization value of the luminance value. (The quantization step size of the luminance value is assumed to be 1).
Therefore, in order to prevent a mismatch, it is necessary to create an encoding device and a decoding device such that the operation error of Equation 2 is sufficiently smaller than 1/255. However, increasing the calculation accuracy generally leads to an increase in the number of digits in the internal representation of a numerical value, which further complicates the calculation process. Since Equations 2 and 5 are calculated extremely many times in the motion compensation processing, complicating this operation has a serious effect on the overall processing amount.

[0016]

There is no restriction that all pixels belonging to the same patch have a common motion vector.
In addition, in the motion compensation method in which the horizontal and vertical components of the pixel motion vector can take values other than an integer multiple of the distance between adjacent pixels, the calculation accuracy required for calculating the conversion function is reduced, and the calculation accuracy of the conversion function is reduced. And a method for preventing occurrence of mismatch of predicted images.

[0017]

The horizontal and vertical components of the motion vector of each pixel used in synthesizing a predicted image are 1 / d1 and 1 / d2 of the distance between adjacent pixels (where d1 and d2 are positive). The above-mentioned object is achieved by specifying that only values that are integral multiples of (integer) are taken.

[0018]

By adopting the rules relating to the motion vector described in "Means for Solving the Problem", if the worst condition described in "Prior Art" is given, a conversion that always leads to an error in the quantization value of the luminance value. The magnitude of the error of the function fi (x, y) is 1 / d1
Becomes For example, if d1 = 4, even if the calculation accuracy of fi (x, y) is reduced by 6 bits compared to the countermeasure described in the "prior art", the mismatch of the prediction image under the worst condition is reduced. The dangers that occur can be kept at almost the same level.

[0019]

EXAMPLE An example of the operation method of Equation 2 will be described with reference to an example in which an affine transformation is used as a transformation function in the same manner as described above. For simplicity, it is assumed that d1 = d2 = d (d is a positive integer). Also, it is assumed that the patch is a triangle, and the motion vectors of the three vertices of the patch are transmitted as motion information.

Hereinafter, description will be made using the example shown in FIG. It is assumed that the patch 102 in the reference image 101 has been moved and deformed to the patch 107 in the current frame 106 (grid points 103, 104, and 105 correspond to grid points 108, 109, and 110, respectively). At this time,
The coordinates of the vertices 103, 104, and 105 of the patch 102 are respectively (x1 ', y1'), (x2 ', y2'), (x3 ', y3'), and the coordinates of the vertices 108, 109, and 110 of the patch 107, respectively. Assuming that (x1, y1), (x2, y2), and (x3, y3) (coordinate values are all non-negative integers), the motion parameter aij of Equation 2 in this patch is

[0021]

(Equation 6)

## EQU2 ## Here, without performing the division operation, aij is defined as aij = aij '/ Di where the denominator and the numerator are integers.
In the form. Then, the calculation results of Expression 2 are all fi (x, y) = fi '(x, y) / Di and gi (x,
y) = gi '(x, y) / Di can be expressed in the form of a fraction with an integer as a denominator and a numerator. Here, "//" is divided by integers (division in which the decimal component of the operation result is truncated)
And ki = Di // 2,

[0023]

(Equation 7)

It is assumed that Fi (x, y) and Gi (x, y) are f
This function rounds i (x, y) and gi (x, y) to the nearest integer multiple of 1 / d. In equation (1), fi (x, y) and gi
By using Fi (x, y) and Gi (x, y) instead of (x, y), the horizontal / vertical components of the motion vector of each pixel can be obtained by multiplying 1 / d of the distance between adjacent pixels by an integer. It can be restricted to take only values. In addition, if Fi (x, y) and Gi (x, y) are used on both the transmitting side and the receiving side, it is possible to prevent a mismatch of a predicted image caused by an error in a conversion function with a relatively low-precision operation. Becomes

FIG. 2 shows the flow of processing for calculating Fi (x, y) and Gi (x, y) when d = 4. First, when the coordinates of the vertices of the patch before and after the deformation are given by 201, 2
Functions fi '(x, y) and gi' (x, y) are defined by 02 and 204, a constant Di is obtained by 203, and a constant ki is obtained by 205. Using these functions and constants, the values of Fi (x, y) and Gi (x, y) are calculated from the coordinates (x, y) for each pixel in the patch. When (x, y) is given by a binary integer representation, first, a product-sum operation is performed at 206 to obtain fi ′ (x, y) and g
The value of i ′ (x, y) is obtained, and the result is shifted by 2 bits to the left by 207 to multiply by 4 (= d). By adding ki to the result at 208 and dividing it by Di at 209 (decimal components of the operation result are discarded), 4
Obtain the values of Fi (x, y) and 4Gi (x, y). This integer 4
For Fi (x, y) and 4Gi (x, y), by placing a decimal point between the second and third digits from the bottom at 210, Fi
The value of (x, y) and Gi (x, y) can be obtained (this has the same meaning as performing the operation of dividing by 4).

The value of d may be defined as a fixed parameter of the encoding system, or may be made variable between the transmitting side and the receiving side before transmitting the image data. FIG. 3 shows an example of a procedure when the transmitting side and the receiving side communicate with each other to determine d. First, the transmitting side 301 informs the receiving side 302 that the upper limit of the allowable d is 4 in communication 303 due to hardware restrictions of the image encoding apparatus. Next, the receiving side informs the communication 304 that the upper limit of d is 2 due to restrictions of the image decoding device. As a result, the transmitting side obtains the optimal d
Is determined to be 2, and the image data transmitted thereafter is d
= 2 is recommended by the communication 305. Immediately after this, the transmitting side transmits the image data through the communication 306. In general, the larger the value of d, the more complicated the hardware of the device. Therefore, the sending side
It is considered appropriate to adopt the lower value of the upper limit of d on the receiving side. In order to realize this method, the image encoding / decoding device must have a function capable of coping with d equal to or less than its own upper limit.

The value of d is considered to be a power of 2 in consideration of ease of multiplication and division. The prediction error decreases as the value of d increases, but the process of synthesizing the predicted image becomes complicated. Considering the prediction characteristics, it is desirable that the value of d is 2 or more. Considering the balance between the prediction characteristics and the complexity of the processing, 2, 4, and 8 are appropriate as specific values of d.

It is apparent that the following modifications are also included in the present invention.

(1) As the interpolation method of the luminance value, in this specification, the primary interpolation is taken, but a function other than this may be used. The more complicated the function, the greater the effect of simplifying the operation.

(2) Although the affine transformation is mainly described in this specification as a type of the transformation function, other transformation functions may be used. The present invention is effective as long as there is a possibility that the operation result may change according to the operation accuracy of the conversion function.

(3) The shape of the patch may be any shape as long as it specifies a set of pixels, and need not be a triangle in particular in the present specification.

(4) In the motion compensation based on the spatial transformation, the motion information transmitted as in the example described in this specification may not be the motion vector of the vertex of the patch. The motion information only needs to specify a conversion function for each patch, and for example, the motion parameter aij of Expression 2 may be transmitted as it is. In the case where the motion parameters are transmitted as they are, by applying the present invention, the accuracy of the transmitted motion parameters is reduced while preventing the mismatch of the predicted image caused by the calculation accuracy of the conversion function (the number of digits is reduced). ) Is possible. As the value of d is smaller, the number of digits of the motion parameter can be reduced, and as a result, the amount of transmission information can be reduced.

(5) In the embodiment, the case where the values of d1 and d2 are equal is illustrated, but both may be different.

(6) In this specification, the method of modifying the patch of the reference image by fixing the patch structure of the current frame has been described. Conversely, the patch of the current frame is modified by fixing the patch structure of the reference image. May be used.

(7) In this specification, the number of reference images used for synthesizing one prediction image has been described as one. However, a method using a plurality of reference images may be used.

[0036]

According to the present invention, there is no restriction that all the pixels belonging to the same patch have a common motion vector, and the horizontal and vertical components of the motion vector of the pixel have a value other than an integral multiple of the distance between adjacent pixels. In the possible motion compensation method,
The calculation accuracy of the conversion function can be reduced while preventing the occurrence of a mismatch in the predicted image. Further, in the method in which the values of d1 and d2 are determined on the transmission side and the reception side before the transmission of the image data, the optimum image quality of the reproduced image can be determined according to the performance of the devices on the transmission side and the reception side.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an example of movement and deformation of a patch in motion compensation based on spatial transformation.

FIG. 2 is a diagram illustrating an example of a method of calculating a conversion function when horizontal and vertical components of a motion vector are limited to an integral multiple of 1/4 (when d = 4).

FIG. 3 is a diagram showing an example of a system in which a value of 1 / d, which is a minimum unit of a motion vector of a pixel, is determined by communication between a transmission side and a reception side before communication of image data.

[Explanation of symbols]

101: one patch in the reference image after motion estimation
102, 107 ... patch, 103-105, 108-1
10: vertex of patch; 106: all patches of the original image of the current frame; 201: coordinate value of vertex of patch; 202
To 210: calculation procedure, 301: image encoding device on the transmission side, 302: image decoding device on the reception side, 303 to 306
... communication procedures.

Continuation of the front page (56) References JP-A-5-49023 (JP, A) JP-A-5-130581 (JP, A) JP-A-6-500212 (JP, A) US Patent 4,937,666 (US, A) Yuichiro Nakaya and Hiroshi Harashima, "Basic Study on Motion Compensation Using Triangular Patches", IEICE Technical Report, IE-106 p. 9-16 Issued in March 1990 Michael Gilge "A HIGH QUALITY VIDEO PHONE CODER USING HIERACHICAL MOTION ESTIMATION AND ST RUCTION CODING OF THE PREDICTION VOL. 1001 Visual Communications and Image Processing '88 (58) Fields surveyed (Int. Cl. ⁷ , DB name) H04N ^7/ 24-7/68 H03M 7/36

Claims (10)

(57) [Claims] 1. An image coding method for performing motion compensation based on a spatial transform, comprising : performing motion compensation based on a spatial transform between a reference image and an original image of a current frame to estimate motion information of a predicted image of the current frame. The accuracy d when the horizontal and vertical components of the motion vector of each pixel of the predicted image are limited to an integer multiple of 1 / d (d is a positive integer of 2 or more) of the distance between adjacent pixels and are obtained from the motion information.
And transmitting the motion information and the information indicating the accuracy d. 2. The image coding method according to claim 1, wherein the value of the precision d is 2 to the power of w (w is a positive integer). 3. The image coding method according to claim 1, wherein the motion information is a motion vector of a patch vertex of the predicted image. 4. An image coding method for performing motion compensation which allows pixels in the same patch to have different motion vectors, wherein said motion compensation is performed between a reference image and an original image of a current frame. Estimate the motion vector of the patch vertex of the predicted image of the above, and make the horizontal and vertical components of the motion vector of each pixel of the predicted image an integer multiple of 1 / d (d is a positive integer of 2 or more) of the distance between adjacent pixels. An image coding method, comprising: determining an accuracy d when obtaining from the motion vector by limiting the motion vector; and transmitting a motion vector of the patch vertex and information indicating the accuracy d. 5. The encoding method according to claim 4, wherein the value of the precision d is 2 to the power of w (w is a positive integer). 6. An image coding apparatus for performing motion compensation based on spatial transformation, wherein motion compensation based on spatial transformation is performed between a reference image and an original image of the current frame to estimate motion information of a predicted image of the current frame. Means for limiting the horizontal and vertical components of the motion vector of each pixel of the predicted image to an integral multiple of 1 / d (d is a positive integer of 2 or more) of the distance between adjacent pixels, and obtaining the motion vector from the motion information. Accuracy d
And a means for transmitting the motion information and the information indicating the accuracy d. 7. The image encoding apparatus according to claim 6, the value of the accuracy d is image coding apparatus characterized by (is w a positive integer) 2 w th power is. 8. An image encoding apparatus according to claim 6, wherein said motion information is a motion vector of a patch vertex of said predicted image. 9. An image coding apparatus for performing motion compensation that allows pixels in the same patch to have different motion vectors, wherein the motion compensation is performed between a reference image and an original image of a current frame. Means for estimating the motion vector of the patch vertex of the predicted image of the above, and the horizontal and vertical components of the motion vector of each pixel of the predicted image are expressed by an integer of 1 / d (d is a positive integer of 2 or more) of the distance between adjacent pixels. An image coding apparatus, comprising: means for determining the precision d when obtaining from the motion vector with limiting to twice, and transmitting a motion vector of the patch vertex and information indicating the precision d. 10. The encoding device according to claim 9, wherein
The encoding apparatus according to claim 1, wherein the value of the precision d is 2 to the power of w (w is a positive integer).