JP2001352549A - Image coding and decoding method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for simplifying arithmetic operations in the processing of global motion compensation where a motion vector field of the entire image is approximated by a few parameters. SOLUTION: Interpolating/extrapolating motion vectors at representative points 602, 603 and 604 having features in a spatial interval obtain motion vectors in global motion compensation. Since division used to composite predicted images in the global motion compensation can be substituted by shift arithmetic operations, the processing by a computer or an exclusive hardware unit can be simplified.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image encoding and decoding method for applying global motion compensation based on linear interpolation / extrapolation or bilinear interpolation / extrapolation to an entire image.

[0002]

2. Description of the Related Art It is known that, in high-efficiency coding of moving images, motion compensation utilizing similarity between temporally adjacent frames has a great effect on information compression. The motion compensation method which is the mainstream of the current image coding technology is H.261, MPEG which is an international standard of the moving image coding method.
1, block matching adopted in MPEG2. In this method, an image to be encoded is divided into a number of blocks, and a motion vector is obtained for each block.

FIG. 1 shows a configuration example 10 of an H.261 encoder.
Indicates 0. H.261 adopts a hybrid coding method (inter-frame / intra-frame adaptive coding method) that combines block matching and DCT (discrete cosine transform) as a coding method. The subtracter 102 outputs the input image (original image of the current frame) 101 and the output image 113 of the inter-frame / intra-frame coding switch 119.
A difference between the error image 103 and an error image 103 is output.
This error image is converted into a DCT coefficient by a DCT converter 104 and then quantized by a quantizer 105 to be quantized DCT.
It becomes the T coefficient 106. The quantized DCT count is output to the communication path as transmission information, and is also used in the encoder to synthesize an inter-frame predicted image. The procedure of predictive image synthesis will be described below. The above-described quantized DCT coefficient 106 is converted into an inverse quantizer 108 and an inverse DCT transformer 109.
, And becomes a decoded error image 110 (the same image as the error image reproduced on the receiving side). In addition, in the adder 111, an inter-frame / intra-frame coding changeover switch 119 is provided.
Of the current frame (the same image as the decoded image of the current frame reproduced on the receiving side) is obtained. This image is temporarily stored in the frame memory 114 and is delayed by a time corresponding to one frame. Therefore, at present, the frame memory 114 outputs the decoded image 115 of the previous frame. The decoded image of the previous frame and the input image 101 of the current frame are input to the block matching unit 116, and the block matching process is performed. In the block matching, the predicted image 117 of the current frame is synthesized by dividing the image into a plurality of blocks and extracting a portion most similar to the original image of the current frame from the decoded image of the previous frame for each block. At this time, it is necessary to perform processing (motion estimation processing) for detecting how much each block has moved between the previous frame and the current frame. The motion vector for each block detected by the motion estimation processing is the motion information 12
It is transmitted to the receiving side as 0. The receiving side can independently synthesize the same predicted image as that obtained on the transmitting side from the motion information and the decoded image of the previous frame. The predicted image 117 is input to the inter-frame / intra-frame coding changeover switch 119 together with the “0” signal 118. This switch selects one of both inputs,
Switching between inter-frame coding and intra-frame coding. When the predicted image 117 is selected (FIG. 2 shows this case), inter-frame encoding is performed. on the other hand,
When the “0” signal is selected, the input image is DCT-coded as it is and output to the communication channel, so that intra-frame coding is performed. In order for the receiving side to correctly obtain a decoded image, it is necessary to know whether inter-frame coding or intra-frame coding has been performed on the transmitting side. Therefore, the identification flag 121 is output to the communication path. Final H.261 encoded bit stream 123
Is a quantized DCT coefficient, a motion vector,
It is obtained by multiplexing the information of the intra-frame / inter-frame identification flag.

FIG. 2 shows a configuration example of a decoder 200 that receives an encoded bit stream output from the encoder of FIG. The received H.261 bit stream 217 is divided by a separator 216 into a quantized DCT coefficient 201 and a motion vector 2.
02, separated into an intra-frame / inter-frame identification flag 203. The quantized DCT coefficient 201 is converted into an error image 20 decoded through an inverse quantizer 204 and an inverse DCT transformer 205.
It becomes 6. This error image is inter-frame /
Output image 2 of intra-frame encoding changeover switch 214
15 is added and the result is output as a decoded image 208.
The interframe / intraframe coding changeover switch is operated in accordance with the interframe / intraframe coding identification flag 203.
Switch the output. A predicted image 212 used for performing inter-frame encoding is synthesized by a predicted image synthesis unit 211. Here, a process of moving the position for each block in accordance with the received motion vector 202 is performed on the decoded image 210 of the previous frame stored in the frame memory 209. On the other hand, in the case of intra-frame coding,
The inter-frame / intra-frame encoding switch is
The "0" signal 213 is output as it is.

[0005] Block matching is currently the most widely used motion compensation method. However, when the entire image is enlarged, reduced, or rotated, motion vectors must be transmitted for all blocks. In this case, a problem that encoding efficiency is deteriorated occurs. To solve this problem, global motion compensation (for example, M.Hotter, &# 34Differential e) expressing the motion vector field of the whole image using few parameters
stimation of the global motion parameters zoom and
pan &# 34, Signal Processing, vol. 16, no.3, pp. 24
9-265, Mar. 1989). This corresponds to the motion vector (ug (x, y), vg (x, x) of the pixel (x, y) in the image.
y)),

[0006]

(Equation 1)

[0007]

[0008]

(Equation 2)

In this method, motion compensation is performed using this motion vector. Where a0-a5, b0-
b7 is a motion parameter. When performing motion compensation,
The same predicted image must be obtained on the transmitting side and the receiving side. To this end, the sender sends a0 to a5 or
Although the values of b0 to b7 may be transmitted directly, there is also a method of transmitting the motion vectors of some representative points instead. Now, it is assumed that the coordinates of the upper left, upper right, lower left, and lower right pixels of the image are represented by (0, 0), (r, 0), (0, s), and (r, s), respectively. (However, r and s are positive integers). At this time, the horizontal and vertical components of the motion vector of the representative points (0, 0), (r, 0), (0, s) are (ua, va), (ub, v
b), (uc, vc), (Equation 1) becomes

[0010]

(Equation 3)

It can be rewritten as This means
Instead of transmitting a0 to a5, ua, va, ub, vb, u
It means that the same function can be realized by transmitting c and vc. This is shown in FIG. Original image 3 of current frame
02 and the reference image 301, it is assumed that global motion compensation has been performed.
304, 305 motion vectors 306, 307, 308
(At this time, the motion vector is defined such that a point in the original image of the current frame is set as a starting point and a corresponding point in the reference image is set as an end point.) Similarly, four representative points (0,0), (r, 0), (0, s), (r,
horizontal and vertical components (ua, va), (u) of the motion vector
b, vb), (uc, vc), (ud, vd),

[0012]

(Equation 4)

It can be rewritten as Therefore,
Instead of transmitting b0 to b7, ua, va, ub, vb, u
The same function can be realized by transmitting c, vc, ud, and vd. In this specification, the method using (Equation 1) is referred to as global motion compensation based on linear interpolation / extrapolation, and the method using (Equation 2) is referred to as global motion compensation based on linear interpolation / extrapolation.

[0014] In the linear form transmitting the motion vector of the representative point,
FIG. 4 shows a configuration example of the motion compensation processing unit 401 of the image encoder employing the global motion compensation method based on extrapolation.
The same numbers as those in FIG. 1 indicate the same things. By replacing the block matching unit 116 in FIG. 1 with the motion compensation processing unit 401, an image coding device that performs global motion compensation can be configured. The global motion compensation unit 402 performs motion estimation regarding global motion compensation between the decoded image 115 of the previous frame and the original image 101 of the current frame, and estimates the values of ua, va, ub, vb, uc, and vc. You. Information 403 on these values is transmitted as part of the motion information 120. The predicted image 404 of the global motion compensation is synthesized using Expression 3, and is supplied to the block matching unit 405. Here, motion compensation is performed by block matching between the predicted image of global motion compensation and the original image of the current frame, and the motion vector information 406 of the block and the final predicted image 117 are obtained. The motion vector information is multiplexed with the motion parameter information in the multiplexing unit 407, and is output as motion information 120.

FIG. 5 shows a configuration example of the motion compensation processing unit 501 different from that of FIG. The same numbers as those in FIG. 1 indicate the same things. By replacing the block matching unit 116 in FIG. 1 with the motion compensation processing unit 501, an image coding device that performs global motion compensation can be configured.
In this example, instead of applying block matching to a predicted image of global motion compensation, either global motion compensation or block matching is applied to each block. Between the decoded image 115 of the previous frame and the original image 101 of the current frame, a global motion compensation unit 502
And the block matching unit 505 performs global motion compensation and block matching in parallel. The selection switch 508 controls the predicted image 5 based on global motion compensation.
03 and a prediction image 506 obtained by block matching, an optimum method is selected for each block. Motion vector 504 of representative point, motion vector 507 for each block, selection information 50 of global motion compensation / block matching
9 is multiplexed by a multiplexing unit 510 and output as motion information 120.

By introducing the above-described global motion compensation, the global motion of an image can be expressed using a small number of parameters, and a higher information compression rate can be realized. However, on the other hand, the processing amount in encoding and decoding is increased as compared with the conventional method. In particular
The division shown in (Equation 3) and (Equation 4) is a major factor complicating the processing.

[0017]

In global motion compensation in which the motion vector field of the entire image is approximated by a small number of parameters, there is a problem that the processing amount for synthesizing the predicted image increases. An object of the present invention is to reduce the amount of calculation by replacing the division process in global motion compensation with a binary shift operation.

[0018]

Means for Solving the Problems By appropriately selecting the coordinates of a representative point when performing global motion compensation, the division processing can be realized by a shift operation.

[0019]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following discussion, the pixel sampling interval is set to 1 in both the horizontal and vertical directions,
The coordinates of the upper right, lower left, and lower right pixels are (0,
0), (r, 0), (0, s), (r, s) (where r and s are positive integers).

Linear interpolation / extrapolation (affine transformation) or 1
When performing motion compensation using next-order interpolation / extrapolation (bilinear transformation), if quantization is performed on a motion vector for each pixel,
Effects such as prevention of mismatch and simplification of calculation can be obtained (Japanese Patent Application No. 06-193970). Hereinafter, it is assumed that the horizontal component and the vertical component of the motion vector of a pixel are integer multiples of 1 / m (m is a positive integer). Also, assuming that global motion compensation using the motion vector of the representative point described in “Prior Art” is performed, the motion vector of each representative point is 1 / k
(K is a positive integer). In this specification, “pixel motion vector” refers to a motion vector used to actually synthesize a predicted image when performing global motion compensation. On the other hand, “representative point motion vector” means a parameter used to calculate a pixel motion vector. Therefore, due to a difference in quantization step size or the like, a case may occur where the motion vector of a pixel does not match the motion vector of a representative point even if they are present on the same coordinates.

First, the case of using linear interpolation / extrapolation will be described with reference to FIG. At this time, the representative point is not set as the point located at the corner of the image 601, as described in the “Prior Art”, and (i, j), (i + p, j), (i, j + q) (Points i, j, p, and q are integers). At this time, the points 602, 603, and 604 may exist inside or outside the image. If the horizontal and vertical components of the motion vector at the representative point are multiplied by k, they are (u0, v0), (u1, v1), and (u2, v2), respectively.
0, v0, u1, v1, u2, v2 are integers), pixel (x, y)
The horizontal and vertical components of the motion vector are multiplied by m (u
(x, y), v (x, y)) can be represented by the following equation (where x, y, u (x, y) and v (x, y) are integers).

[0022]

(Equation 5)

However, “//” is a division in which the result of a normal division operation is not an integer and is rounded to a neighboring integer, and the priority as an operator is equivalent to multiplication / division.
In order to reduce the calculation error, it is desirable that the non-integer value be rounded to the nearest integer. In this case, 1 /
The rounding method of adding 2 is (1) rounding toward 0, (2) rounding away from 0, (3)
If the dividend is negative, it is rounded toward 0, if it is positive, it is rounded away from 0 (the divisor is always positive). (4) If the dividend is negative, it is rounded away from 0. If the dividend is positive, it is rounded. Rounding in a direction approaching 0 (the divisor is always positive) can be considered. Among these (3)
Since (4) and (4) do not change the rounding direction irrespective of the sign of the dividend, it is advantageous in terms of the processing amount by the amount that the sign judgment is not necessary. The high-speed processing using (3) is realized by the following equation.

[0024]

(Equation 6)

However, "#" is an integer division which rounds off the decimal part in the direction of 0, and is a division which is most easily realized by a computer. Here, L and M are numbers for keeping the dividend dividend positive at all times, and are sufficiently large positive integers. The term (pqk # 2) is used to round the division result to the nearest integer.

The conversion of the processing into integers contributes to the reduction of the processing amount. Here, p, q, k are each raised to the power of 2 to the power of α, β, h0 (α, β are positive integers, h0 is a negative integer). In this case, since the division of (Equation 5) can be realized by a shift operation of α + β + h0 bits, the amount of processing in a computer and dedicated hardware can be greatly reduced. Further, if m is 2 to the power of h1, (h1 is a non-negative integer, h1
<Α + β + h0), (Equation 6) is

[0027]

(Equation 7)

(“X &# 60 &# 60
“α” shifts x to the left by α bits and puts 0 in the lower α bits, and “x &# 62 &# 62α” shifts x to the right by α bits and puts 0 or 1 in the upper α bits (x is 2 In the case of the complement expression of x, it is assumed that when the most significant bit of x is 1, it is 1, and when it is 0, 0 is inserted.) It is assumed that the precedence of these operators is intermediate between addition and subtraction and multiplication and division.) , The operation can be further simplified.

When linear interpolation / extrapolation is used, (i +
Assuming that the horizontal and vertical components of the motion vector of the representative point located at (p, j + q) are multiplied by k, (u3, v3), (Equation 5) is (i, j), (i + p, j) , (I + p, j + q) as a representative point,

[0030]

(Equation 8)

If (i, j), (i, j + q) and (i + p, j + q) are representative points,

[0032]

(Equation 9)

(I + p, j), (i, j + q), (i + p, j +
If q) is the representative point,

[0034]

(Equation 10)

The processing amount can be similarly reduced by setting the values of p, q, k, and m to positive integer powers of 2.

When the first-order interpolation and extrapolation are used, the representative points (i, j), (i + p, j), (i, j + q), (i + p, j + q) The horizontal and vertical components of each motion vector are multiplied by k (u0, v0), (u1, v1), (u2, v2), (u3, v3).
(U (x, y), v (x, y)) can be expressed by the following equation.

[0037]

[Equation 11]

This equation also takes the values of p, q, k, and m as 2 to the power of α, β, h0, and h1, respectively.

[0039]

(Equation 12)

The processing amount can be reduced in the same manner as above.

In order to obtain the same global motion compensation predicted image on the transmitting side and the receiving side, it is necessary to transmit information about the motion vector at the representative point to the receiving side in some form. Although there is a method of transmitting the motion vector of the representative point as it is, there is also a method of transmitting the motion vector of the point at the corner of the image and calculating the motion vector of the representative point from this value. For this method,
This will be described below.

First, the case where linear interpolation / extrapolation is used will be described. The three points (0,0), (r,
Assuming that the motion vectors of (0) and (0, s) can only take values that are integral multiples of 1 / n, these horizontal and vertical components are multiplied by n (u
(00, v00), (u01, v01), (u02, v02) are transmitted. At this time, the points (i, j), (i + p, j), (i, j +
q), (i + p, j + q) horizontal of each motion vector,
The vertical component is multiplied by k (u0, v0), (u1, v
1), (u2, v2), (u3, v3)

[0043]

(Equation 13)

Is defined as Where u '(x, y), v'
(x, y) is obtained by transforming (Equation 5),

[0045]

[Equation 14]

Is defined as At this time, "///" is a division in which the result of the ordinary division is not an integer and the result is rounded to a neighboring integer, and the precedence as an operator is equivalent to the multiplication / division. (u0, v0), (u1, v1), (u2, v2), (u
3, v3), and performing global motion compensation using these as representative points, (0, 0), (r, 0), (0,
Global motion compensation with s) as a representative point can be approximated. Of course, at this time, if p and q are set to a positive integer power of 2, the processing can be simplified as described above. In order to reduce the calculation error, "//
It is desirable to round the non-integer value to the nearest integer. At this time, as a method of rounding a value obtained by adding 考 え to an integer, the methods (1) to (4) described above can be considered.
However, compared with the case of (Equation 5) (calculated for each pixel),
(Equation 14) (calculation only three times for one image) has a small number of times the operation is executed, so even if the method (1) or (2) is selected, it does not significantly affect the total amount of operation. .

Even when three points different from the example of (Expression 13) are selected as the corner points of the image, the same processing can be realized by modifying Expressions 8 to 10. In addition to the above example, if the horizontal and vertical components of the motion vector at the corner point (r, s) of the image are multiplied by n, then (u03, v03), (Equation 14) becomes (u00, v00) ), (U01, v01), (u03,
v03) is transmitted,

[0048]

(Equation 15)

And (u00, v00), (u02, v02), (u03, v
03) is transmitted,

[0050]

(Equation 16)

And (u01, v01), (u02, v02), (u03, v
03) is transmitted,

[0052]

[Equation 17]

Can be rewritten as

The same applies to the case where primary interpolation and extrapolation are performed. As above, the four representative points (0,0), (r,
Assuming that the motion vectors of (0), (0, s), and (r, s) can only take values that are integral multiples of 1 / n, these horizontal and vertical components are multiplied by n (u00, v00), (u01, v01). , (U02, v02), (u03,
v03) is transmitted. The representative point (i,
j), (i + p, j), (i, j + q), and (i + p, j + q) are obtained by multiplying the horizontal and vertical components by k times (u0, v0), (u1, v1), (u2, v2), (u3, v3) are given by (Equation 13) similarly to the above. Where u '(x,
y), v '(x, y) is obtained by transforming (Equation 11),

[0055]

(Equation 18)

Is defined as

The advantage of the method of transmitting the motion vector at the corner point of the image and performing the interpolation / extrapolation on the motion vector to obtain the motion vector at the representative point is that the range of the motion vector for each pixel is easily limited. Is a point. For example, in the bilinear interpolation / extrapolation given by (Equation 4), when (x, y) is a point in the image, the value of ug (x, y) is the maximum of ua, ub, uc, ud It does not exceed the value nor fall below the minimum value. Therefore, at the time of global motion estimation, ua, ub, u
If constraints are added so that the values of c and ud fall within a certain limit range (for example, within a range of ± 32 pixels), the values of ug (x, y) must be within the same limit range for all pixels. (Of course, this also holds for vg (x, y)). This makes it possible to clarify the number of digits required for the operation, which is convenient for designing software or hardware. However, the above discussion is for the case where all calculations are performed by floating point arithmetic, so care must be taken in actual processing. In the operation for obtaining the motion vector of the representative point from the motion vector of the point at the corner of the image (Equation 18), rounding to an integer exists.
It is necessary to consider the possibility that the motion vector obtained by (Equation 12) will be out of the above-mentioned limited range. In particular, care must be taken when the representative point is located inside the image. This is because, for pixels located outside the rectangle surrounded by the representative point, a motion vector is obtained by extrapolation, so that a rounding error may be amplified. FIG. 7 shows an example in which a motion vector is obtained by extrapolation. For the image 701, the representative points 702, 703, 70
When global motion compensation is performed using 4,705, a motion vector is calculated by extrapolation of a hatched portion in the image. This is because the hatched portion is outside the rectangle 706 surrounded by the representative point.

As a countermeasure against this problem, a method of arranging four representative points such that a rectangle surrounded by the representative points includes the entire image is effective. This example is shown in FIG. Representative point 80
A rectangle 806 surrounded by 2, 803, 804, and 805 includes an image 801. In this case, since the motion vectors of all the pixels are obtained by interpolation from the representative point, the influence of the rounding error at the representative point is not amplified in the image. Therefore, an error larger than the rounding error of the representative point does not occur in the image, and the upper limit of the error becomes clear. However, if the rectangle surrounded by the representative point is too large, the range of values that the motion vector of the representative point can take is widened, and the number of digits required for the calculation increases, which is disadvantageous in mounting.

From the above discussion, it is desirable that the value of p is not less than r and the value of q is not less than s in order to reduce the influence of the rounding error. Even when p and q are smaller than r and s, respectively, it is desirable to take values as large as possible. Further, it is desirable that the values of i, j are set so that a portion as large as possible in the image falls within a region surrounded by the representative points.

As described above, when the first-order interpolation and extrapolation are used for global motion compensation, each component of the motion vector of the pixel included in the rectangle surrounded by the four representative points is represented by the representative point. Has the property that only a value between the maximum value and the minimum value of each component of the motion vector can be taken. On the other hand, when linear interpolation / extrapolation is used, the motion vectors of the pixels in the triangle surrounded by the three representative points have similar properties. Therefore, when performing global motion compensation using linear interpolation / extrapolation, the motion vectors at the four corners of the image are transmitted, and global motion is independently performed on two right triangles divided by the diagonal of the image. A method of performing motion compensation is effective. In this way, the restrictions on the range of the motion vector for the four corner points can be applied to the motion vectors of all the pixels in the image as they are. At this time, i, j,
The value of p, q may be different between the two right triangles. Also, from the viewpoint of calculation error, in order to avoid the case of calculating a motion vector of a pixel by extrapolation,
It is preferable that the triangle surrounding the representative point includes a right triangle to be subjected to global motion compensation. This example is shown in FIG. Points 909, 903, and 90 at the four corners of image 901
8,910 motion vectors are transmitted and points 909,90
3,910, a right-angled triangle and a point 90
Global motion compensation is independently performed on each of the right triangles constituted by 9, 910 and 908. Therefore, if a restriction is imposed on the range of the motion vector of the vertex, the motion vectors of all the pixels in the image will fall within this restriction range. A right triangle formed by points 909, 903, and 910 is represented as a point 902 as a representative point.
Using 903 and 904, a right triangle composed of points 909, 910 and 908 is represented by points 906 and 906 as representative points.
907 and 908 are used. The triangles formed by the representative points each include a right-angled triangle to be subjected to global motion compensation. Therefore, the influence of the rounding error of the motion vector at the representative point is not amplified at a point in the image. In this example, the two triangles formed by the representative points are similar but need not be so.

[0061]

According to the present invention, it becomes possible to substitute the division processing in the prediction image synthesis processing of the global motion compensation with the shift operation, and to simplify the processing by software and dedicated hardware.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration example of an H.261 image encoder.

FIG. 2 is a diagram illustrating a configuration example of an H.261 image decoder.

FIG. 3 is a diagram illustrating an example of global motion compensation for transmitting a motion vector at a representative point.

FIG. 4 is a diagram illustrating a motion compensation processing unit of an image encoder that performs block matching on a predicted image of global motion compensation.

FIG. 5 is a diagram illustrating a motion compensation processing unit of an image encoder that selects global motion compensation and block matching for each block.

FIG. 6 is a diagram showing an example of arrangement of representative points for performing high-speed processing.

FIG. 7 is a diagram showing an area in an image for which a motion vector is obtained by extrapolation.

FIG. 8 is a diagram showing a case where motion vectors of all pixels in an image are obtained by interpolation from motion vectors of representative points.

FIG. 9 is a diagram illustrating an example in which an image is divided into two right-angled triangles, and global motion compensation by interpolation from a motion vector at a representative point is applied to each of the triangles.

[Explanation of symbols]

100: image encoder, 101: input image, 102: subtractor, 103: error image, 104: DCT converter, 10
5. DCT coefficient quantizer, 106, 201 ... Quantized DC
T coefficient, 108, 204... DCT coefficient inverse quantizer, 10
9, 205: inverse DCT converter, 110, 206: decoded error image, 111, 207: adder, 112: decoded image of current frame, 113, 215: output image of inter-frame / intra-frame encoding changeover switch, 114 , 209 ...
Frame memory, 115, 210... Decoded image of previous frame, 116, 405, 505.
117, 212... Predicted image of the current frame, 118, 21
3: "0" signal, 119, 214 ... Inter-frame / intra-frame coding changeover switch, 120, 202, 40
6,507... Motion vector information, 121, 203... Inter-frame / intra-frame identification flags, 122, 407, 51
0: multiplexer, 123: transmission bit stream, 200
... image decoder, 208 ... output image, 211 ... predicted image synthesizing unit, 216 ... separator, 301 ... reference image, 302 ...
Original image of the current frame, 303, 304, 305, 60
2,603,604,702,703,704,70
5,802,803,804,805,902,90
4,906,907 ... representative points, 306,307,308
... motion vectors of representative points, 401, 501 ... motion compensation processing units for performing global motion compensation, 402, 502 ... global motion compensation units, 403, 504 ... motion parameters,
404, 503: predicted image of global motion compensation, 50
6: predicted image by block matching, 508: block matching / global motion compensation changeover switch, 509: selection information of block matching / global motion compensation, 601, 701, 801, 901: image to be subjected to global motion compensation, 706 ... Rectangles surrounded by the representative points, 903, 908... Points that also serve as representative points and image corner points, and 909, 910... Image corner points.

Claims (28)

[Claims] When calculating motion vectors for all pixels in an image by performing linear interpolation and extrapolation on the motion vectors at three representative points, the sampling intervals of the pixels are set in both the horizontal and vertical directions. 1 and the sampling point is on a point where both the horizontal and vertical components of the coordinates are integers, and the coordinates are (i, j), (i + p, j), (i, j + q), (i + p, j +
q) 3 points out of 4 points are used as representative points.
(i and j are integers, p and q are positive integers), and p and q are 2 α powers and 2 β powers (α and β are positive integers), respectively. Compensation method. 2. When calculating motion vectors for all pixels in an image by performing linear interpolation and extrapolation on motion vectors at four representative points, the sampling intervals of pixels are set to be horizontal and vertical. The coordinates of the four representative points are (i, j) and (i +), assuming that the sampling point exists on a point where both directions are 1 and the horizontal and vertical components of the coordinates are both integers. p, j), (i,
j + q) and (i + p, j + q) (i and j are integers), and p and q are 2 to the power of α and 2 to the power of β (α and β are positive A global motion compensation method characterized by being an integer. 3. The sampling interval of pixels in an image is 1 in both the horizontal and vertical directions, and the sampling point exists on a point where both the horizontal and vertical components of the coordinates are integers. The horizontal and vertical components of the motion vector in (i, j), (i + p, j), and (i, j + q) take only integer multiples of 1 / k (where i and j are integers, p and q are positive integers, k is 2 to the power of h0, and h0 is a non-negative integer), and the horizontal and vertical components of the motion vector of each pixel are 1 / m
(I.e., m is 2 to the power of h1 and h1 is a non-negative integer), the representative points (i, j), (i + p, j), (i, j) + u) is obtained by multiplying the horizontal and vertical components of the motion vector by k times (u
0, v0), (u1, v1), (u2, v2) (where u
0, v0, u1, v1, u2, and v2 are integers), and the horizontal and vertical components of the motion vector of the pixel (x, y) are multiplied by m (u (x, y), v (x, y) ) Is (but
x, y, u (x, y) and v (x, y) are integers), u (x, y) = (u0 · p · q + (u1-u0) (xi) · q + (u2-u0) ( yj) ・ p) m // (p
・ Q ・ k) v (x, y) = (v0 ・ p ・ q + (v1-v0) (xi) ・ q + (v2-v0) (yj) ・ p) m // (p
(Q / k) (however, "//" is a division in which the result of a normal division operation is not an integer and is rounded to a nearby integer, and the precedence as an operator is equivalent to multiplication / division) , And p and q are 2 to the power of α and 2 to the power of β (α and β are positive integers), respectively. 4. A pixel sampling interval in an image is 1 in both the horizontal and vertical directions, and the sampling point is located on a point where both the horizontal and vertical components of the coordinate are integers. The horizontal and vertical components of the motion vector at (i, j), (i + p, j) and (i + p, j + q) take only integer multiples of 1 / k (where i and j are Integers, p and q are positive integers, k is 2 to the power of h0, and h0 is a non-negative integer), and the horizontal and vertical components of the motion vector of each pixel are 1 / m
(Where m is 2 to the power of h1 and h1 is a non-negative integer), the representative points (i, j), (i + p, j), (i + p , j + q) obtained by multiplying the horizontal and vertical components of the motion vector by k times.
Using (u0, v0), (u1, v1), (u3, v3) (where
u0, v0, u1, v1, u3, v3 are integers) and the horizontal and vertical components of the motion vector of pixel (x, y) are multiplied by m (u (x, y), v (x, y) ) Is (but
x, y, u (x, y) and v (x, y) are integers), u (x, y) = (u0 · p · q + (u1-u0) (xi) · q + (u3-u1) ( yj) ・ p) m // (p
・ Q ・ k) v (x, y) = (v0 ・ p ・ q + (v1-v0) (xi) ・ q + (v3-v1) (yj) ・ p) m // (p
(Q / k) (however, "//" is a division in which the result of a normal division operation is not an integer and is rounded to a nearby integer, and the precedence as an operator is equivalent to multiplication / division) , And p and q are 2 to the power of α and 2 to the power of β (α and β are positive integers), respectively. 5. A sampling interval of pixels in an image is 1 in both horizontal and vertical directions, and a sampling point is located on a point where both horizontal and vertical components of coordinates are integers. The horizontal and vertical components of the motion vector at (i, j), (i, j + q), (i + p, j + q) take only integer multiples of 1 / k (where i, j are Integers, p and q are positive integers, k is 2 to the power of h0, and h0 is a non-negative integer), and the horizontal and vertical components of the motion vector of each pixel are 1 / m
(I.e., m is 2 to the power of h1 and h1 is a non-negative integer), the representative points (i, j), (i, j + q), (i + p , j + q) obtained by multiplying the horizontal and vertical components of the motion vector by k times.
Using (u0, v0), (u2, v2), (u3, v3) (where
u0, v0, u2, v2, u3, v3 are integers) and the horizontal and vertical components of the motion vector of pixel (x, y) are multiplied by m (u (x, y), v (x, y) ) Is (but
x, y, u (x, y) and v (x, y) are integers), u (x, y) = (u0 · p · q + (u3-u2) (xi) · q + (u2-u0) ( yj) ・ p) m // (p
・ Q ・ k) v (x, y) = (v0 ・ p ・ q + (v3-v2) (xi) ・ q + (v2-v0) (yj) ・ p) m // (p
(Q / k) (however, "//" is a division in which the result of a normal division operation is not an integer and is rounded to a nearby integer, and the precedence as an operator is equivalent to multiplication / division) , And p and q are 2 to the power of α and 2 to the power of β (α and β are positive integers), respectively. 6. A pixel sampling interval in an image is 1 in both the horizontal and vertical directions, and a sampling point exists on a point where both horizontal and vertical components of coordinates are integers. (i + p, j), (i, j + q), (i + p, j + q)
Take only values of the horizontal and vertical components of the motion vector at integer multiples of 1 / k (where i and j are integers, p and q are positive integers, k is 2 to the power of h0, and h0 is a negative And the horizontal and vertical components of the motion vector of each pixel are 1 / m
(Where m is 2 to the power of h1 and h1 is a non-negative integer), the representative points (i + p, j), (i, j + q), (i + p, j + q) by multiplying the horizontal and vertical components of the motion vector by k times (u1, v1), (u2, v2), (u3, v3) (where u1, v1, u2 , V2, u3, and v3 are integers), and the horizontal and vertical components of the motion vector of the pixel (x, y) are multiplied by m (u (x, y), v (x, y)) are
x, y, u (x, y) and v (x, y) are integers), u (x, y) = (u3pq + (u2-u3) (p-x + i) q + (u1 -u3) (q-y + j) ・ p) m
// (p ・ q ・ k) v (x, y) = (v0 ・ p ・ q + (v2-v3) (p-x + i) ・ q + (v1-v3) (q-y + j) ・ p ) m
// Represented by (p, q, k) (however, "//" is a division in which the result of a normal division operation is not an integer and is rounded to a nearby integer. And p and q are 2 to the power of α and 2 to the power of β (α and β are positive integers), respectively. 7. A sampling interval of pixels in an image is 1 in both horizontal and vertical directions, and a sampling point is located on a point where both horizontal and vertical components of coordinates are integers. (i, j), (i + p, j), (i, j + q), (i +
p, j + q), the horizontal and vertical components of the motion vector are 1
/ K only (where i and j are integers,
p and q are positive integers, k is 2 to the power of h0, and h0 is a non-negative integer), and the horizontal and vertical components of the motion vector of each pixel are 1 / m
(Where m is 2 to the power of h1 and h1 is a non-negative integer), the representative points (i, j), (i + p, j), (i, j) + q), (i + p, j +
q) are obtained by multiplying the horizontal and vertical components of the motion vector by k times (u0, v0), (u1, v1), (u2, v2), (u3,
v3) (where u0, v0, u1, v1, u2, v
2, u3 and v3 are integers), and the horizontal and vertical components of the motion vector of pixel (x, y) are multiplied by m (u (x, y), v (x, y))
x, y, u (x, y) and v (x, y) are integers), u (x, y) = ((u0 (p-x + i) + u1 (xi)) (q-y + j ) + (u2 (p-x + i) + u3 (x
-i)) (yj)) m // (pqqk) v (x, y) = ((v0 (p-x + i) + v1 (xi)) (q-y + j) + ( v2 (p-x + i) + v3 (x
-i)) (yj)) m // (p ・ q ・ k) (where `` // '' is a division that rounds the result of a normal division operation to a neighboring integer when the result is not an integer. , And the priority as an operator is equivalent to multiplication / division), and p and q are 2 to the power of α and 2 to the power of β (α and β are positive integers). Synthesis method. 8. The coordinates of pixels at the upper left, upper right, lower left and lower right corners of the image are (0,0), (r, 0), (0, s),
When represented by (r, s) (where r and s are positive integers), the horizontal and vertical positions of the motion vector at the points (0,0), (r, 0), (0, s) at the corners of the image Assuming that the vertical component takes an integer multiple of 1 / n (where n is a positive integer), these motion vectors are multiplied by n (u00, v00), (u01, v0).
1), (u02, v02) (where u00, v00, u01, v01,
u02, v02 are integers), and the points (i, j), (i + p, j), (i, j + q), (i + p, j + q)
The horizontal and vertical components of the motion vector are multiplied by k.
(u0, v0), (u1, v1), (u2, v2), (u3, v3) are given by u '(x, y) = (u00 · rs · s + (u01-u00) x · s + (u02− u00) y ・ r) k /// (r ・
s ・ n) v '(x, y) = (v00 ・ r ・ s + (v01-v00) x ・ s + (v02-v00) y ・ r) k /// (r ・
s ・ n) u0 = u '(i, j) v0 = v' (i, j) u1 = u '(i + p, j) v1 = v' (i + p, j) u2 = u '(i , j + q) v2 = v '(i, j + q) u3 = u' (i + p, j + q) v3 = v '(i + p, j + q) (where "/ //) is a division that rounds the result of a normal division operation to an adjacent integer when the result is not an integer. The precedence as an operator is equivalent to multiplication / division.) 3 points out of these 4 points and their movement 7. The method according to claim 1, wherein the vector is used as a representative point and a motion vector thereof. 9. The coordinates of the upper left, upper right, lower left and lower right pixels of the image are (0,0), (r, 0), (0, s),
When represented by (r, s) (where r and s are positive integers), the horizontal and vertical positions of the motion vector at the points (0,0), (r, 0), (r, s) at the corners of the image Assuming that the vertical component takes an integer multiple of 1 / n (where n is a positive integer), these motion vectors are multiplied by n (u00, v00), (u01, v0).
1), (u03, v03) (where u00, v00, u01, v01,
u03, v03 are integers), and the points (i, j), (i + p, j), (i, j + q), (i + p, j + q)
The horizontal and vertical components of the motion vector are multiplied by k.
(u0, v0), (u1, v1), (u2, v2), (u3, v3) are given by u ′ (x, y) = (u00 · rs · s + (u01−u00) x · s + (u03− u01) y ・ r) k /// (r ・
s ・ n) v '(x, y) = (v00 ・ r ・ s + (v01-v00) x ・ s + (v03-v01) y ・ r) k /// (r ・
s ・ n) u0 = u '(i, j) v0 = v' (i, j) u1 = u '(i + p, j) v1 = v' (i + p, j) u2 = u '(i , j + q) v2 = v '(i, j + q) u3 = u' (i + p, j + q) v3 = v '(i + p, j + q) (where "/ //) is a division that rounds the result of a normal division operation to an adjacent integer when the result is not an integer. The precedence as an operator is equivalent to multiplication / division.) 3 points out of these 4 points and their movement 7. The method according to claim 1, wherein the vector is used as a representative point and a motion vector thereof. 10. The coordinates of the upper left, upper right, lower left and lower right pixels of the image are (0,0), (r, 0), (0, s),
When represented by (r, s) (where r and s are positive integers), the horizontal and vertical components of the motion vector at points (0,0), (0, s), (r, s) are 1 / N
(Where n is a positive integer), and these motion vectors are multiplied by n. (U00, v00), (u02, v02), (u03, v0)
3) (where u00, v00, u02, v02, u03, and v03 are integers), the points (i, j), (i + p, j), (i, j + q), (i + p , j + q)
The horizontal and vertical components of the motion vector are multiplied by k.
(u0, v0), (u1, v1), (u2, v2), (u3, v3) are: u '(x, y) = (u00 · rs · (u03-u02) x · s + (u02− u00) y ・ r) k /// (r ・
s ・ n) v '(x, y) = (v00 ・ r ・ s + (v03-v02) x ・ s + (v02-v00) y ・ r) k /// (r ・
s ・ n) u0 = u '(i, j) v0 = v' (i, j) u1 = u '(i + p, j) v1 = v' (i + p, j) u2 = u '(i , j + q) v2 = v '(i, j + q) u3 = u' (i + p, j + q) v3 = v '(i + p, j + q) (where "/ //) is a division that rounds the result of a normal division operation to an adjacent integer when the result is not an integer. The precedence as an operator is equivalent to multiplication / division.) 3 points out of these 4 points and their movement 7. The method according to claim 1, wherein the vector is used as a representative point and a motion vector thereof. 11. The coordinates of the upper left, upper right, lower left, and lower right pixels of the image are (0,0), (r, 0), (0, s),
When represented by (r, s) (where r and s are positive integers), the horizontal and vertical components of the motion vector at points (r, 0), (0, s), (r, s) are 1 / N
(Where n is a positive integer), and these motion vectors are multiplied by n. (U01, v01), (u02, v02), (u03, v0)
3) (where u01, v01, u02, v02, u03, and v03 are integers), the points (i, j), (i + p, j), (i, j + q), (i + p , j + q)
The horizontal and vertical components of the motion vector are multiplied by k.
(u0, v0), (u1, v1), (u2, v2), (u3, v3) are u '(x, y) = (u03urs ・ (u01-u03) (rx) ・ s + ( u02-u03) (sy) ・
r) k /// (r ・ s ・ n) v '(x, y) = (v03 ・ rs ・ s + (v01-v03) (rx) ・ s + (v02-v03) (sy) ・
r) k /// (r ・ s ・ n) u0 = u '(i, j) v0 = v' (i, j) u1 = u '(i + p, j) v1 = v' (i + p , j) u2 = u '(i, j + q) v2 = v' (i, j + q) u3 = u '(i + p, j + q) v3 = v' (i + p, j + q (Where "///" is a division in which the result of ordinary division is rounded to the nearest integer when the result of the operation is not an integer, and the precedence as an operator is equivalent to multiplication / division.) 7. The method according to claim 1, wherein three of the points and their motion vectors are used as representative points and their motion vectors. 12. The coordinates of the upper left, upper right, lower left and lower right pixels of the image are (0,0), (r, 0), (0, s),
When represented by (r, s) (where r and s are positive integers), the motion vectors at points (0,0), (r, 0), (0, s), (r, s) Assuming that the horizontal and vertical components take an integer multiple of 1 / n (where n is a positive integer), these motion vectors are multiplied by n (u00, v00), (u01, v01),
(u02, v02), (u03, v03) (However, u00, v00, u0
U ′ (x, y) = ((sy) (u00 · (rx) + u01 · x) + y (u02 · (rx) +) using 1, v01, u02, v02, u03, and v03 are integers. u03 ・
x)) k /// (r ・ s ・ n) v '(x, y) = ((sy) (v00 ・ (rx) + v01 ・ x) + y (v02 ・ (rx) + v03 ・
x)) k /// (r ・ s ・ n) u0 = u '(i, j) v0 = v' (i, j) u1 = u '(i + p, j) v1 = v' (i + p, j) u2 = u '(i, j + q) v2 = v' (i, j + q) u3 = u '(i + p, j + q) v3 = v' (i + p, j + (u0, v0), (u1, v1), (u2, v2), (u3,
v3) (where "///" is a division in which the result of normal division is not an integer and is rounded to a nearby integer, and the precedence as an operator is equivalent to multiplication / division).
(i, j), (i + p, j), (i, j + q), (i + p, j + q) are characterized by using the horizontal and vertical components of the motion vector multiplied by k. The method for synthesizing an inter-frame predicted image according to claim 2 or 7. 13. An image which is divided by its diagonal 2
12. The method according to claim 1, wherein the image is divided into a plurality of right-angled triangles, and global motion compensation based on linear interpolation / extrapolation is independently performed on pixels included in each of the triangles. A method for synthesizing an inter-frame prediction image. 14. The method according to claim 8, wherein “///” is defined as an operation for rounding an operation result obtained by ordinary division to the nearest integer. 15. When "///" is a value obtained by adding 1/2 to an integer as a result of an operation performed by ordinary division, the value is defined as an operation of rounding this value toward zero. The method for synthesizing an inter-frame predicted image according to claim 14. 16. When "///" is a value obtained by adding 1/2 to an integer as a result of an operation by ordinary division, the value is defined as an operation of rounding this value away from 0. The method for synthesizing an inter-frame predicted image according to claim 14. 17. The expression “//” represents an operation result by ordinary division,
17. The method according to claim 1, wherein the method is defined as an operation of rounding to the nearest integer. 18. When "//" is a value obtained by adding 1/2 to an integer as a result of a normal division operation, the value is defined as an operation of rounding this value toward zero. A method for synthesizing an inter-frame prediction image according to claim 17. 19. When "//" is a value obtained by adding 1/2 to an integer as a result of a normal division operation, the value is defined as an operation of rounding this value away from zero. A method for synthesizing an inter-frame prediction image according to claim 17. 20. When "//" is a value obtained by adding 1/2 to an integer as a result of a normal division operation, the dividend is closer to 0 when the dividend is negative, and away from 0 when the dividend is positive. 18. The method for synthesizing an inter-frame predicted image according to claim 17, wherein the operation is defined as an operation of rounding to an image. 21. When "//" is a value obtained by adding 1/2 to an integer as a result of the normal division, the direction of the dividend is away from 0 if the dividend is negative, and the direction of approaching 0 if the dividend is positive. 18. The method for synthesizing an inter-frame predicted image according to claim 17, wherein the operation is defined as an operation of rounding to an image. 22. Assuming that r is the horizontal length of the image and s is the vertical length of the image, p is equal to or less than r, 2p is greater than r, q is equal to or less than s, and 22. The method according to claim 1, wherein 2q is larger than s. 23. Assuming that r is the horizontal length of the image and s is the vertical length of the image, p / 2 is smaller than r, p is not smaller than r, and q / 2 is smaller than s, 22. The method according to claim 1, wherein q is equal to or larger than s. 24. An image coding method and an image decoding method using the inter-frame prediction image synthesizing method according to claim 1. Description: 25. An image coding method using an inter-frame predicted image synthesizing method according to claim 1, wherein information relating to a motion vector at a representative point is directly encoded. 26. The method for synthesizing an inter-frame predicted image according to claim 1, wherein information on a motion vector of a representative point directly encoded as encoded data is reproduced and used. Image decoding method. 27. An image code using the method for synthesizing an inter-frame predicted image according to claim 8, wherein information relating to a motion vector at a corner point of the image is directly encoded. Method. 28. The apparatus according to claim 8, wherein information relating to a motion vector at a corner point of an image directly encoded as encoded data is reproduced and used.
An image decoding method using the method for synthesizing an inter-frame predicted image according to any one of Items 4 to 16.