JP3520845B2 - Image encoding apparatus and image encoding method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image coding and decoding method for applying global motion compensation based on linear interpolation / extrapolation or bilinear interpolation / extrapolation to an entire image.

[0002]

2. Description of the Related Art In high-efficiency coding of moving images, it is known that motion compensation utilizing the similarity between temporally adjacent frames has a great effect on information compression. The motion compensation method, which is the mainstream of the current image encoding technology, is H.261, which is an international standard of the moving image encoding method, MPEG.
1 is the block matching adopted in MPEG2. In this method, an image to be encoded is divided into a large number of blocks, and the motion vector is obtained for each block.

FIG. 1 shows a configuration example 10 of an H.261 encoder.
Indicates 0. H.261 adopts a hybrid coding system (interframe / intraframe adaptive coding system) that combines block matching and DCT (discrete cosine transform) as a coding system. The subtractor 102 outputs the input image (original image of the current frame) 101 and the output image 113 of the inter-frame / intra-frame coding switch 119.
The difference with (described later) is calculated, and the error image 103 is output.
This error image is converted into DCT coefficients by the DCT converter 104 and then quantized by the quantizer 105 to obtain quantized DC.
The T coefficient is 106. This quantized DCT count is output to the communication path as transmission information, and at the same time, used in the encoder to synthesize the inter-frame predicted image. The procedure of predictive image composition will be described below. The quantized DCT coefficient 106 described above is obtained by using the inverse quantizer 108 and the inverse DCT converter 109.
And becomes a decoded error image 110 (the same image as the error image reproduced on the receiving side). In addition, in the adder 111, an inter-frame / intra-frame coding changeover switch 119
Output image 113 (described later) is added to obtain a decoded image 112 of the current frame (the same image as the decoded image of the current frame reproduced on the receiving side). This image is temporarily stored in the frame memory 114 and delayed by the time for one frame. Therefore, at present, the frame memory 114 is outputting the decoded image 115 of the previous frame. The decoded image of the previous frame and the input image 101 of the current frame are input to the block matching unit 116, and the block matching processing is performed. In block matching, an image is divided into a plurality of blocks, and a portion of each block that most resembles the original image of the current frame is extracted from the decoded image of the previous frame to synthesize the predicted image 117 of the current frame. At this time, it is necessary to perform processing (motion estimation processing) for detecting how much each block has moved between the previous frame and the current frame. The motion vector for each block detected by the motion estimation process is the motion information 12
It is transmitted to the receiving side as 0. The receiving side can independently synthesize the same predicted image as that obtained on the transmitting side from the motion information and the decoded image of the previous frame. The predicted image 117 is input to the inter-frame / intra-frame coding changeover switch 119 together with the “0” signal 118. This switch, by selecting one of both inputs,
Switches between interframe coding and intraframe coding. When the predicted image 117 is selected (FIG. 2 shows this case), interframe coding is performed. on the other hand,
When the "0" signal is selected, the input image is DCT-encoded as it is and output to the communication path, so that intra-frame encoding is performed. In order for the receiving side to correctly obtain a decoded image, it is necessary for the transmitting side to know whether inter-frame coding or intra-frame coding has been performed. Therefore, the identification flag 121 is output to the communication path. Final H.261 encoded bitstream 123
Is a quantized DCT coefficient, a motion vector,
It is obtained by multiplexing the information of the intra-frame / inter-frame identification flag.

FIG. 2 shows an example of the configuration of a decoder 200 that receives the encoded bit stream output from the encoder shown in FIG. The received H.261 bit stream 217 is quantized DCT coefficient 201 and motion vector 2 by the separator 216.
02, an intra-frame / inter-frame identification flag 203 is separated. The quantized DCT coefficient 201 passes through the inverse quantizer 204 and the inverse DCT converter 205, and is decoded into the error image 20.
It becomes 6. This error image is added by the adder 207 between frames /
Output image 2 of intraframe coding changeover switch 214
15 is added and the decoded image 208 is output.
The inter-frame / intra-frame coding changeover switch operates according to the inter-frame / intra-frame coding identification flag 203.
Switch the output. The predicted image 212 used when performing inter-frame coding is synthesized in the predicted image synthesis unit 211. Here, a process of moving the position of each block in accordance with the received motion vector 202 is performed on the decoded image 210 of the previous frame stored in the frame memory 209. On the other hand, in the case of intraframe coding,
The interframe / intraframe coding switch is
The “0” signal 213 is output as it is.

Block matching is the most widely used motion compensation method at present, but when the entire image is enlarged, reduced or rotated, the motion vector must be transmitted to all blocks. However, there arises a problem that the coding efficiency is deteriorated. To solve this problem, global motion compensation is used to represent the motion vector field of the entire image using a small number of parameters (eg, M. Hotter, "Differential esti
mation of the globalmotion parameters zoom and pa
n ", Signal Processing, vol. 16, no. 3, pp.249-265,
Mar. 1989) has been proposed. This is the motion vector (ug (x, y), vg (x, y)) of the pixel (x, y) in the image.
To

[0006]

[Equation 1]

[0007]

[0008]

[Equation 2]

This is a system for performing motion compensation using this motion vector. Where a0-a5 and b0-
b7 is a motion parameter. When performing motion compensation,
The same predicted image must be obtained on the transmitting side and the receiving side. To this end, the sender sends to the receiver a0-a5 or
Although the values of b0 to b7 may be directly transmitted, there is also a method of transmitting the motion vectors of some representative points instead. Now, assume that the coordinates of the pixels at the upper left corner, upper right corner, lower left corner, and lower right corner of the image are represented by (0,0), (r, 0), (0, s), and (r, s), respectively. (However, r and s are positive integers). At this time, the horizontal and vertical components of the motion vectors of the representative points (0,0), (r, 0) and (0, s) are (ua, va) and (ub, v), respectively.
b), (uc, vc), (Equation 1) becomes

[0010]

[Equation 3]

It can be rewritten as This is
Instead of transmitting a0 to a5, ua, va, ub, vb, u
It means that the same function can be realized by transmitting c and vc. This state is shown in FIG. Original image of the current frame 3
02 and the reference image 301, global motion compensation is performed, and instead of the motion parameter, the representative points 303,
Motion vectors 306, 307, 308 of 304, 305
(At this time, the motion vector is defined by defining the point of the original image of the current frame as the starting point and the corresponding point of the reference image as the ending point). Similarly, four representative points (0,0), (r, 0), (0, s), (r,
horizontal and vertical components (ua, va), (u) of the motion vector of
b, vb), (uc, vc), (ud, vd)

[0012]

[Equation 4]

It can be rewritten as Therefore,
Instead of transmitting b0 to b7, ua, va, ub, vb, u
Similar functions can be realized by transmitting c, vc, ud, and vd. In this specification, the method using (Equation 1) will be referred to as global motion compensation based on linear interpolation / extrapolation, and the method using (Equation 2) will be referred to as global motion compensation based on bilinear interpolation / extrapolation.

In the linear form for transmitting the motion vector of the representative point
FIG. 4 shows a configuration example of the motion compensation processing unit 401 of the image encoder that employs the global motion compensation method based on extrapolation.
The same numbers as in FIG. 1 indicate the same things. By replacing the block matching unit 116 in FIG. 1 with the motion compensation processing unit 401, an image coding apparatus that performs global motion compensation can be configured. The global motion compensation unit 402 performs motion estimation related to global motion compensation between the decoded image 115 of the previous frame and the original image 101 of the current frame, and estimates the values of ua, va, ub, vb, uc, vc. It Information 403 about these values is transmitted as part of the motion information 120. The predicted image 404 for global motion compensation is combined using Equation 3 and supplied to the block matching unit 405. Here, motion compensation by block matching is performed between the predicted image of global motion compensation and the original image of the current frame, and the motion vector information 406 of the block and the final predicted image 117 are obtained. This motion vector information is multiplexed with the motion parameter information in the multiplexing unit 407 and output as motion information 120.

FIG. 5 shows an example of the configuration of the motion compensation processing section 501 different from that shown in FIG. The same numbers as in FIG. 1 indicate the same things. By replacing the block matching unit 116 in FIG. 1 with the motion compensation processing unit 501, an image coding apparatus that performs global motion compensation can be configured.
In this example, instead of applying block matching to the prediction image of global motion compensation, either global motion compensation or block matching is applied to each block. Between the decoded image 115 of the previous frame and the original image 101 of the current frame, the global motion compensation unit 502
And block matching unit 505 perform global motion compensation and block matching in parallel. The selection switch 508 is used for the prediction image 5 by the global motion compensation.
03 and the predicted image 506 by block matching, the optimum method is selected for each block. Motion vector 504 of representative point, motion vector 507 of each block, selection information 50 of global motion compensation / block matching
9 is multiplexed by the multiplexing unit 510 and output as motion information 120.

By introducing the global motion compensation described above, it is possible to represent the global motion of an image using a small number of parameters, and a higher information compression rate can be realized. However, on the other hand, the amount of processing in encoding and decoding increases as compared with the conventional method. In particular
The division shown in (Equation 3) and (Equation 4) becomes a large factor that complicates the processing.

[0017]

In the global motion compensation in which the motion vector field of the entire image is approximated by a small number of parameters, there is a problem that the amount of processing for synthesizing the predicted image increases. An object of the present invention is to reduce the amount of calculation by replacing the division process in global motion compensation with a binary shift calculation.

[0018]

The division processing can be realized by a shift operation by properly selecting the coordinates of a representative point when performing global motion compensation.

[0019]

BEST MODE FOR CARRYING OUT THE INVENTION In the following discussion, the sampling interval of pixels is set to 1 in both the horizontal and vertical directions, and the upper left corner of the image,
The coordinates of the upper right, lower left, and lower right pixels are (0,
0), (r, 0), (0, s), (r, s) (where r and s are positive integers).

Linear interpolation / extrapolation (affine transformation) or co-1
When performing motion compensation using the next interpolation / extrapolation (co-linear transformation), if the motion vector for each pixel is quantized,
It is possible to obtain effects such as prevention of mismatch and simplification of calculation (Japanese Patent Application No. 06-193970). In the following, it is assumed that the horizontal and vertical components of the pixel motion vector are integral multiples of 1 / m (m is a positive integer). Further, assuming that the global motion compensation using the motion vector of the representative point described in “Prior Art” is performed, the motion vector of each representative point is 1 / k.
(K is a positive integer). In this specification, the “pixel motion vector” refers to a motion vector used for actually synthesizing a predicted image when global motion compensation is performed. On the other hand, “representative point motion vector” means a parameter used to calculate a pixel motion vector. Therefore, there may be a case where the motion vector of the pixel and the motion vector of the representative point do not match even if they exist on the same coordinate due to a difference in quantization step size or the like.

First, the case of using linear interpolation / extrapolation will be described with reference to FIG. At this time, the representative points are not positioned at the corners of the image 601 as described in "Prior Art", and (i, j), (i + p, j), (i, j + q) Points 602, 603, and 604 (i, j, p, and q are integers). At this time, the points 602, 603, and 604 may exist inside or outside the image. Let (u0, v0), (u1, v1), and (u2, v2) be the horizontal and vertical components of the motion vector of the representative point multiplied by k, respectively.
0, v0, u1, v1, u2, v2 are integers), pixel (x, y)
M of the horizontal and vertical components of the motion vector of
(x, y), v (x, y)) can be expressed by the following formula (where x, y, u (x, y) and v (x, y) are integers).

[0022]

[Equation 5]   u (x, y) = (((u1-u0) (x-i) q + (u2-u0) (y-j) p + u0pq) m) // (pqk)   v (x, y) = (((v1-v0) (x-i) q + (v2-v0) (y-j) p + v0pq) m) // (pqk) ... (Equation 5)

However, "//" is a division in which the result of the usual division is not an integer and is rounded to the nearest integer, and the priority as an operator is equivalent to the multiplication / division.
To reduce the calculation error, it is desirable that the non-integer value be rounded to the nearest integer. At this time, 1 /
The rounding method of the value which added 2 is (1) round toward the direction of 0, (2) round toward the direction of 0, (3)
When the dividend is negative, it is rounded toward 0, and when it is positive, it is rounded away from 0 (divisor is always positive). (4) When the dividend is negative, it is moved away from 0. Rounding in the direction of approaching 0 (assuming that the divisor is always positive) can be considered. Among these (3)
Since (4) and (4) do not change the rounding direction regardless of whether the dividend is positive or negative, it is advantageous in terms of processing amount as much as there is no need to determine whether the dividend is positive or negative. High-speed processing using (3) is realized by the following equation.

[0024]

[Equation 6]

However, "#" is an integer division in which the fractional part is rounded down to the direction of 0, and is generally the easiest division type to be realized by a computer. Here, L and M are numbers for always keeping the dividend of division positive, and are sufficiently large positive integers. The term (pqk # 2) is used to round the division result to the nearest integer.

Although making the processing into an integer contributes to the reduction of the processing amount itself, here, p, q, and k are respectively raised to the power of 2 by α, β, h0 (α and β are positive integers, and h0 is negative). Is not an integer), the division of (Equation 5) can be realized by a shift operation of α + β + h0 bits, so that the processing amount in a computer or dedicated hardware can be greatly reduced. If m is the power of 2 (h1 is a non-negative integer, h1
<Α + β + h0), (Equation 6) is

[0027]

[Equation 7]

Can be rewritten as follows (“x << α” is x
Shift left by α bits and put 0 in the lower α bits,
"X >>α" shifts x to the right by α bits and puts 0 or 1 in the upper α bits (when x is a two's complement expression, when the most significant bit of x is 1, it is 1,0 Enter 0)
It means that the priority of these operators is intermediate between addition and subtraction and multiplication and division), and the operation can be further simplified.

When linear interpolation / extrapolation is used, (i +
(u3, v3) is obtained by multiplying the horizontal and vertical components of the motion vector of the representative point located at (p, j + q) by (u3, v3), and (Equation 5) is (i, j), (i + p, j) , (I + p, j + q) are the representative points,

[0030]

[Equation 8]   u (x, y) = (((u1-u0) (x-i) q + (u3-u1) (y-j) p + u0pq) m) // (pqk)   v (x, y) = (((v1-v0) (x-i) q + (v3-v1) (y-j) p + v0pq) m) // (pqk) ... (Equation 8)

If (i, j), (i, j + q), and (i + p, j + q) are the representative points,

[0032]

[Equation 9]   u (x, y) = (((u3-u2) (x-i) q + (u2-u0) (y-j) p + u0pq) m) // (pqk)   v (x, y) = (((v3-v2) (x-i) q + (v2-v0) (y-j) p + v0pq) m) // (pqk) ... (Equation 9)

(I + p, j), (i, j + q), (i + p, j +)
If q) is the representative point,

[0034]

[Equation 10]   u (x, y) = (((u2-u3) (i + p-x) q + (u1-u3) (j + q-y) p + u3pq) m) // (pqk)   v (x, y) = (((v2-v3) (i + p-x) q + (v1-v3) (j + q-y) p + v3pq) m) / (pqk) ... (Equation 10)

The processing amount can be similarly reduced by rewriting as follows and setting the values of p, q, k and m to the powers of 2 as positive integers.

When co-linear interpolation / extrapolation is used, representative points (i, j), (i + p, j), (i, j + q), (i + p, j + q) The horizontal and vertical components of each motion vector are multiplied by k (u0, v0), (u1, v1), (u2, v2), (u3, v3).
(U (x, y), v (x, y)) can be expressed by the following equation.

[0037]

[Equation 11]   u (x, y) = (((j + q-y) (i + p-x) u0 + (x-i) u1 + (y-j) ((i + p-x) u2 + (x-i) u3)) m) // (pqk)   v (x, y) = (((j + q-y) (i + p-x) v0 + (x-i) v1 + (y-j) ((i + p-x) v2 + (x-i) v3)) m) // (pqk)                                                             ... (Equation 11)

Also in this equation, the values of p, q, k and m are respectively set to the powers of 2 of α, β, h0 and h1,

[0039]

[Equation 12]

It can be rewritten as follows, and the processing amount can be reduced as in the above case.

In order to obtain the same global motion-compensated prediction image on the transmitting side and the receiving side, it is necessary to convey information about the motion vector of the representative point to the receiving side in some form. There is also a method of transmitting the motion vector of the representative point as it is, but also a method of transmitting the motion vector of the point at the corner of the image and calculating the motion vector of the representative point from this value. Regarding this method,
This will be described below.

First, the case where linear interpolation / extrapolation is used will be described. Three points (0,0), (r,
Assuming that the motion vectors of (0) and (0, s) can take only values of 1 / n integer multiples, these horizontal and vertical components are multiplied by n (u
It is assumed that 00, v00), (u01, v01), and (u02, v02) are transmitted. At this time, points (i, j), (i + p, j), (i, j +)
q), (i + p, j + q) horizontal of each motion vector,
The vertical component is multiplied by k (u0, v0), (u1, v
1), (u2, v2), (u3, v3)

[0043]

[Equation 13]

It is defined as However, u '(x, y), v'
(x, y) is a modification of (Equation 5),

[0045]

[Equation 14]   u '(x, y) = (((u01-u00) xs + (u02-u00) yr + u00rs) k) /// (rsn)   v '(x, y) = (((v01-v00) xs + (v02-v00) yr + v00rs) k) /// (rsn) ... (Equation 14)

It is defined as At this time, "///" is a division in which the result of the ordinary division is rounded to the nearest integer when the result is not an integer, and the priority as an operator is equivalent to the multiplication / division. (u0, v0), (u1, v1), (u2, v2), (u
If three points are selected from (3, v3) and global motion compensation is performed with those points as representative points, (0, 0), (r, 0), (0,
Global motion compensation, whose representative point is s), can be approximated. Of course, if p and q are set to be positive integer powers of 2 at this time, the processing can be simplified as described above. To reduce the calculation error, use "//
It is desirable that "/" round the non-integer value to the nearest integer. At this time, as the method for rounding the value obtained by adding 1/2 to the integer, the methods (1) to (4) described above can be considered.
However, compared to the case of (Equation 5) (calculation for each pixel),
(Equation 14) (Calculation only 3 times for one image) does not significantly affect the overall calculation amount even if the method of (1) or (2) is selected because the number of calculations is small. .

Even when three points different from the example of (Equation 13) are selected as the corner points of the image, the same processing can be realized by modifying Equations 8 to 10. In addition to the above example, if (u03, v03) is obtained by multiplying the horizontal and vertical components of the motion vector of the point (r, s) at the corner of the image by (u03, v03), (Equation 14) becomes (u00, v00 ), (U01, v01), (u03,
v03) is transmitted,

[0048]

[Equation 15]   u '(x, y) = (((u01-u00) xs + (u03-u01) yr + u00rs) k) /// (rsn)   v '(x, y) = (((v01-v00) xs + (v03-v01) yr + v00rs) k) // (rsn) ... (Equation 15)

And (u00, v00), (u02, v02), (u03, v
03) is transmitted,

[0050]

[Equation 16]   u '(x, y) = (((u03-u02) xs + (u02-u00) yr + u00rs) k) /// (rsn)   v '(x, y) = (((v03-v02) xs + (v02-v00) yr + v00rs) k) // (rsn) ... (Equation 16)

And (u01, v01), (u02, v02), (u03, v
03) is transmitted,

[0052]

[Equation 17]   u '(x, y) = (((u02-u03) (r-x) s + (u01-u03) (s-y) r + u03rs) k) /// (rsn)   v '(x, y) = (((v02-v03) (r-x) s + (v01-v03) (s-y) r + v03rs) k) /// (rsn) ... (Equation 17)

It is rewritten as

The same applies when co-primary interpolation / extrapolation is performed. Similar to the above, four representative points (0,0), (r,
0), (0, s), (r, s) motion vectors can take only values of 1 / n integer multiples, and the horizontal and vertical components are multiplied by n (u00, v00), (u01, v01) , (U02, v02), (u03,
v03) is transmitted. The representative point (i,
j), (i + p, j), (i, j + q), (i + p, j + q), which are the horizontal and vertical components of the motion vector multiplied by k (u0, v0), (u1, v1), (u2, v2), (u3, v3) are given by (Equation 13) as in the above. However, u '(x,
y), v '(x, y) is a modification of (Equation 11)

[0055]

[Equation 18]   u '(x, y) = (((s-y) ((r-x) u00 + xu01) + y ((r-x) u02 + xu03)) k) /// (rsn)   v '(x, y) = (((s-y) ((r-x) v00 + xv01) + y ((r-x) v02 + xv03)) k) ///(rsn)...(Equation 18)

Is defined as

The advantage of the method of transmitting the motion vector of the point at the corner of the image and performing the interpolation / extrapolation on the motion vector to find the motion vector of the representative point is to easily limit the range of the motion vector for each pixel. It is a point. For example, in colinear extrapolation given by (Equation 4), when (x, y) is a point in the image, the value of ug (x, y) is the maximum of ua, ub, uc, ud. It does not exceed the value or fall below the minimum value. Therefore, at the time of global motion estimation, ua, ub, u
If you add a constraint that the values of c and ud fall within a certain limit range (for example, within ± 32 pixels), keep the values of ug (x, y) within the same limit range for all pixels. (Of course, this also holds for vg (x, y)). This makes it possible to clarify the number of digits required for calculation, which is convenient for designing software or hardware. However, since the above discussion is for the case where all calculations are performed by floating point arithmetic, caution is required in the actual processing. The calculation (Equation 18) for calculating the motion vector of the representative point from the motion vector of the corner point of the image has rounding to the integer, so it is affected by calculation error.
It is necessary to consider the possibility that the motion vector obtained by (Equation 12) may fall out of the above-mentioned limited range. In particular, care should be taken when the representative point is located inside the image. This is because the motion vector is obtained by extrapolation for the pixels located outside the rectangle surrounded by the representative points, and thus the rounding error may be amplified. FIG. 7 shows an example in which the motion vector is obtained by extrapolation. Representative points 702, 703, 70 for the image 701
When global motion compensation is performed using Nos. 4,705, the motion vector is calculated by extrapolation in the shaded portion in the image. This is because the shaded area is outside the rectangle 706 surrounded by the representative points.

As a measure against this problem, a method of arranging four representative points so that the rectangle surrounded by the representative points includes the entire image is effective. This example is shown in FIG. Representative point 80
A rectangle 806 surrounded by 2, 803, 804 and 805 contains an image 801. In this way, since the motion vectors of all pixels are obtained by interpolation from the representative point, the effect of the rounding error at the representative point is not amplified in the image. Therefore, an error larger than the rounding error of the representative point does not occur in the image, and the upper limit of the error becomes clear. However, if the rectangle surrounded by the representative points is made too large, the range of values that the motion vector of the representative point can take becomes wide, and the number of digits required for the calculation increases, which is disadvantageous for implementation.

From the above discussion, it is desirable that the value of p is r or more and the value of q is s or more in order to reduce the influence of the rounding error. Even if p and q are smaller than r and s, respectively, it is desirable to take as large a value as possible. Further, it is desirable that the values of i and j are values such that the widest part in the image falls within the area surrounded by the representative points.

As described above, when the first-order interpolation / extrapolation is used for global motion compensation, each component of the motion vector of the pixel included in the rectangle surrounded by the four representative points is the representative point. Has the property that it can only take values between the maximum and minimum values of each component of the motion vector. On the other hand, when linear interpolation / extrapolation is used, the motion vectors of the pixels within the triangle surrounded by the three representative points have the same property. Therefore, when performing global motion compensation using linear interpolation / extrapolation, the motion vectors of the four corners of the image are transmitted, and global motion is independently performed for the two right triangles divided by the diagonal lines of the image. A method of performing motion compensation is effective. By doing so, the restrictions on the range of the motion vector for the four corner points can be applied to the motion vectors of all the pixels in the image as they are. At this time, i, j,
The values of p and q may differ between the two right triangles. Also, from the viewpoint of calculation error, in order to avoid the case of calculating the pixel motion vector by extrapolation,
It is preferable that the triangle surrounding the representative point includes a right-angled triangle that is the target of global motion compensation. An example of this is shown in FIG. Points 909, 903, 90 which are the four corners of the image 901
8,910 motion vectors are transmitted and the points 909,90
A right-angled triangle formed by 3,910 and a point 90
Global motion compensation is independently performed for each right-angled triangle formed by 9, 910 and 908. Therefore, if a constraint is placed on the range of the motion vector of the apex, the motion vectors of all the pixels in the image will also fall within this constraint range. The right-angled triangle formed by the points 909, 903, and 910 is the representative point 902.
The right-angled triangle formed by the points 909, 910, and 908 using 903 and 904 is the point 906 as a representative point.
907 and 908 are used. Each of the triangles formed by the representative points includes a right-angled triangle that is a target of global motion compensation. Therefore, the effect of the rounding error of the motion vector of the representative point is not amplified at the point in the image. In addition, in this example, the two triangles formed by the representative points are similar, but it is not always necessary.

[0061]

As described above, according to the present invention, it is possible to substitute the division processing in the predictive image composition processing of the global motion compensation by the shift operation, and it is possible to simplify the processing by software or dedicated hardware.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration example of an H.261 image encoder.

FIG. 2 is a diagram showing a configuration example of an H.261 image decoder.

FIG. 3 is a diagram showing an example of global motion compensation for transmitting a motion vector of a representative point.

[Fig. 4] Fig. 4 is a diagram illustrating a motion compensation processing unit of an image encoder that performs block matching on a predicted image for global motion compensation.

[Fig. 5] Fig. 5 is a diagram illustrating a motion compensation processing unit of an image encoder that selects global motion compensation and block matching for each block.

FIG. 6 is a diagram showing an example of arrangement of representative points for performing high-speed processing.

FIG. 7 is a diagram showing a region where a motion vector is obtained by extrapolation in an image.

FIG. 8 is a diagram showing a case where motion vectors of all pixels in an image are obtained by interpolation from motion vectors of representative points.

FIG. 9 is a diagram showing an example in which an image is divided into two right-angled triangles and global motion compensation is applied to each by interpolation from the motion vector of a representative point.

[Explanation of symbols]

100 ... Image encoder, 101 ... Input image, 102 ... Subtractor, 103 ... Error image, 104 ... DCT converter, 10
5 ... DCT coefficient quantizer, 106, 201 ... Quantized DC
T coefficient, 108, 204 ... DCT coefficient inverse quantizer, 10
9, 205 ... Inverse DCT converter, 110, 206 ... Decoding error image, 111, 207 ... Adder, 112 ... Decoded image of current frame, 113, 215 ... Output image of inter-frame / intra-frame coding changeover switch, 114 209 ...
Frame memory, 115, 210 ... Decoded image of previous frame, 116, 405, 505 ... Block matching unit,
117, 212 ... Predicted image of current frame, 118, 21
3 ... "0" signal, 119, 214 ... Interframe / intraframe coding changeover switch, 120, 202, 40
6, 507 ... Motion vector information, 121, 203 ... Inter-frame / intra-frame identification flag, 122, 407, 51
0 ... Multiplexer, 123 ... Transmission bit stream, 200
... Image decoder, 208 ... Output image, 211 ... Predictive image synthesizing unit, 216 ... Separator, 301 ... Reference image, 302 ...
Original image of the current frame, 303, 304, 305, 60
2,603,604,702,703,704,70
5,802,803,804,805,902,90
4, 906, 907 ... Representative points, 306, 307, 308
... Motion vector of representative point, 401, 501 ... Motion compensation processing unit for performing global motion compensation, 402, 502 ... Global motion compensation unit, 403, 504 ... Motion parameter,
404, 503 ... Predicted image for global motion compensation, 50
6 ... Predicted image by block matching, 508 ... Block matching / global motion compensation selection switch, 509 ... Block matching / global motion compensation selection information, 601, 701, 801, 901 ... Global motion compensation target image, 706 ... Rectangle surrounded by representative points, 903, 908 ... Points at corners of image and points that also serve as representative points, 909, 910 ... Points at corners of image.

Claims (16)

(57) [Claims] 1. A frame memory for storing a decoded image obtained by decoding an already encoded frame, and motion estimation by global motion compensation between the decoded image and an input image of the current frame, and the current frame. In the image coding apparatus having the global motion compensation unit for synthesizing the predicted image of the present invention, the global motion compensation unit is a motion vector of a current frame corner point obtained by motion estimation by global motion compensation, and the pixel interval is 1. The above-mentioned corner points are coordinates (0,0), (r, 0), and (0,
s), the coordinates (i, j), (i + p, j), and
means for obtaining motion vectors of three representative points having (i, j + q) (where r, s, i and j are integers, p and q values are powers of 2 and positive integers), An image code having means for obtaining a motion vector for each pixel of the predicted image from a motion vector of a representative point, and means for synthesizing the predicted image from the motion vector for each pixel and the decoded image Device. 2. The image coding apparatus according to claim 1, wherein the means for obtaining the motion vector of the representative point obtains the motion vector of the representative point by linear interpolation / extrapolation from the motion vector of the frame corner point. An image coding apparatus, wherein the means for obtaining the motion vector for each pixel obtains the motion vector for each pixel by linear interpolation / extrapolation from the motion vector of the representative point. 3. A frame memory for storing a decoded image obtained by decoding an already encoded frame, and motion estimation by global motion compensation between the decoded image and an input image of the current frame, and the current frame. In the image coding apparatus having the global motion compensation unit for synthesizing the predicted image of the present invention, the global motion compensation unit is a motion vector of a current frame corner point obtained by motion estimation by global motion compensation, and the pixel interval is 1. The above-mentioned corner points are coordinates (0,0), (r, 0), and (0,
s), the coordinates (i, j), (i + p, j), and
means for obtaining motion vectors of three representative points having (i, j + q) (where r, s, i and j are integers, p and q values are powers of 2 and positive integers), And a means for obtaining a motion vector for each pixel of the predicted image from a motion vector of a representative point, and a means for synthesizing the predicted image from the motion vector for each pixel and the decoded image, The horizontal and vertical components of the vector are integer multiples of 1 / n, and the motion vectors of the above corner points are multiplied by n to obtain (u00, v00), (u01, v01), (u02, v02), respectively Horizontal and vertical components of the motion vector of 1 / m
Is multiplied by an integer, and the horizontal and vertical components of the motion vector of the pixel having the coordinates (x, y) are multiplied by m to obtain (u (x, y), v
(x, y)), and the horizontal and vertical components of the motion vector of the above representative points (i, j), (i + p, j), (i, j + q) are integer multiples of 1 / k, When the horizontal and vertical components of the motion vector of each representative point are multiplied by k to be (u0, v0), (u1, v1), (u2, v2), the means for obtaining the motion vector of the representative point is u '(x, y) = ((u00rs + (u01-u00) xs + (u02-u00) yr) k) /// (rsn) v' (x, y) = ((v00rs + (v01-v00) xs + ( v02-v00) yr) k) /// (rsn) u0 = u '(i, j), v0 = v' (i, j) u1 = u '(i + p, j), v1 = v' ( i + p, j) u2 = u '(i, j + q), v2 = v' (i, j + q) to convert the above (u0, v0), (u1, v1), (u2, v2) The means for obtaining the motion vector for each pixel is u (x, y) = ((u0pq + (u1-u0) (xi) q + (u2-u0) (yj) p) m) // (pqk) v (x, y) = ((v0pq + (v1-v0) (xi) q + (v2-v0) (yj) p) m) // (pqk) by the above (u (x, y), v (x, y )) Is calculated. (However, u00, v00, u01, v01,
u02, v02, u (x, y), v (x, y), u0, v0, u1, v1,
u2, v2 are integers, k, m, n are powers of 2, and "///" and "//" are rounded to neighboring integers when the result of ordinary division is not an integer. By division,
(The precedence as an operator is equivalent to multiplication and division.) 4. The image coding apparatus according to claim 3, wherein the calculation result by the calculation of “///” and “//” is 1 for an integer.
The rounding method in the case of adding / 2 is (1) round to an integer that moves away from 0, (2) round to the direction of 0, (3) round to the direction of 0 when the dividend is negative, and the dividend is If positive, round toward 0, or (4)
An image coding apparatus, wherein when the dividend is positive, the rounding is performed in a direction away from 0, and when the dividend is negative, the rounding is performed in a direction toward 0. 5. A frame memory for storing a decoded image obtained by decoding an already encoded frame, and motion estimation by global motion compensation between the decoded image and the input image of the current frame, and the current frame. In the image coding apparatus having the global motion compensation unit for synthesizing the predicted image of the present invention, the global motion compensation unit is a motion vector of a current frame corner point obtained by motion estimation by global motion compensation, and the pixel interval is 1. The above corner points are coordinates (0,0), (r, 0), (0, s) and
From the one with (r, s), the coordinates (i, j), (i + p, j),
A means for obtaining motion vectors of four representative points having (i, j + q) and (i + p, j + q) (provided that r, s, i and j are integers, and values of p and q are 2) a means for obtaining a motion vector for each pixel of the predicted image from the motion vector of the representative point, and a means for synthesizing the predicted image from the motion vector for each pixel and the decoded image An image coding apparatus comprising: 6. The image encoding device according to claim 5, wherein the means for obtaining the motion vector of the representative point is a motion vector of the representative point by bilinear interpolation / extrapolation from the motion vector of the frame corner point. The image coding apparatus is characterized in that the means for obtaining the motion vector for each pixel obtains the motion vector for each pixel by bilinear interpolation / extrapolation from the motion vector of the representative point. 7. A frame memory for storing a decoded image obtained by decoding an already encoded frame, and motion estimation by global motion compensation between the decoded image and an input image of the current frame, and the current frame. In the image coding apparatus having the global motion compensation unit for synthesizing the predicted image of the present invention, the global motion compensation unit is a motion vector of a current frame corner point obtained by motion estimation by global motion compensation, and the pixel interval is 1. The above corner points are coordinates (0,0), (r, 0), (0, s) and
From the one with (r, s), the coordinates (i, j), (i + p, j),
A means for obtaining motion vectors of four representative points having (i, j + q) and (i + p, j + q) (provided that r, s, i and j are integers, and values of p and q are 2) a means for obtaining a motion vector for each pixel of the predicted image from the motion vector of the representative point, and a means for synthesizing the predicted image from the motion vector for each pixel and the decoded image And the horizontal and vertical components of the motion vector of the corner point are integer multiples of 1 / n, and n times the motion vector of the corner point are (u00, v00), (u01, v01), (u02, v02), (u0
3, v03), and the horizontal and vertical components of the motion vector for each pixel are 1 / m
Of the horizontal motion vector of pixel (x, y)
The vertical component multiplied by m is (u (x, y), v (x, y)), and the representative points (i, j), (i + p, j), (i, j + q) and (i
+ P, j + q) motion vector horizontal / vertical components are 1 /
Taking an integer multiple of k and multiplying the horizontal and vertical components of the motion vector of each representative point by k, (u0, v0), (u1, v1), (u2,
v2) (u3, v3), the means for obtaining the motion vector of the representative point is u '(x, y) = (((sy) ((rx) u00 + xu01) + y ((rx) u02 + xu03)) k) // (rsn) v '(x, y) = (((sy) ((rx) v00 + xv01) + y ((rx) v02 + xv03)) k) /// (rsn) u0 = u '(i, j), v0 = v' (i, j) u1 = u '(i + p, j), v1 = v' (i + p, j) u2 = u '( i, j + q), v2 = v '(i, j + q) u3 = u' (i + p, j + q), v3 = v '(i + p, j + q) , v0), (u1, v1), (u2, v2), (u
3, v3) and the motion vector for each pixel is u (x, y) = (((j + qy) ((i + px) u0 + (xi) u1 + (yj) ((i + px) u2 + (xi) u3)) m) // (pqk) v (x, y) = (((j + qy) ((i + px) v0 + (xi) v1 + (yj) ((i + px) v2 + (xi) v3)) m) // (pqk) is used to obtain the above (u (x, y), v (x, y)). An image coding apparatus (however, u00, v00, u01) , v01,
u02, v02, u03, v03, u (x, y), v (x, y), u0, v
0, u1, v1, u2, v2, u3, and v3 are integers, k, m,
n is an integer power of 2, and "//" and "///" are divisions that round the result to the nearest integer when the result of ordinary division is not an integer. The priority as an operator is multiplication / division. Equivalent to). 8. The image coding apparatus according to claim 7, wherein the calculation result of the calculation of “///” and “//” is 1 for an integer.
The rounding method in the case of adding / 2 is (1) round to an integer that moves away from 0, (2) round to the direction of 0, (3) round to the direction of 0 when the dividend is negative, and the dividend is If positive, round toward 0, or (4)
An image coding apparatus, wherein when the dividend is positive, the rounding is performed in a direction away from 0, and when the dividend is negative, the rounding is performed in a direction toward 0. 9. An image encoding method for performing motion estimation by global motion compensation between a decoded image obtained by decoding an already encoded frame and an input image of the current frame, and synthesizing a predicted image of the current frame. In the motion vector of the corner point of the current frame obtained by the motion estimation, the corner point has coordinates (0,0), (r, 0), and (0,0, s), the motion vectors of three representative points having coordinates (i, j), (i + p, j), and (i, j + q) are obtained (provided that r, s, i, and j is an integer, p and q are powers of positive integers of 2, and the motion vector for each pixel of the predicted image is obtained from the motion vector of the representative point. The motion vector for each pixel and the decoded image An image coding method comprising synthesizing the above predicted image from and. 10. The image coding method according to claim 9, wherein the motion vector of the representative point is obtained by linear interpolation / extrapolation from the motion vector of the frame corner point, and the motion vector of each pixel is An image coding method characterized by linearly interpolating / extrapolating from a motion vector of a representative point. 11. An image encoding method for performing motion estimation by global motion compensation between a decoded image obtained by decoding an already encoded frame and an input image of the current frame, and synthesizing a predicted image of the current frame. In the motion vector of the corner point of the current frame obtained by the motion estimation, the corner point has coordinates (0,0), (r, 0), and (0,0, s), the motion vectors of three representative points having coordinates (i, j), (i + p, j), and (i, j + q) are obtained (provided that r, s, i, and j is an integer, p and q are powers of positive integers of 2, and the motion vector for each pixel of the predicted image is obtained from the motion vector of the representative point, and the motion vector for each pixel and the decoded image are obtained. And the predicted image is synthesized from, and the horizontal and vertical components of the motion vector of the corner point are 1 / Take an integer multiple of n and multiply the above corner point motion vector by n.
(u00, v00), (u01, v01), (u02, v02), and the horizontal and vertical components of the motion vector for each pixel are 1 / m
Of the horizontal motion vector of pixel (x, y)
The vertical component multiplied by m is (u (x, y), v (x, y)), and the above representative points (i, j), (i + p, j), and (i, j + q) The horizontal / vertical components of the motion vector take an integer multiple of 1 / k, and the horizontal / vertical components of the motion vector at each representative point are multiplied by (u0, v0), (u1, v1), (u2, v2). ), The motion vector of the representative point is u '(x, y) = ((u00rs + (u01-u00) xs + (u02-u00) yr) k) /// (rsn) v' (x, y) = ((v00rs + (v01-v00) xs + (v02-v00) yr) k) /// (rsn) u0 = u '(i, j), v0 = v' (i, j) u1 = u ' (i + p, j), v1 = v '(i + p, j) u2 = u' (i, j + q), v2 = v '(i, j + q) The above (u0, v0 ), (U1, v1), (u2, v2)
Then, the motion vector for each pixel is u (x, y) = ((u0pq + (u1-u0) (xi) q + (u2-u0) (yj) p) m) / (pqk) v (x, y) = ((v0pq + (v1-v0) (xi) q + (v2-v0) (yj) p) m) / (pqk) The above (u (x, y), v (x, y)) Image coding method (provided that u00, v00, u
01, v01, u02, v02, u (x, y), v (x, y), u0, v0,
u1, v1, u2, and v2 are integers, k, m, and n are powers of 2, and "///" and "//" are adjacent if the result of ordinary division is not an integer. Rounded to an integer and the operator precedence is equivalent to multiplication and division). 12. The image coding method according to claim 11, wherein the calculation result by the calculation of “///” and “//” is 1 for an integer.
The rounding method in the case of adding / 2 is (1) round to an integer that moves away from 0, (2) round to the direction of 0, (3) round to the direction of 0 when the dividend is negative, and the dividend is If positive, round toward 0, or (4)
The image coding method is characterized in that when the dividend is positive, it is rounded in a direction away from 0, and when the dividend is negative, it is rounded in a direction toward 0. 13. An image encoding method for performing motion estimation by global motion compensation between a decoded image obtained by decoding an already encoded frame and an input image of the current frame, and synthesizing the predicted image of the current frame. In the motion vector of the corner point of the current frame obtained by the motion estimation, the corner point has coordinates (0,0), (r, 0), (0, s ) And (r, s), the coordinates (i, j), (i + p, j), (i, j + q) and (i +
p, j + q) to obtain the motion vectors of four representative points (where r, s, i and j are integers, and the values of p and q are powers of two positive integers). An image coding method, wherein a motion vector for each pixel of the predicted image is obtained from the motion vector, and the predicted image is synthesized from the motion vector for each pixel and the decoded image. 14. The image coding method according to claim 13, wherein the motion vector of the representative point is obtained from the motion vector of the frame corner point by bilinear interpolation / extrapolation, and the motion vector of each pixel is An image coding method, characterized in that the motion vector of the representative point is obtained by co-linear interpolation / extrapolation. 15. An image encoding method for performing motion estimation by global motion compensation between a decoded image obtained by decoding an already encoded frame and an input image of the current frame, and synthesizing a predicted image of the current frame. In the motion vector of the corner point of the current frame obtained by the motion estimation, the corner point has coordinates (0,0), (r, 0), (0, s ) And (r, s), the coordinates (i, j), (i + p, j), (i, j + q) and (i +
p, j + q) to obtain the motion vectors of four representative points (where r, s, i and j are integers, and the values of p and q are powers of two positive integers). A motion vector for each pixel of the predicted image is obtained from the motion vector, the predicted image is synthesized from the motion vector for each pixel and the decoded image, and the horizontal and vertical components of the motion vector of the corner point are 1 / n. Take the integer multiples and multiply the corner point motion vectors by n times (u00, v00), (u01, v0).
1), (u02, v02), (u03, v03), and the horizontal and vertical components of the motion vector for each pixel are 1 / m
Of the horizontal motion vector of pixel (x, y)
The vertical component multiplied by m is (u (x, y), v (x, y)), and the representative points (i, j), (i + p, j), (i, j + q) and (i
+ P, j + q) motion vector horizontal / vertical components are 1 /
Taking an integer multiple of k and multiplying the horizontal and vertical components of the motion vector of each representative point by k, (u0, v0), (u1, v1), (u2,
v2) (u3, v3), the motion vector of the representative point is u '(x, y) = (((sy) ((rx) u00 + xu01) + y ((rx) u02 + xu03) ) k) /// (rsn) v '(x, y) = (((sy) ((rx) v00 + xv01) + y ((rx) v02 + xv03)) k) // (rsn) u0 = u '(i, j), v0 = v' (i, j) u1 = u '(i + p, j), v1 = v' (i + p, j) u2 = u '(i, j + q), v2 = v '(i, j + q) u3 = u' (i + p, j + q), v3 = v '(i + p, j + q) (u0, v0) , (U1, v1), (u2, v
2) and (u3, v3), the motion vector for each pixel is u (x, y) = (((j + qy) ((i + px) u0 + (xi) u1 + (yj) ((i + px) u2 + (xi) u3)) m) // (pqk) v (x, y) = (((j + qy) ((i + px) v0 + (xi) v1 + (yj) ((i + px ) v2 + (xi) v3)) m) // (pqk) is used to obtain the above (u (x, y), v (x, y)) image coding method (however, u00, v00, u
01, v01, u02, v02, u03, v03, u (x, y), v (x,
y), u0, v0, u1, v1, u2, v2, u3, and v3 are integers, k, m, and n are integer powers of 2, and "//" and "///"
Is a division in which the result of normal division is rounded to the nearest integer when the result is not an integer, and the priority as an operator is equivalent to multiplication and division.) 16. The image coding method according to claim 15, wherein the calculation result by the calculation of “///” and “//” is 1 for an integer.
The rounding method in the case of adding / 2 is (1) round to an integer that moves away from 0, (2) round to the direction of 0, (3) round to the direction of 0 when the dividend is negative, and the dividend is If positive, round toward 0, or (4)
The image coding method is characterized in that when the dividend is positive, it is rounded in a direction away from 0, and when the dividend is negative, it is rounded in a direction toward 0.