JPH09266574A - Image encoding device and image decoding device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To transmit and store an image of excellent picture quality even at a low encoding rate by providing global movement compensation which is small in resolution deterioration and facilitates matching between predicted images on an encoding side and a decoding side. SOLUTION: A local movement compensation part 409 performs macroblock movement compensation for an image which is one frame precedent. An image in a template memory 405 which is updated in longer cycles. on the other hand, is processed by global movement compensation by affine transformation through a representative vector interpolation movement compensation part 404. A selector 407 selects an optimum prediction mode in macroblock units according to an input instruction. Consequently, global movement compensation is performed by interpolating moving vectors, so arithmetic precision can be held high and the predicted images on the encoding and decoding sides can easily be matched with each other.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image coding apparatus and an image decoding apparatus for transmitting and storing an image with a small coding amount.

[0002]

2. Description of the Related Art If the movement of the entire screen caused by swaying the camera from side to side or zooming can be compensated, it is expected that the image coding efficiency will be improved. As a conventional high-efficiency image coding method using such motion compensation for the entire screen, Kazuto Uekura, Yutaka Watanabe: "Global motion compensation method in moving image coding", IEICE
B-1, Vol. J76-BI, No. 12, pp. 944-952, 1993.

In this method, the movement of the entire screen of the image caused by the pan / zoom operation of the camera is estimated and compensated.
Hereinafter, motion compensation of the entire screen is called global motion compensation. FIG. 5 shows a block diagram of an image coding apparatus that executes this conventional motion compensation method. In FIG. 5, 501 is a residual coding unit, 502 is a residual decoding unit, 503 is a frame memory, 504 is a motion prediction unit 1, 505 is a three-parameter prediction unit, 506 is a global motion compensation unit, and 507 is a local motion. Compensation units 1 and 508 are motion prediction units 2, 509 are motion compensation units 2, 510 are comparators, 511 are difference units, 512 is an adder, and 513 is a motion information multiplexing unit.

For easy understanding, 501, 50
2, 512, 503, 504, 507, and 511 are MPEG1 ("A Video Compression Standard for Multimedia Applications" Le Gall, D .: "MPEG: A Video Compression Standa
rd for Multimedia Applications ", Trans. ACM, 199
1, described in April)).

An image coding apparatus and a decoding apparatus which will be described later as embodiments of the present invention are also apparatuses based on MPEG1. Therefore, the residual encoding unit 501 is composed of an MPEG1-compliant digital cosine transform (DCT), a quantizer, and a Huffman encoder. Also, the residual decoding unit is M
It is composed of a PEG1 compliant inverse quantizer and a digital cosine inverse converter. In this conventional example, the motion vector is obtained by block matching, as in the embodiment described later. This is shown in FIG. FIG. 6 is a diagram for explaining block matching common to a conventional example and an embodiment described later.

In FIG. 6, a region R is a block region of 16 pixels × 16 pixels. The motion prediction unit 1 (504) performs block matching on the input image one frame before as a reference image, and the motion prediction unit 2 (508) uses a later-described global motion-compensated image as a reference image. In block matching, the sum of absolute error values shown in Equation 1 is calculated, and as shown in Equation 2, the deviation (p, q) that results in the minimum absolute error value.
Is the measured motion vector of the block. Equation 1
In, gt (x, y) represents the luminance value at the frame t and the screen position (x, y). And Equation 3
As shown in, the deviation with the smallest value is searched to obtain the original estimation (p, q). This (p, q) is obtained with 1/2 pixel precision.

[0007]

[Equation 1]

[0008]

[Equation 2]

In MPEG1, a motion vector is obtained between the decoding result one frame before and the current input frame,
Motion compensation is performed by parallel movement of 16 × 16 blocks (hereinafter referred to as macroblocks). This is executed by the local motion compensation unit 1 (507). In addition to this, in the conventional example, the lateral motion compensation of the camera and the global motion compensation by the zoom are realized.

Now, assuming that the zoom coefficient of the entire screen is Z, the translation amount is H0 horizontally, V0 vertically, and the zoom center is (x0, y0),
The global motion amount at the arbitrary position (x, y) of the entire screen in the input image can be expressed by Equations 3 and 4.

[0011]

(Equation 3)

[0012]

(Equation 4)

Further, assuming that the center position of the macroblock in the j-th row and the i-th column is (xij, yij), the motion vector (uij, vij) at the center of the macroblock can be expressed as the formula 7 from the definitions of the formulas 5 and 6.

[0014]

(Equation 5)

[0015]

(Equation 6)

[0016]

(Equation 7)

In the conventional example, the motion vector of two or more macroblocks is taken out from the output of the motion prediction unit 1 (504), and the equation 8 is executed by the three-parameter prediction unit (505).

Expression 8 is a calculation expression that minimizes the sum of squared errors between the motion vector observed in each macroblock and the predicted motion vector.

P in the equation 8 is a real symmetric matrix obtained from the position of the macro block taken out as shown in the equation 9, and can be calculated in advance.

[0020]

(Equation 8)

[0021]

[Equation 9]

This calculated global parameter (Z,
H, V) is used to perform the transformation shown in Expression 3 by the global motion compensation unit (506). This transformation is performed at each pixel position in the input image.
For (x, y), calculate the motion vector according to Equation 3,
This is executed by obtaining the pixel value of the pixel position indicated by the motion vector obtained from the frame memory (503). The motion vector for global motion compensation is obtained with 1/4 pixel accuracy, and is obtained as linear interpolation of 4 pixel values at neighboring integer positions. Further, the motion prediction unit 2 (508) performs motion prediction on a macroblock basis on the transformed image. The comparator (510) is an MPEG unit for each MPEG.
A prediction method with a small error is selected from the macroblock motion-compensated image similar to 1 and the macroblock motion-compensated image after global motion compensation of the previous frame.

In FIG. 5, the motion vector information includes the selection result SW in macroblock units, the motion vector mv1 or mv2,
Then, the global parameter g is set to the motion information multiplexing unit (5
It is multiplexed and sent in 13). In this conventional example, the motion vector for each block calculated by the block matching method is used, and the parameters of the entire screen are calculated only by a simple product-sum operation without performing iterative calculation. When applying global motion compensation to video coding,
An adaptive ON / OFF control method for each block is proposed.

[0024]

[Problems to be Solved by the Invention]

[Problem 1] In the above-described conventional example, it is possible to compensate for the movement caused by horizontal and vertical swinging and zooming of the camera. However, the above-described configuration of global motion compensation has a problem that the spatial resolution of the predicted image is reduced for each frame unless residual coding is performed.

The pixel-based motion vector obtained by global motion compensation is generally not an integer value. Therefore, the pixel value is calculated from the neighboring pixel value by interpolation. As a result, unless the pixel value is updated by residual coding,
The spatial resolution of the predicted image decreases each time the frame is updated. If residual coding is performed, high resolution can be maintained, but the code amount for that purpose increases. In order to code a moving image with high efficiency by global motion compensation, a motion compensation method with less resolution reduction is required.

[Problem 2] In predictive image encoding, the predictive images must match on the encoding side and the decoding side. In the above-mentioned conventional example, the motion vector for global motion compensation has to match at the pixel position with ¼ pixel precision. For this reason, the encoding accuracy of the term Z relating to the zoom of Expression 3 needs to be about 1/4 pixel accuracy at each pixel position. However, this accuracy depends on the size of the image because it is a multiplier related to the position. This is not desirable considering future expansion of the encoder. In addition, the address calculation for global motion compensation requires high calculation accuracy as the image becomes larger.

[0027]

A first aspect of the present invention is directed to the first aspect, in which a frame sequence of an image is input, and a decoded frame temporally before and after a frame to be encoded is referred to for a short period of time. A short-term frame memory for storing as a frame; a local motion vector detecting means for detecting a local correspondence between the encoding target frame and the short-term reference frame as a local motion vector; Short-term motion compensation means for reading an image from a short-term frame memory to generate a predicted image, and a decoded frame having a longer update interval than the short-term reference frame for a frame to be encoded is stored as a long-term reference frame. Correspondence of all screens between the long-term frame memory, the encoding target frame, and the long-term reference frame is arbitrary on the screen. Full-screen motion detection means for detecting a motion vector of a position as a parameter of a polynomial function having the position as a variable, and full-screen motion for generating a predicted image by reading an image from the long-term frame memory using the parameter of the polynomial function An image coding apparatus, which comprises a compensating means, adaptively selects the short-term motion compensating means and the full-screen motion compensating means to perform predictive coding, and a coded image as an input to a frame to be coded. On the other hand, a short-term frame memory that stores decoded frames that are temporally before and after in time as a short-term reference frame, and a local correspondence between the frame to be coded and the short-term reference frame is decoded from the input of a local motion vector. Local motion vector decoding means for converting the image, and reading an image from the short-term frame memory using the local motion vector Short-term motion compensation means for generating a prediction image, a long-term frame memory for storing a decoded frame having a longer update interval than the short-term reference frame for a frame to be encoded as a long-term reference frame, and the encoding A full-screen motion decoding means for decoding the correspondence between the target frame and the long-term reference frame from the input as a parameter of a polynomial function whose position is a variable and which is a motion vector at an arbitrary position on the screen; Full-screen motion compensating means for reading out an image from the long-term frame memory using a parameter of a polynomial function to generate a predicted image, and selecting the short-term motion compensating means and the full-screen motion compensating means in accordance with an instruction from an input. An image decoding device for performing predictive decoding.

A second aspect of the present invention is directed to the problem 2, wherein a frame memory that receives a frame sequence of an image and stores a decoded frame temporally before and after a frame to be encoded as a reference frame is provided. , Full-screen motion detection means for detecting the correspondence of the entire screen between the encoding target frame and the reference frame as the motion vector of the representative point on the screen, and the motion vector of the representative point on the screen with a predetermined accuracy. An image coding apparatus comprising: a representative point motion vector coding means for coding; and a full-screen motion compensating means for generating a predicted image by interpolating the motion vector of the representative point. A frame memory that stores a decoded frame that is temporally before and after a frame to be encoded as a reference frame, using the encoded image as an input; Representative point motion vector decoding means for decoding the motion vector of the representative point which is the output of the plane motion detecting means, and the predicted image by interpolating the motion vector of the representative point which is the output of the full-screen motion detecting means. And a full-screen motion compensating means for generating

A third aspect of the present invention is directed to the problem 2, wherein a frame memory that receives a frame sequence of an image and stores a decoded frame temporally before and after the frame to be encoded as a reference frame is provided. , Full-screen motion detection means for detecting the correspondence of the entire screen between the encoding target frame and the reference frame as a motion vector of a representative point on the screen, and a motion vector of the representative point on the screen as a difference vector Representative point motion vector difference encoding means for encoding with the accuracy of 1., and full-screen deforming means for deforming an image by switching the operation of the image deforming operation according to the motion vector difference of the representative points. An image encoding device and an encoded image as an input, and a decoded frame temporally before and after the decoding target frame as a reference frame A frame memory to be memorized, a representative point motion vector difference decoding means for decoding a motion vector difference of a representative point, and a motion vector difference of the representative point to switch the operation of the image deformation calculation to deform the image. An image decoding apparatus comprising means.

[0030]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first invention is to solve the above-mentioned problem 1, in which a frame sequence of an image is input, and a decoded frame temporally before and after a frame to be coded is referred to for a short period of time. A short-term frame memory for storing as a frame; a local motion vector detecting means for detecting a local correspondence between the encoding target frame and the short-term reference frame as a local motion vector; Short-term motion compensation means for reading an image from a short-term frame memory to generate a predicted image, and a decoded frame having a longer update interval than the short-term reference frame for a frame to be encoded is stored as a long-term reference frame. Correspondence of all screens between the long-term frame memory, the encoding target frame, and the long-term reference frame is arbitrary on the screen. Full-screen motion detection means for detecting a motion vector of a position as a parameter of a polynomial function having the position as a variable, and full-screen motion for generating a predicted image by reading an image from the long-term frame memory using the parameter of the polynomial function An image coding apparatus comprising a compensating means, and adaptively selecting the short-term motion compensating means and the full-screen motion compensating means to perform predictive coding.

Further, a short-term frame memory that stores a decoded frame that is temporally before and after the encoded frame as a short-term reference frame with the encoded image as an input, the encoded frame, and the encoded frame. A local motion vector decoding means for decoding a local motion vector from an input for a local correspondence between short-term reference frames, and an image is read from the short-term frame memory using the local motion vector to generate a predicted image. Short-term motion compensation means, a long-term frame memory that stores a decoded frame that is updated with respect to the encoding target frame and has a longer interval than the short-term reference frame as a long-term reference frame, the encoding target frame, and the encoding target frame. Correspondence of all screens between long-term reference frames Motion vector of arbitrary position on the screen Variable that position A full-screen motion decoding means for decoding from an input as a parameter of a polynomial function to perform, and a full-screen motion compensation means for reading an image from the long-term frame memory using the parameters of the polynomial function to generate a predicted image, This is an image decoding apparatus for selecting short-term motion compensation means and full-screen motion compensation means in accordance with an instruction from an input and performing predictive decoding.

A second invention is to solve the above-mentioned problem 2, and is a frame in which a frame sequence of an image is input, and a decoded frame temporally before and after a frame to be coded is stored as a reference frame. A memory, full-screen motion detection means for detecting the correspondence of the entire screen between the encoding target frame and the reference frame as a motion vector of a representative point on the screen, and a motion vector of the representative point on the screen to a predetermined value. An image coding apparatus comprising: a representative point motion vector coding means for coding with precision; and a full-screen motion compensating means for generating a predicted image by interpolating the motion vector of the representative point. And a frame memory that receives a coded image as input and stores a decoded frame temporally before and after a frame to be coded as a reference frame. Representative point motion vector decoding means for decoding the motion vector of the representative point which is the output of the full screen motion detecting means, and interpolation of the motion vector of the representative point which is the output of the full screen motion detecting means An image decoding apparatus, comprising: a full-screen motion compensation unit that generates a predicted image.

In the image coding apparatus according to the first aspect of the invention, the short-term frame memory stores the decoded frame temporally before and after the frame to be coded as a short-term reference frame. The local motion vector detecting means detects a local correspondence between the short-term reference frame and the encoding target frame as a local motion vector. Then, the short-term motion compensation means reads the image from the short-term frame memory using the local motion vector to generate a predicted image.

On the other hand, the long-term frame memory stores, as a long-term reference frame, a decoded frame having an updated interval longer than that of the short-term frame memory with respect to the encoding target frame. The full-screen motion detection means detects the correspondence between the long-term reference frame and the full screen of the encoding target frame, using a motion vector at an arbitrary position on the screen as a parameter of a polynomial function having the position as a variable.

The full-screen motion compensating means uses the parameters of the polynomial function to read the image from the long-term frame memory and generate a predicted image. The image coding apparatus adaptively selects the local motion-compensated image from the short-term memory and the motion-compensated image from the long-term memory to perform predictive coding. Full-screen motion compensation is performed using frame data with a long update interval.

In the image decoding apparatus of the first aspect of the invention, similarly, the short-term frame memory stores the decoded frame temporally before and after the frame to be coded as a short-term reference frame. The local motion vector is obtained by the local motion vector decoding means, and the short-term motion compensation means uses the local motion vector to read the image from the short-term frame memory and generate the predicted image.

On the other hand, the long-term frame memory stores the frame as a long-term reference frame at a longer interval than the short-term frame memory with respect to the frame to be encoded. The parameters of the polynomial function are obtained by the full-screen motion decoding means, and the full-screen motion compensation means uses the parameters of the polynomial function to read the image from the long-term frame memory and generate a predicted image.

The image decoding apparatus adaptively selects the local motion-compensated image from the short-term memory and the motion-compensated image from the long-term memory to perform predictive decoding.

In the image coding apparatus of the second invention, the frame memory stores the decoded frame temporally before and after the frame to be coded as a reference frame. The full screen motion detection means detects the correspondence of the full screen between the encoding target frame and the reference frame as a motion vector of a representative point on the screen. The full-screen motion compensator generates a predicted image by interpolating the motion vector of the representative point. That is, the motion vector at each pixel position is obtained as an interpolation of the transmitted motion vector of the representative point.

The representative point motion vector coding means codes the motion vector of the representative point on the screen with a predetermined accuracy.

In the image decoding apparatus of the second invention, the frame memory stores, as a reference frame, a decoded frame which is temporally before and after the frame to be coded, with the output of the image coding apparatus as an input. To do.

The representative point motion vector decoding means decodes the motion vector of the representative point on the screen. The full-screen motion compensator generates a predicted image by interpolating the motion vector of the representative point.

A third aspect of the present invention is directed to the problem 2, wherein a frame memory that receives a frame sequence of an image and stores a decoded frame temporally before and after a frame to be encoded as a reference frame is provided. , Full-screen motion detection means for detecting the correspondence of the entire screen between the encoding target frame and the reference frame as a motion vector of a representative point on the screen, and a motion vector of the representative point on the screen as a difference vector Representative point motion vector difference encoding means for encoding with the accuracy of 1., and full-screen deforming means for deforming an image by switching the operation of the image deforming operation according to the motion vector difference of the representative points. An image encoding device and an encoded image as an input, and a decoded frame temporally before and after the decoding target frame as a reference frame A frame memory to be memorized, a representative point motion vector difference decoding means for decoding a motion vector difference of a representative point, and a motion vector difference of the representative point to switch the operation of the image deformation calculation to deform the image. An image decoding apparatus comprising means.

In the image coding apparatus of the third invention, the frame memory stores, as a reference frame, a decoded frame that is temporally before and after the frame to be coded.

The full-screen motion detection means detects the correspondence of the entire screen between the encoding target frame and the reference frame as a motion vector of a representative point on the screen.

The representative point motion vector difference encoding means encodes the motion vector of the representative point on the screen as a difference vector with a predetermined accuracy.

In the image decoding apparatus of the third invention, the frame memory stores, as a reference frame, a decoded frame temporally before and after the frame to be coded, with the output of the image coding apparatus as an input. To do.

The representative point motion vector difference decoding means decodes the motion vector difference of the representative point. The full-screen transforming unit switches the image transforming operation based on the motion vector difference of the representative point to transform the image.

(First Embodiment) An embodiment of the present invention will be described below with reference to the drawings. The present invention (claim 1
An embodiment of the image coding apparatus of 3) is described with reference to FIG.
An embodiment of the image decoding apparatus of the present invention (claims 2 and 4) will be described with reference to FIG.

FIG. 1 is a block diagram of the image coding apparatus.
1 is a residual coding unit, 102 is a residual decoding unit, 103 is a frame memory, 104 is a motion prediction unit 1, 105 is an affine parameter interframe estimation unit, 106 is an affine parameter updating unit, 1
07 is a template memory, 108 is an affine motion compensation unit, 1
09 is a local motion compensation unit 1, 110 is a motion prediction unit 2, 111 is a motion compensation unit 2, 112 is a comparator, 113 is an adder, 114 is a difference unit, 1
Reference numeral 15 is a switch, and 116 is a motion information multiplexing unit.

FIG. 2 is a block diagram of the image decoding apparatus.
1 is a residual decoding unit, 202 is a frame memory, 203 is a local motion compensation unit 1, 204 is an affine motion compensation unit, 205 is a local motion compensation unit 2, 206 is a motion information separation unit, 207 is a selector, and 208 is A template memory, 209 is a switch, and 210 is an adder.

In this embodiment, as its basic configuration, an encoding device and a decoding device based on MPEG1 as in the conventional example are adopted.

In the configuration of the image coding apparatus shown in FIG. 1, a residual coding section (101), a residual decoding section (102) and an adder (11
3), the frame memory (103), the motion prediction unit 1 (103), the local motion compensation unit 1 (109), the difference unit (114), and the residual decoding unit (in the configuration of the image decoding apparatus shown in FIG. 201), adder (2
10), frame memory (202), local motion compensation unit 1 (20
3) has the same configuration as the MPEG1 based encoding device and decoding device cited in the conventional example. Therefore, the residual encoding unit (103) is composed of an MPEG1-compliant digital cosine transform (DCT), a quantizer and a Huffman encoder. In addition, the residual decoding unit (102, 201) is MPEG1.
It consists of a compliant inverse quantizer and a digital cosine inverse transformer.

The input image of the encoding device and the output image of the decoding device are composed of horizontal and vertical 240 × 352 pixels. The block for performing the correlation calculation for detecting the motion vector is composed of 16 pixels in the vertical direction and 16 pixels in the horizontal direction.

The first embodiment configured as described above will be described by showing the differences from the MPEG1 and the conventional example.

The local motion prediction unit 1 (104) reads out the block information R of the input image as shown in FIG. 6 and, at the same time, changes the deviation (u, v) pixel by pixel, while the frame memory (103) Then, the block information R ′ of the previous frame is read out and the correlation calculation shown in Expression 1 is performed. This is the same operation as the local motion prediction unit 1 (504) in the conventional example. As a result, as shown in Equation 2, the deviation having the smallest value is searched to obtain the original estimation (p, q). This (p, q) is obtained with 1/2 pixel precision.

The above calculation is performed for each block, and the result is sent to the local motion compensator (109) to perform the macroblock motion compensation similar to MPEG1.

In this embodiment, unlike the conventional example, equation 1
Global motion compensation is performed by affine transformation represented by 6 parameters of 0. In equation 10, a0 to a5
Is the affine parameter, the pixel position of the x, y input image, (u ^
(x, y,), v ^ (x, y)) is a motion vector at the pixel position. The process of obtaining this affine parameter will be described below.

[0059]

(Equation 10)

In the process of obtaining the minimum deviation for each block by the correlation calculation shown in Expression 1, the array S of absolute value errors near the minimum deviation shown in Expression 11 is calculated. () ^T represents the transpose of the matrix. The (p, q) and S of each block are sent to the affine parameter interframe estimation section (105).

[0061]

[Equation 11]

Affine parameter interframe estimation unit (10
In 5), the correlation parameters are calculated by performing the calculations of Expression 12 to Expression 17.

[0063]

(Equation 12)

[0064]

(Equation 13)

[0065]

[Equation 14]

[0066]

(Equation 15)

[0067]

(Equation 16)

[0068]

[Equation 17]

The above calculation of the correlation parameter is performed for each block. The correlation parameters have the following meanings. Since the coefficients of Expressions 12 to 17 are obtained, a quadratic function in which the deviation (u, v) shown in Expression 18 is the absolute value error of each block from the input image to the image one frame before is used as a variable. Can be expressed as

Ei, j shown in Expression 18 is expressed as a Taylor expansion at the position (p, q) of the absolute value error of the block. The correlation is evaluated by this function.

[0071]

(Equation 18)

Here, the affine motion compensator (108) transforms the entire screen by the affine transformation expressed by the equations (19) and (20). (u (x, y), v (x, y)) is a motion vector at the position (x, y), and Expression 20 represents a vector composed of a sequence of affine transformation parameters.

[0073]

[Equation 19]

[0074]

(Equation 20)

Since the motion vector of each block is described by the affine parameters as shown in Expressions 19 and 20, as shown in Expression 21 as a necessary condition for minimizing the total sum of the absolute value error functions by the variational principle. It is possible to derive an Euler equation in which the partial differential of the affine transformation parameter a must be a zero vector. This can be expressed by the matrix of Expression 22.

Affine parameter interframe estimation unit (10
In step 5), the equations (23) (6 × 6 matrix) and equation (24) (6 × 1 matrix) are obtained from equation (18), and finally affine transformation parameters are obtained from equation (25).

In equations (23) and (24), (xj, yi) is the center position of block i, j.

[0078]

(Equation 21)

[0079]

(Equation 22)

[0080]

(Equation 23)

[0081]

(Equation 24)

[0082]

(Equation 25)

On the other hand, the switch (115) is closed in N (N ≧ 1) frame units, and the contents of the frame memory (103) are copied to the template memory (107).

In this embodiment, N = 10. That is, the frame memory (103) is updated every frame, whereas the template memory (107) is updated every 10 frames. The operation of the affine parameter inter-frame estimation unit (105) described above is to obtain the corresponding position of the decoded image one frame before as seen from the input image as an affine parameter. The affine motion compensator (108) is stored in the template memory (107) when viewed from the input image.
Since the decoded image in the range from the previous frame to the 9th frame is transformed, the output of the affine parameter inter-frame estimation unit (105) cannot be used as it is.
Therefore, the affine transformation is performed by synthesizing the affine parameters between the preceding and succeeding frames by utilizing the fact that the pixel position of the input image is linear as shown in Expression 10. This is realized by the affine parameter updating unit (106).

The affine parameter updating section (106) has a template memory (107) viewed from the input image one frame before.
The affine transformation parameters for the image inside are stored as a ~ 0 ~ a ~ 5. This parameter is reset to 0 every time an image is newly stored in the template memory (107) in cooperation with the switch (115). a ~ 0 ~ a ~ 5 and a0 ~ obtained by the affine parameter interframe estimation section (105)
Using a5, affine transformation parameters a'0 to a'5 for the decoded image stored in the template memory (107) viewed from the input image are obtained by the equations 26 to 31. Got a '
Substituting 0 to a'5 into a to 0 to a to 5 and updating repeatedly.

[0086]

(Equation 26)

[0087]

[Equation 27]

[0088]

[Equation 28]

[0089]

(Equation 29)

[0090]

[Equation 30]

[0091]

(Equation 31)

The affine motion compensation unit (108) has a'0 to a'5.
It is executed by calculating the motion vector shown in Expression 10 for all pixel positions of the input image using, and obtaining the pixel value of the pixel position of the motion vector moved obtained on the image in the template memory (107). . The motion vector for global motion compensation is obtained with 1/4 pixel precision, and is obtained as a linear interpolation of 4 pixel values in the neighborhood integer position. Further, the motion prediction unit 2 (110) performs motion prediction on a macroblock basis on the transformed image. Local motion compensation unit (111),
Comparator (112), difference unit (114), residual coding unit (10
The operations of 1), the residual decoding unit (102) and the adder (113) are the same as those of the conventional example.

As the motion vector information, the selection result SW in macroblock units, the motion vector mv1 or mv2, and the affine parameter a are multiplexed by the motion information multiplexing unit (116) and transmitted.

Next, the operation of the decoding device corresponding to the encoding device of FIG. 1 will be described with reference to FIG. Motion information separation unit (20
6) receives the output of the motion information multiplexing unit (116) as an input,
Selection result SW in macroblock units, motion vector mv1
Alternatively, mv2 and the affine parameter a are separately decoded.

The operation of the motion compensation unit 1 (203) or the affine motion compensation unit (204) and the local motion compensation unit (20) is performed by SW.
The combination operation of 5) is selected. A selector (207) selects any output based on SW.

In another configuration of the image decoding apparatus, the residual decoding unit (201), the frame memory (202), the local motion compensation unit 1
(203), affine motion compensation unit (204) local motion compensation unit 2
(205) Template memory (208), switch (209),
The operation of the adder (210) is the same as the block of the same name in FIG. The switch (209) updates the contents of the template memory (208) every 10 frames in synchronization with the switch (115) on the encoding device side.

The present embodiment described above has the following features as compared with the conventional example. (1) In the conventional example, the output of the motion prediction unit 1 (504) is only the motion vector, and the parameter describing the motion of the entire screen is calculated based on this motion vector. However, in the local correlation calculation, the local motion vector becomes indefinite in the area where the luminance does not change. Further, even if there is a change in brightness, the motion vector has a degree of freedom in the tangential direction of the contour around the monotonous contour, and the local motion vector becomes similarly undefined. Therefore, a large estimation error is feared in the two-step parameter estimation method shown in the conventional example. On the other hand, in the method shown in the present embodiment, the degree of freedom is quantitatively expressed by a quadratic function, and the sum of the quadratic functions is minimized to suppress the influence of the local motion vector, which would otherwise be indefinite. It is expected that the parameters can be estimated more stably.

(2) Since the global motion compensation is executed on the basis of the template updated at intervals of 10 frames, the problem that the spatial resolution of the prediction image without residual coding gradually decreases is different from the conventional example. It is reduced in comparison.

Note that it is difficult to directly obtain the affine parameters of the input image and the images separated by a maximum of 9 frames on the time axis by block correlation because the motion vector at each pixel may become large. Therefore, in the present embodiment, the affine parameters are obtained between the preceding and following frames, and these are combined to obtain the affine parameters with long frame intervals.

(Second Embodiment) Next, FIG. 3 shows an embodiment of an image coding apparatus according to the present invention (claims 5, 6, and 8).
An embodiment of the image decoding apparatus of the present invention (claims 7 and 9) will be described with reference to FIG.

FIG. 3 is a block diagram of the image coding apparatus.
301 is a residual coding unit, 302 is a residual decoding unit, 303 is a frame memory, 304 is a motion prediction unit 1, 305 is a representative point motion vector interframe estimation unit, 306 is a representative point motion vector updating unit, and 307 is Template memory, 308 is representative vector interpolation motion compensation unit, 309 is local motion compensation unit 1, 310 is switch, 31
1 is a motion information multiplexing unit, 312 is a comparator, 313 is an adder, 314
Is a differentiator.

FIG. 4 is a block diagram of the image decoding apparatus.
1 is a residual decoding unit, 402 is a frame memory, 403 is a local motion compensation unit 1, 404 is a representative vector interpolation motion compensation unit, 405 is a template memory, 406 is a motion information separation unit, 407 is a selector, and 408 is A switch and 409 are adders.

Except for the motion information multiplexing unit (311), the components of FIG. 3 operate in the same manner as the components of FIG. 1 if they have the same name. Similarly, except for the motion information separation unit (406), as shown in FIG.
The operation of the components is the same as the components in FIG.

The differences between the second embodiment and the first embodiment are: (1) Local motion compensation after global motion compensation is not performed. (2) Update and transmit affine parameters with motion vectors of three representative points. (3) Global motion compensation is performed by interpolating the motion vectors of the three representative points. That is three points. This difference will be described below.

First, in the configuration of FIGS. 3 and 4, in the encoding device, the motion information multiplexing unit (311) causes the motion vector mv,
Difference representation (s0, t0), (s1, t1), (s2, t2) of the representative point motion vector quadrupled and quantized to an integer value, and the selection result SW in macroblock units are multiplexed and supported. Then, in the decoding apparatus of FIG. 4, the motion information separation unit (406) is set to mv, (s0, t0), (s1, t1),
The operation of separating and decoding three pieces of information (s2, t2), SW is different.

The affine parameters are estimated by the following method. As shown in Expression 32, the affine parameter a can be obtained if the movements of three points o, h, and v that are not on the same straight line are determined. Po is a matrix for the position of o shown in Equation 33, and Vo is Equation 3
It is the motion vector at the o position shown in 4. The same applies to h and v.

[0107]

(Equation 32)

[0108]

[Equation 33]

[0109]

(Equation 34)

In the present embodiment, the positions of o, h, and v are set to the positions of (0,0), (H, 0), (0, V) on the input image.

H and V are image sizes, and H = 352 and V = 240. Next, as shown in Fig. 7 on the screen, TL, TR, C, B are newly added.
Five points L and BR are selected as observation positions, and the representative point motion vector interframe estimation unit (305) uses the motion vector prediction unit 1 (3
In the output of 04), the affine parameters are calculated using the results of the above five block correlations. This procedure is shown in FIG. The affine parameter a is calculated by the equation 25 by sequentially selecting a combination of 3 points from the name of 5 points. Then, by using the obtained affine parameters, the motion vector at the positions of o, h, v is 1 /
Calculated with 4-pixel accuracy and stored.

For all combinations, the motion vectors are o,
Obtain the positions of h and v, and use the intermediate value as the result. This is V
Letting o, Vh, Vv be different representations of affine parameters during one frame. In the first embodiment, the affine parameters of the input image and the template are obtained by combining the affine parameters.

In the present embodiment, the representative point motion vector updating unit (306) determines the correspondence between the image in the template memory (307) as seen from the input image one frame before and the representative point motion vector V ~ o, V ~. It is stored as h, V to v.

The representative point motion vector is reset to 0 vector every time an image is newly stored in the template memory (307) in cooperation with the switch (310). V ~ o, V
~ H, V ~ v and representative point motion vector interframe estimation unit (305)
Using Vo, Vh, Vv obtained by the above, affine transformation parameters V'o, V'h, V'to the decoded image stored in the template memory (307) viewed from the input image according to Equations 35 to 37. ask for v The obtained V'o, V'h, V'v are substituted into V ~ o, V ~ h, V ~ v for each frame, and the iterative update is performed.

Then, the sequence of values (U4o, V4o), (U4h, V4h), and (U4v, V4v) obtained by multiplying V'o, V'h, V'v by 4 and converting them into integers is given by s0, s1, s2, t0, t1, t2) and transmit.

[0116]

(Equation 35)

[0117]

[Equation 36]

[0118]

(37)

Here, the difference formats are as shown below. s0 = U4o s1 = U4h-U4o s2 = U4v-U4o t0 = V4o t1 = V4h-V4o t2 = V4v-V4o This difference data is converted into the variable length code shown in Table 1 and encoded.

For this, the still image encoding standard JPEG "J-PEG Technical Specification" (ISO / IEC JTC1 / SC29 / WG10: JPEG Techn) by the International Organization for Standardization (ISO) is used.
icalSpeicification, JPEG8-R8, August 1998). In this encoding method, the category SSSS and Huffman code for the difference data are obtained. Here, SSSS is a variable for grouping the difference data, and when the difference is positive, it is the number of the least significant bit where "1" stands first from the upper bit when the difference is expressed in binary. When the difference is negative, it is the number of the least significant bit in which "0" is first set when the difference value -1 is expressed in binary as seen from the upper bits.

By transmitting the Huffman code of (Table 1) corresponding to the category, the category of the difference value can be known, but since the difference data cannot be specified only by this, additional bits are added after the Huffman code. SSSS
Corresponds to the bit length of this additional bit. If the difference is positive, the SSSS bit is added from the least significant bit of the difference, and if the difference is negative, the SSSS bit is added from the least significant bit of the difference value -1. Becomes

In this way, for each of (s0, s1, s2, t0, t1, t2), the Huffman code of category SSSS and the additional bits are transmitted as a variable length code. This encoding is performed by the motion information multiplexing unit (311).

[0123]

[Table 1]

The motion information separating section (406) finds a Huffman code corresponding to SSSS in the bit stream and decodes the category SSSS. Subsequently, additional bits for SSSS bits are read, and it is determined that the difference is positive when the first bit of the additional bits is 1 and negative when the first bit of the additional bits is 0.

If it is negative, the additional bit is expressed as a binary number, 1 is added to this, and then 1 is added above the SSSS + 1 bit to convert it to a negative number.

In the first embodiment, the affine parameter a
Then, the motion vector of each pixel position on the input screen was calculated by transmitting Eq. In this embodiment, equivalently (U4o, V
4o), (U4h, V4h), (U4v, V4v) are interpolated and calculated.

According to the equation 32, the affine parameter o can be obtained from the equations 38 and 39 from the motion vectors of the three points o, h and v.

[0128]

(38)

[0129]

[Equation 39]

If expanded, the motion vector can be obtained as equations 40 and 41. This means that the motion vectors of the three points o, h, and v are divided and obtained according to the position.

[0131]

(Equation 40)

[0132]

[Equation 41]

1/4 precision is guaranteed for each pixel position,
Moreover, in order to make the prediction image by the global motion compensation the same on the encoding side and the decoding side, it is conceivable to perform the equations shown in the equations 42 and 43 as integer operations.

[0134]

(Equation 42)

[0135]

[Equation 43]

In Expressions 42 and 43, (X, Y) is a pixel position on the input screen, V4 (X, Y), V4 (X, Y) are quadruple integer motion vectors at that position, and (U4o , V4o) is the quadrupled position o
, (U4h, V4h), (U4v, V4v) are the motion vectors at the upper right corner and the lower left corner of the screen, respectively.

In practice, this calculation shows that (U4o, V
4o), (U4h, V4h), (U4v, V4v) is the difference format (s0, s1, s2, t0,
It is used to be transmitted at t1, t2).

That is, in the representative vector interpolation motion compensator (308), equations 42 and 43 are realized as equations 44 and 45.

[0139]

[Equation 44]

[0140]

[Equation 45]

At this time, if it is found that all of s1, s2, t1 and t2 are zero, the calculation of equations 44 and 45 is simplified. That is, only parallel movements determined by s0 and t0 are s1, s2,
The calculation for global motion compensation regarding t1 and t2 is omitted.

The features of this embodiment are summarized below. (1) Since motion estimation after global motion compensation is omitted in the second embodiment as compared with the first embodiment, the block matching process, which is relatively heavy in the image encoding device, is performed. It is on the same scale as MPEG1 described in the conventional technique.

(2) The processing load is light because the entire screen movement is obtained by the five block correlations. In addition, even if there is a motion different from the entire screen by selecting three block correlations from the five block correlations, calculating the motion vector of the representative points o, h, v from this and selecting the median value of the components. It has a structure that is unlikely to be affected by it.

(3) By selecting the representative points o, h, v at orthogonal positions and transmitting the motion vector at this point, coordinate calculation at the time of motion compensation is simplified. Thereby, high calculation accuracy can be compensated.

(4) By performing variable-length coding on the representative point in the difference format, if the affine transformation is a simple parallel movement, (s
The parallel movement is represented by (0, t0), and the other s1, s2, t1, t2 are all zero. Therefore, assuming that the parallel movement is one pixel horizontally, the affine transformation parameter can be transmitted with a total of 15 bits.

In (Table 1), a motion with ¼ pixel precision is -5.
It can be expressed in the range of 11.75 to +512 pixels, but if this is performed by fixed length coding, 12 bits × 6 = 72 bits are required. Compared to this, the amount of generated bits can be suppressed in the range of normal motion vectors. Further, it is possible to easily judge from the difference expression whether the image can be deformed only by parallel movement or whether general affine transformation needs to be performed. As a result, there is an advantage that the address calculation for global motion compensation can be simplified.

In the method of transmitting global motion data described in the present embodiment, the case in which the global motion compensation is the affine transformation has been described, but it is expanded to a plane motion model under the transformation by the bilinear interpolation or the perspective transformation. be able to.
In any case, it is a conversion of 8 parameters and can be realized by newly transmitting the 1/4 pixel precision motion vector (U4hv, V4hv) at the coordinate position (H, V) of the lower right corner of the screen. When using bilinear interpolation, as shown in Equation 46 and Equation 47, 8
The movement of the whole screen is described by one parameter.

[0148]

[Equation 46]

[0149]

[Equation 47]

Here, s0 = U4o s1 = U4h-U4o s2 = U4v-U4o s3 = U4o-U4h-U4v + U4hv t0 = V4o t1 = V4h-V4o t2 = V4v-V4o t3 = V4o-V4h-V4v + h Then, when using bilinear interpolation, the motion vector is

[0151]

[Equation 48]

[0152]

[Equation 49]

It can be expressed by Where (s3, t3)
= (0,0), the same process as affine transformation, and if all except (s0, t0) are zero, the same process as parallel movement. In the case of the plane motion model in perspective transformation, the calculation is a little complicated. In either case, whether (s3, t3) = (0,0) or not (s0, t0) is zero. Affine transformation,
Since automatic switching to parallel movement can be performed, the advantage of the motion vector transmission of the image end point in the above-described difference format is great.

[0154]

As described above, according to the first aspect of the present invention, it is possible to reduce the reduction in the resolution of the predicted image caused by the global motion compensation, as compared with the prior art. This compensates for the movement caused by the camera swinging and zooming, so that the image can be transmitted with a smaller amount of information transmission.

According to the second invention, since the motion vector is interpolated and the global motion compensation is performed, the complexity of the address calculation is reduced and the calculation accuracy can be kept high.
Therefore, it is possible to easily match the prediction images on the encoding side and the decoding side, and as a result, it is possible to perform efficient prediction image encoding by global motion compensation.

According to the third invention, the code amount of the representative point motion vector can be reduced by the differential expression while maintaining the effect of the second invention. Furthermore, the zero determination of the difference vector makes it possible to automatically switch the deformation model of the image to a simpler model.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an image encoding device according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram of an image decoding apparatus according to the first embodiment.

FIG. 3 is a configuration diagram of an image encoding device according to a second embodiment of the present invention.

FIG. 4 is a configuration diagram of an image decoding apparatus according to the second embodiment.

FIG. 5 is a configuration diagram of a conventional image encoding device.

FIG. 6 Principle diagram of block correlation method

FIG. 7 is a diagram showing a representative point and an observation point in the second embodiment.

FIG. 8 is an operation diagram of a representative point motion vector interframe estimation unit according to the second embodiment.

[Explanation of symbols]

101, 301 Residual coding unit 102, 201, 302, 401 Residual decoding unit 103, 202, 303, 402 Frame memory 104, 110, 304 Local motion prediction unit 105 Affine parameter Interframe estimation unit 106 Affine parameter updating unit 107, 208, 307, 405 Template memory 108, 204 Affine motion compensation unit 109, 110, 203, 205, 309, 403 Local motion compensation unit 112, 312 Comparator 113, 210, 313, 409 Adder 114, 314 Difference device 116, 311 Motion information multiplexing unit 115, 209, 310, 408 Switch 206, 406 Motion information separating unit 207, 407 Selector 305 Representative point motion vector inter-frame estimation unit 306 Representative point motion vector updating unit 308, 404 Representative vector Interpolation motion compensator SW Prediction selection result in macro block units SW mv1, mv2 macro block motion vector g Global motion compensation parameter a Affine motion compensation parameter U4, V4 Representative point motion vector quantized to an integer value of 4

Claims (9)

[Claims] 1. A short-term frame memory, which receives a frame sequence of an image and stores decoded frames temporally before and after the frame to be coded as short-term reference frames, the frame to be coded and the short-term frame. Local motion vector detection means for detecting a local correspondence between reference frames as a local motion vector, and short-term motion compensation for reading an image from the short-term frame memory using the local motion vector to generate a predicted image Means, a long-term frame memory for storing a decoded frame having a longer interval of updating than the short-term reference frame as a long-term reference frame with respect to the encoding target frame, the encoding target frame and the long-term reference frame Between all the screens, and the motion vector at any position on the screen as the variable The short-term motion compensation includes: a full-screen motion detection unit that detects as a parameter of a polynomial function; and a full-screen motion compensation unit that reads an image from the long-term frame memory using the parameter of the polynomial function to generate a predicted image. Means for adaptively selecting a means and a full screen motion compensation means for predictive coding. 2. A short-term frame memory, which receives a coded image as an input, and stores a decoded frame temporally before and after a frame to be coded as a short-term reference frame; Local motion vector decoding means for decoding a local motion vector from an input for local correspondence between short-term reference frames, and a predicted image by reading an image from the short-term frame memory using the local motion vector described above. Short-term motion compensation means for generating, a long-term frame memory for storing a decoded frame having a longer update interval than the short-term reference frame as a long-term reference frame for the coding target frame, and the coding target frame Correspondence of the whole screen between the long-term reference frames is calculated by changing the position of the motion vector at any position on the screen. A full-screen motion decoding means for decoding from an input as a parameter of a polynomial function, and a full-screen motion compensating means for reading an image from the long-term frame memory using the parameters of the polynomial function to generate a predicted image. An image decoding apparatus for performing predictive decoding by selecting the short-term motion compensation means and the full-screen motion compensation means according to an instruction from an input. 3. The full-screen motion compensating means reads out an image from a long-term frame memory according to a linear polynomial function whose position is a motion vector at an arbitrary position on the screen to generate a predicted image, and the full-screen motion is calculated. Using the output of the local motion vector detection means as the configuration of the detection means, a linear polynomial function parameter calculation means for calculating the correspondence of the entire screen between the encoding target frame and the short-term reference frame using the parameters of the linear polynomial function, A linear polynomial function parameter synthesizing means for obtaining the correspondence of the entire screen between the encoding target frame and the long-term reference frame as a synthesis of the linear polynomial function by utilizing the linearity of the linear polynomial function. The image coding apparatus according to claim 1. 4. The full-screen motion compensating means generates a predicted image by reading an image from a long-term frame memory according to a linear polynomial function having a position of a motion vector at an arbitrary position on the screen as a variable. Item 2. The image decoding device according to item 2. 5. A short-term frame memory, which receives a frame sequence of an image and stores decoded frames temporally before and after the frame to be coded as short-term reference frames, and a representative point of the frame to be coded. Representative point motion vector detection means for detecting a local correspondence between the short-term reference frames as a representative point motion vector, and decoding for a frame to be coded having a longer update interval than the short-term reference frame. A long-term frame memory that stores a selected frame as a long-term reference frame, and the representative point motion vector are combined to obtain the correspondence of the entire screen between the encoding target frame and the long-term reference frame as a representative point motion vector. An image is read from the long-term frame memory using the detection means and the output of the full-screen motion detection means. The image coding apparatus is characterized in that a full screen motion compensation means for generating a measurement image. 6. A frame memory which receives as input a frame sequence of an image, and stores as a reference frame a decoded frame temporally before and after the frame to be coded; a frame memory of the frame to be coded and the reference frame; A full-screen motion detection means for detecting the correspondence between all screens as a motion vector of a representative point on the screen, and a representative point motion vector coding means for encoding the motion vector of the representative point on the screen with a predetermined accuracy. An image coding apparatus comprising: a full-screen motion compensating unit that generates a predicted image by interpolating the motion vector of the representative point. 7. A frame memory for receiving a coded image as an input and storing a decoded frame temporally before and after a frame to be coded as a reference frame, and full-screen motion detection according to claim 6. A representative point motion vector decoding unit that decodes the motion vector of the representative point that is the output of the unit, and a predicted image is generated by interpolating the motion vector of the representative point that is the output of the full-screen motion detection unit. An image decoding apparatus comprising: a full-screen motion compensation means. 8. A frame memory for storing, as a reference frame, a decoded frame that is temporally before and after a frame to be coded, and a full-screen correspondence between the frame to be coded and the reference frame. Full-screen motion detection means for detecting as a motion vector of a representative point on the screen, representative point motion vector difference encoding means for encoding a motion vector of the representative point on the screen as a difference vector with a predetermined accuracy, and the representative point An image coding apparatus, comprising: a full-screen transforming unit that transforms an image by switching the image transforming operation according to the motion vector difference. 9. A frame memory, which receives a coded image as an input, stores a decoded frame temporally before and after a decoding target frame as a reference frame, and decodes a motion vector difference of a representative point. An image decoding apparatus comprising: a representative point motion vector difference decoding means and a full-screen transforming means for transforming an image by switching the image transforming operation according to the motion vector difference of the representative points.