JPH08116542A - Image coder, image decoder and motion vector detector - Google Patents

Abstract

PURPOSE: To encode/decode an image representing luminance and transparency being components of hierarchical images separated in the relation of the depth in the line of light directions. CONSTITUTION: This coder is provided with a modification analyzer 201 that takes correlation between a luminance plane and an α plane representing a transparency and extracts a modification parameter expressed by affine transformation and movement of a block and with a modification compositing unit 202 producing a predicted image from the result of decoding of a preceding frame and the result of the modification analyzer 201. Then a predicted image consists of the luminance plane and the α plane and a difference of them is subject to error coding and an outputted bit stream consists of an affine transformation parameter, a block movement parameter, a luminance plane error code and an α plane error code.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion vector detecting device used for format conversion such as image coding and frame frequency conversion, and an image coding device and an image decoding device for transmitting and recording an image with a small coding amount. Is.

[0002]

2. Description of the Related Art One and Adelson (J. Wang a
nd E. Adelson) has proposed a method of decomposing a moving image into hierarchical images and encoding the moving image as shown in FIG. 23 for the purpose of efficiently transmitting and recording the moving image.

A document disclosing this method, "Layered representation for image sequence coding" (J. Wang and E. Ad.
elson: "Layered Representa
tion for Image Sequence C
oding ”, Proc. IEEE Int. Con
f. Acoustic Speech Signal
Processing '93, pp. V221-V22
4, 1993) and the document "Layered representation for motion analysis" (J.
Wang and E. Adelson: "Layer
ed Representation for Moti
on Analysis ”, Proc. Comput
erVision and Pattern Reco
gnition, pp. 361-366, 1993), the following image processing (1) to (3) is performed. (1) An area described by the same motion parameter (affine transformation parameter in the conventional example) is extracted from the moving image. (2) The same motion area is overlapped to generate a hierarchical image. Each hierarchical image is represented by the transparency and brightness of each pixel, which indicates the occupation of the overlapped areas. (3) The vertical relationship between the hierarchical images in the line-of-sight direction is examined and ordered.

[0004] Here the affine transformation parameters, the horizontal and vertical position in the image (x, y), when the horizontal and vertical components of the motion vector and (u, v), of a ₀ ~a ₅ shown in Equation 1 Means a coefficient.

[0005]

[Equation 1]

It is known that the movement of a rigid projection image at a sufficient distance from the camera can be approximated by an affine transformation parameter. Utilizing this, they combine several tens to several hundreds of frames of moving images while deforming several types of hierarchical images composed of one frame by affine transformation. The information necessary for transmitting and recording this moving image is the image that is the source of deformation for each hierarchical image (hereinafter referred to as template).
Since only the affine transformation parameter and the hierarchical relationship between the hierarchical images are used, the moving image can be recorded and transmitted with extremely high coding efficiency. It should be noted that the template is represented by the transparency and the brightness of each pixel indicating the occupation of the area for image synthesis.

[0007]

In the motion image representation of Wang and Adelson, only the motion of a rigid body whose projected image can be described by affine transformation is dealt with. Therefore, their moving image representation cannot deal with the case where the movement of the projected image cannot be described by the affine transformation. For example, when the person shown in FIG. 23 makes a non-rigid body motion, it cannot be applied when the distance between camera objects is small and the nonlinear term of perspective transformation cannot be ignored. Further, their method of obtaining the movement of the projected image as an affine transformation parameter is composed of the following two-stage processing.

1. A method based on the relational expression of the spatiotemporal gradient of the luminance that the temporal change of the luminance can be approximated by the inner product of the spatial luminance gradient and the motion vector (B. Lucas and T. et al.
Kanade: "An Iterative Image
e Registration Technique
with Application to Ster
eo Vision ”, Proc. Image Und
erstanding Workshop, pp. 12
1-130, April 1981), a local motion vector is obtained at each position on the screen.

2. The obtained motion vectors are clustered to obtain affine transformation parameters.

However, the above method cannot be applied to the case where there is a large motion in the moving image in which the relational expression of the spatiotemporal gradient of the brightness does not hold. Furthermore, the two-step method of estimating the affine transformation parameter from the obtained motion vector causes a large estimation error if the motion vector that is the basis of the parameter estimation is incorrect. The motion vector is indefinite in a region where there is no change in brightness or a region where there is a change in brightness even if there is a change in brightness.
The two-step estimation method requires special processing for motion vectors in these uncertain regions. In summary, the following problems 1 and 2 have not been solved.

Problem 1: Efficient encoding of an image (template) of luminance and transparency having irregular deformation Problem 2: Robust estimation of affine transformation parameters The present invention is to solve the above problems and to solve the above problems. An image encoding device for highly efficiently encoding and decoding images of luminance and transparency that form a hierarchical image separated in the context of
An object is to provide an image decoding device and a motion vector detecting device.

[0012]

An image coding apparatus according to a first invention for solving the above-mentioned problem 1 is constituted by luminance and transparency by inputting a series of images constituted by luminance and transparency of an object. Predicting means for predicting the image of the luminance and the transparency of the encoding target image by the correspondence between the partial areas from the reference image,
Predictive encoding means for encoding the correspondence between the partial regions in the predicting means as a predictive code; error calculating means for obtaining a difference in luminance and transparency between the predicted image and the image to be encoded as an error image; An error coding means for coding an error image as an error image code is provided, and an image sequence is transmitted and recorded as an error image code and a prediction code for the reference image.

An image decoding apparatus of a second invention for solving the above-mentioned problem 1 holds the same reference image as that of the first image coding apparatus, and decodes a predictive code for decoding the correspondence between partial regions from the predictive code. Conversion means, prediction image generation means for generating a prediction image from a reference image based on the correspondence between the partial areas, error image decoding means for decoding an error image by an error image code, the prediction image and the error image Is added to obtain an image composed of brightness and transparency, and an image composed of brightness and transparency is decoded as an output of the predicted image generation means or the addition means.

The image coding apparatus of the third invention for solving the above-mentioned problem 1 classifies an area into two areas, a transparent area and an opaque area, using an image composed of the brightness and the transparency of an object as an input, and opaque. The area has the brightness of the object, and the transparent area has a superimposing means for generating a brightness image on which the brightness and transparency information is superimposed so as to take a predetermined value outside the brightness range. This is a configuration for encoding a luminance image on which information is superimposed.

An image decoding apparatus according to a fourth aspect of the present invention for solving the above-mentioned problem 1 is a transparent area when the brightness value is out of the range, and a transparency image and a brightness image when the brightness value is within the range. It is configured such that it has a separating means for separating into and, and decodes an image of luminance and transparency.

In the image coding apparatus of the fifth invention for solving the above-mentioned problem 1, when the original image is expressed hierarchically by the front-and-rear relationship on the line-of-sight axis and the transparency of the region in addition to the luminance, this hierarchical structure is used. Layered image encoding means for encoding the luminance and the transparency as a layered image code for each layered image, and a layer for obtaining the layered image decoded from the result of the layered image encoding means. Image image decoding means, synthesizing means for synthesizing the plurality of decoded hierarchical images according to their context, luminance, and transparency, and an error image between the original image and the synthesized image is obtained and encoded. An error image encoding means is provided, and the original image is transmitted and recorded by an error code between a plurality of hierarchical image codes and the original image.

An image decoding apparatus according to a sixth aspect of the present invention for solving the above-mentioned problem 1 is a hierarchical image decoding means for decoding a hierarchical image consisting of a luminance, a transparency and a front-rear relationship on a line-of-sight axis from a plurality of hierarchical image codes. And a composition means for generating a composite image from the hierarchical image, and an error image decoding means for decoding an error image from an error code, wherein the image is decoded by adding the error image to the composite image. is there.

The image coding apparatus of the seventh invention for solving the above-mentioned problem 1 has a reference image coding means for transmitting and recording a plurality of reference images in advance, and a luminance between the input image and the plurality of reference images. Approximate the deviation or deformation of the corresponding position as a polynomial function with the position on the screen as a variable, and obtain the approximation error between images and the reference image with a small approximation error from the plurality of reference images. A reference image selecting unit for outputting a selected reference image identifier and a coefficient of a polynomial function, the reference image encoding unit encoding a plurality of reference images, and selecting at least the input image sequence. It is configured to be transmitted and recorded as an identifier for the generated reference image and a coefficient of the polynomial function.

An image decoding apparatus of an eighth invention for solving the above-mentioned problem 1 is a reference image decoding means for reconstructing a plurality of reference images in advance, and a reference image included in an input from the plurality of reference images. Reference image selecting means for selecting a reference image corresponding to the identifier for, and a deformation of the image is determined based on the coefficient of the polynomial function included in the input, the polynomial function having the position on the screen as a variable, The configuration is such that the selected reference image has a reference image transforming unit that transforms the image, and the image is decoded using the reference image transformed by the reference image transforming unit.

A motion vector detecting device of a ninth invention for solving the above-mentioned problem 2 is to input a plurality of images composed of the brightness and the transparency of an object, and add the multiplication of the transparency to a predetermined value and, if necessary, Threshold value processing is performed to convert the value range, and the converted value is added to the luminance to generate a luminance image in which the information of the luminance and the transparency is superimposed, and a partial area of the two images by the correlation of the luminance. The image analysis means for obtaining the correspondence between the plurality of images is converted by the superimposing means into an image composed of only the brightness and the transparency, and the image analysis means is provided between the converted plurality of images. Is used to obtain the correspondence of the partial areas.

A motion vector detecting device of a tenth invention for solving the above-mentioned problem 2 is a device for expressing a motion vector at an arbitrary position on the screen as a polynomial function having the position as a variable, and dividing an image. For a plurality of partial areas obtained as a result, the correspondence between the partial areas of two different images is calculated as an error, and the error calculating means for calculating the deviation between the partial areas having the minimum error and the error value in the vicinity thereof; An error function calculating means for obtaining a quadratic error function having a deviation as a variable from a deviation which becomes an error and an error value in the vicinity thereof, and a total sum or a partial sum of the quadratic error function using a coefficient of a polynomial function as a variable. The configuration is such that there is an optimizing means for expressing and minimizing the total sum or partial sum for the coefficient, and the motion vector between different images is output as the coefficient of the polynomial function.

[0022]

In the image coding apparatus according to the first aspect of the present invention, the predicting means associates the reference image (that is, the template) with the partial region of the image to be coded to thereby obtain the luminance and the transparency of the image to be coded from the reference image. Is predicted and a predicted image is generated. Correspondence of the partial areas is output as a prediction signal by the predictive coding means. The difference between the brightness and the transparency of the predicted image and the image to be encoded is obtained by the error calculating means and encoded by the error encoding means.

The image decoding apparatus according to the second invention holds the same reference image as that of the image coding apparatus according to the first invention, and the predictive code decoding means and the predictive image generating means are provided between the predictive code and the partial area. A prediction image is generated from the reference image by decoding the correspondence. On the other hand, the error image decoding means decodes the error image from the error image code. Then, the adding means adds the predicted image and the error image to obtain an image composed of brightness and transparency.

In the above two inventions, on the encoding side, the difference between the brightness and the transparency of the predicted image and the image to be encoded is calculated and encoded. On the other hand, on the decoding side, the difference between the transparency and the brightness is decoded. As a result, it is possible to encode a hierarchical image that allows irregular template deformation.

In the image coding apparatus of the third invention, the superimposing means classifies the area into two areas, a transparent area and an opaque area, with the image composed of the brightness and the transparency of the object as an input, and regarding the opaque area, A luminance image in which luminance and transparency information is superimposed so that the luminance of an object takes a predetermined value outside the luminance range for a transparent region is generated and then encoded.

In the image decoding apparatus of the fourth invention, the separating means from the decoded image transmits the image as a transparent area when the luminance value is a predetermined value outside the range, and transmits the image as the luminance value when inside the range. Degree image and luminance image are separated. In the above two inventions,
By converting the two pieces of information of the brightness and the transparency which form the template into one brightness image, the deformation of the template can be treated as the change of the brightness image.

In the image coding apparatus of the fifth aspect of the invention, the original image is hierarchically represented by the front-rear relationship on the line-of-sight axis and the transparency of the area in addition to the luminance. The image encoding device receives the plurality of layered images and encodes the luminance and the transparency as a layer image code by the layer image encoding means for each layer image. On the other hand, the layered image decoding means obtains the layered image decoded from the result of the layered image encoding means, and a plurality of layered images decoded by the synthesizing means are synthesized based on their context, brightness, and transparency. This allows
The result of synthesizing the hierarchical images in the decoding device will be estimated. Then, the error image encoding means obtains an error image between the original image and the estimated composite image, and encodes this.

In the image decoding apparatus of the sixth aspect of the invention, the hierarchical image decoding means decodes the hierarchical image consisting of the luminance, the transparency, and the front-rear relationship on the line-of-sight axis from the plurality of hierarchical image codes, and the synthesizing means decodes the hierarchical image. Generate a composite image from. Then, the error image decoding means decodes the error image from the error code. Finally, the image is restored by adding the error image to the composite image. In the above two inventions, even if the template is deformed irregularly, the composite of the hierarchical images is not used as the final result, but the predicted image is used, and the difference between the predicted image and the original image is transmitted and recorded, so that the image is enlarged. It can be transmitted and recorded without visual deterioration.

In the seventh invention, the template is transmitted and recorded in advance by the reference image coding means. The correspondence between the input image and the plurality of templates is approximated by the inter-image correspondence approximating means as a polynomial function having the position on the screen as a variable. The minimum distortion reference image selection means obtains a reference image with a small approximation error from the plurality of reference images in the plurality of templates regardless of the time order, and selects the identifier of the selected reference image and the coefficient of the polynomial function. Output. Providing a plurality of templates improves the degree of approximation with the polynomial function.

In the image decoding apparatus of the eighth invention, the plurality of templates are reconstructed in advance by the reference image decoding means. The reference image selection means selects a template corresponding to the inputted template identifier, and the reference image transformation means transforms the image based on the inputted coefficient of the polynomial function. On the side of the encoding device, since it is guaranteed that the deformation result of the template by the polynomial function is similar to the input image, the image can be decoded with a small encoding amount.

In the motion vector detecting device of the ninth invention, which inputs a plurality of images composed of the brightness and the transparency of the object, the superimposing means calculates the transparency by adding and multiplying the transparency by a predetermined value. Threshold processing is performed to convert the value range, and the converted value is added to the brightness to generate a brightness image in which brightness and transparency information is superimposed. Then, the image analysis means obtains the correspondence between the partial areas of the two images by the correlation of the brightness. This makes it possible to detect a motion vector using not only the luminance but also the correlation of the transmittance.

In the motion vector detecting apparatus of the tenth aspect of the present invention which represents a motion vector at an arbitrary position on the screen as a polynomial function having the position as a variable, the error calculating means has a plurality of partial regions obtained by dividing an image. With respect to, the correspondence between the partial areas of two different images is calculated as an error, and the deviation between the partial areas having the minimum error and the error value in the vicinity thereof are obtained. The error function calculating means obtains a quadratic error function having the deviation as a variable from the deviation having the minimum error and the error value in the vicinity thereof. The optimizing means expresses the sum or partial sum of the quadratic error function by using the coefficient of the polynomial function as a variable, and minimizes the sum or partial sum of the coefficients. In the present invention, the coefficient of a polynomial function having a position as a variable (an affine transformation is an example thereof) so as to minimize the total sum or partial sum of the quadratic error function having a deviation as a variable, not from a motion vector. Can be determined.

[0033]

In each of the embodiments of the present invention, in order to easily understand the operation of the apparatus, the image is composed of 288 pixels by 352 pixels in the horizontal and vertical directions, and the hierarchy does not lose generality. Shall consist of Further, it is assumed that the block for performing the correlation calculation for detecting the motion vector is composed of 16 pixels in the vertical direction and 16 pixels in the horizontal direction.

Embodiments of the present invention will be described below with reference to the drawings. The first embodiment of the present invention (claims 1, 2, 9, 10) is shown in FIG.
~ It demonstrates using FIG. Here, FIG. 2 is a structural example of the first invention, FIG. 3 is a structural example of the second invention, FIG. 4 is a structural example of the ninth invention, and FIG. 5 is a structural example of the tenth invention. is there. FIG. 1 is a block diagram of a hierarchical coding system for explaining the operations of the image coding apparatus and the image decoding apparatus. The hierarchical image is composed of two frames of brightness and transparency. These are called a luminance plane and an α plane, respectively. The hierarchical image synthesizer 107 performs the synthesis represented by Expression 2.

[0035]

[Equation 2]

In equation 2, (x, y) is the horizontal and vertical position, and g _f and g _b are the brightness values [0, 255] of the foreground and background, respectively.
, Α represents the transparency [0, 1] of the foreground. g is the synthesized luminance value. In this embodiment, since there are two layers for simplification, the transparency of the background is all 1's. The hierarchical image encoders 101 and 102 encode the respective moving images of the foreground and background luminance planes and the α plane, and the bit stream multiplexed by the multiplexer 103 is sent to the decoding device. In the decoding device, the data of each hierarchical image is separated by the demultiplexer 104, and the hierarchical images are reconstructed by the hierarchical image decoders 105 and 106. The reconstructed hierarchical image is the hierarchical image synthesizer 10
It is synthesized in 7. Hereinafter, this encoding system will be described in order.

FIG. 2 shows the present invention (claim 1) in FIG.
FIG. 3 is a configuration diagram of hierarchical image encoders 101 and 102 in one embodiment. In FIG. 2, 201 is a deformation analyzer, 2
Reference numeral 02 is a modified combiner, 203 and 204 are differentiators, 205 is a predictive code encoder, 206 is a luminance plane error encoder, 207 is an α plane error encoder, 208 is a luminance plane error decoder, and 209 is α plane error decoder, 210 is a multiplexer, 211, 212 are adders, 213, 214 are frame delays. The operation of the hierarchical image encoder configured as above will be described below.

First, the deformation analyzer 201 determines which position of the luminance plane and α plane, which is the decoding result of the previous frame, corresponds to each position of the luminance plane and α plane currently input. Correspondence information of this position is encoded by the predictive code encoder 205 as an affine transformation parameter and a block parallel movement component described later. The transformation / combiner 202 receives this correspondence information, transforms the luminance plane and the α plane that are the decoding results of the previous frame, and makes the difference signals in the difference units 203 and 204. As far as the luminance plane is concerned, this is generally CCITT Recommendation H.264. The image coding apparatus described in H.261 corresponds to a process called “motion compensation”. The difference here is that not only block movement that minimizes the sum of error absolute values (abbreviated as SAD) in a block of 16 × 16 pixels but also motion compensation that combines affine transformation of the entire screen is performed. This is shown in FIG.
Will be explained.

FIG. 6 shows the configuration of the modification synthesizer 202 of FIG. 2, in which 601 is a luminance image memory, 602 is an α image memory, 603 is a demultiplexer, and 604 and 605.
Are affine transformation units, 606 and 607 are affine transformation image memories, and 608 and 609 are image block transformation units.
The correspondence information (deformation parameter) is composed of an affine transformation parameter (see Expression 1) and a parallel movement component (see FIG. 12) of a block divided into 18 × 22 vertical and horizontal blocks. The steps of motion compensation are as follows.

1. Pixel values of the luminance plane and the α plane are loaded into the luminance image memory 601 and the α image memory 602, respectively. At the same time, the demultiplexer 603 separates the affine transformation parameter and the block translation component.

2. The image is shifted by the affine transformation units 604 and 605 with respect to the movement amount (u, v) shown in Expression 1 by the affine transformation parameter. The result is stored in the affine transformed image memories 606 and 607.

3. Affine transformed image memory 606, 60
The image stored in FIG.
The parallel movement is performed for each of the blocks vertically and horizontally divided into 18 × 22 blocks by 8 and 609 in units of blocks each having a size of 16 × 16 pixels.

To do this, the deformation analyzer 20 of FIG.
1 must extract affine transformation parameters and block movement components. FIG. 4 shows the deformation analyzer 2 of FIG.
It is a block diagram of 01. The operation of the deformation analyzer 201 will be described using this. The configuration of the deformation analyzer 201 shown in FIG. 4 is an example of the configuration of the motion vector detecting device of the present invention (claim 9).

In FIG. 4, 401 and 402 are luminance image memories, 403 and 404 are α image memories, and 405 and 40.
6 is a luminance / α superimposing unit, 407 is an affine transformation unit, 408
Is an affine transformation coefficient calculation unit, 409 is an affine transformation image memory, 410 is a block correlation calculation unit, and 411 is a multiplexer. In the modification analyzer 201 shown in FIG. 4, the luminance image memory 401 and the α image memory 403 buffer the results of the previous frame as reference images. The luminance image memory 402 and the α image memory 404 hold the image of the current input frame which is the encoding target image. The luminance / α superimposing units 405 and 406 perform the processing shown in Expression 3 to generate one luminance image. In Equation 3, for the horizontal and vertical position (x, y) of the pixel, h (x, y) is the combined luminance image, and g is the luminance value of the luminance plane [0, 25].
5] and α are values of the α plane [0, 1].

[0045]

(Equation 3)

In Expression 3, the brightness value is superimposed in the opaque area and the appropriate negative value (-100) is superimposed in the transparent area according to the value of α. As a result, a luminance image on which transparency information is superimposed is generated. This is shown in FIG. Further, instead of such threshold processing, the experimentally determined superposition coefficient γ may be multiplied to perform superposition as shown in Expression 4.

[0047]

[Equation 4]

In this embodiment, since the luminance / α superimposing section is used for other purposes in another embodiment described later, the operation defined by the equation 3 is performed. In this way, the correlation calculation defined in Equations 5 and 6 is performed on the image in which the luminance and the α value are superimposed.

[0049]

(Equation 5)

[0050]

(Equation 6)

In Equation 5, h _t-1 represents the superimposed pixel value of the previous frame (FIG. 4, reference image) and h _t represents the superimposed pixel value of the current frame (FIG. 4, target image). R represents an area of 16 × 16 pixels, and (u, v) is the deviation to the corresponding block area as shown in FIG. According to Equation 6, SAD
The smallest deviation of is obtained as the motion vector. The block correlator 410 performs this calculation on the affine transformed reference superimposed image and the target superimposed image. Note that the output of the α image memory 404 is input to the block correlating unit 410 because the calculation is omitted because the motion vector is undefined in the region where the α plane of the target image is all transparent. The block correlator 410 is 18 ×
The minimum deviation (p, q) for 22 blocks is output as a motion vector. The affine transformation unit 407 and the affine transformation image memory 409 perform the same operation as that of the block having the same name already described in FIG.

Next, the affine transformation coefficient calculation unit 408 will be described. FIG. 5 is a configuration diagram of the affine transformation coefficient calculation unit 408, which is an example of the configuration of the motion vector detection device of the present invention (claim 10). In FIG. 5, 501 is a block correlation calculation unit, 502 is an SAD phase approximation unit, 503 is an error function parameter storage memory, and 504 is an affine transformation parameter calculation unit. The operation of the block correlation calculation unit 501 is almost the same as that of the block correlation unit 410 in FIG.
The difference is that in addition to the minimum deviation (p, q) for the 18 × 22 block, the block correlation calculation unit 501
The SAD minimum value at that position and the SAD values near 8 are output. This is shown in Equation 7. In Expression 7, t represents the transpose of the matrix.

[0053]

(Equation 7)

In response to this, the SAD phase approximating unit 502 performs the operations of Expressions 8 to 13. The calculation result is stored in the error function parameter storage memory 503. The calculations of Expressions 8 to 13 are equivalent to performing the second-order Taylor expansion near the minimum deviation (p, q), considering the SAD value as a function of the deviation (u, v). When the positions of the blocks of 18 × 22 in the vertical and horizontal directions are expressed as i and j, each quadratic error function can be expressed by Expression 14, except for the transparent region.

[0055]

(Equation 8)

[0056]

[Equation 9]

[0057]

[Equation 10]

[0058]

[Equation 11]

[0059]

(Equation 12)

[0060]

(Equation 13)

[0061]

[Equation 14]

Here, since the motion vector of each block is described by the affine transformation parameters as shown in equations 15 and 16, equation 17 is used as a necessary condition for minimizing the total sum of the SAD error functions according to the variation principle. As shown, it is possible to derive the Euler equation in which the partial differential of the affine transformation parameter a must be a zero vector. This is Equation 18
Can be expressed as a matrix. Affine parameter calculator 504
For this purpose, Equation 19 (6 × 6 matrix) and Equation 20 (6 × 1 matrix) are first obtained, and then the affine transformation parameter is calculated by Equation 21.

In equations 19 and 20, (x _j ,
y _i ) is the center position of blocks i and j.

[0064]

(Equation 15)

[0065]

[Equation 16]

[0066]

[Equation 17]

[0067]

(Equation 18)

[0068]

[Formula 19]

[0069]

(Equation 20)

[0070]

[Equation 21]

The deformation analyzer 201 configured as described above.
In (FIG. 4), a motion vector can be obtained from both pieces of information by performing a correlation operation on an image in which luminance and α are superimposed. Negative value of transparent area defined by Equation 3 (-100)
If the absolute value of is increased, a motion vector in which the contour information of the opaque region is emphasized can be obtained. This is particularly effective when there are no cues such as edges and patterns effective for motion estimation inside the region. The affine transformation coefficient calculation unit 408 shown in FIG. 5 obtains the affine transformation parameter by performing a quadratic function approximation instead of performing a local correlation calculation. In the local correlation calculation, the motion vector often has a degree of freedom in the tangent direction of the contour around the monotonous contour. In this case, a large estimation error is expected in the two-step affine transformation parameter estimation shown in the conventional example, but in the method shown in this embodiment, the degrees of freedom are expressed by a quadratic function, and the quadratic function of From the minimization of summation,
It is expected that the parameters can be estimated more stably.

Further, the quadratic function approximation using the deviation of the SAD correlation as a variable has an advantage that the undetermined parameter can be easily derived because the Euler equation represented by the equation 17 has a linear form with respect to the undetermined parameter. This can be said in common when a more general polynomial is used. For example, the motion vector formulas shown in Formulas 22 and 23 can represent the motion vector generated from the projected image of the planar object under perspective transformation.

[0073]

[Equation 22]

[0074]

(Equation 23)

Also in this case, the Euler equation of Equation 24 is calculated similarly to the affine transformation, and the parameters can be easily estimated by the procedure similar to Equations 19 to 21.

[0076]

[Equation 24]

The hierarchical image encoder 10 shown in FIG. 2 has been described above.
The deformation analyzer 201 and the deformation synthesizer 202 of Nos. 1 and 102 have been described. At the same time, a configuration example of the motion vector detecting device of the present invention (claims 9 and 10) is shown. Hereinafter, the remaining blocks of FIG. 2 will be described.

Luminance data by the subtracters 203 and 204,
The difference of the transparency data is the luminance plane error encoder 2
06, sent to the α-plane error encoder 207 and encoded independently. Each encoder has the configuration shown in FIGS. 7 and 8. FIG. 7 is a block diagram of the luminance plane error encoder. 701 is a DCT operation unit, 702 is a quantization unit 1, 70.
Reference numeral 3 is a variable length coding unit 1.

FIG. 8 is a block diagram of the α-plane error encoder, in which 801 is a Haar transform operation unit, 802 is a quantization unit 2, and
Reference numeral 803 is the variable length coding unit 2. DCT calculation unit 701
Performs a discrete cosine transform on a block of 8 × 8 pixels,
The transformed DCT coefficient is quantized by the quantizer 702,
The variable length coding unit 703 scans the cosine transform coefficient and performs two-dimensional Huffman coding with a combination of the zero coefficient length and the quantized coefficient. This process is based on CCITT Recommendation H.264. Since it is almost the same as the technology disclosed in H.261, detailed description will be omitted.

The α-plane error encoder 207 uses Haar transform on a block of 8 × 8 pixels instead of discrete cosine transform. Here, the Haar transform is realized by vertically and horizontally performing a one-dimensional Haar transform in which an 8 × 1 column vector is multiplied from the right of Expression 25 on an 8 × 8 pixel block.

[0081]

(Equation 25)

The luminance plane encoder 206 differs from the luminance plane encoder 206 in that the quantization table and the Huffman table are slightly changed because the Haar transform is used instead of the discrete cosine transform. However, since the basic operation is the same, detailed description will be omitted. Returning to FIG.

The outputs of the luminance plane error encoder 206 and the α plane error encoder 207 described so far are multiplexed by the multiplexer 210 and output. On the other hand, the above-mentioned output is input to the luminance plane error decoder 208 and the α plane error decoder 209 in order to generate the predicted image of the next frame. Each decoder has the configuration shown in FIGS. 9 and 10.

FIG. 9 is a block diagram of a luminance plane error decoder. 901 is a variable length decoding unit, 902 is an inverse quantization unit, and 9
Reference numeral 03 is an inverse DCT calculation unit. FIG. 10 is a block diagram of an α-plane error decoder, in which 1001 is a variable length decoding unit,
Reference numeral 02 is an inverse quantization unit, and 1003 is an inverse Haar transform calculation unit. The variable length decoding unit 901 Huffman-decodes the combination of the zero coefficient length and the quantized coefficient to restore the cosine transform coefficient, and the inverse quantization unit 902 replaces the quantization index with the representative value, and finally, the inverse DCT operation unit 903. By 8 × 8
The image of the pixel block is reproduced. This processing is similar to that of the luminance plane error encoder 206, and CCITT Recommendation H.264. Two
Since it is almost the same as the technique disclosed in No. 61, detailed description will be omitted.

The inverse Haar transform calculation unit 906 of FIG.
It is realized by taking out a column vector of 8 × 1 vertically and horizontally with respect to the Haar coefficient of 8 and multiplying the matrix shown in Expression 26 from the left. The operations of the variable length decoding unit 1001 and the dequantization unit 1002 correspond to the α plane error encoder 207, and only the contents of the table are different from those of the block of the luminance plane decoder 209. The description is omitted.

[0086]

(Equation 26)

Next, the hierarchical image decoders 105 and 106 constituting the hierarchical coding system of FIG. 1 will be described with reference to FIG. FIG. 3 is a configuration diagram of hierarchical image decoders 105 and 106 corresponding to an example of the configuration of the image decoding device of the present invention (claim 2).

In FIG. 3, 301 is a demultiplexer, 302 is a luminance plane error decoder, 303 is an α plane error decoder, 304 is a predictive code decoder, and 30.
5 is a modification synthesizer, 306 and 307 are adders, 308 and 3
Reference numeral 09 is a frame delay device. Demultiplexer 301
In the input of, a deformation parameter composed of a luminance error, an α error, an affine transformation parameter, and a movement vector for a block divided into 18 × 22 vertical and horizontal blocks is multiplexed. These are separated and output to the luminance plane error decoder 302, the α plane error decoder 303, and the prediction code decoder 304, respectively.

Here, the luminance plane error decoder 302,
The α-plane error decoder 303 and the modified combiner 305 perform exactly the same operations as the luminance plane decoder 208, the α-plane decoder 209, and the modified combiner 202 of FIG. 2, respectively.
As described above, the hierarchical encoder 10 that constitutes the hierarchical encoding of FIG.
The first embodiment has been described in which the configurations of the first and second hierarchical decoders 105 and 106 are composed of FIGS. 2 and 3, respectively. The present embodiment is characterized in that the template is updated as sequential interframe coding. In the template, the difference in transparency is multi-valued waveform information, and this is converted and encoded. By independently encoding the α plane, a moving image of a semitransparent object such as frosted glass can be handled, unlike the second embodiment described later. Since the images are hierarchized, there are more cases where the foreground or background can be described only by the affine transformation parameters.

In this case, it is not necessary to transmit only the affine transformation parameter and encode the block moving component, other luminance error image, and α plane error image, so that the encoding efficiency is greatly improved. Further, when the object is deformed and cannot be described by the affine transformation parameter, the template is updated with the block movement component, another luminance error image, and the α plane error image, so that a large image quality deterioration does not occur.

By the way, the affine parameter calculation unit 504
It is not necessary to perform the calculation of Equations 19 to 21 performed in step 1) for the entire image. Estimating the estimated affine transformation parameters by excluding the block having a large error value using Equation 14 enables matching to the motion of a large number of blocks instead of the entire block set divided into 18 × 22 blocks. Efficient affine transformation parameters can be estimated. This makes it possible to locally encode the block movement component for template correction, another luminance error image, and the α plane error image. Further, in the present embodiment, the block correlation calculation is SAD, but other evaluation measures such as sum of squared errors (SSD) and correlation coefficient can be used.

Next, a second embodiment of the present invention (claims 3 and 4) will be described below with reference to FIGS. 1, 11, 13, 14, and 15.
This will be described with reference to FIGS. Also in the second embodiment, the configuration of the hierarchical image coding system is the same as that in FIG. The hierarchical image encoders 101 and 102 in FIG. 1 have the configuration of FIG. 13 in this embodiment. Also, the configuration of the hierarchical image decoders 105 and 106 differs from that of the first embodiment in that the configuration of FIG. 14 is adopted. Here, FIG. 13 corresponds to a configuration example of the third invention, and FIG. 14 corresponds to a configuration example of the fourth invention.

In FIG. 13, 1301 is a deformation analyzer.
1302 is a deformation synthesizer, 1303 is a luminance / α separation unit, 1
304 is a luminance / α superimposing unit, 1305 is a frame delay unit,
1306 is a difference unit, 1307 is an adder, 1308 is a luminance / α superimposition error encoder, 1309 is a predictive code encoder,
Reference numeral 1310 is a luminance / α superposition error encoder, and 1311 is a multiplexer. The configuration of FIG. 13 is basically the same as the configuration of the hierarchical encoder shown in FIG.

Of the blocks constituting FIG. 13, the deformation analyzer 1301, the frame delay unit 1305, and the difference unit 13 are included.
The operations of 06, adder 1307, and predictive encoder 1309 are exactly the same as those of the block having the same name in FIG. In the first embodiment, the luminance plane and the α plane are encoded separately, but in the present embodiment, the luminance / α superimposing unit 1304 uses the value of α as shown in FIG. Then, an appropriate negative value (-10) is superimposed. This is shown in Equation 27.

[0095]

[Equation 27]

As a result, a luminance image on which transparency information is superimposed is generated. On the contrary, the brightness / α separation unit 1303 calculates the brightness and α from the brightness information superimposed by Expressions 28 and 29.
Are separated.

[0097]

[Equation 28]

[0098]

[Equation 29]

The constant -10 in the equation 27 is the luminance in the equation 28 and the equation 29 due to the quantization error caused by the encoding / decoding.
It is a value set in consideration of α separation. The operation of the modification synthesizer 1302 is the modification synthesizer 202 in FIG.
The image is almost the same except that the image treated as is a luminance / α-superimposed image. The structure is shown in FIG.

FIG. 15 is a block diagram of the modification synthesizers 1302 and 1405, in which 1501 is a luminance / α memory and 1502.
Is a demultiplexer, 1503 is an affine transformation unit, 15
Reference numeral 04 is an affine transformed image memory, and 1505 is an image block transformation unit. The transformation combiner 1302 operates by inputting the affine transformation parameter and the correspondence information (transformation parameter) formed by the parallel movement component of the block divided into 18 × 22 vertical and horizontal blocks. Brightness / α memory 15
Reference numeral 01 is a memory for buffering the brightness / α superposed image. The operation of the other blocks in FIG. 15 is the same as that of the block having the same name in FIG.

Next, the luminance / α superposition error encoder 1308 in FIG. 13 will be described below. Fig. 16 shows brightness / α
In the configuration diagram of the superposition error encoder 1308, 1601 is a region boundary determination unit, 1602 and 1609 are switches, 1603 is a DCT operation unit, 1604 is a quantization unit, 1605 is a variable length encoding unit, 1606 is a Haar transform operation unit, Reference numeral 1607 is a quantizer, 1608 is a variable length encoder, and 1610 is a multiplexer. Region boundary determining section 1601 outputs the control and switching information bits shown in (Table 1).

[0102]

[Table 1]

The luminance plane error encoder 206 described with reference to FIG. 7 is operated in the intra-object region where the value of all α in the 8 × 8 block is 1, and at least one α in the 8 × 8 block is operated. In the contour area where the value of is 0, the operation of the α-plane error encoder 207 described in FIG. 8 is performed.

The operations of the blocks having the same names are the same in FIGS. 16 and 7 and 8. The quantizing unit 1607 and the variable length coding unit 1608 in FIG. 16 use the quantizing unit 80 in order to code the multivalued pattern around the contour on which the luminance is superimposed.
2. It differs from the variable length coding unit 803 in that the quantization table and the Huffman table are slightly changed. With the above configuration, the conversion code and the switching information bit are multiplexed and output from the multiplexer 1610.

Generally, for luminance information, DCT is superior in coding efficiency to Haar transform, but a ripple called mosquito noise is generated in a region including a steep edge. This is not preferable for reproducing the α value around the contour.
Therefore, the Haar transform that does not generate the mosquito noise is used around the contour.

Next, the above-mentioned luminance / α superposition error encoder 1
Luminance / α error decoders 1310, 14 corresponding to 308
The configuration of No. 02 will be described with reference to FIG. Figure 17 shows the brightness
It is a block diagram of alpha superposition error decoder 1310, 1402, 1701 is a demultiplexer, 1702 is a switching control part, 1703, 1710 is a switch, 1704 is a variable length decoding part, 1705 is a dequantization part, 1706 is an inverse. DCT
An arithmetic unit, 1707 is a variable length decoding unit, 1708 is an inverse quantization unit, and 1709 is an inverse Haar transform arithmetic unit. The switching control unit 1702 sets the switches 1803 and 1710 corresponding to (Table 1) by the switching information bits separated by the demultiplexer 1701 to 0 so that the inverse DCT operation unit 1706 is selected if the bit is 1. For example, the inverse Haar transform 1709 is controlled to be selected. The operations of the blocks having the same names shown in FIGS. 17, 9, and 10 are the same.

The variable length decoding unit 1707 and the dequantization unit 1708 in FIG. 17 are respectively the quantization unit 1 in FIG.
Inverse processing corresponding to 607 and the variable length coding unit 1608 is performed.

Next, the structure of the hierarchical image decoders 105 and 106 in FIG. 1 will be described with reference to FIG. In FIG. 14, reference numeral 1401 denotes a demultiplexer, 1402 denotes luminance,
α convolutional error decoder, 1403 is a predictive code decoder, 1
Reference numeral 404 is an adder, 1405 is a modification combiner, 1406 is a frame delay unit, and 1407 is a luminance α separation unit. The demultiplexer 1401 receives the outputs of the hierarchical image encoders 101 and 102 configured as shown in FIG. 13, and separates the transformation parameter and the bit sequence of the luminance / α error image. The separated data are output to the luminance / α superposition error decoder 1402 and the prediction code decoder 1403, respectively. The predictive code decoder 1403 uses the affine transformation parameters and the vertical and horizontal 18 × 2.
The motion vector of the second block is output. The operation of the other blocks is the same as the block of the same name described in FIG.

The second embodiment has been described above. In the second embodiment, unlike the first embodiment, the information of the α plane is degenerated from multi-valued [0, 1] to binary. Instead, α
By superimposing the plane information on the luminance plane,
It is possible to treat the deformation of the template as the difference of the luminance information. In the present embodiment, the brightness value of the transparent area is set to −10 as shown in Expression 27, but when the brightness of the opaque area is close to 255, a large discontinuity of the brightness occurs in the contour of the object and the code amount increases. It is expected that. in this case,
It is also easy to expand the operation of the brightness / α superimposing unit 1304 adaptively to Expression 27 or Expression 30, and further change the operation of the brightness / α separating units 1303 and 1407 to Expressions 31 and 32.

[0110]

[Equation 30]

[0111]

[Equation 31]

[0112]

[Equation 32]

It is also conceivable to extend the luminance handled in this embodiment into a vector and superimpose the α plane information on the luminance plane in the vector space. For example, consider a three-dimensional color vector c composed of three primary colors or luminance and a color difference signal. The average c- and variance Σ of this color vector are calculated, and cc-normalized by the variance Σ as shown in Expression 33.
The value range Th of the quadratic form of is calculated.

[0114]

[Expression 33]

Th of Expression 33 for one template
And the data of the variance, the average vector, and the range Th are attached, it is possible to superimpose the information of the α plane by using an arbitrary vector e in which the value of the quadratic form is larger than the threshold value as in Expression 34. it can.

[0116]

(Equation 34)

This separation can be performed by the equations 35 and 36.

[0118]

[Equation 35]

[0119]

[Equation 36]

According to this, it is possible to reduce the strength of the step edge generated near the object contour in the separated luminance image.

Next, a third embodiment of the present invention (claims 5 and 6) will be described below with reference to FIGS. FIG.
Is a block diagram of an image encoding apparatus corresponding to the configuration example of the present invention (claim 5), in which 1801 is a foreground memory, 1802 is a background memory, 1803 is a foreground separator, 1804 and 1805 are hierarchical image encoders, 1806. Is a multiplexer, 180
Reference numeral 7 is a demultiplexer, 1808 and 1809 are hierarchical image decoders, 1810 is a hierarchical image synthesizer, 1811 is a predictive image memory, 1812 is a differentiator, 1813 is a luminance plane error encoder, and 1814 is a multiplexer.

FIG. 19 is a block diagram of an image decoding apparatus corresponding to the configuration example of the present invention (claim 6).
Is a demultiplexer, 1913 and 1914 are hierarchical image decoders, 1915 is a hierarchical image synthesizer, 1916 is a predictive image memory, 1917 is a luminance plane error decoder, 19
18 is an adder. Among the above blocks, the hierarchical image encoders 1804 and 1805, the multiplexer 1806, the demultiplexers 1807 and 1912, the hierarchical image decoders 1808, 1809, 1913 and 1914, and the hierarchical image synthesizers 1810 and 1915 are either the first or the second. 1 in the embodiment of FIG. The luminance plane error encoder 1813 uses the luminance plane error encoder 207 of FIG. 2, and the luminance plane error decoder 1917 uses the luminance plane error decoder 208 of FIG.

In the image coding device and the image decoding device configured as described above, the background image is captured in advance and stored in the background image memory 1802. The foreground separator 1803 separates the foreground by determining the α value according to Expression 37.

[0124]

(37)

In Equation 37, g is the luminance value of the camera input, g _b is the luminance value of the background memory, and T is the threshold value [0, 1] experimentally determined. The result is input to the foreground memory 1801. After that, the hierarchical image is subjected to the processing described in the first embodiment or the second embodiment, and the processing result is output to the prediction image memory 1811.

In the two embodiments described so far, this is used as the output of the reconstructed image, but in this embodiment,
Further, the difference between the output of the predictive image memory 1811 and the original image is obtained by the differentiator 1812, this is subjected to error coding, and the coded result is multiplexed and outputted from the multiplexer 1814. Decoding of this output is sent to the demultiplexer 1912 via the demultiplexer 1911, and the same processing as the first and second conventional examples is performed. The other is sent to the adder 1918 via the luminance plane error decoder 1917. Adder 19
In 18, the predicted image re-synthesized by the hierarchical image synthesizer 1915 and the above-mentioned error image are added and the image data is output. In the present embodiment, when the foreground separation succeeds and the object in the foreground makes a motion that can be described by the affine transformation parameter, it is possible to obtain the same high coding efficiency as in the first and second embodiments. In addition, by performing error coding on the synthesis result of the hierarchical image, even if the result of the foreground separation contains an error or the template update is defective due to the limitation of the coding amount, the visual deterioration is caused. It is possible to perform transmission recording of images with few images.

Next, a fourth embodiment of the present invention (claims 7 and 8) will be described below with reference to FIGS. 20, 21, 22 and 1, 5, 13, 14. In this embodiment as well as in the second embodiment, the hierarchical image is held in a format in which the brightness and the transparency are superimposed (see Expression 27). In the present embodiment, the transmission and recording of a plurality of templates is performed using the hierarchical image encoder and the hierarchical image decoder described in FIGS. 13 and 14 used in the second embodiment in advance.

FIG. 22 is a conceptual diagram of hierarchical image coding using multiple templates. The images shown as template A and template B in FIG. 22 are obtained by directly connecting the hierarchical image encoder and the hierarchical image decoder in the second embodiment. One template is selected from several frames or several tens of frames and transmitted. As a result, the template encoding is realized as "inter-template encoding" utilizing the correlation between templates, and efficient template transmission can be performed.

FIG. 20 is a block diagram of an image coding apparatus according to an embodiment of the present invention (claim 7). 2001 is an affine transform coefficient operation unit, 2002 is a luminance / α separation unit, 2003 is a template storage memory, and 2004 is a template storage memory. Is an affine distance shortest template determination unit, 2005 is a predictive code encoder, 200
6 is a multiplexer.

FIG. 21 is a block diagram of an image decoding apparatus according to an embodiment of the present invention (claim 8). 2101 is a demultiplexer, 2102 is a predictive code encoder, 2103 is a template reading circuit, 2104 is a storage memory. Reference numeral 2105 is an affine transformation unit, and 2106 is a luminance / α superimposing unit. After transmitting the templates of the foreground and the background, the hierarchical image encoder shown in FIG. 20 is the hierarchical image encoders 101 and 102 of FIG. 1, and the hierarchical image decoder shown in FIG. 21 is the hierarchical image decoder of FIG. It is used as a container 105, 106.

The transmitted template is stored in the template memories 2003 and 2104 with the same identifier on the encoding side and the decoding side, respectively. The affine transformation coefficient calculation unit 2001 is the affine transformation coefficient calculation unit 4 described with reference to FIG.
In addition to the operation of 08, the deviation of each block is calculated from Expression 15 using the obtained affine transformation coefficient, the deviation of each block is substituted into Expression 14, and the error sum of each block is calculated to obtain the shortest affine distance as an approximation error. It is output to the template determination unit 2004. Affine distance shortest template determination unit 200
4 selects the smallest template from the obtained approximation errors, and outputs the deformation parameter together with its identifier.

The predictive code encoder encodes the affine transformation parameters. The multiplexer 2006 multiplexes the encoded affine transformation parameter and the template identifier to generate an output bitstream of the hierarchical image encoder. On the decoding side, the input bit stream is separated by the demultiplexer 2101 and the predictive code decoder 210
At 2, the affine transformation parameters are decoded. The template reading circuit 2103 reads the template corresponding to the input template identifier and outputs it to the affine transformation unit 2105. The processing of the affine transformation unit 2105 is the same as that of the affine transformation unit 407 in FIG. The operations of the brightness / α separation units 2002 and 2106 are the same as the operations of the brightness / α separation units 1303 and 1407 in FIGS. 13 and 14.

According to the above processing, as shown in FIG. 22, since the image can be transmitted based on the template that can be approximated by the affine transformation, the image can be transmitted with a very small code amount. In particular, high efficiency can be expected because the template to be used can be selected without depending on the order of time. Further, instead of synthesizing an image from a single template as in the conventional example, a plurality of templates are selectively used, so that it is possible to deal with the case where the image has irregular deformation. Note that the transformation used here is not limited to the affine transformation, but general polynomial transformation description such as the transformation of the planar object shown in Expression 22 can be used. Although the template is handled according to the second embodiment, it may be changed to the template used in the first embodiment.

[0134]

The image encoding device of the first invention and the second invention
According to the image decoding apparatus of the present invention, the deformation of the reference image (template) composed of the brightness and the transparency of the object can be treated as the prediction by the correspondence between the templates and the difference between the prediction results.

According to the image coding apparatus of the third invention and the image decoding apparatus of the fourth invention, the conventional high-efficiency coding technique is realized by superposing the brightness and the transparency of the object and treating them as a brightness image. It can be applied, and template deformation can be handled efficiently.

According to the image coding apparatus of the fifth aspect of the invention and the image decoding apparatus of the sixth aspect of the invention, the synthesis of the hierarchical images is not the final result but the predicted image, and the predicted image and the original image are combined. By transmitting and recording the difference, even if there is an error in the foreground / background separation processing, the image can be transmitted and recorded without significant visual deterioration.

According to the image coding apparatus of the seventh invention and the image decoding apparatus of the eighth invention, a plurality of templates are transmitted in advance, and the minimum distortion template for the input image is sequentially selected and selected regardless of the time order. Since the image sequence is reproduced by the modification of the created template, the image can be transmitted with extremely high efficiency.

The motion vector detecting device of the ninth invention can treat the correspondence problem of the image composed of the luminance and the transparency as the correspondence problem of the luminance image in which the information of the luminance and the transparency is superimposed. As a result, it is possible to obtain the correspondence result, that is, the motion vector, in which both the brightness and the transparency are taken into consideration.

The motion vector detecting apparatus of the tenth aspect of the invention does not perform the polynomial approximation in two steps after once obtaining the motion vector when performing the polynomial approximation in which the correspondence between the images is the position variable. It can be obtained directly from the corresponding error function, and motion vector estimation robust to noise can be performed.

Hierarchical coding, in which the projected image of the object is separated according to the context and individually coded, can reduce the amount of code generated due to the movement of different regions, the appearance of concealed regions, etc., and high efficiency. Coding properties are expected. recent years,
Hierarchical coding has a high industrial value because hierarchical images and computer graphics generated by chromakey are frequently used for image synthesis. According to the invention described above, efficient coding of luminance and transparency images having irregular deformations and robust estimation of motion vectors related thereto necessary for hierarchical image coding are performed. Can be done and its effect is great.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a hierarchical coding system according to an embodiment of the present invention.

FIG. 2 is a configuration diagram of a hierarchical image encoder according to the first embodiment.

FIG. 3 is a configuration diagram of a hierarchical image decoder in the first embodiment.

FIG. 4 is a configuration diagram of a deformation analyzer according to the present embodiment.

FIG. 5 is a configuration diagram of an affine transform coefficient calculation unit according to the present embodiment.

FIG. 6 is a configuration diagram of a modified combiner of the present embodiment.

FIG. 7 is a configuration diagram of a luminance plane error encoder according to the present embodiment.

FIG. 8 is a configuration diagram of an α-plane error encoder according to the present embodiment.

FIG. 9 is a configuration diagram of a luminance plane error decoder according to the present embodiment.

FIG. 10 is a configuration diagram of an α-plane error decoder according to the present embodiment.

FIG. 11 is a diagram showing the operation of the brightness / α superimposing unit of the present embodiment.

FIG. 12 is a block correlation diagram of this embodiment.

FIG. 13 is a configuration diagram of a hierarchical image encoder according to the second embodiment of the present invention.

FIG. 14 is a configuration diagram of a hierarchical image decoder in the second embodiment.

FIG. 15 is a block diagram of a modified synthesizer of this embodiment.

FIG. 16 is a configuration diagram of a luminance / α superposition error encoder according to the present embodiment.

FIG. 17 is a block diagram of a luminance / α superposition error decoder according to the present embodiment.

FIG. 18 is a configuration diagram of an image encoding device according to a third embodiment.

FIG. 19 is a configuration diagram of an image decoding apparatus according to a third embodiment.

FIG. 20 is a configuration diagram of a hierarchical image encoder according to the fourth embodiment.

FIG. 21 is a configuration diagram of a hierarchical image decoder in the fourth embodiment.

FIG. 22 is a conceptual diagram of hierarchical image coding by the multiple template according to the present embodiment.

FIG. 23 is a conceptual diagram of conventional hierarchical image coding.

[Explanation of symbols]

 101, 102 Hierarchical image encoder 103 Multiplexer 104 Demultiplexer 105, 106 Hierarchical image decoder 107 Hierarchical image synthesizer 201 Deformation analyzer 202 Deformation synthesizer 203, 204 Difference device 205 Predictive code encoder 206 Luminance plane error code 207 α plane error coder 208 luminance plane error decoder 209 α plane error decoder 210 multiplexers 211, 212 adders 213, 214 frame delay unit 301 demultiplexer 302 luminance plane error decoder 303 α plane error Decoder 304 Predictive code decoder 305 Deformation synthesizer 306, 307 Adder 308, 309 Frame delay device 401, 402 Luminance image memory 403, 404 α image memory 405, 406 Luminance / α superimposing unit 407 Fin transform unit 408 Affine transform coefficient computing unit 409 Affine transform image memory 410 Block correlation computing unit 411 Multiplexer 501 Block correlation computing unit 502 SAD phase approximation unit 503 Error function parameter storage memory 504 Affine transformation parameter computing unit 601 Luminance image memory 602 α image Memory 603 Demultiplexer 604, 605 Affine transformation unit 606, 607 Affine transformation image memory 608, 609 Image block transformation unit 701 DCT calculation unit 702 Quantization unit 703 Variable length coding unit 801 Haar transformation calculation unit 802 Quantization unit 803 Variable length Encoding unit 901 Variable length decoding unit 902 Inverse quantization unit 903 Inverse DCT operation unit 1001 Variable length decoding unit 1002 Inverse quantization unit 1003 Inverse Haar transform operation unit 1301 Modification analyzer 1302 Modified combiner 1303 Luminance / α separation unit 1304 Luminance / α superposition unit 1305 Frame delay unit 1306 Difference unit 1307 Adder 1308 Luminance / α superposition error encoder 1309 Prediction code encoder 1310 Luminance / α superposition error encoder 1311 multiplexer 1401 demultiplexer 1402 luminance / α superimposition error decoder 1403 prediction code decoder 1404 adder 1405 modified combiner 1406 frame delayer 1407 luminance α separation unit 1501 luminance / α memory 1502 demultiplexer 1503 affine transformation unit 1504 affine Transformed image memory 1505 Image block transformation unit 1601 Region boundary determination unit 1602, 1609 Switch 1603 DCT calculation unit 1604 Quantization unit 1605 Variable length coding unit 1606 Haar transform calculation unit 1 07 Quantizer 1608 Variable length encoder 1610 Multiplexer 1701 Demultiplexer 1702 Switching controller 1703, 1710 Switch 1704 Variable length decoder 1705 Inverse quantizer 1706 Inverse DCT calculator 1707 Variable length decoder 1708 Inverse quantizer 1709 Inverse Haar transform calculation unit 1801 Foreground memory 1802 Background memory 1803 Foreground separator 1804, 1805 Hierarchical image encoder 1806 Multiplexer 1807 Demultiplexer 1808, 1809 Hierarchical image decoder 1810 Hierarchical image combiner 1811 Predictive image memory 1812 Difference device 1813 Luminance plane error encoder 1814 Multiplexer 1911, 1912 Demultiplexer 1913, 1914 Hierarchical image decoder 1915 Hierarchical image synthesizer 191 6 Prediction image memory 1917 Luminance plane error decoder 1918 Adder 2001 Affine transform coefficient calculation unit 2002 Luminance / α separation unit 2003 Template storage memory 2004 Affine distance shortest template determination unit 2005 Prediction code encoder 2006 Multiplexer 2101 Demultiplexer 2102 Prediction Code encoder 2103 Template reading circuit 2104 Storage memory 2105 Affine transformation unit 2106 Luminance / α superposition unit

Claims (10)

[Claims] 1. An image of brightness and transparency of an image to be coded by inputting a sequence of images composed of brightness and transparency of an object, and from a reference image composed of brightness and transparency to a correspondence between partial regions. Predicting means, a predictive coding means for coding the correspondence between the partial areas in the predicting means as a predictive code, and a difference in brightness and transparency between the predicted image and the image to be coded as an error image. An image code having an error calculating means for obtaining and an error coding means for coding the error image as an error image code, and transmitting and recording an image sequence as an error image code and a prediction code for the reference image. Device. 2. A device for holding the same reference image as the image coding device according to claim 1, and decoding the output of the image coding device, wherein the correspondence between partial regions is decoded by a predictive code. Predictive code decoding means, a predicted image generation means for generating a predicted image from a reference image based on the correspondence between the partial areas, an error image decoding means for decoding an error image from an error image code, and the predicted image And an addition means for adding the error image to obtain an image composed of brightness and transparency, and decoding an image composed of brightness and transparency as an output of the prediction image generation means or the addition means. An image decoding device characterized by: 3. An image which is composed of the brightness and the transparency of an object is input and the area is classified into two areas, a transparent area and an opaque area. The brightness of the object is determined for the opaque area and the brightness value is determined for the transparent area. A luminance image in which luminance and transparency information is superimposed so as to take a predetermined value outside the range, and a luminance image in which the luminance and transparency information is superimposed is encoded. Image encoding device. 4. An apparatus for decoding the output of the image coding apparatus according to claim 3, wherein the image is used as a transparent area when the brightness value is out of the range and as a brightness value when the brightness value is within the range. An image decoding apparatus having a separating unit for separating a transparency image and a brightness image, and decoding the brightness and transparency images. 5. When the original image is hierarchically expressed by the front-and-rear relationship on the line-of-sight axis and the transparency of the region in addition to the luminance, the plurality of hierarchized images are input and the luminance is calculated for each hierarchical image. A hierarchical image coding means for coding the transparency and the transparency as a hierarchical image code, and a hierarchical image image decoding means for obtaining the hierarchical image decoded from the result of the hierarchical image coding means,
There are provided a synthesizing unit for synthesizing the plurality of decoded hierarchical images according to their context, luminance and transparency, and an error image coding unit for obtaining an error image between the original image and the synthesized image and encoding the error image. An image coding apparatus is characterized in that the original image is transmitted and recorded by an error code between a plurality of hierarchical image codes and the original image. 6. A device for decoding the output of the image coding device according to claim 5, wherein a hierarchical image consisting of luminance, transparency, and a front-back relationship on a line-of-sight axis is decoded from a plurality of hierarchical image codes. An image by adding an error image to the synthetic image, and a hierarchical image decoding unit for generating a synthetic image from the hierarchical image, and an error image decoding unit for decoding an error image from an error code. An image decoding apparatus, which is characterized by decoding. 7. A reference image coding means for transmitting and recording a plurality of reference images in advance, and a deviation or deformation of a position where luminance is corresponding between the input image and the plurality of reference images is variable as a position on the screen. And a reference image with a small approximation error is obtained from the plurality of reference images, and the identifier of the selected reference image and the coefficient of the polynomial function are output. Minimum distortion reference image selecting means for encoding a plurality of reference images by the reference image encoding means, and transmitting an input image sequence as at least an identifier for the selected reference image and a coefficient of the polynomial function. An image encoding device for recording. 8. An apparatus for decoding the output of the image encoding apparatus according to claim 7, comprising: reference image decoding means for reconstructing a plurality of reference images in advance; and a plurality of the reference images. Reference image selection means for selecting a reference image corresponding to the identifier for the reference image included in the input, and a polynomial function whose image position is a variable for image deformation is determined based on the coefficient of the polynomial function included in the input. An image decoding apparatus comprising: a reference image deforming unit that deforms the selected reference image by the polynomial function, and decodes an image using the reference image deformed by the reference image deforming unit. . 9. A plurality of images composed of the brightness and the transparency of an object are input, and the transparency is subjected to addition and multiplication of predetermined values, threshold processing is performed as necessary, and a range is converted, and the converted value is obtained. Is added to the luminance to generate a luminance image in which the information on the luminance and the transparency is superimposed, and image analysis means for obtaining the correspondence between the partial areas of the two images by the correlation of the luminance, and the superimposing means. According to the above, an image composed of brightness and transparency is converted into an image composed of brightness only, and correspondence between partial areas is obtained between the converted plurality of images by using the image analysis means. Motion vector detection device. 10. A device for expressing a motion vector at an arbitrary position on a screen as a polynomial function having the position as a variable, wherein a plurality of partial regions obtained by dividing an image are different two image parts. An error calculating means that calculates the correspondence between regions as an error and obtains the deviation between the partial regions having the minimum error and the error value in the vicinity thereof, and the deviation from the deviation having the minimum error and the error value in the vicinity thereof. An error function calculating means for obtaining a quadratic error function as a variable and the sum or partial sum of the quadratic error function are expressed by using the coefficient of the polynomial function as a variable, and the sum or the partial sum is minimized for the coefficient. A motion vector detecting device having an optimizing means and outputting a motion vector between different images as a coefficient of a polynomial function.