JP3405788B2 - Video encoding device and video decoding device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture coding apparatus and a moving picture decoding apparatus therefor.

[0002]

2. Description of the Related Art Image coding technology has been used for image communication,
It is used for broadcasting and storage. Image encoding is carried out for the purpose of transmitting as fast as possible for still image transmission such as facsimile, and for moving image communication such as video conferencing and videophone, it uses a narrow band or low bit rate as much as possible. It is done for the purpose of transmission. Further, when recording image information on a disk or a memory, an image encoding technique is used to efficiently record as many images as possible.

FIG. 5 shows a block diagram of a conventional moving picture coding apparatus. The input image signal is divided into a plurality of blocks, and a motion vector indicating the motion of the image of the previous frame is estimated for each block. The motion vector is used for motion compensation of the image of the previous frame, and the difference from the input image is obtained. This difference signal is DCT (discrete cosine transform)
Quantized, variable length coded and output.

As shown in the DCT here, in the conventional transform coding, the transform method (transform matrix) is performed using only a single transform on the assumption that the image is stationary. It was Further, in the adaptive KL transform coding method in which a plurality of transform matrices are prepared and switched, a method with poor calculation efficiency such as full search for fixed quantization in order to select a transform matrix has been studied. The optimum method for moving images has not been established. Further, as a method of selecting an optimum transformation from the KL transformations prepared in advance, there are a method of using a distance in an input autocorrelation matrix space, a method of assuming that an image always has directionality, and the like. Since it is not the optimal method from the viewpoint of coding error minimization, its performance has deteriorated.

On the other hand, in the conventional image coding system shown in FIG. 5, even if the optimum conversion method is selected, the input image signal is always translated in block units to perform motion compensation. The prediction error was converted. Therefore, at a low bit rate, the prediction error cannot be sufficiently transmitted, so that block-shaped distortion occurs and the image quality is deteriorated. In addition, the background portion that appears from the back of the moving object cannot be predicted, so that the coding efficiency is reduced.

[0006]

As described above, in the conventional image coding system, the motion compensation is performed by moving the blocks in parallel. Therefore, if a block straddles the static region and the dynamic region, it is either moved in parallel to the dynamic region or left as it is in the static region. In either case, the difference signal in one area becomes large. Further, even when the subject is deformed, such as a person's eyes or mouth, sufficient movement compensation cannot be performed only by the parallel movement of the blocks. Also, the subject rotates,
The same applies when the subject is enlarged or reduced by zooming the camera.

The present invention has been made in order to solve the above-mentioned problems, and an object of the present invention is to provide a moving picture coding apparatus capable of sufficiently compensating the motion of a subject and obtaining a high coding efficiency. .

Further, to provide a moving picture decoding apparatus for decoding the encoded data.

[0009]

According to a first aspect of the present invention , a moving object is extracted from an input image signal and its movement is analyzed to obtain the input image signal and a reconstructed image subjected to motion compensation. A moving object analysis unit that outputs a residual signal and a motion parameter that is the difference between the moving object, a residual coding unit that codes the residual signal from the moving body analyzing unit to form residual coding information, and and motion parameters from the moving object analyzing means and the residual coding information from the residual encoding means
Reconstructing means for reconstructing a reconstructed image signal using the variable parameter coding for variable-length coding the motion parameter from the moving object analyzing means and the residual coding information from the residual coding means. A moving picture coding apparatus having means,
The moving object analysis means is a part of the input image signal.
Global motion parameters related to the motion of the entire motion region
Global motion estimation means for estimating
A polygonal pattern is added to the moving area in the signal or reconstructed image signal.
By hitting the
By estimating the polygon patch motion parameters,
A polygonal patch for estimating fine movements in the writing area
Motion estimation means and the global motion estimation means
Fixed global motion parameters and the polygon patch
Polygonal patch motion parameters estimated by motion estimation means
Motion compensation for motion compensation of the reconstructed image signal
A moving picture coding device characterized by comprising a compensation means
is there. According to a second aspect of the present invention, the moving body analyzing means is
Equipped with moving area extraction means for extracting moving areas from input image signals
The moving picture coding apparatus according to claim 1, wherein
Is. The invention of claim 3 is the motion of the reconstructed image signal.
A feature point extracting means for extracting feature points from the region is provided.
Place the polygon patch so that the vertices of the polygon patch come to the sign
The moving picture coding apparatus according to claim 2, wherein the moving picture coding apparatus is applied.
It is a place. According to a fourth aspect of the present invention, the polygon patch motion estimation is performed.
The determining means is the motion extracted by the feature point extracting means.
Apply a small polygon patch around the feature points in the region.
Estimate the polygon patch parameters by and
Applies a larger polygonal patch to the moving area to
Claim characterized by estimating rectangular patch parameters
The moving picture coding device according to item 3. Invention of Claim 5
The moving body analysis means temporally analyzes the input image signal.
Take a difference and search for a large difference area from the top, bottom, left, and right of the image.
The search is performed, and the area is set as a moving area.
4 is a moving picture coding device according to item 4. The invention of claim 6 is
A background image forming a background image signal from the reconstructed image signal
Image signal composing means, wherein the motion compensating means comprises the background
According to the background image signal formed by the image signal forming means,
Also, compensate for the background that appears from behind the moving area.
The moving picture coding device according to claim 1.
The invention of claim 7 receives the variable length coded data, decodes this <br/> encoded data, the entire moving region in the image signal
Global motion parameters related to body motion , said image
Variable-length decoding means for separating into a polygon patch motion parameter and a residual code related to the motion of a part of the moving region in the image signal, and the residual code output from this variable-length decoding means is decoded and left. Residual code decoding means for outputting a difference signal and the decoded image of the previous frame accumulated in the frame memory
Motion compensation of the entire motion region from the signal and the global motion parameter output from the variable length decoding means,
The global motion compensation means for outputting the global motion compensation image signal , the global motion compensation image output from the global motion compensation means, and the polygon patch motion parameter output from the variable length decoding means A polygon patch compensating means for locally performing motion compensation and outputting a motion-compensated predicted image signal , a motion-compensated predicted image signal output from the polygon patch compensating means, and the residual code decoding moving image by adding the residual code output from the unit, constitutes a decoded image signal, and wherein the more becomes possible and adding means for sending the decoded image in the frame memory
It is a decoding device. The invention of claim 8 is an input image signal
By extracting the moving body from these and analyzing this movement,
Between the input image signal and the motion compensated reconstructed image
An animal-body solution that outputs a residual signal that is a difference and a motion parameter
Analysis step and the residual in the animal body analysis step
Residual code that encodes the difference signal to form residual coded information
And the motion pattern in the moving object analysis step.
Parameter and the residual in the residual encoding step
Reconstruction for reconstructing a reconstructed image signal using encoded information
And the movement in the moving body analysis step.
Parameters and residual code in the residual coding step
Variable-length coding step for variable-length coding the coding information;
A moving image encoding apparatus having:
The step is the entire moving area in the input image signal.
A group for estimating global motion parameters related to motion.
The global motion estimation step and the input image signal or
Applies a polygonal patch to the moving area in the reconstructed image signal.
The polygonal pattern associated with the movement of the moving area.
The motion region by estimating the motion parameters
Polygon patch motion estimation for estimating small motions in a tree
And the global motion estimation step
Wherein the estimated global motion parameters were polygonal package
H. Polygon patch motion estimated in the motion estimation step
Motion compensation of the reconstructed image signal according to the
A moving image comprising a motion compensation step
This is an encoding method. According to a ninth aspect of the present invention, a variable length coding data is used.
Data, decodes this encoded data, and
Movement related to the movement of the entire moving area in
Parameter, the motion of the part of the moving area in the image signal
-Related polygonal patch motion parameters and residual codes
Variable length decoding step to separate into
The residual code output in the step is decoded and the residual signal is decoded.
Error code decoding step for outputting the
The decoded image signal of the previous frame accumulated in
The global motion pattern output in the long decoding step
The motion compensation of the whole moving area is performed from the parameter and
Global motion compensation system that outputs a motion-compensated image signal.
And output in this global motion compensation step
Global motion compensated image and the variable length decoding
Polygon patch motion parameters output in step
Motion compensation is performed by performing local motion compensation inside the motion area from
A polygon patch compensation step of outputting a predicted image signal,
The motion output in this polygon patch compensation step
Compensated prediction image signal and the residual code decoding step
Then, the residual code output by
And add this decoded image to the frame memory.
The video decoding method is characterized in that
It

[0010]

[Operation] A moving picture coding apparatus according to the first aspect of the present invention will be described.

The moving body analyzing means extracts a moving body from the input image signal and analyzes the movement of the moving body to output a residual signal from the reconstructed image and a movement parameter.
The residual coding means codes the residual signal from the moving object analyzing means to form residual coding information. Reconstructing means reconstructs an image from the motion parameters from the moving object analyzing means and the residual coding information from the residual coding means. Then, the variable length coding means variable length codes the motion parameter from the moving object analysis means and the residual coding information from the residual coding means.

As a result, a moving area is extracted from the input image signal and motion compensation is applied only to that area, so that the efficiency of motion compensation near the boundary of the moving object is improved. Also, by applying a polygonal patch to the moving area and performing fine motion estimation,
It is possible to perform motion compensation capable of coping with eye and mouth deformations.

A moving picture decoding apparatus according to the invention of claim 7 will be described.

The variable length decoding means receives the coded data from the moving picture coding device, decodes this coded data which is a variable length code, and decodes the global motion parameter, the polygon patch motion parameter and the residual code. To separate.

Residual code decoding means decodes the residual code output from the variable length decoding means and outputs a residual signal.

The global motion compensating means performs motion compensation for the entire moving region from the decoded image of the previous frame accumulated in the frame memory and the global motion parameter output from the variable length decoding means, and obtains the global motion compensating image. Output.

The polygon patch compensating means uses the global motion compensated image output from the global motion compensating means and the polygon patch motion parameters output from the variable length decoding means to locally compensate the motion inside the moving area. And output a motion compensation predicted image.

The adding means adds the motion compensation prediction image output from the polygon patch compensating means and the residual code output from the residual code decoding means to create a decoded image and decode the decoded image. The image is output and sent to the frame memory.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Invention) An embodiment of the moving picture coding apparatus of the first invention is shown in FIG.
~ It demonstrates using FIG.

FIG. 1 is a block diagram of a moving picture coding apparatus.

The image signal input from the input terminal 1 is divided into a moving area and a stationary area by the moving object analyzing section 2, and the moving area is extracted and the motion parameters are estimated. Motion compensation of the reconstructed image is performed by the motion parameter, and the difference from the input image is obtained. The residual signal is used as the residual coding unit 4
Is encoded with. The motion parameter information and the residual coding information are variable length coded by the variable length coder (VLC) 5 and output. Further, the reconstructing unit 3 reconstructs an image using the motion parameter information and the residual coding information.

Any method can be used as the method of the residual encoding unit 4 as long as it can efficiently express a waveform. For example, conditional pixel supplementation in which a residual signal is quantized and a non-zero pixel value is addressed and sent. A method, vector quantization in which several pixels are collectively expressed by one code, a method in which discrete cosine transform is performed and then quantization is performed, and the like can be considered.

FIG. 2 is a block diagram showing an example of the moving body analysis unit 2.

A moving area of the image signal input from the input terminal 1 is extracted by the moving area extracting unit 11. The global motion estimation unit 12 estimates the motion parameters of the entire moving area. Then, the polygon patch motion estimation unit 13 applies a polygon patch to the moving area and estimates a fine movement in the moving area. The reconstructed image input from the input terminal 10 is
The motion compensation unit 14 performs motion compensation based on the global motion parameter and the polygon motion parameter. The difference between the input image and the motion-compensated image is calculated and output from the output terminal 16 as a residual signal.

As a method of extracting the moving area in the moving area extracting unit 11, a temporal difference of the input image, that is, a difference between frames is taken, a pixel larger than the threshold value is set as the moving area, and an isolated point is used to remove a noise component. A moving area is extracted by performing removal or integration of areas. But,
In this method, since there is almost no difference in the flat part of the moving object, the moving regions are separated and divided into many moving regions as shown in FIG. 6, and there is a problem that the region information increases. . Another method for extracting the moving area is shown in FIG.
As described above, first, the difference between frames is calculated, the edge of the moving object is searched from the left and right, and the inside thereof is set as the moving area. According to this method, the moving areas are not separated. In addition, the moving region can be more accurately extracted by searching the edges of the moving object not only from the left and right but also from the top, bottom, left, and right.

The global motion estimation in the global motion estimator 12 is performed by mapping a point (x, y) to another point (x ', y') as shown in FIG. 8 and the following equation (1). , The motion parameters a to f are obtained.

However, a to d are parameters expressing rotation, enlargement, reduction and deformation, and e and f are parameters expressing parallel movement.

[0028]

[Equation 1] In the polygon patch motion estimation in the polygon patch motion estimation unit 13, as shown in FIG. 9, the image of the previous frame is mapped by the global motion parameter, and then the polygon patch is applied to the moving area. Here, a triangular patch is used. The vertices of each triangle are moved so that the error from the current frame image is minimized. The image in the triangle is mapped by the affine transformation. The affine transformation formula is the same as the formula (1), and the parameters are different for each triangle. The only information to be transmitted is the amount of movement of each vertex.

When the moving object moves, the degree of light hit may change, and the brightness may change even at the same corresponding point. To compensate for this, it is necessary to move not only the vertices of the triangular patch in the two-dimensional plane, but also in the luminance value direction. One point (x, y) becomes another point (x ',
If the brightness correction amount in the brightness value direction when moving to y ′) is z ′, then z ′ = kx + ly + m (2) The parameters k, l, and m can be obtained from the amount of movement of each vertex of the triangular patch in the brightness value direction.

FIG. 3 is a block diagram showing another embodiment of the moving body analysis unit 2.

The difference from FIG. 2 is that there is a feature point extraction section 17 and a person detection section 18. The feature point extraction unit 17 extracts feature points such as eyes and mouth of a person's face from the reconstructed image. Around the feature point, the size of the triangular patch is reduced as shown in FIG. 10, so that it is possible to cope with a finer movement. Alternatively, as shown in FIG. 11, motion compensation may be performed by applying a triangular patch having a shape matching a feature point, and further applying a fixed-shape triangular patch. The feature points may be extracted from the input image. However, if the feature points are extracted from the reconstructed image, the reception side can do the same, and it is not necessary to send the position information of the feature points.

This feature point extraction can be performed with a high probability when it can be modeled to some extent like a human face, but is difficult to extract for general images. Therefore, whether the person detection unit 18 automatically determines whether or not it is a person image,
Alternatively, the mode may be manually switched, and in the case of a general image, the feature point extraction may be stopped and a fixed triangle patch may be used. Alternatively, an encoder for person images and an encoder for general images may be prepared and switched.

FIG. 4 is a block diagram showing still another example of the moving body analysis unit of the present invention. The difference from FIG. 2 is that there is a background image composing unit 19. Pixels that have been stationary for a long time are written in the background memory from the reconstructed image to form the background image. This background image is used when the background appears from behind the moving object. The background that appears from the back of the moving body is the part that has become the stationary region this time in the front rotation region. Since this background compensation area is known to the receiving side, it is not necessary to send area information.

There is a difference chain code as a method of transmitting the information on the boundary line of the area. This is a method in which, as shown in FIG. 12, the eight directions that can be traveled are numbered, the numbers are sent first, and then the difference from the previous direction is sent. When it is not necessary to send the boundary line so accurately, as shown in FIG. 13, there is also a method of sending only the positions of the representative points and interpolating between them with a curve such as a spline curve. When this method is used, in the case of a moving image, only the shift amount of the representative point needs to be sent, so that the amount of information can be further reduced.

When the still area, the background compensation area, and the moving area are combined, if they are combined by cutting and pasting, the boundary line is emphasized, which is visually unfavorable. FIG. 14 is a one-dimensional representation of the background and moving area signals. The background and moving areas are blunted by camera blur and signal processing,
It is connected gently. When the moving area moves to the left, if the moving area is cut and pasted on the background, the border becomes sharp and becomes more conspicuous than necessary. Therefore, by smoothly attenuating the vicinity of the boundary and overlapping them, a smooth connection can be achieved.

FIG. 15 is a block diagram for communicating face images, in which (a) is a transmitter and (b) is a receiver.

In the transmitting unit, the code 100 is an encoding unit generally called a parameter encoder, and is an encoder for separating the moving area of the present invention, an encoder for separating by a contour, It is a model-based coder, an analysis-synthesis coder or an intelligent coder. Reference numeral 102 is a conventional waveform encoder. Then, the input signal is used as a parameter detector 1
It is detected by 04 whether or not a face image is included, and either the automatic or manual selection is made by the selector 106 to switch the two. If the face image is included, the parameter encoder 100 sends it. , If not included, the waveform encoder 102 sends the waveform.

Similarly, in the receiving section, the selector 108 is automatically or manually selected to switch between the two. If a face image is included, the parameter decoder 110 receives the image.
If not included, it is received by the waveform decoder 112.

(Second Invention) A moving picture decoding apparatus 200 which is an embodiment of the second invention will be described with reference to the block diagram of FIG.

This is a device for decoding the coded data coded by the moving picture coding device shown in FIG.

The coded data from the moving picture coding apparatus is
It is input from the input terminal 201 of the moving image decoding apparatus 200, the variable length decoder (VLD) 202 decodes the variable length code, and the motion parameter and the residual code are output. The motion parameters include motion area information, global motion parameters, and polygon patch motion parameters.

The residual code decoder 204 decodes the residual code and outputs a residual signal.

The global motion compensation circuit 207 receives the decoded image of the previous frame stored in the frame memory 208, the moving area information and the global motion parameter, performs the motion compensation of the entire moving area, and outputs the global motion compensated image. .

The parameters are input, local motion compensation is performed inside the moving area, and a motion-compensated predicted image is output.

The adder 205 adds the motion-compensated prediction image and the residual code to create a decoded image, outputs the decoded image from the output terminal 209, and also the frame memory 208.
Send to.

As a result, the coded data coded by the moving picture coding apparatus shown in FIG. 1 can be decoded.

(Third Invention) FIG. 17 is a block diagram of the third embodiment of the present invention. The input image is subjected to prediction difference by the differentiator 301 by the inter-frame or inter-field prediction signal generated by the motion prediction circuit 309 from the temporally previous image data stored in the frame memory 308. For example, the time direction prediction is 1
It can be obtained by searching for a block having the minimum sum of power or sum of absolute values of prediction difference errors by matching with respect to a block range of 6 pixels × 16 lines. The prediction difference signal is selected by the selecting means 302 from the set of orthogonal transforms prepared in advance, and the selection information is sent to the transform calculator 303. In the selection means 302, for example, the autocorrelation matrix of the input signal-in this case, the prediction residual signal-is obtained, and calculation is performed as a criterion for selecting the one having the closest Euclidean distance to the autocorrelation matrix corresponding to the orthogonal transformation prepared in advance. I can do it. The conversion is performed by dividing the image into small blocks G of NxN pixels and performing a matrix conversion represented by F = TvGTn by a vertical conversion matrix Tv of NxN and a horizontal conversion matrix Tn of NxN. The converted data is quantized by the quantizer 304 and, if necessary, the variable length encoder 310.
Variable-length coded and transmitted. On the other hand, the quantization result is inversely quantized by the inverse quantizer 305. In this case, instead of dequantizing the quantized result as shown in FIG. 18, the dequantized result may be directly generated. The inverse quantization result is sent from the selecting means 302 as an operation at the time of conversion, and the inverse converter 306 performs an operation of inverse conversion corresponding to the operation. The transformation is stored in the memory as the coefficient of the matrix, and the coefficient used for the transformation is read by the instruction of the selecting means. At this time, the conversion coefficient for inverse conversion may be read and sent to the inverse converter 306. Since this orthogonal transform is an orthonormal transform, the inverse transform is a transposed matrix. Therefore, the coefficient read out for transform can be used as it is by simply changing the arrangement with respect to the vertical and horizontal directions, so that the configuration can be simplified. The inverse conversion result is the motion prediction circuit 30.
9 is added to the prediction signal generated from 9 by the adder 307,
The contents of the frame memory 308 are rewritten. A flag signal representing the transformation selected by the selection means 302 is transmitted together with the quantized and encoded signal. The above orthogonal transform is optimal for the KL transform, but other approximate orthogonal transforms can be used.

FIG. 19 shows a second embodiment of the present invention. The decision unit 311 uses the temporal prediction signal, the prediction residual signal, the input signal, and the like obtained by the motion prediction circuit 309 to instruct switching of the coding mode. That is, first, the power sum or the absolute value sum of the input signal and the prediction residual signal is compared, and the smaller one is selected. This signal is switched by the changeover switch 312. When the sum of powers in blocks or the sum of absolute values of the selected signals is equal to or less than a certain threshold value, the following conversion coding is not performed and a code indicating this is transmitted. If it is larger than the threshold value, adaptive conversion is performed, and the code indicating this and the result of conversion, quantization, and coding are transmitted. In this case, a flag signal indicating the result of selecting a desired conversion from a plurality of conversions is also transmitted. On the other hand, in the local decoding as well, based on the above determination result, when the input signal is selected instead of the prediction residual signal, the switch 31
According to 3, the 0 signal is added to the inverse transformation result instead of the motion compensation prediction signal. In addition, the contents of the frame memory 308 are not changed for the blocks for which the input signal power to the conversion calculation is less than the threshold value and the conversion encoding is not performed.

FIG. 20 shows a third embodiment of the present invention. Prior to encoding, the control unit of the transmitter communicates with the receiver to make an arrangement to limit the conversions to be used from the conversions prepared in advance. For example, a total of 17 types of 16 types of orthogonal transformation and one DCT are prepared in advance, all 17 types are used, 16 types of orthogonal transformation are used, or only DCT is used. Also DC
It is also possible to prepare 16 types of conversions including T in total and select from among 16 types, 8 types, 4 types, 2 types, and only one type of DCT depending on the hardware conditions. At this time, the selection information indicating which matrix has been selected can be made more efficient by changing it according to the number of kinds of conversions and the contents thereof which are initially agreed.

FIG. 21 shows a fourth embodiment of the present invention. In the present embodiment, a plurality of conversions are sequentially switched and converted, and after conversion, quantization and variable length coding are performed, and an optimum conversion is selected while maintaining a predetermined functional relationship between a coding error and a code amount, The present invention relates to a variable matrix adaptive conversion in which a signal representing the selection result is transmitted together, and relates to a method of selecting an optimum conversion. It tries all the conversions prepared in advance and selects the most efficient conversion from them. Therefore, first, the error is calculated by the coding error coefficient unit 314 which compares the converted data with the quantized data. Specifically, the sum of squares of the difference between the converted and quantized data of each block may be calculated. Next, the code amount after variable length coding is counted by the counter 315. A smaller error is better and a smaller code amount is better, but they are in a conflicting relationship, and it is necessary to make a comprehensive judgment by the R / D judging device 316. FIG. 22 shows the operating principle of the R / D judging device 316. That is, when the bit rate is plotted on the horizontal axis and the error is plotted on the vertical axis, connecting points at which the bit rate and the error are equivalent to each other in terms of efficiency are connected to form a group of curves in FIG. It is assumed that the points on which the data measured by the counters 314 and 315 are written on this curve are points A, B, C, D, and E. First, points A and B have the same value, but points below this point have higher efficiency because the error is smaller at the same bit rate. Therefore, the efficiency is higher at points D and E than at point C. The points D and E are equivalent, and either one may be selected. At this time, it is also possible to further determine a rule to select either D point or E point. For example, it is easily conceivable to prioritize one having a rate within a certain range, one having a lower rate, or to prepare a determination map considering both rates.

FIG. 23 shows a fifth embodiment of the present invention. The present embodiment has an autocorrelation matrix corresponding to a plurality of transforms and a distance relation network between the autocorrelation matrices, and provides means for calculating the distance between the autocorrelation matrix of the input image signal and the autocorrelation matrix corresponding to the transform. A matrix variable adaptive conversion is performed, which is characterized by performing a minimum distance search of an input and conversion autocorrelation matrix on the distance network. Since the distance relationship of the autocorrelation matrix corresponding to the transformation matrix is networked as shown in FIG. 24, the optimum one is not selected from all the transformation matrices, and only a part of the transformation matrix needs to be searched. , The amount of calculation is greatly reduced. First, an autocorrelation matrix of the input signal is created in 317, and is searched by the network searcher 320. The autocorrelation matrix corresponding to the coefficient memory is networked and stored in the memory 31.
8 is stored. In the network, as shown in FIG. 24, the autocorrelation matrices are arranged based on the distance relationship. For example, in the first stage, distance comparison is performed with the input and the initial value of the first autocorrelation matrix and the autocorrelation matrices 1 to 7 in the vicinity thereof. If 4 is the smallest, the search is performed for 1, 5, 8, 9, 10, and 3 in the four neighborhoods. When 4 is the minimum, this can be determined as the optimum conversion. Since it is networked by distance, it does not fall into local optimum. In addition, it has the feature that it can reach the optimum point faster than a full search.

FIG. 25 shows a sixth embodiment of the present invention. This is because after obtaining the autocorrelation matrix corresponding to the transform that is at the minimum distance from the input autocorrelation matrix, the transform coding result is locally applied to the set of partial transforms including the transform corresponding to the autocorrelation matrix. The optimum conversion is selected as shown in FIG.
This is an example of a general combination of the embodiment of FIG. 1 and the embodiment of FIG. After performing the search in the autocorrelation matrix space, the candidates for the optimal conversion such as when performing the variable length coding are narrowed down. The optimum conversion is searched from the viewpoint of R / D for the limited conversion candidates.

(Fourth Invention) An embodiment of the fourth invention will be described below with reference to the drawings. FIG. 26 is a block diagram of a moving picture coding apparatus according to an embodiment of the present invention. The image signal input from the input terminal 321 receives the motion compensation circuit 322 and the sub-band division circuit 32.
Input to 3. The motion compensation circuit 322 performs motion analysis from the input image signal and the previous frame image signal to perform motion compensation on the previous frame image. The motion analysis may be a block matching method that obtains a translation amount for each block, or, as described later, a method that first extracts a moving area and obtains the movement of the entire moving area, and further obtains a finer movement. This motion-compensated image signal is output to the sub-band division circuit 32.
At 4, the input image signal is divided into a plurality of subbands. FIG. 27 shows an example of subband division. This is first divided into two frequency regions in each of the horizontal and vertical directions, and LL,
Divide into HL, LH, and HH into four. Furthermore, low band LL
Is divided into 4 into LLL, LHL, LLH and LHH. Next, the adaptive prediction circuit 327 selects from the motion-compensated subband image, the previous frame subband image, the background subband image, and no signal, and the difference between the input subband image and the selected prediction signal is used as the residual code. The coding circuit 328 performs coding. Next, the residual coding information, the adaptive prediction information, and the motion analysis information are variable-length coded by the variable-length coding circuit 333 and output from the output terminal 334. At the same time, the residual coded information is decoded by the residual decoding circuit 329 and added by the adder 330.
Is added with the adaptive prediction signal. The added signal is input to the background prediction circuit 331 and the frame memory 332.
The background prediction circuit 331 determines whether or not the background portion
Accumulate only the background part. The previous frame sub-band image signal output from the frame memory 332 is composed of each frequency component in the sub-band combining circuit 325 to reconstruct the previous frame image.

FIG. 28 is a block diagram showing an example of the motion compensation circuit of the present invention. A moving area is extracted by the moving area extraction unit 343 from the image signal input from the input terminal 341. As one method of the moving area extraction unit, first, a difference between frames is obtained, an edge of a moving object is searched from the left and right, and the inside thereof is set as a moving area. According to this method, the moving areas are not separated. In addition, the moving region can be more accurately extracted by searching the edges of the moving object not only from the left and right but also from the top, bottom, left, and right. The global motion estimation unit 344 estimates the motion parameters of the entire moving area. The global motion estimation considers a map in which a certain point (x, y) moves to another point (x ′, y ′) as shown in the following expression (3), and obtains the motion parameters a to f. a to d are rotations,
Parameters that represent enlargement, reduction, and transformation, and e and f are
This is a parameter that represents parallel movement.

[0055]

[Equation 2] In the polygon patch motion estimation unit 345, a polygon patch is applied to the moving area to estimate a fine movement in the moving area.
As shown in FIG. 29, the polygon patch motion estimation is performed by mapping an image one frame before with global motion parameters, and
Apply a polygon patch to the motion area. Here, a triangular patch is used. The vertices of each triangle are moved so that the error from the current frame image is minimized. The image in the triangle is mapped by the affine transformation. The affine transformation formula is (3)
It is the same as the formula, and the parameters are different for each triangle. The only information to be transmitted is the amount of movement of each vertex. The previous frame image input from the input terminal 342 is motion-compensated by the motion compensating unit 326 according to the global motion parameter and the polygonal motion parameter, and is output from the output terminal 348.

FIG. 30 is a block diagram showing an example of the background prediction circuit of the present invention. The still time measuring unit 353 measures the time during which the signal sub-band image input from the input terminal 351 is stationary for each pixel or for each block of a plurality of pixels. When the stationary time is a certain time or more, a write signal is generated in the write control unit 354 and the decoded subband image is written in the background memory 355. Also,
When the adaptive prediction selection signal input from the input terminal 352 indicates no signal, that is, when the intra-frame coding is performed, the write signal is generated. This is because the intra-frame coding is selected when a new image signal that cannot be predicted by the past signal appears, and therefore that portion is likely to be the background.

FIG. 31 is a block diagram showing an example of the adaptive prediction circuit of the present invention. The input subband image is input from the input terminal 361, the motion compensation subband image is input from the input terminal 362, the previous frame subband image is input from the input terminal 363, and the background subband image is input from the input terminal 364. The evaluation circuit 365 calculates the evaluation values of the motion-compensated subband image, the previous frame subband image, the background subband image, and no signal for the input subband image. As the evaluation value, the sum of absolute differences or the sum of squared differences is calculated. When the evaluation value of the previous frame subband image is less than or equal to the set value, the previous frame subband image is selected by the selector 366 as a prediction signal and output from the output terminal 368. Otherwise, the one with the smallest evaluation value is selected. When no signal is selected, the prediction signal becomes 0, and intraframe coding is performed.

FIG. 32 is a block diagram showing an example of the residual coding circuit of the present invention. The prediction residual signal has almost no correlation between pixels, but a high correlation may remain locally when intraframe coding is selected.
Such a local correlation can be used to further reduce the residual signal. The prediction residual signal input from the input terminal 371 is predicted by the prediction unit 374 using the pixels already encoded, and the prediction error is quantized by the quantizer 373 and output from the output terminal 377. In addition, the quantized signal and the prediction signal are added by the adder 376 and the memory 375 is added.
And used for prediction of the next pixel. Figure 3
3 shows an example of processing with a 4 × 4 block. The shaded portion is the portion that has already been encoded. First, the pixel X (4,4) at the lower right of the block is predicted by the equation 2 by using the already coded pixel, and the prediction error is quantized.

P (4, 4) = (ah (4, 4) X (0, 4) + av (4, 4) X (4, 0)) / 2 (4) Quantized prediction error And the predicted value are added to X (4,4)
Is confirmed. Next, the pixels X (2,4) and X (4,2) are
The prediction error is quantized by prediction using the following equations (5) and (6).

P (2, 4) = ah (2, 4) (X (0, 4) + X (4, 4)) / 2 (5) P (4, 2) = av (4, 2) ) (X (4,0) + X (4,4)) / 2 (6) Next, the pixel X (2,2) is predicted by the equation (7), and the prediction error is quantized.

P (2,2) = (ah (2,2) (X (0,2) + X (4,2)) + av (2,2) (X (2,0) + X (2 , 4))) / 4 (7) Similarly, the pixels in between are also encoded. Here, the prediction coefficients (ah (i, j), av (i, j) are determined by the inter-pixel correlation. However, it is not realistic to calculate the inter-pixel correlation for each input image, so that the standard image The inter-pixel correlation of each band with respect to is checked, and the prediction coefficient may be determined by the value.

The quantized prediction error may be coded separately depending on whether the quantized value is 0 or not. In FIG. 34 (a), the black portion shows a pixel whose quantized value is not 0. This position information is represented by a quadtree as shown in FIG. 34 (b) and variable length coding is performed. Then, variable length coding is performed only for the quantized value that is not 0.

(Fifth Invention) A moving picture decoding apparatus according to an embodiment of the fifth invention will be described with reference to the block diagram of FIG.

This is a device for decoding coded data coded by the moving picture coding device of the fourth invention.

Coded data is input from the input terminal 401, a variable length decoder 402 decodes a variable length code, and a motion parameter, an adaptive prediction selection signal, and a prediction error quantized signal are output.

The prediction error quantized signal is the inverse quantizer 403.
Is inversely quantized and is added to the output of the predictor 405 and the adder 404 to form a residual decoded signal. This residual decoded signal is
Written to memory 406 and used to predict the next decoded pixel.

The predictor 405 predicts the next decoded pixel using the already decoded pixels in the vicinity of the next decoded pixel.

The selection circuit 408 selects one from four signals of no signal, a motion-compensated subband image signal, which will be described later, a previous frame subband image signal, and a subband background prediction signal.
The two signals are selected by the adaptive prediction selection signal and output as the adaptive prediction signal.

The residual decoded signal is added to the adaptive prediction signal from the selection circuit 408 by the adder 407 to obtain a subband decoded image. Then, this sub-band decoded image is output to the frame memory 409 and the background prediction circuit 410.

The sub-band decoded image is stored in the frame memory 4
Written in 09.

The background prediction circuit 410 selectively writes a newly sent part or a part which has been stationary for a long time from the subband decoded image in the background memory in the background prediction circuit 410. Then, it is output to the selection circuit 408 as a subband background prediction signal.

The subband decoded image from the frame memory 409 is synthesized with the baseband decoded image by the subband synthesizer 413 and output from the output terminal 414. Further, the selection circuit 40 selects the previous frame sub-band image signal.
8 is output.

The baseband decoded image is the motion compensation circuit 4
Motion compensation is performed at 12 based on the motion parameter, sub-band division circuit 411 divides into sub-bands, and the result is output to the selection circuit 408 as a motion-compensated sub-band image signal.

If the moving picture coding apparatus side has two stages of global motion compensation and polygon patch motion compensation as shown in FIG. 28, the moving picture decoding apparatus side also needs to have two stages of motion compensation. .

If the background coding circuit is not included in the moving picture coding apparatus side, the moving picture decoding apparatus side is not necessary either.

[0076]

According to the present invention, the moving area is extracted from the input image signal and the motion compensation is applied only to the area.
The efficiency of motion compensation near the boundary of the moving object is improved. Further, by applying a polygonal patch to the moving region and performing fine motion estimation, it is possible to perform motion compensation that can cope with eye and mouth deformations.

According to the seventh and ninth aspects of the present invention, the motion of the subject is sufficiently compensated, and the data is divided into sub-bands for processing, so that the coded data with high coding efficiency can be decoded.

[Brief description of drawings]

FIG. 1 is a block diagram of a moving picture coding apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an example of a moving image analysis unit of the present invention.

FIG. 3 is a block diagram showing another example of the moving image analysis unit of the present invention.

FIG. 4 is a block diagram showing still another example of the moving image analysis unit of the present invention.

FIG. 5 is a block diagram showing an example of a conventional moving image encoding device.

FIG. 6 is an explanatory diagram of moving area extraction by removing isolated points of inter-frame differences and area integration.

FIG. 7 is an explanatory diagram of moving area extraction by a method of setting the inside of a moving area boundary as a moving area.

FIG. 8 is a diagram illustrating global motion estimation.

FIG. 9 is a diagram for explaining triangle patch motion estimation.

FIG. 10 is a diagram illustrating a method of reducing the size of a triangular patch around a feature point to perform motion compensation.

FIG. 11 is a diagram illustrating a method of performing motion compensation by matching the shape of a triangular patch with a feature point.

FIG. 12 is a diagram illustrating differential chain encoding.

FIG. 13 is a diagram showing a boundary line by sending only representative points and interpolating between them.

FIG. 14 is a diagram illustrating a method of synthesizing a background and a moving area.

FIG. 15 is a block diagram in the case of communicating a face image, in which (a) is a transmission unit and (b) is a reception unit.

FIG. 16 is a block diagram of a moving picture decoding apparatus according to a second invention.

FIG. 17 is a block diagram of the first embodiment of the third invention.

FIG. 18 is a diagram showing the positions of a quantizer and an inverse quantizer according to the first embodiment of the third invention.

FIG. 19 is a block diagram of a second embodiment of the third invention.

FIG. 20 is a block diagram of a third embodiment of the third invention.

FIG. 21 is a block diagram of a fourth embodiment of the third invention.

FIG. 22 is a diagram showing the operating principle of the R / D determiner of the fourth embodiment of the third invention.

FIG. 23 is a block diagram of a fifth embodiment of the third invention.

FIG. 24 is a diagram showing a distance relation network of an autocorrelation matrix according to the fifth embodiment of the third invention.

FIG. 25 is a block diagram of a sixth embodiment of the third invention.

FIG. 26 is a block diagram of a moving picture coding apparatus according to a fourth invention.

FIG. 27 is a diagram showing an example of subband division of the fourth invention.

FIG. 28 is a block diagram of a motion compensation circuit according to a fourth invention.

FIG. 29 is a diagram for explaining triangle patch motion estimation according to the fourth invention.

FIG. 30 is a block diagram of a background prediction circuit of a fourth invention.

FIG. 31 is a block diagram of an adaptive prediction circuit of a fourth invention.

FIG. 32 is a block diagram of a residual encoding circuit according to a fourth invention.

FIG. 33 is a diagram for explaining the residual encoding process of the fourth invention.

FIG. 34 is a diagram for explaining encoding of pixel positions whose quantized value is not 0 according to the fourth invention.

FIG. 35 is a block diagram of a moving picture decoding apparatus of the fifth invention.

[Explanation of symbols]

1 ... Input terminal 2… Animal analysis department 3 ... Image reconstruction unit 4 ... Residual coding unit 5 ... Variable length coding unit 6 ... Output terminal 11 ... Moving area extraction unit 12 ... Global motion estimation unit 13 ... Polygon patch motion estimation unit 14 ... Motion compensation unit 17 ... Feature point extraction unit 19 ... Background image composition section 301 ... Differentiator 302 ... Selection means 303 ... Conversion calculator 304 ... Quantizer 305 ... Inverse quantizer 306 ... Inverse converter 307 ... Adder 308 ... Frame memory 309 ... Motion prediction circuit 310 ... Variable length encoder 311 ... Judgment device 312, 313 ... Changeover switch 314 ... Encoding error counter 315 ... Code amount counter 316 ... R / D judging device 317 ... Autocorrelation matrix 318 ... Coefficient memory 319 ... Corresponding autocorrelation matrix network 320 ... Network searcher 321 ... Input terminal 322 ... Motion compensation circuit 323, 324 ... Subband division circuit 325 ... Subband synthesis circuit 326 ... Subtractor 327 ... Adaptive prediction circuit 328 ... Residual coding circuit 329 ... Residual decoding circuit 330 ... Adder 331 ... Background prediction circuit 332 ... Frame memory 333 ... Variable-length coding circuit 334 ... Output terminal

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Naoaki Kuratate 1-30 Oyonaka, Kita-ku, Osaka-shi, Osaka Umeda Sky Building Tower West Co., Ltd. Toshiba Kansai Branch Office (72) Inventor Mitsunori Ogawa 1-30, Oyodo Naka, Kita-ku, Osaka-shi, Osaka Umeda Sky Building Tower West Co., Ltd. Toshiba Kansai Branch Company (72) Inventor Shogo Yamaguchi 1-30, Oyodo-naka, Kita-ku, Osaka-shi, Osaka City Umeda Sky Building Tower West Co., Ltd. Toshiba Kansai Branch Office (72) Inventor Kazuo Ozeki 1-1-30 Oyodo Naka, Kita-ku, Osaka City, Osaka Prefecture Umeda Sky Building Tower West Co. Ltd. Toshiba Kansai Branch (56) References Yuichiro Nakaya (1 other person), Motion compensation method based on region division by integration of triangular patches, 1991 Image coding Symposium Beam (PCSJ91) 6th Symposium documentation, Japan, The Institute of Image Electronics Engineers, October 7, 1991, PCSJ91, P. 57-60 Hans Georg (2 others), OBJECT-ORIENTED ANA LYSIS-SYNTHESIS CO DING OF MOVING IMA GES, Signal processing, Image, Communication, Elsevier Sr. 1989. 1, No. 2, p. 117-138 Dirk Adolph (1 person outside), 1.15 Mbit / s coding of video signals in clogging global motion compensation, Signal processing, vulcanization, corn vulcanization, ed. 3, Nos. 2-3, p259-274 (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N ^7/ 24-7/68

Claims (9)

(57) [Claims] 1. A moving object is extracted from an input image signal and its movement is analyzed to detect the movement of the input image signal and the movement of the moving object.
A moving object analysis unit that outputs a residual signal and a motion parameter that is a difference between the reconstructed image that has been compensated , and the residual signal from the moving object analysis unit is encoded to form residual encoded information. Reconstructing using the residual encoding means for performing the motion parameter from the moving object analyzing means, and the residual encoding information from the residual encoding means.
Reconstruction means for reconstructing an image signal , and variable length coding means for variable length coding the motion parameter from the moving object analysis means and the residual coding information from the residual coding means. A moving image encoding apparatus, wherein the moving object analyzing means relates to movement of the entire moving region in the input image signal.
Global motion estimating estimated global motion parameters
Estimation means and motion in the input image signal or reconstructed image signal
By applying a polygonal patch to the area,
Estimate motion-related polygon patch motion parameters
By doing so, it is possible to estimate fine movement in the moving area.
And a global patch estimated by the global motion estimation means.
Motion parameter and the polygon patch motion estimation means.
The polygon patch motion parameters estimated from
A moving picture coding apparatus comprising: a motion compensating means for compensating a motion of a constituent image signal . 2. The moving body extraction means extracts a moving area from the input image signal.
Video encoding according to claim 1, characterized in that it comprises
apparatus. 3. A feature point is extracted from a moving area of the reconstructed image signal.
A feature point extracting means for extracting the polygonal pattern is provided to the feature points.
Especially to shed a polygon patch as the apex of the pitch will come
The moving picture coding apparatus according to claim 2, wherein 4. The polygon patch motion estimation means is a feature in the motion area extracted by the feature point extraction means.
By applying a small polygon patch around the trait,
Estimate polygon patch parameters, and
Apply a new polygon patch to the moving area to
4. The motion according to claim 3, wherein the parameter is estimated.
Image coding device. 5. The moving object analyzing means calculates a temporal difference from an input image signal to obtain a large difference.
The area is searched from the top, bottom, left, and right of the image, and the area is set as the moving area.
Moving picture encoding apparatus according to claim 4, wherein the that. 6. A background image signal is constructed from the reconstructed image signal.
A background image signal composing means for generating the background image signal.
The background image signal composed of
2. The background appearing from the above is also compensated.
On-board video coding device. 7. A receives the variable length coded data, decodes the encoded data, the whole moving region in the image signal dynamic
Global motion parameters associated with come, the image signal
Variable length decoding means for separating into a polygonal patch motion parameter and a residual code related to the motion of a portion of a moving area in the inside, and a residual code outputted by the variable length decoding means for decoding the residual signal. Motion compensation for the entire moving region from the residual code decoding means for outputting the decoded image signal of the previous frame stored in the frame memory and the global motion parameter output from the variable length decoding means. And a global motion compensation means for outputting a global motion compensation image signal , a global motion compensation image output from the global motion compensation means, and a polygon patch motion parameter output from the variable length decoding means. perform local motion compensation in the internal region, a polygonal patch compensating means for outputting a motion compensated prediction image signal, outputted from the polygon patch compensating means Is a motion compensated prediction image signal, by adding the residual code output from the residual code decoding means constitutes a decoded image signal, the more becomes possible and adding means for sending the decoded image in the frame memory Characteristic moving picture decoding apparatus. 8. A moving object is extracted from an input image signal and
By analyzing the motion, the input image signal and the motion
The residual signal, which is the difference from the reconstructed image that has been compensated, and the motion pattern.
A moving object analysis step for outputting parameters, Coding the residual signal in the moving object analysis step
A residual coding step of forming residual coding information by Motion parameters in the moving object analysis step,
The residual coding information in the residual coding step
A reconstruction step of reconstructing a reconstructed image signal using With the motion parameter in the moving object analysis step,
Allows the residual coding information in the residual coding step.
Video with variable length coding step for variable length coding
An encoding device, The moving body analysis step, Related to the movement of the entire moving area in the input image signal
Global motion estimating estimated global motion parameters
Estimation step, Motion in the input image signal or the reconstructed image signal
Apply a polygon patch to the area. By doing so,
Estimate motion-related polygon patch motion parameters
By doing so, it is possible to estimate fine movement in the moving area.
Polygon patch motion estimation step The motion estimated in the global motion estimation step
The global motion parameters and the polygon patch motion estimation
Polygon patch motion parameters estimated at step
Motion compensation step for performing motion compensation of the reconstructed image signal by
And a moving picture coding method. 9. Variable-length coded data is received and encoded.
The data is decoded and the motion of the entire motion area in the image signal is
Global motion parameters associated with
Polygonal patch related to the movement of the part of the moving area in the
A variable length decoding stream that is separated into motion parameters and residual codes.
Tep, Residual code output in this variable length decoding step
Residual code decoding step for decoding and outputting a residual signal
When, Decoded image signal of previous frame stored in frame memory
And the glob output in the variable length decoding step.
Global motion parameters and motion compensation for the entire motion area.
Global, which outputs a motion compensated image signal
A motion compensation step, The output of the global motion compensation step
The global motion compensation image and the variable length decoding step
Motion from the polygon patch motion parameters output in
Motion compensated prediction image by performing local motion compensation inside the region
A polygon patch compensation step for outputting a signal, The motion output in this polygon patch compensation step
Compensated prediction image signal and the residual code decoding step
Then, the residual code output by
And add this decoded image to the frame memory.
And a moving picture decoding method.