JP2005323252A - Image encoding device and image decoding device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem in which inefficient encoding is performed when residual processing etc., is performed in inter-frame prediction in object encoding while a gap part (uncovered area) where a texture generated in a motion compensation result is not present is left as it is. <P>SOLUTION: A block area dividing means 8 divides the motion compensation result including the uncovered area into blocks. A motion estimating means 9 performs motion prediction from a frame on a non-compensation side (predicted side) to a frame on a compensation side (side divided and used for prediction) reversely to the beginning when the number of pixels of the uncovered area in a block is less than a nearly half of the number of pixels in the block. An uncovered area filling means 10 searches for a texture of a frame on a compensation side of a part most similar to the uncovered area through the motion prediction and fills the uncovered area with the texture. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image encoding device and an image decoding device, and in particular, when compressing and encoding a digital moving image signal, after performing segmentation of the current frame, the division result (segment) is used to determine the next frame. The present invention relates to an image encoding apparatus and an image decoding apparatus that employ an image encoding method (hereinafter referred to as “object-based encoding”) in an expected form.

  The current high-efficiency compression coding method for moving picture signals is represented by MPEG (Moving Picture Experts Group), which reduces the redundancy in an image frame by discrete cosine transform (DCT) and uses inter-image compensation using motion compensation. Waveform coding combined with reduction of redundancy between image frames by prediction can be said to be mainstream. In most of the coding of this waveform coding system, the processing unit is a block in both intra-frame and inter-frame processing. For this reason, it is often called “block-based encoding”.

  On the other hand, as another high-efficiency compression encoding method for moving image signals, generally, the scene is divided into arbitrary shapes according to the shape of the subject, and their relative positional relationships are changed before reconstruction. Object-based coding based on a method for performing motion compensation is also known (see, for example, Non-Patent Document 1 and Non-Patent Document 2). In this object-based encoding, a part having similar texture characteristics or a part surrounded by one closed contour is generally handled as an object (also referred to as a segment).

  Object-based coding performs motion estimation / compensation in units of regions that match the shape of the subject, so it has excellent coding efficiency, and problems with mosquito noise and block distortion that occur with block-based coding such as conventional MPEG1-4 Can be avoided.

H. G. Musmann, M. Hotter, J. Ostermann: “Object-Oriented Analysis Synthesis Coding of Moving Images”, Signal Process: Image Commun., 1, 2, pp. 117-138 (Oct. 1989). J.Y.A.Wang, et. Al. “Applying Mid-level Vision Techniques for Video Data Compression and Manipulation”, M.I.T.Media Lab.Tech.Report No.263, (Feb.1994). (Http://www-bcs.mit.edu/people/adelson/pub_pdfs/wang_tr263.pdf)

  Motion compensation in object-based coding is done by selecting the part of the past frame that is most similar to each block of the current frame, and in any way a leak-free prediction is made over the entire frame. You can say that.

  However, in this object-based encoding, a certain intra-frame encoded image (intra-encoded image; key frame) is divided into a plurality of objects, and motion compensation of other frames is performed using this frame. As the object moves, the hidden background appears as an uncovered area (uncovered background). If this portion is left as it is and the subsequent residual processing or the like is performed, there is a problem that the encoding becomes inefficient.

  The present invention has been made in view of the above points, and realizes highly accurate prediction, and as a result, the burden of residual processing is reduced, and the image coding apparatus and the coding that can improve the overall coding efficiency An object of the present invention is to provide an image decoding apparatus for decoding a signal.

  In order to achieve the above object, the image signal encoding apparatus according to the first aspect of the present invention divides a frame image into arbitrarily shaped regions, changes the relative positional relationship of the region dividing results, and reconstructs the motion. In the image coding apparatus of the object-based coding method for performing compensation, a first temporal frame image signal of the first frame that is temporally divided among first and second frames in a moving image is divided into arbitrary shape regions. Based on the dividing unit, the division result of the arbitrary shape region obtained by the first dividing unit, and the image signals of the first and second frames, the second frame is predicted for each division result of the arbitrary shape region. A motion estimation / compensation unit that estimates a motion vector and performs motion compensation for the second frame from the motion vector, and a void portion that is obtained by the motion estimation / compensation unit and has no texture A second division unit that divides a motion compensation result including each block having a predetermined shape, and a gap portion in the block for each block of the motion compensation result divided by the second division unit The most similar texture portion in the first frame is searched for by inter-frame prediction, and a gap portion is compensated with the searched texture portion, and the image signal after the compensation output from the compensation means And encoding means for outputting a residual signal of the image signal of the second frame and a motion vector obtained by the motion estimation / compensation means as an encoded bit stream.

  In order to achieve the above object, an image decoding apparatus according to a second aspect of the present invention includes an encoded bit stream obtained by the image encoding apparatus according to the first aspect of the present invention, and first and second frames in a moving image. Among these, an image decoding apparatus that decodes an image signal of a second frame from an image signal of a first frame that has been decoded in the past, receives an encoded bit stream as an input, Decoding means for decoding the motion vector and residual signal of the second frame constituting the coded bit stream, first dividing means for dividing the image signal of the first frame into arbitrary shape regions, Motion compensation means for performing motion compensation of the second frame from the motion vector from the decoding means and the image signal of the first frame, for each arbitrary shape region result obtained by dividing by the dividing means, The second compensation unit that divides the motion compensation result obtained by the compensation unit including the void portion where the texture does not exist for each block having a predetermined specific shape, and the motion compensation divided by the second division unit For each block of the result, a filling means for searching for the most similar texture portion in the first frame for the gap portion in the block by inter-frame prediction, and filling the gap portion with the searched texture portion; And a residual processing means for outputting a decoded video signal of the second frame based on the image signal after compensation outputted from the compensation means and the residual signal obtained by the decoding means, To do.

  In the image coding device of the first invention and the image decoding device of the second invention, the motion compensation result of the second frame that is temporally future out of the first and second frames of the moving image sequence is blocked. It is divided every time, and the most similar texture part in the first frame is searched for by the inter-frame prediction for the gap part where the texture in each block does not exist, and the above-mentioned gap part is filled with the searched texture part. Therefore, the introduction of the backward direction motion estimation for performing motion prediction from the second frame on the compensated side (predicted side) to the first frame on the compensation side (side that is divided and used for prediction) It is possible to compensate for voids that do not exist in the texture included in the motion compensation result using the texture of the first frame on the compensation side of the best match part .

  According to the present invention, in inter-frame prediction in object-based coding, from the second frame on the compensated side (predicted side) to the first frame on the compensating side (side divided and used for prediction). With the introduction of motion estimation in the reverse direction to perform motion prediction, the void portion without the texture included in the motion compensation result of the second frame is used by using the texture of the first frame on the compensation side of the portion that is the best match Since the compensation is made, it is possible to realize highly accurate prediction without consuming a new bit amount. As a result, the burden of residual processing can be reduced, and the overall coding efficiency can be improved.

  Next, each embodiment of the present invention will be described with reference to the drawings. Note that the present invention encodes a moving image signal by an object-based encoding method, but basically relates to a part related to inter-frame prediction. It is assumed that a so-called P frame (P picture) encoding method, which is an inter-frame forward prediction encoded image between a frame and a key frame), is basically considered.

  FIG. 1 shows a block diagram of an embodiment of an image encoding apparatus according to the present invention. In the figure, the frame memories 1 and 2 store digital image signals of moving images of two frames (two frames in the moving image) in the moving image sequence, respectively. The digital image signal of the previous frame that is one frame past from the frame memory 2 is stored, and the digital image signal of the current frame is stored in the frame memory 2. Here, unless the digital image signal of the previous frame stored in the frame memory 1 is not a key frame, it is assumed that the image signal of one frame before is input through a series of processes shown in FIG.

  The image signal of the previous frame read out from the frame memory 1 is supplied to the arbitrary shape region dividing means 3, where the image of the previous frame is divided into arbitrary shapes. Regarding the norm of division, it is basically free to use the statistical property of the pixel value of the image or to detect the edge. The divided image signal of arbitrary shape (region division result) divided by the arbitrary shape region dividing means 3 is temporarily stored in the frame memory 4.

  The motion estimation unit 5 estimates a motion vector for each segment of the region division result read from the frame memory 4 so as to predict the digital image signal of the current frame read from the frame memory 2. Since the motion vector estimated by the motion estimation unit 5 is transmitted as a bit stream, the motion vector is supplied to the bit stream encoding unit 12 and subjected to processing for generating a bit stream with a residual signal described later. , Supplied to the motion compensation means 6 in the encoder, and used together with the result of arbitrary region segmentation.

  The motion compensation unit 6 obtains a motion compensation result for each region division result based on the motion vector from the motion estimation unit 5 and the region division result from the frame memory 4. As described above, the motion compensation result includes a void portion (hereinafter referred to as an uncovered area) where there is no texture, which is a substantial prediction failure. The motion compensation result including this uncovered area is temporarily stored in the frame memory 7.

  The block area dividing unit 8 divides the motion compensation result including the uncovered area read from the frame memory 7 into, for example, blocks each having 8 pixels vertically and horizontally, and supplies each block to the motion estimating unit 9. In general, the size of the arbitrary shape divided by the arbitrary shape region dividing unit 3 is equal to or larger than the block divided by the block region dividing unit 8, but the arbitrary shape may be smaller than the block. In addition, the above-described block has a rectangular shape of 8 pixels in each of the vertical and horizontal directions, but the block is not limited to this shape, and may have a predetermined specific shape, and may have another shape such as a triangle.

  For each block divided by the block area dividing unit 8, the motion estimation unit 9 determines that the previous frame read from the frame memory 1 when the number of pixels in the uncovered area is about half or less of the number of pixels in the block. Motion estimation is performed based on the motion compensation result.

  This is because, as described above, the uncovered area is a void portion where there is no texture that is substantially a prediction error, so only when the number of pixels in the uncovered area is about half or less of the number of pixels in the block. , Motion estimation is performed by recognizing the reliability of the texture in the block, but if the number of pixels in the uncovered area is larger than about half the number of pixels in the block, the texture in the block is not reliable. This is to prevent the occurrence of an error by not performing (processing as a motion vector (0, 0)).

  In addition, as described above, the motion estimation unit 9 performs the motion compensation of the current frame on the compensated side in the direction opposite to the normal direction in which the motion estimation of the image information of the current frame from the previous frame is performed for each block. Based on the result, motion estimation is performed by inter-frame prediction on the previous frame on the compensation side. Note that the motion vector as a result of the motion estimation here does not need to be transmitted as a bit stream.

  The uncovered area compensation means 10 is based on the motion estimation result for each block by the motion estimation means 9, the motion compensation result from the frame memory 7, and the image signal of the previous frame from the frame memory 1. A part of the image of the previous frame that is the best match in the motion estimation result is used for compensation.

  Here, for example, an image signal of one block which is supplied to the uncovered area compensation means 10 and is motion-compensated and divided into blocks is schematically shown at 14 in FIG. Assuming that the frame image is one in which the uncovered area 16 is present in the image 15 estimated to be circular, the uncovered area compensation means 10 performs the motion estimation in the reverse direction as schematically shown in FIG. By supplementing the image 15b of the previous frame most similar to the portion of the uncovered area 16 obtained by performing the process, the image 15a of the prediction side frame and the image 15b of the previous frame compensated are substantially the same as the current frame. A predicted image can be obtained.

  Thus, the image signal that has been uncovered by the uncovered area compensation means 10 is supplied to the residual processing means 11 where the difference from the image signal of the current frame read from the frame memory 2 is obtained. The calculated difference value is input to the bit stream encoding means 12 as a residual signal. The bit stream encoding unit 12 generates a bit stream from the motion vector estimated by the motion estimation unit 5 and the residual signal read from the frame memory 2 and outputs the bit stream.

  Next, details of the above series of processing will be described using an actual image as an example. The image size is 640 × 480 pixels. When the image shown in FIG. 3 is the image of the previous frame and the image shown in FIG. 4 is the image of the current frame, the result of the arbitrary shape area division of the previous frame output from the arbitrary shape area dividing means 3 is the result of the image shown in FIG. It becomes like this. Note that the lower left image portion in the current frame image shown in FIG. 4 has moved slightly to the right as compared to the previous frame image shown in FIG.

  The motion compensation unit 6 uses the image (texture) of the previous frame shown in FIG. 3 based on the result of the arbitrary shape region division of the image shown in FIG. 5 to convert the current frame image shown in FIG. The motion compensation is performed based on the motion compensation image shown in FIG. In the image shown in FIG. 6, the uncovered area is black and cracked.

  The motion compensated image shown in FIG. 6 is divided into 8 × 8 pixel blocks by the block area dividing means 8 and the uncovered area is compensated by the uncovered area compensating means 10 to compensate for the uncovered area. The resulting image is shown in FIG. The image after covering the uncovered area shown in FIG. 7 is predicted to be quite close to the image of the current frame shown in FIG. 4, and the residual can be reduced.

  As described above, according to the present embodiment, in the inter-frame prediction in the object-based coding, the backward direction motion estimation is introduced to compensate for the uncovered area. Therefore, it is possible to reduce the burden of residual processing and improve the overall coding efficiency.

  Next, the image coding method of the present invention will be described. FIG. 8 shows a flowchart of an embodiment of an image encoding method according to the present invention. In this embodiment, an inter-frame predictive encoding operation using an object-based encoding method similar to that of the image encoding device shown in FIG. 1 is performed. First, the image of the previous frame is divided into arbitrary shapes (step S1). Regarding the norm of division, it is basically free to use the statistical property of the pixel value of the image or to detect the edge.

  Next, for each segment obtained as a result of area division of an arbitrary shape, a motion vector is estimated so as to predict the digital image signal of the current frame (step S2). While this motion vector is transmitted as a bit stream, it is used together with the previous frame image and the arbitrarily shaped region segmentation result to obtain a motion compensation result (step S3). As described above, the motion compensation result includes an uncovered area that is a substantial prediction error.

  Subsequently, after the motion compensation result is divided into, for example, blocks of 8 × 8 pixels (step S4), it is determined whether or not the number of pixels in the uncovered area for each block is about half or less of the number of pixels in the block ( In step S5), motion estimation is performed on the previous frame only for blocks in which the number of pixels in the uncovered area is about half or less of the number of pixels in the block (step S6). A block in which the number of pixels in the uncovered area is larger than about half the number of pixels in the block is processed as a motion vector (0, 0) without performing motion estimation (step S7).

  Subsequent to the processing of step S6 or S7, an uncovered area in which a part of the image of the previous frame of the portion that is the best match in the motion estimation in step S6 in the block that is the motion compensation result is compensated in the uncovered area. Compensation processing is performed (step S8). That is, the uncovered area is replaced with the texture of the similar part in the previous frame. It is not necessary to transmit a motion vector as a result of motion estimation here as a bit stream.

  The image covered with the uncovered area is output as a bit stream together with the motion vector obtained by the motion estimation in step S2 after the residual processing in which the difference from the current frame is calculated (step S9). .

  Also in this encoding method, as in the encoding apparatus shown in FIG. 1, in the inter-frame prediction in the object-based encoding, the backward motion estimation is introduced to compensate for the uncovered area. Therefore, a high prediction can be realized without consuming a new bit amount. As a result, the burden of residual processing can be reduced, and the overall coding efficiency can be improved.

  Next, an embodiment of an image decoding apparatus according to the present invention will be described. FIG. 9 shows a block diagram of an embodiment of an image decoding apparatus according to the present invention. Here, since this is basically a part related to inter-frame prediction, a so-called P frame decoding method is basically considered. In FIG. 9, in the frame memory 21, an image one frame before in time is held through a series of processes in this figure except when the previous frame is not a key frame.

  On the other hand, the bit stream generated by the bit stream encoding unit 12 shown in FIG. 1 is input to the bit stream decoding unit 22 via an arbitrary transmission medium. Here, the motion estimation unit 5 of FIG. 1 is decoded into the residual signal generated by the residual processing means 11 of FIG. 1, the motion vector is supplied to the motion compensation means 25, and the residual signal is supplied to the residual processing means 30. Is done.

  The arbitrary shape area dividing unit 23 divides the image signal of the previous frame read from the frame memory 21 into an arbitrary shape, and stores the division result in the frame memory 24. Regarding the norm of division, it is basically free to use the statistical property of the pixel value of the image or to detect the edge.

  The motion compensation means 25 is a result of the arbitrary shape area division read from the frame memory 24, the motion vector of each segment decoded by the bit stream decoding means 22, and the image of the previous frame read from the frame memory 21. A signal is received as an input, and the motion compensation of the image of the previous frame is performed for each result of dividing the arbitrarily shaped region. The motion compensation result obtained in this way includes the uncovered area which is the above-described substantial prediction error. This motion compensation result is temporarily stored in the frame memory 26.

  The block area dividing unit 27 divides the motion compensation result including the uncovered area read from the frame memory 26 into, for example, blocks each having 8 pixels vertically and horizontally, and supplies each block to the motion estimating unit 28. Normally, the size of the arbitrary shape divided by the arbitrary shape region dividing unit 23 is equal to or larger than the block divided by the block region dividing unit 27, but the arbitrary shape may be smaller than the block.

  For each block divided by the block area dividing unit 27, the motion estimation unit 28 determines that the previous frame read from the frame memory 21 when the number of pixels in the uncovered area is about half or less of the number of pixels in the block. Motion estimation is performed on the image signal based on the motion compensation result. For the same reason as the motion estimation means 9. Again, for blocks where the number of pixels in the uncovered area is greater than about half the number of pixels in the block, motion estimation is not performed and processing is performed as a motion vector (0, 0).

  The uncovered area compensation unit 29 is a best match based on the motion estimation result based on the motion estimation result from the motion estimation unit 28, the motion compensation result from the frame memory 26, and the image signal of the previous frame from the frame memory 21. A part of the image of the previous frame of a certain part is used for compensation. That is, the uncovered area compensation unit 29 estimates the motion of the previous frame based on the motion compensation result of the current frame in the direction opposite to the normal direction in which the motion estimation of the image information of the current frame from the previous frame is performed. Based on the result, an operation of filling the uncovered area with the image signal of the previous frame is performed.

  The image signal that has been uncovered by the uncovered area filling unit 29 is supplied to the residual processing unit 30 where it is added to the residual component decoded from the received bitstream by the bitstream decoding unit 22. As a result, the image signal is reconstructed. The reconstructed image signal is output as a decoded image signal and supplied to the frame memory 21 as an image signal of the previous frame and stored.

  Next, the image decoding method of the present invention will be described. FIG. 10 shows a flowchart of an embodiment of an image decoding method according to the present invention. In this embodiment, an inter-frame predictive decoding operation using an object-based encoding method similar to that of the image decoding apparatus shown in FIG. 9 is performed. First, the bit stream generated by the bit stream encoding means 12 shown in FIG. 1 is input via an arbitrary transmission medium, where the motion vector obtained by the motion estimation means 5 of FIG. The residual signal generated by the first residual processing means 11 is decoded (step S11). On the other hand, the image of the previous frame is divided into arbitrary shapes (step S12). Regarding the norm of division, it is basically free to use the statistical property of the pixel value of the image or to detect the edge.

  Next, for each segment of the region division result having an arbitrary shape, a motion vector is estimated so as to predict the digital image signal of the current frame (step S13). This motion vector is used together with the previous frame image and the arbitrarily shaped region division result to obtain a motion compensation result (step S14). As described above, the motion compensation result includes an uncovered area that is a substantial prediction error.

  Subsequently, after the motion compensation result is divided into, for example, blocks of 8 × 8 pixels (step S15), it is determined whether or not the number of pixels in the uncovered area for each block is about half or less of the number of pixels in the block ( Step S16) Motion estimation is performed on the previous frame only for blocks in which the number of pixels in the uncovered area is about half or less of the number of pixels in the block (step S17). A block in which the number of pixels in the uncovered area is larger than about half of the number of pixels in the block is processed as a motion vector (0, 0) without performing motion estimation (step S18).

  Subsequent to the processing in step S17 or S18, an uncovered area in which a part of the image of the previous frame of the portion that is the best match in the motion estimation in step S13 in the block that is the motion compensation result is filled in the uncovered area. Compensation processing is performed (step S19). That is, the uncovered area is replaced with the texture of the similar part in the previous frame.

  The uncovered area-compensated image signal is added to the residual component decoded from the received bit stream in step S11, thereby reconstructing the image signal (step S20). The reconstructed image signal is output as a decoded image signal and used as the image signal of the previous frame in the next step S12.

  The present invention is not limited to the above embodiment, and includes a computer program for realizing the apparatus and method of the above embodiment by a computer. That is, it includes an encoding computer program and a decoding computer program. This computer program may be taken into a computer from a recording medium, or may be distributed through a network and taken into a computer.

It is a block diagram of one embodiment of an image coding device of the present invention. It is an image schematic diagram of an example for description of operation | movement of the uncovered area compensation means in FIG. It is a figure which shows an example of the image of a front frame. It is a figure which shows an example of the image of the present flame | frame. It is a figure which shows an example of the image of the arbitrary shape area | region division | segmentation result of the front frame in FIG. It is a figure which shows the image of an example of the motion compensation result in FIG. It is a figure which shows an example of the image after uncovered area filling in FIG. It is a flowchart of one Embodiment of the image coding method of this invention. It is a block diagram of one embodiment of an image decoding device of the present invention. It is a flowchart of one Embodiment of the image decoding method of this invention.

Explanation of symbols

1, 2, 4, 7, 21, 24, 26 Frame memory 3, 23 Arbitrary shape area dividing means 5, 9, 28 Motion estimating means 6, 25 Motion compensating means 8, 27 Block area dividing means 10, 29 Uncovered area Compensation means 11, 30 Residual processing means 12 Bit stream encoding means 14 Block 15a Predicted frame image 15b Complemented previous frame image 16 Uncovered area 22 Bit stream decoding means

Claims (2)

In an image coding apparatus of an object-based coding method that performs motion compensation by dividing a frame image into arbitrary shape regions and reconstructing after changing the relative positional relationship of the region division results.
A first dividing unit that divides the image signal of the first frame in the past among the first and second frames in the moving image into arbitrary-shaped regions;
Based on the division result of the arbitrary shape area obtained by the first division means and the image signals of the first and second frames, the second frame is predicted for each division result of the arbitrary shape area. Motion estimation / compensation means for estimating a motion vector and performing motion compensation for the second frame from the motion vector;
A second dividing unit that divides the motion compensation result obtained by the motion estimation / compensation unit, including a void portion where no texture exists, for each block having a predetermined specific shape;
For each block of the motion compensation result divided by the second dividing means, search for the most similar texture part in the first frame for the gap part in the block by inter-frame prediction, Filling means for filling the void portion with the searched texture portion;
Output a residual signal between the image signal after compensation output from the compensation means and the image signal of the second frame, and the motion vector obtained by the motion estimation / compensation means as an encoded bit stream. An image encoding apparatus comprising: an encoding unit. The encoded bit stream obtained by the image encoding device according to claim 1, and an image signal of a first frame in the past that has already been decoded among the first and second frames in a moving image. An image decoding device for decoding the image signal of the second frame from:
Decoding means for receiving the encoded bitstream as an input and decoding the motion vector and residual signal of the second frame constituting the encoded bitstream;
First dividing means for dividing the image signal of the first frame into arbitrarily shaped regions;
For each of the arbitrary shape area results obtained by dividing by the first dividing means, the motion compensation of the second frame is performed from the motion vector from the decoding means and the image signal of the first frame. Motion compensation means for performing
A second dividing unit that divides a motion compensation result obtained by the motion compensating unit including a void portion where no texture exists, for each block having a predetermined specific shape;
For each block of the motion compensation result divided by the second dividing means, search for the most similar texture part in the first frame for the gap part in the block by inter-frame prediction, Filling means for filling the void portion with the searched texture portion;
Residual processing means for outputting the decoded video signal of the second frame based on the image signal after compensation output from the compensation means and the residual signal obtained by the decoding means; An image decoding apparatus characterized by that.

Images