JP2005333569A - Image encoding apparatus and image decoding apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem that it is difficult to encode a moving image in which shape information for each frame is not present, or a moving image in which an object concealing relationship is remarkably changed, in an object-based encoding system image encoding apparatus. <P>SOLUTION: A key frame which becomes a base of motion compensating prediction is divided into areas to create an initial area, and the initial area is disposed on a corresponding position in a plurality of frames different from the key frame (S1, S3). Characteristics of texture in the disposed initial area are compared with characteristics of peripheral adjacent texture (S4) and when a result of comparison closer than a threshold can be obtained, the area of peripheral adjacent texture is added to/integrated with the initial area to form a new area (S5, S6). Thus, accuracy of motion compensation is improved and the number of areas wherein motion compensation is not performed can be reduced. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image encoding device and an image decoding device, and in particular, an image signal indicating a scene is divided into arbitrary shapes according to the shape of a subject, and reconfiguration is performed after changing their relative positional relationship. The present invention relates to an image encoding apparatus and an image decoding apparatus of an image encoding (hereinafter referred to as object-based encoding) system that performs motion compensation.

  The current high-efficiency compression coding method for moving image signals is a reduction in redundancy in image frames and motion compensation by discrete cosine transform (DCT), represented by the Moving Picture Experts Group (MPEG). It can be said that waveform coding combined with reduction of redundancy between image frames by the inter-image prediction used is the mainstream. In most of the coding of this waveform coding system, the processing unit is a block in both intra-frame and inter-frame processing. For this reason, it is often called “block-based encoding”.

  On the other hand, as another high-efficiency compression encoding method for moving image signals, generally, the scene is divided into arbitrary shapes according to the shape of the subject, and their relative positional relationships are changed before reconstruction. Object-based coding based on a method for performing motion compensation is also known (see, for example, Non-Patent Document 1 and Non-Patent Document 2). In this object-based encoding, a part having similar texture characteristics or a part surrounded by one closed contour is generally handled as an object (also referred to as a segment).

  Object-based coding performs motion estimation / compensation in units of regions that match the shape of the subject, so it has excellent coding efficiency, and problems with mosquito noise and block distortion that occur with block-based coding such as conventional MPEG1-4 Can be avoided.

  However, in this object-based encoding, a certain intra-frame encoded image (intra-encoded image; key frame) is divided into a plurality of objects, and motion compensation of other frames is performed using this frame. As the object moves, the hidden background appears as an uncovered area (uncovered background), which is a void that is not compensated for motion.

  This will be described with reference to FIGS. FIG. 17 is a key frame that is a substantially L-shaped image in which a vertically long rectangular object 101 and a horizontally long rectangular object 102a are in contact with each other. Assume that segmentation in units of objects is performed. An image. Here, as shown in FIG. 18, it is assumed that a vertically long rectangular object 101 is stationary, and a horizontally long rectangular object partially concealed is moved to the right side as shown by 2b. .

  Since the object 102a corresponding to the horizontally long rectangle in the key frame has the shape shown in FIG. 19, motion compensation is performed using the object 102a. As a result, as shown in FIG. It looks like an image made up of a rectangular object 102c and a portion 103 shown in black. In FIG. 20, a portion 103 shown in black is a void portion (uncovered area) that has not been subjected to motion compensation.

  The void portion 103 where no motion compensation is performed and no predicted value (texture) is present is a portion to which predictive coding for encoding the difference between the predicted value and the actual pixel value is basically not applicable. The actual pixel value needs to be encoded as it is. Therefore, the fact that the uncompensated portion occupies a large area leads to deterioration in coding efficiency. Therefore, a conventional image coding apparatus is known that improves the efficiency of background prediction using shape information for a scene composed of a relatively simple object and background (see, for example, Patent Document 1). .

H. G. Musmann, M. Hotter, J. Ostermann, “Object-Oriented Analysis Synthesis Coding of Moving Images”, Signal Process, Image Commun., 1, 2, pp. 117-138 (Oct. 1989). J.Y.A.Wang, et. Al, “Applying Mid-level Vision Techniques for Video Data Compression and Manipulation”, M.I.T.Media Lab.Tech.Report No.263, (Feb.1994). (Http://www-bcs.mit.edu/people/adelson/pub_pdfs/wang_tr263.pdf) JP-A-9-98425

  However, the conventional image encoding device described in Patent Document 1 has a relatively large area and uses motion compensation on an explicitly determined background that does not have a large change in time. area (uncovered background)), and is difficult to apply to a moving image in which shape information for each frame does not exist and a moving image in which the object concealment relationship changes greatly.

  The present invention has been made in view of the above points, and an image coding apparatus and an image that can improve the coding efficiency even for moving images in which shape information about each frame does not exist and moving images in which the object concealment relationship changes greatly An object of the present invention is to provide a decoding device.

  In order to achieve the above object, the image encoding device of the present invention divides an image signal indicating a scene into an arbitrary shape according to the shape of the subject, changes the relative positional relationship of the division result, and then calculates the division result. In an object-based coding image coding apparatus that performs motion compensation to be reconfigured, an initial region creating unit that creates an initial region by dividing a key frame that is a basis of motion compensated prediction, and an initial region as a key frame Is placed on the corresponding position in different frames, and whether the characteristics of the texture of the arranged initial area and the characteristics of the texture in the vicinity are similar based on similar preset criteria When a comparison result is obtained by the comparison means for comparison and comparison, and a similar comparison result is obtained, the texture area in the vicinity is integrated with the initial area to form a new area. And a that integrating means, a signal of a new region formed by integrating means, characterized in that use as a reference picture signal in motion compensation image coding.

  In the present invention, a key frame that is the basis of motion compensation prediction is not divided into regions that do not have an overlap portion in a state where an image is divided into regions as used in motion compensation of conventional object-based coding. The initial area created in this way is placed on the corresponding positions in a plurality of frames different from the key frame, and the texture characteristics of the arranged initial area and the characteristics of the texture in the vicinity are respectively set to similar Based on the judgment criteria, whether or not they are similar is compared, and when similar comparison results are obtained, the texture area in the vicinity is integrated into the initial area to form a new area. The accuracy of compensation can be improved and the area to be applied can be expanded.

  In order to achieve the above object, the image decoding apparatus of the present invention includes a signal of a new region of a key frame that is a basis of motion compensation prediction obtained by the image encoding apparatus of the present invention, and a decoding target. An image decoding device that receives object shape data and motion vectors of a frame and a residual signal that is a difference between a key frame and a decoding target frame and restores an image signal of the original scene from these signals Then, based on the motion vector of the object of the frame to be decoded, the signal of the new area of the key frame is moved, and based on the object shape data of the frame to be decoded, the new key frame is moved. By extracting the signal of the area, the motion compensated decoded signal of the frame to be decoded is used by using the texture of the signal of the new area of the key frame. Motion compensation means, and a residual processing means for performing a residual process for adding the decoded signal of the decoding target frame obtained by the motion compensation means and the residual signal to obtain a decoded image signal. Features.

  According to the present invention, the initial region created by dividing the key frame into regions is arranged at corresponding positions in a plurality of frames different from the key frame, and the texture characteristics of the arranged initial region and the texture in the vicinity When a comparison result closer to the threshold value is obtained by comparing and comparing with the characteristics of each, the texture area near the periphery is integrated into the initial area to form a new area, thereby improving motion compensation accuracy As a result, it is possible to reduce the area where motion compensation is not performed. As a result, it is possible to improve the coding efficiency for a moving image in which shape information for each frame does not exist and a moving image in which the object concealment relationship changes greatly.

  Next, embodiments of the present invention will be described with reference to the drawings. In the present invention, the accuracy of motion compensation and the enlargement of the region to be performed are realized by using an overlapping layered region expanded by referring to a plurality of frames.

  FIG. 1 is a flowchart for explaining the outline of an embodiment of an image encoding method according to the present invention. In the figure, first, an initial region is created by dividing a key frame, which is the basis of motion compensation prediction, into regions based on, for example, the average / dispersion of luminance (step S1). Note that the image signal of the key frame is usually an intra-frame encoded signal. As an example of a known method of region segmentation based on luminance average and variance, see, for example, the literature (SCZhu and A. Yuille: “Region competition: unifying snakes, region growing, and Bayes / MDL for multiband image segmentation”. IEEE Transactions On There is a region competition method described in Pattern Analysis and Machine Intelligence. Vol.18, No.9 (1996)).

  This area competition method is a kind of energy functional optimization method, and grows and competes for areas where normal distribution statistics such as average and variance of luminance are similar. At this time, the competition is automatically adjusted so that unnatural unevenness does not occur at the region boundary.

  Here, the divided area is referred to as an initial area. For each initial region, the following steps S3 to S5 are performed for two frames before and after (step S2). That is, after placing the initial region on the corresponding position in the frame M (where M is four natural numbers represented by N-2, N-1, N + 1, N + 2 where N is an arbitrary natural number) ( In step S3), the texture characteristics of the arranged initial region and the texture characteristics in the vicinity of the surroundings are compared and collated based on the similar criteria set in advance (step S4). If it is determined that the texture is close and similar, the neighboring portion is newly added to the initial region as the same region and integrated (step S5). Thereafter, the area expanded by integration is stored as a layer (step S6).

  Here, as an example of the means for texture analysis in step S4 described above, there is a method using the average and standard deviation of the luminance values of each region. In addition, as an example of a criterion for determining the similarity of the texture at this time, for example, when the image is 8 bits (0 to 255 in the numerical value), the absolute value of the average difference is within 10 and the absolute value of the standard deviation difference is 35. A numerical value of within can be mentioned.

  As described above, in the present embodiment, with respect to each initial area of the divided key frame, the area corresponding to the initial area is first divided into areas by referring to two frames before and after the key frame, for a total of four frames. If the texture is of a different shape from the initial area and contains a texture of the same quality that was not included in the initial area determined by the initial area division, the part not included in the initial area is included in the initial area. Since a new area fused with the initial area determined by the division is formed and this new area is used for motion compensation, a portion where motion compensation is not performed can be reduced.

  Next, the integration in step S5 will be described on the premise that the object is moved as shown in the sequence of FIG. 2 in the scene composed of the two rectangular objects 101 and 102a shown in FIG. Assuming that frame 0 shown in FIG. 2 is a key frame, the initial area of the rectangular object 102a of the key frame has a shape as shown in FIG. FIG. 3 shows the initial region overlapped with the corresponding part of Frame 1 shown in FIG.

  A rectangular object 102b of Frame 1 shown in FIG. 2 includes an initial region portion 102a and a rectangular object portion 104 evaluated as similar to the initial region 102a as shown in FIG. Here, the part 104 of the object is a part that is evaluated as similar according to the above-mentioned criteria when the average / standard deviation of the texture is evaluated. Then, these portions 102a and 104 are integrated into a layer, and the same operation is performed in the next Frame2.

  Here, if the layer obtained by integrating 102a and 104 is set to 102b and overlapped with the corresponding portion 102d of Frame 2 shown in FIG. 2, the rectangular object portion 105 is evaluated as similar as shown in FIG. These layers 102b and 105 are integrated. In this example, two frames after the key frame are shown, but the same operation is performed for the past. A layer obtained by repeating this operation for all the initial regions is used for motion compensation.

  Next, an image encoding apparatus according to the present invention based on the above image encoding method will be described. FIG. 5 shows a block diagram of an embodiment of an image encoding apparatus according to the present invention. This embodiment divides an image signal indicating a scene into an arbitrary shape according to the shape of the subject, changes the relative positional relationship of the division result, and then performs object compensation that reconstructs the division result. This is an encoding method image encoding apparatus.

  In FIG. 5, if the key frame that is the basis of motion compensation prediction is the Nth frame (frame N), the frame memories 1, 2, 3, and 4 store the N-frames 2 frames before (the past) the key frame. 2nd frame (frame N-2), N-1 frame before frame 1 (frame N-1), 1 frame after (future) N + 1 frame (frame N + 1), 2 frames after The image signals of the (N + 2) th frame (frame N + 2) are stored. The input image signal of the key frame is segmented (region division) by the segmentation means 5, and a region having the segmentation result is stored in the frame memory 6 as an initial state of the layer, that is, as an initial layer.

  The initial layer images read from the frame memory 6 are the frame N-2, frame N-1, frame N + 1, and frame N + 2 images read from the frame memories 1, 2, 3, and 4, respectively. Motion estimation means 7, 8, 9, 10 are evaluated by motion estimation means 7, 8, 9, 10, respectively, and the motion estimation results obtained thereby are respectively obtained by texture matching means 11, 12, 13, 14. Are matched with the texture around the corresponding portion, and the respective matching results are supplied to the additional / integrated region determining means 15, 16, 17, 18 respectively, where it should be added to the initial layer based on the matching results or not. Are determined independently of each other.

  The image signal of the frame N-2 in the region to be added / integrated obtained by the addition / integration region determining means 15 is added / integrated to the image of the initial layer by the adding / integrating unit 20 in the frame memory 19. Similarly, the image signal of the frame N-1 of the region to be added / integrated obtained by the addition / integration region determining means 16 is added / added to the image from the integration unit 20 by the addition / integration unit 21 in the frame memory 19. Integrated. Further, the frame N + 1 image signal of the region to be added / integrated obtained by the addition / integration region determination means 17 is added / integrated to the image from the add / integration unit 21 by the addition / integration unit 22 in the frame memory 19. The image signal of the frame N + 2 of the area to be added / integrated obtained by the addition / integration area determining means 18 is added / integrated to the image from the add / integration section 22 by the addition / integration section 23 in the frame memory 19. Is done.

  As a result, a decoding motion compensation layer 24 is obtained from the addition / integration unit 23, and this is used for motion compensation in the image coding apparatus. Note that the addition / integration operation in the addition / integration units 20 to 23 is realized by processing of rewriting the frame memory 19 storing the initial layer in terms of hardware.

  Next, the operation of the image coding apparatus will be described in detail with a more specific example. Consider a part of the moving image in which the triangle 201 and the hexagon 202 are translated behind the rectangle 201 and the hexagon 202 as shown in FIG. 6, and frame N is divided into regions as described above. Frame and key frame.

  At this time, the initial region of the triangular portion 203c obtained as a result of dividing the frame N into regions is concealed by the rectangle 201 and the hexagon 203, and thus has a shape as shown in FIG. FIG. 8 shows the result of motion compensation based on the initial region of the triangular portion 203c, and the black portions of the moving triangular portions 203a ′ to 203e ′ are portions that have not been substantially compensated for motion. .

  In order to prevent this, the image coding apparatus of the above embodiment refers to the past and future frames, and as shown in FIG. N-2 and frame N-1 images are integrated with a similar texture indicated by 204 in FIG. 9 by performing motion estimation, texture matching, and integration region determination with the initial layer (initial region image). Fusion) Generates candidate area images, performs future frame N + 1, frame N + 2 images for key frames, performs motion estimation with initial layers (initial area images), texture matching, and integrated area determination Then, an image of the integration (fusion) candidate region having a similar texture indicated by 205 in FIG. 9 is generated, and these are integrated with the image of the initial region.

  As a result, a layer having a triangular shape as shown in FIG. 10 is configured as the decoded motion compensation layer 24. The decoded motion compensation layer 24 is used as a reference image for motion compensation in the image coding apparatus, thereby realizing motion compensation without a hole as in the input sequence of FIG. Note that the motion compensation layer 24 is intra-coded separately from a normal key frame, for example, and transmitted to the image decoding apparatus.

  When the common texture portion is integrated, if a part of the adjacent area remains as shown by 301 and 302 in FIG. 11 due to a segmentation error, the common texture portion is also integrated in FIG. As shown, false contours 401 and 402 are generated. In order to suppress this, as shown by reference numerals 501 and 502 in FIG. 13, a method of taking the added / integrated portion (fusion candidate region) large so as to include the segmentation error portions 301 and 302 and continuously changing the mixing degree is available. Conceivable. According to this method, the mixing ratio of the initial region 503 in which a part of the adjacent region remains can be reduced, and as a result, it can be expected that the false contour becomes inconspicuous.

  As described above, when merging a part that was not included in the initial region as a result of referring to multiple frames with the initial region, both textures are blended at a ratio that continuously changes in the vicinity of the boundary. False contours due to errors can be reduced.

  Next, an example of an image encoding method corresponding to the image encoding device of the present invention will be described with reference to FIG. In the encoder, the key frame (frame N) is segmented (region division) (step S11), and the region having the segmentation result is first stored as the initial state of the layer, that is, the initial layer (step S12).

  On the other hand, the motions between the above segmentation results and frame N-2, frame N-1, frame N + 1, and frame N + 2 are estimated separately (steps S13, S14, S15, and S16). This corresponds to step S3 in FIG. 1 and is the same as placing the initial region on the corresponding position in frame M.

  Subsequently, texture matching is performed to compare the texture characteristics of the arranged initial region and the textures around the corresponding portion (steps S17, S18, S19, S20). For example, when the comparison result of the texture characteristics is closer than the threshold value, it is determined that addition and integration should be performed (steps S21, S22, S23, and S24). These correspond to steps S4 and S5 in FIG.

  Subsequently, the areas for which addition / integration is determined are sequentially added / integrated to the initial layer (steps S25, S26, S27, S28). The region added / integrated to the initial layer is set as one layer and output as a decoded motion compensation layer for use in motion compensation at the decoder (step S29). This corresponds to step S6 in FIG.

  As described above, according to the present embodiment, a plurality of frames are referred to in units of each initial area, instead of using an area obtained by dividing an image as used in the conventional object-based coding motion compensation. Thus, by forming a layer in which the region is expanded and using this, it is possible to improve the accuracy of motion compensation, reduce the region where motion compensation is not performed, and consequently improve the coding efficiency.

  Next, an image decoding apparatus according to the present invention will be described. FIG. 15 shows a block diagram of an embodiment of an image decoding apparatus according to the present invention. This image decoding device is a device that decodes the encoded signal obtained by the above image coding device, and decodes the key frame that is the basis of the motion compensated prediction obtained by the above image coding device. A compensation layer, object shape data and motion vectors of a decoding target frame (current frame), and a residual signal that is a difference between the key frame and the decoding target frame are input via a transmission medium. .

  Here, the decoding motion compensation layer of the key frame is transmitted by being intra-coded separately from a normal key frame, for example, as a part of the key frame information and stored in the frame memory 31. Further, the shape information of the object corresponding to the decoding target frame (current frame) of the non-key frame is stored in the frame memory 32.

  In other words, the current frame of the non-key frame is transmitted after being subjected to arithmetic coding after the image of the current frame is segmented, and similar to MPEG-4 binary arbitrary shape coding, etc., and is stored in the frame memory 32. Is done. Further, the object motion vector of the current frame is stored in the frame memory 33.

  The decoded motion compensation layer and the current frame object shape information read from the frame memories 31 and 32 are supplied to the motion compensation means 34 together with the motion vector 33 of the object of the current frame. Based on the motion vector, the decoding motion compensation layer of the key frame is moved, and the decoded motion compensation layer after the movement is cut out based on the object shape information of the current frame. Is used to decode the motion-compensated signal of the current frame.

  The motion-compensated decoded signal obtained by the motion compensation unit 34 is supplied to the residual processing unit 35. On the other hand, in the image coding apparatus, the difference between the above frame and the actual current frame is taken and this difference (residual) signal is transmitted by DCT. Therefore, in the image decoding apparatus, this signal is IDCT. Thus, a residual signal is obtained. In the residual processing means 35, residual processing for adding both signals similar to MPEG or the like is performed on the basis of the residual signal and the motion compensated decoded signal obtained by the motion compensation means 34. The frame memory 36 stores the decoded image signal of the original scene, and the decoding process ends.

  Next, an image decoding method corresponding to an embodiment of the image decoding apparatus of the present invention shown in FIG. 15 will be described with reference to FIG. The decoding motion compensation layer input as a part of the key frame information in the decoder is shown in FIG. 15 based on the shape information of the corresponding object of the current frame of the non-key frame and the object motion information of the current frame. Motion compensation similar to that of the motion compensation means 34 is performed (step S31). Thereafter, residual processing similar to MPEG or the like for adding the residual signal to the motion compensated decoded signal is performed (step S32), and the residual processing result is stored in the memory as a decoded image signal (step S33). .

  Note that the present invention is not limited to the above-described embodiment, and the present invention executes, for example, each unit of the image encoding device and each unit of the image decoding device of the above-described embodiment by a computer. The computer program to be included is also included. That is, it includes an encoding computer program and a decoding computer program. These programs may be taken into a computer from a recording medium, or may be distributed via a network and taken into a computer.

It is a flowchart for outline | summary description of one Embodiment of the image coding method of this invention. It is a figure which shows an example of the sequence which consists of two rectangles by which the movement of an object is performed. It is explanatory drawing of the layer integration in Frame1 of FIG. It is explanatory drawing of the layer integration in Frame2 of FIG. It is a block diagram of one embodiment of an image coding device of the present invention. It is a figure which shows an example of the input sequence of FIG. 5 with which an object is moved. It is an initial region of the triangular portion in FIG. It is explanatory drawing of the result of the motion compensation by a conventional method. It is a figure which shows the addition and the integration result which referred the several flame | frame by this invention apparatus. It is a figure which shows the triangular layer without a chip | tip obtained by this invention apparatus. It is a figure which shows an example of the error of a segmentation. It is a figure which shows an example of a false outline. It is explanatory drawing of the false contour suppression by blending. It is explanatory drawing of one Embodiment of the image coding method of this invention. It is a block diagram of one embodiment of an image decoding device of the present invention. It is explanatory drawing of one Embodiment of the image coding method of this invention. It is a figure which shows an example of the image of a key frame. It is a figure which shows an example of the image of the flame | frame of prediction object. It is a figure which shows an example of the object used for prediction. It is a figure which shows an example of the prediction result by the conventional image coding method.

Explanation of symbols

1-4, 6, 19, 31, 32, 36 Frame memory 5 Segmentation means 7-10 Motion estimation means 11-14 Texture matching means 15-18 Addition / integration area determination means 20-23 Addition / integration section 24 Decoding motion compensation Layer 34 motion compensation means 35 residual processing means

Claims (2)

An image code of an object-based coding method that divides an image signal indicating a scene into an arbitrary shape according to the shape of the subject, changes the relative positional relationship of the division result, and performs motion compensation to reconstruct the division result In the conversion device,
An initial region creating means for creating an initial region by dividing a key frame serving as a basis for motion compensation prediction;
The initial area is arranged on corresponding positions in a plurality of frames different from the key frame, and the texture characteristics of the arranged initial area and the texture characteristics in the vicinity of the surrounding areas are set in advance as similar judgment criteria. Comparing means for comparing and collating based on
An integration unit that forms a new region by integrating the texture region in the vicinity of the periphery when a similar comparison result is obtained by the comparison unit, and formed by the integration unit. An image coding apparatus characterized in that the signal of the new area is used as a reference image signal in motion compensation of image coding. The signal of the new area of the key frame that is the basis of motion compensated prediction obtained by the image encoding device according to claim 1, the object shape data and motion vector of the frame to be decoded, the key frame, and the key frame A residual signal that is a difference from a frame to be decoded is input, and an image decoding device that restores an image signal of an original scene from these signals,
Based on the motion vector of the object of the frame to be decoded, the signal of the new area of the key frame is moved, and based on the object shape data of the frame to be decoded, the new key frame is moved. A motion compensation unit that obtains a motion-compensated decoded signal of a frame to be decoded using a texture of a signal of a new region of the key frame by cutting out a signal of a new region;
And a residual processing unit that performs a residual process of adding the decoded signal of the decoding target frame obtained by the motion compensation unit and the residual signal to obtain a decoded image signal. Image decoding device.

Images