WO2005125218A1

WO2005125218A1 - Prediction error based segmentation refinement within a forward mapping motion compensation scheme

Info

Publication number: WO2005125218A1
Application number: PCT/IB2005/051901
Authority: WO
Inventors: Reinier B. M. Klein Gunnewiek
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2004-06-16
Filing date: 2005-06-09
Publication date: 2005-12-29

Abstract

A method, a decoder and a computer readable medium having embodied thereon a computer program for optimizing an encoding/decoding process of a video sequence having at least three frames are disclosed. Based on a discontinuity in the residue information comprised in said video sequence a decision is taken if a setting of an encoder or decoder in said coding process is chosen for optimal efficiency of the process. According to an embodiment residue information from coding one frame is used to steer segmentation and to enhance coding efficiency. According to another embodiment residue information is used to steer foreground/background decisions for segments of the frames. When segmentation or a foreground/background decision is made for a first frame, the residue information of a subsequent second frame is used to confirm or to adjust the decision or segmentation in the subsequent third frame.

Description

PREDICTION ERROR BASED SEGMENTATION REFINEMENT WITHIN A FORWARD MAPPING MOTION OMPENSATION SCHEME

This invention pertains in general to the field of video coding, and more particularly to a video coding method having improved efficiency by optimizing an encoding/decoding process of a video sequence.

In recent years, the use of digital storage and distribution of video signals has become increasingly prevalent. In order to reduce the bandwidth required to transmit digital video signals, it is well known to use efficient digital video encoding comprising video data compression whereby the data rate of a digital video signal may be substantially reduced. 10 In order to ensure interoperability, video encoding standards have played a key role in facilitating the adoption of digital video in many professional- and consumer applications. Most influential standards are traditionally developed by either the International Telecommunications Union (ITU-T) or the MPEG (Motion Pictures Experts Group) committee of the ISO/IEC (the International Organization for Standardization/the 15 International Electrotechnical Committee). The ITU-T standards, known as recommendations, are typically aimed at real-time communications (e.g. videoconferencing), while most MPEG standards are optimized for storage (e.g. for Digital Versatile Disc (DVD)) and broadcast (e.g. for Digital Video Broadcast (DVB) standard). Currently, one of the most widely used video compression techniques is

20 known as the MPEG-2 (Motion Picture Expert Group) standard. MPEG-2 is a block based compression scheme wherein a frame is divided into a plurality of blocks each comprising eight vertical and eight horizontal pixels. For compression of luminance data, each block is individually compressed using a Discrete Cosine Transform (DCT) followed by quantization which reduces a significant number of the transformed data values to zero.

25 Frames based only on intra-frame compression are known as Intra Frames (I- Frames). In addition to intra-frame compression, MPEG-2 uses inter-frame compression to further reduce the data rate. Inter-frame compression includes generation of predicted frames (P-frames) based on previous I- frames. In addition, I and P frames are typically interposed by Bidirectional predicted frames (B-frames), wherein compression is achieved by only transmitting the differences between the B-frame and surrounding I- and P-frames. In addition, MPEG-2 uses motion estimation wherein the image of macro-blocks of one frame found in subsequent frames at different positions are communicated simply by use of a motion vector. Motion estimation is performed to determine the parameters for the process of motion compensation or, equivalently, inter prediction. As a result of these compression techniques, video signals of standard TV studio broadcast quality level can be transmitted at data rates of around 2-4 Mbps. Recently, a new ITU-T standard, known as H.26L, has emerged. H.26L is becoming broadly recognized for its superior coding efficiency in comparison to the existing standards such as MPEG-2. Although the gain of H.26L generally decreases in proportion to the picture size, the potential for its deployment in a broad range of applications is undoubted. This potential has been recognized through formation of the Joint Video Team (JVT) forum, which is responsible for finalizing H.26L as a new joint ITU-T/MPEG standard. The new standard is known as H.264 or MPEG-4 AVC (Advanced Video Coding). Furthermore, H.264-based solutions are being considered in other standardization bodies, such as the DVB and DVD Forums. The H.264/AVC standard employs similar principles of block-based motion estimation as MPEG-2. However, H.264/AVC allows a much increased choice of encoding parameters. For example, it allows a more elaborate partitioning and manipulation of 16x16 macro-blocks whereby e.g. a motion compensation process can be performed on divisions of a macro-block as small as 4x4 in size. Another, and even more efficient extension, is the possibility of using variable block sizes for prediction of a macro-block. Accordingly, a macro-block (still 16x16 pixels) may be partitioned into a number of smaller blocks and each of these sub-blocks can be predicted separately. Hence, different sub-blocks can have different motion vectors and can be retrieved from different reference pictures. Also, the selection process for motion compensated prediction of a sample block may involve a number of stored, previously-decoded frames (or images), instead of only the adjacent frames (or images). Also, the resulting prediction error following motion compensation may be transformed and quantized based on a 4x4 block size, instead of the traditional 8x8 size. Generally, existing encoding standards such as MPEG 2 and H.264/ AVC exploit temporal correlation by a block based motion estimation and compensation. Thus, the motion estimation and compensation algorithms are based on the encoding blocks of the video standard. Although this provides for an efficient encoding of video signals, it is desirable to provide an even more efficient video encoding wherein a higher quality to data rate ratio can be achieved. An option that promises improved encoding performance is to provide an image segment based motion estimation and compensation. For example, image segments corresponding to players in a sports arena may be determined and used for motion estimation. However, such segmentation is not always optimal. Segmentation errors, e.g. erroneous estimations of a segments position in subsequent frames of a video sequence, for instance based on a wrong decision concerning the segment being in the foreground or in the background. Hence, an improved system for video coding would be advantageous and in particular a system enabling or facilitating proper segmentation at the decoder for increasing the coding efficiency. Increased coding efficiency means for instance that a given picture quality is maintained at a lower bit rate or that a better picture quality/resolution is obtained at the same bitrate compared with existing coding/decoding schemes. Therefore, a number of advantages are offered by increased coding efficiency, among others more effective use of existing transmission channels, transmission of video sequences via transmission channels with lower bandwidth without a loss of quality, etc.

The present invention seeks to overcome the above-identified deficiencies in the art and solves at least the above identified problems singly or in any combination by providing a method, a decoder and a computer readable medium according to the appended patent claims. The general solution according to the invention is to use residue information as a feedback input for improving the coding. More precisely, the invention is based on the finding that a discontinuity in the residue signal is an indication that a setting of the encoder or decoder is not optimally chosen. Hence, this setting is modified for frames of a video sequence that still have to be coded, wherein the modification is based on the input residue signal. This surprising insight was revealed during experiments performed by the applicant. It was found out that the segmentation is not always optimal, as mentioned above. Further, it was found out that in this connection locally much residue information exists in images that have been segmented sub-optimally. Moreover, this high amount of residue information increases the bit rate dramatically. Furthermore, in motion compensation processes, i.e. shifting thus segmented segments from one frame to the other, false foreground background detection, in areas where segments overlap, results as well in more residue information and therefore a higher bit rate. As mentioned above, proper segmentation and ordering of segmentation is essential to get the desired high coding efficiency. Therefore, the residue information is used as a feedback input to optimize the efficiency of the coding process of a video sequence having a plurality of frames. According to aspects of the invention, a method, a decoder, and a computer-readable medium for optimizing a coding process of a video sequence are disclosed. According to one aspect of the invention, a method is provided for optimizing an encoding/decoding process comprising segmentation of a video sequence having at least three frames. The method comprises the step of deciding if a setting of an encoder or decoder in said encoding/decoding process is chosen for optimal efficiency of said encoding/decoding process, wherein the decision is based on a discontinuity in the residue signal of said video sequence. Preferably this setting is modified based on said residue signal without transmitting any additional information from the encoder to the decoder for the frames that still have to be coded. According to a preferred embodiment the method comprises the steps of segmenting a first frame of the video sequence including a foreground/background decision for each segment. Furthermore, residue information is derived from a second frame, subsequent to said first frame of the video sequence. Then the decision already made is adjusted based on the residue information derived. According to a further preferred embodiment, the method comprises the steps of subdividing a first frame of the video sequence into segments, deriving residue information from a second frame subsequent to the first frame of the sequence, and adjusting the segment subdivision based on the derived residue information. According to another aspect of the invention, a decoder is provided for optimizing a coding process of a video sequence having at least three frames. The decoder is adapted and configured to perform the method according to the invention. According to a further aspect of the invention, a computer-readable medium having embodied thereon a computer program for processing by a computer is provided. The computer program comprises at least one code segment for optimizing a coding process of a video sequence having at least three frames. The code segments are adapted to perform the method according to the invention. The present invention has the advantage over the prior art that it allows to steer decisions at the decoder side without putting additional information in the bit stream, whereby the coding efficiency is improved.

Further objects, features and advantages of the invention will become apparent from the following description of embodiments of the present invention, reference being made to the accompanying drawings, in which Fig. 1 is a block diagram depicting a schematic overview of an improved segmentation based decoder, comprising foreground background detection and residue deviation detection according to an embodiment of the invention; Fig. 2 is an illustration of a video coding method in accordance with an embodiment of the invention; Fig. 3 is a sample image of a video sequence showing a segmented frame of said sequence; Fig. 4 is the sample image of Fig. 3, wherein areas that are double addressed by motion vectors are shown, for which proper foreground background detection is required; Fig. 5 is the subsequent frame of the image shown in Fig. 3 and Fig. 4, having no residue steering applied; and Fig. 6 is the subsequent frame of the image shown in Fig. 5, having residue steering according to the invention applied.

The following description focuses on a specific embodiment of the invention but it will be appreciated that the invention is not limited to this specific application. According to the present embodiment of the invention a method is provided that uses residue information from coding one frame to steer the segmentation of the next frame. With residue the quantized difference signal is meant as defined hereinafter. For each block/segment (k) of a frame at a time instance t a motion estimation is performed, and the resulting vectors, d_k, are used to form a prediction for each corresponding block/segment:

Ipredicted(X_k, t) = Idecoded(X +d_k, t - 1). After this prediction, the predicted block/segment is subtracted from the current corresponding block/segment: Idifference(X_k,t)= Ipredicted(X ,t) - Icurrent(X_k,t).

Subsequently, the difference is quantized with a Qparam_k (Quantization parameter that is also transmitted to the decoder) and VLC (Variable Length Coding) encoded. The quantized difference is what is referred to as the "residue" within this specification:

Iresidue(X_k,t) = Idifference(X_k,t)/Qparam_k

Fig. 1 shows a block diagram depicting a schematic overview of a segmentation based decoder 1 according to an embodiment of the invention, making use of H264 like basic building blocks 2 and 3, in combination with segmentation. The decoder comprises foreground background detection 4 and residue deviation detection 5,6. The incoming bit-stream is decoded, firstly by an H264 like residue infra decoder 2, similarly like a regular infra decoder. This frame is then segmented using an identical segmentation process as used at the encoder side. The resulting segmentation map is used in the motion compensation process. Since the segmentation map is generated at the decoder 1 side, and thus we have the segmentation map of the previous frame, we have to motion compensate from the previous to the current image in order to form a prediction from the current picture. This direction/ way of motion compensation is sometimes referred to as forward (shift) projection/compensation of segments. Due to this forward motion compensation some segments will overlap, as will be explained below. After the motion compensation process, the overlapping segments are then identified and it is determined which segment should be used. In a conventional decoder, this determination would be based on information in the bit- stream or on the basis of a predetermined criteria, e.g. the foreground background map that is produced by the segmentation process. However, in this embodiment the foreground/background decision is based on the method described hereinafter, i.e. deviations detected in the residue signal are used as a basis for the foregroundbackground decision. No additional information has to be transmitted in the bit-stream. This results in the 'Final Projection' 7 that still contains unreferenced areas, the so-called holes. These areas are also due to the forward motion estimation process. Typically uncovering areas will not be addressed in the predicted version of the current frame since these areas were not visible in the previous frame, since those areas were behind a foreground object. Therefore, the decoder detects 8 these holes next and these holes are filled with an identical process 9 as used at the encoder side. Meanwhile an H264 like residue decoder 3 has decoded the bit-stream using an inverse process than that has been used at the encoder side. The resulting residue signal is then added to the outcome of the hole filling process. This frame can now be displayed and should be segmented again for the next frame to decode. Hereinafter an exemplary embodiment of the method performed by decoder 1 is given in order to illustrate this method. In the embodiment illustrated in Fig. 2, we suppose to have a fast moving object in the foreground of a video sequence. This is for instance the tree moving to the left in the foreground of the sequence in the example images given below in Figs. 5-8. In the motion estimation/compensation process of the decoder 1 this fast moving object in the foreground will result in areas that are not referenced, due to the fact that objects (here: fast moving tree in the foreground vs. background) are moving at various speeds, some areas will appear from behind an object moving in the foreground, so-called de- occluding or uncovering areas. These not referenced areas are the above mentioned holes and will give much residue information since the content of the areas that will be uncovered is not known, hence it can not be predicted from information from the past. One can observe a sharp edge in the amount of residue information when going from un-referenced to referenced areas. This sharp edge will be an edge of a segment or of an area comprising several segments. In the example images, this occurs on the right side of the tree 30, where background emerges from behind the tree 30. On the other side of this fast moving object areas are obtained that are multiple referenced. This is due to the fact that the fast moving object that is in the foreground (exemplary tree) will cover/occlude some areas of the background that were visible in the previous frame, the so-called covering or occlusion of background. Normally the occlusion will not end at a segment boundary of the background, but at a segment boundary of the foreground since it is always "in the front" and thus the foreground will not be occluded.

Hence, a foreground segment (of the tree in the example images) will have one motion vector and a background segment will have a different motion vector. This background segment will partly be covered by the foreground segment. In those areas, most likely a part of the foreground and a part of the background segment, one will observe that the motion vector of that foreground segment and the motion vector of that background segment are pointing to it. When the segmentation 20 of a first frame is done, the codec (Coder/Decoder, an encoder at the transmitter side for compressing the data to be send, and a decoder 1 at the receiver side for decompressing the data to be further processed at the receiver side) tries to determine 21 which segment is in the foreground and which segment is in the background. Then, in the next frame (second frame, step 22) the residue signal is analysed. If the codec makes the wrong decision in step 21, a high residue signal will result in the next frame in step 23, otherwise the residue signal will be low. The invention takes advantage of the insight that a high residue signal is a clear indication that the codec made the wrong decision for the previous frame. By means of this knowledge it is possible to give a feedback to the codec and it can modify its choice/setting (step 25) in the subsequent frame (third frame, step 24). The third frame, and every other frame, are the frames that benefit from this method. This method may be applied in case the segmentation map is transmitted as well as in case the segmentation map is regenerated at the decoder side. However, in the latter case, the method according to the invention is of best use as the segmentation map needs not to be transmitted and thus bitrate/bandwidth is saved in the transmission channel. Accordingly, in the latter case this embodiment of the present invention will influence the choice that will be made in the next frame. This embodiment is further elucidated by way of a non- limiting example. According to a first example, the foreground/background issue is contemplated. As mentioned above, it is believed that segmentation helps achieving a better coding efficiency. For the sake of coding efficiency a segmentation map is calculated at the decoder side, as well as in the local decoder at the encoding side. Since the segmentation map is calculated at the decoder side, shift motion compensation has to be performed, i.e. motion compensating segments from the locally decoded image to the current/to be predicted image. Due to this shift motion estimation some areas overlap, some areas are not addressed at all. In order to decide which segment should be in the foreground and which one should be in the background, foreground/background detection is required. This detection may be performed by using the level of residue information. In the first frame an arbitrarily choice is made. If the residue in the overlapping areas is low, comparable to its neighbourhood, this is a strong indication that the initial choice was the right choice. For the next frame, this choice is maintained. On the other hand, if the residue in the overlapping area is high, whereas in its neighbourhood it is low, this is a strong indication that the initial arbitrarily made choice was wrong. For the next frame, the other choice is used. In the next run this will most likely result in a reduction of the bit rate. If not, it is switched back to the previous decision for the next frame. A further embodiment of the method according to the invention is given now, wherein the segmentation issue is contemplated. As mentioned above, it is believed that segmentation helps achieving a better coding efficiency. For the sake of coding efficiency the segmentation map is calculated at the decoder side, and in the local decoder at the encoding side. According to this embodiment, the segmentation uses a segmentation map generated in the past. In certain cases the colour segmentation does not result in a good segmentation map. In these cases the coding efficiency suffers, there will locally be a high residue signal. Due to the fact that the segmentation uses the map from the past the coding efficiency keeps on suffering. It is not favourable to send bits to the decoder in order to signal this, as it would increase the bit rate and deteriorate the coding efficiency. However, we do have the residue signal at the decoder and the encoder side. Hence, we see locally within a part of a segment (a segment on color/texture basis should be constant), a strong increase in residue. This indicates that something is not going as it should, the segmentation of the segment has to be erraneous. Therefore, for the next segmentation process it is stated that this part of the segment belongs to the other segment. Hence, during the next segmentation step this will aid the segmentation. The above method is illustrated with the sample images given in Figs. 3-6. Fig. 3 is a sample image of a video sequence showing a segmented frame of this sequence. The figure depicts a frame from a video sequence at the stage of segmentation for that frame. A tree 30 can be seen in the middle of the image. The tree 30 is moving to the left in the sequence at a difference speed than the background. Typically at the boundary places between the foreground (tree) and the background either not referenced areas (holes, e.g. to the right of the tree's trunk) or overlapping areas (for instance to the left of the tree's trunk) can be observed. In this case the overlapping areas are at the left side of the tree and the holes on the right side. Furthermore, one can see that the tree is not properly segmented. The segmentation process can be steered based on residue information according to the invention. As mentioned above, the residue signal in a segment should be more or less constant. If this is not the case and the segment is not addressed by two or more vectors, the segment should be segmented further until a uniform residue signal for the entire segment is obtained. This is done according to the above described embodiment treating the segmentation issue. Fig. 4 is an image showing the residue signal values of the next frame of the sample video sequence image of Fig. 3, wherein areas that are double addressed by motion vectors are shown, for which proper foreground background detection is required. The white areas 40 in the picture show the areas that are double addressed by motion vectors. This is the place where proper foreground background detection is required. Fig. 5 is the subsequent frame of the image shown in Fig. 3 and corresponds to the Fig. 4, i.e. having no residue steering applied. If no residue steering is applied, e.g. in the first frame, one can clearly see areas where there is much more difference signal, residue. Especially around the tree. At the righthand side it is due to uncovering. At the lefthand side of the tree 30, as can be seen from the picture in Fig. 5, areas are motion compensated more than once. As can be seen from the picture in Fig. 4, the residue in these areas is much higher than in the other areas, due to the wrong foreground background decision. Fig. 6 is the subsequent frame of the image shown in Fig. 5, having residue steering applied according to the above described method by the above described decoder. The picture in Fig. 6 shows what happens after the residue steering is applied in the following/next frame, based on the residue signal information. The picture in Fig. 6 shows clearly the advantage of the residue steering that is applied in the following/next frame. Picture quality is enhanced without additional information in the bit-stream. The above embodiments have been described with reference to a decoder only. However, the method may, and in most cases will, be applied correspondingly both at the encoder and at the decoder side. Applications and use of the above described coding method according to the invention are various and include all fields and any kind of digital device where video compression is required, ranging from professional broadcasting, satellite TV broadcast and Set Top Boxes, interactive TV, to mobile phones equipped with filming capability and digital home servers to portable storage devices. The invention can be implemented in any suitable form including hardware, software, firmware or any combination of these. However, preferably, the invention is implemented as computer software running on one or more data processors and/or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit or may be physically and functionally distributed between different units and processors. Although the present invention has been described in connection with the preferred embodiment, it is not intended to be limited to the specific form set forth herein. Rather, the scope of the present invention is limited only by the accompanying claims. In the claims, the term comprising does not exclude the presence of other elements or steps. Furthermore, although individually listed, a plurality of means, elements or method steps may be implemented by e.g. a single unit or processor. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and the inclusion in different claims does not imply that a combination of features is no feasible and/or advantageous. In addition, singular references do not exclude a plurality. Thus references to "a", "an", "first", "second" etc do not preclude a plurality. Reference signs in the claims are provided merely as a clarifying example shall not be construed as limiting the scope of the claims in any way.

Claims

CLAIMS:

1. A method for optimizing an encoding/decoding process of a video sequence having at least three frames comprising the step of deciding if a setting of an encoder or decoder in said encoding/decoding process is chosen for optimal efficiency of said encoding/decoding process, - said decision being based on a discontinuity in the residue information comprised in said video sequence.

2. The method according to claim 1, further comprising the step of modifying said setting based on said residue information, without transmitting any additional information from said encoder to said decoder for subsequent frames that still have to be coded.

3. The method according to claims 1 or 2, comprising segmenting a first frame of said sequence comprising a foreground/background decision for each segment, - deriving residue information from a second frame, subsequent to said first frame of said sequence, adjusting said decision based on said residue information.

4. The method according to claim 3, said adjusting comprising maintaining said decision in case said residue information is below a predetermined threshold.

5. The method according to claim 3, said adjusting comprising reversing said decision in case said residue information exceeds a predetermined threshold.

6. The method according to claim 5, comprising adjusting segments in the subsequent third frame.

7. The method according to any of claims 3-6, wherein said foreground/background decision for segments of said first frame is being made based on an arbitrarily choice for an overlapping area, and if the residue in the overlapping area of said second frame is low, comparable to the residue of its neighbourhood, the foreground/background decision is maintained in at least a third frame subsequent to said second frame, or - if the residue in the overlapping area of said second frame is high whereas the residue in its neighbourhood is low, the initial foreground background decision is reversed in a third frame subsequent to said second frame.

8. The method according to claim 7, further comprising reversing said decision made in said third frame if the bitrate of a frame subsequent to said third frame is not reduced.

9. The method according to claims 1 or 2, comprising subdividing a first frame of said sequence into segments, - deriving residue information from a second frame, subsequent to said first frame of said sequence, and adjusting said segment subdivision based on said residue information.

10. The method according to claim 9, wherein said segment subdivision is adjusted based on an increase in residue within a part of a segment, and wherein said part of said segment is allocated to another, adjacent segment in a third frame subsequent to said second frame.

11. The method according to any preceding claims, wherein said residue information is the bitrate of said segments, part of said segments or said overlapping areas.

12. The method according to any of claims 1-10, comprising deriving said residue information from a quantized difference signal of said sequence.

13. A video decoder (1) adapted and configured to perform the method according to any of claims 1 to 12.

14. A computer-readable medium having embodied thereon a computer program for processing by a computer, the computer program comprising at least one code segment for performing the method according to any of claims 1 to 12.

15. Use of the video decoder (1) according to claim 13 to process the computer program comprised on the computer-readable medium according to claim 14.