KR20050031460A - Method and apparatus for performing multiple description motion compensation using hybrid predictive codes - Google Patents

Abstract

An improved multiple description coding (MDC) method and apparatus is provided which extends multi-description motion compensation (MDMC) by allowing for multi-frame prediction and is not limited to only I and P frames. Further, the coding method of the invention extends MDMC for use with any conventional predictive codec, such as, for example, MPEG2/4 and H.26L. The improved MDC permits the use of any conventional predictive coder for use as a top and bottom predictive encoder. Further, the top and bottom predictive coders can advantageously include B-frames and multiple prediction motion compensation. Still further, any of the top, middle and bottom predictive encoders can be a scalable encoder (e.g., FGS-like or data-partitioning like where the motion vectors (MVs) are sent first, temporal scalability etc.).

Description

METHOD AND APPARATUS FOR PERFORMING MULTIPLE DESCRIPTION MOTION COMPENSATION USING HYBRID PREDICTIVE CODES}

The present invention generally relates to multiple representation coding (MDC) of data, voice, audio, image, video and other types of signals for transmission over a network or other type of communication medium.

Much of the information flowing through today's networks is useful even in a deteriorating state. Examples include voice, audio, still images and video. If such information suffers from packet loss, retransmissions may not be possible due to real time constraints. Good performance in terms of total rate, distortion, and delay can often be achieved by adding extras to the bit stream rather than repeating lost packets.

Redundancy can be added to the bit stream in one direction through multiple representation coding (MDC), where the data is split into several streams with some redundancy in the stream. Once all streams are received, low distortion can be guaranteed at the expense of having a slightly higher bit rate than a system designed purely for compression. On the other hand, if only a portion of the stream is received, the quality of reconstruction is gracefully degraded, which is rarely the case for systems designed purely for compression. Unlike multiple resolutions or layered source coding, there is no representation scheme, so multiple representation coding is suitable for deleting channels or packet networks without provisioning priority.

Multiple representation coding can be implemented in a number of ways. One approach is to separate the incoming video stream into any subset of the channel by separately collecting odd and even frame sequences at the encoder and independently coding the resulting temporal sub-sampled sequence. Upon receiving one of the sub-sampled sequences at the decoder, the video stream may be decoded at half the frame rate. Due to the correlated nature of the video stream, receiving only one of the sub-sampled sequences allows for recovery of the intermediate frame using motion compensated error cancellation techniques. This technique is described in great detail in Wenger et al., "Error Resilience Support in H.263 +," published in IEEE 1998 pages 867-877 on circuits and systems for video technology in November 1998.

To achieve error resilience, Wang and Lin's "Error Resilient Video Coding Using Multiple Representation Motion Compensation," published in IEEE 2002, 12, page 4348 to page 4352, on Circuits and Systems on Video Technology, June 2002. Describes one way to implement multiple representation coding. According to this approach, the temporal predictor allows the encoder to use both past even and odd frames during encoding, resulting in a mismatch between the encoder and the decoder when only one representation is received at the decoder. Mismatch errors are explicitly encoded to overcome this problem. The main advantage of allowing the encoder to use both odd and even frames for prediction is in terms of coding efficiency. By changing the temporal filter tap, the extra amount can be controlled. This method disclosed provides just flexibility between the extra amount and the error resilience.

The drawback of the Wang and Lin approach is that it is limited to only I and P frames (not B-frames). Another drawback of this approach is that it does not allow multiple frame prediction as used in H.26L. These drawbacks limit the coding efficiency of MDMC and also require a completely proprietary implementation instead of using the available codec modules.

1 illustrates an MDMC encoder in accordance with an embodiment of the present invention.

The present invention provides an improved multiple representation coding (MDC) method and apparatus that overcomes the aforementioned drawbacks. In particular, the coding method of the present invention allows for multiple frame prediction, thereby extending the multiple representation motion compensation (MDMC) and is not limited to only I and P frames. The coding method of the present invention also extends MDMC for use with any conventional predictive codec such as MPEG2 / 4 and H.26L.

According to a first aspect of the present invention, there is provided an improved MDMC encoder comprising three predictive coders: top, middle and bottom coders. The input frame is fed to the encoder as three separate inputs. The input frame is fed to a central encoder. In addition, the input frame is divided or divided into two sub-streams of the frame, the first sub-stream includes only odd frames, and the second sub-stream includes only even frames. The first sub-stream containing odd frames is provided as an input to be encoded by the upper encoder to produce an encoded odd frame sequence, and the second sub-stream comprising even frames contains an encoded even frame sequence. It is provided as an input to be encoded by the lower encoder to produce. Another embodiment may divide a frame using different standards, such as unbalanced splitting, for example, in which two of the three frames are encoded by the upper encoder and every third frame is encoded by the lower encoder. It is noted that it can. The original undivided input stream of the frame is applied to a central encoder that calculates the prediction of odd frames from even frames. In addition, the central encoder separately calculates prediction of even frames from odd frames. The remainder of the prediction is then calculated between the central encoder and the first and second side encoders, respectively. The MDMC encoder of the present invention outputs the first calculated prediction residual corresponding to the prediction of the even frame together with the output of the upper encoder, and the remainder of the second calculated prediction corresponding to the prediction of the odd frame together with the output of the lower encoder. Output

According to a second aspect of the invention, a method is provided for encoding a video signal representing a sequence of frames, the method comprising dividing a sequence of frames into a first sub-sequence and a second subsequence, the first sub Applying a sequence to the first side encoder, applying a second sub-sequence to the second side encoder, applying an original undivided sequence of frames to the central encoder, output of the first side encoder and Calculating a first prediction remainder between the center encoder, calculating a second prediction remainder between the output of the second side encoder and the central encoder, subtracting the output of the first prediction remainder and the first side encoder with the first data sub- Combining into a stream, combining the second prediction residual and the output of the second side encoder into a second data sub-stream, and opening the first and second data sub-streams. And a step of transmitting to the enemy.

Advantages of the present invention include the following:

(1) Any conventional predictive coder can be used for the upper and lower encoders. The upper and lower predictive coders may also advantageously include B-frames and multiple predicted motion compensation.

(2) Any upper, middle and lower prediction encoders are scalable encoders (e.g., FGS or data-partitioning, temporal scalability, where a motion vector (MV) is sent first). And the like. For example, if only the intermediate encoder is a scalable encoder, the intermediate encoder will only send as much information as the channel allows. In extreme cases where the available bandwidth is determined to be very low, only the information encoded by the side coder is transmitted. As the additional bandwidth becomes available, as many mismatched signals as the channel allows, are sent using the scalable intermediate encoder.

(3) To limit the complexity of the system, predictions from the odd / even frame sequences of the current even / odd frame to determine the mismatched signal can be made into B-frames.

(4) Compute and code the side prediction error (i.e., the error between even and odd frames with respect to the side coder), as well as the mismatch between the side prediction error and the center error (i.e., the current frame and the previous two Instead of calculating and coding the error between predictions from the frame), the central error is alternatively calculated.

The same reference numbers now refer to the drawings which indicate corresponding parts throughout the drawings.

Multiple representation coding (MDC) refers to a form of compression that aims to code an input signal into multiple discrete bit streams, which are often referred to as multiple representations. These individual bit streams all have decodable characteristics independently of each other. If the decoder receives in particular any single bit stream, it can decode the bit stream to produce a useful signal (without requiring access to any remaining bit stream). MDC has the additional characteristic that the quality of the decoded signal is improved when more bit streams are received correctly. For example, suppose that a video is coded into a total of N streams via MDC. As long as the decoder receives any of these N streams, the decoder can decode a useful version of the video. If the decoder receives two streams, it can decode an improved version of the video as compared to receiving only one stream. This improvement in quality continues until the receiver has received all N streams, in which case the receiver can reconstruct the maximum quality.

There are many different approaches to achieving MDC coding of video. One of them is to independently code different frames into different streams. For example, each frame of a video sequence may be coded as a single frame (independent of the remaining frames) using only intra frame coding, such as JPEG or JPEG-2000, or only I-frame encoding, for example. It can be coded as any video coding standard (eg MPEG-1 / 2/4, H.26-1 / 3) to use. Thereafter, different frames may be sent in different streams. For example, all even frame sequences can be sent to stream 1 and all odd frames can be sent to stream 2. Since each frame can be decoded independently from the remaining frames, each bit stream can also be decoded independently from the remaining bit streams. This simple form of MDC video coding has the characteristics described above, but is not very efficient in terms of compression due to the lack of inter-frame coding.

Before describing FIG. 1 in detail, recall the hierarchical arrangement of pixels in the digitized picture and some definitions regarding prediction strategies as used in the MPEG2 standard. Both luminance and chrominance samples (pixels) are each grouped into blocks consisting of an 8x8 matrix (eight rows of eight pixels each); A certain number of luminance and chrominance blocks (e.g., four blocks of luminance data and two corresponding blocks of chrominance data) form a macro block; The digitized picture then contains a matrix of macroblocks whose size depends on the selected profile (i.e. for resolution) and the power supply frequency, for example at a power supply of 50 kW, the size is at least 18 × 32 macros. It can range from blocks up to 72 × 120 macroblocks. The picture may alternately have a frame structure (pixels in subsequent rows belong to different fields) or field structures (all pixels belong to the same field). As a result, the macro block may also have a frame or field structure. The pictures are in turn organized into groups of pictures, with the first picture always being an I picture, followed by a number of B pictures (bidirectionally inserted pictures, forward or backward prediction, or both predictions, 'Forward' means that the prediction is based on the previous reference picture, and 'Rear' means that the prediction is based on the future reference picture), and then the P picture used for the prediction of the B picture It is encoded immediately after the I picture.

Referring now to FIG. 1, a not shown source is a sequence of frames 201 (ie, frame structure) already placed in coding order, i.e., making the reference picture available before the picture uses the reference picture for prediction. To the encoder 200. The full frame sequence 201 is received by a motion estimation unit (not shown) that calculates and emits a cost or error associated with each vector of one or more motion vectors for each macro-block in the picture to be coded. The encoder 200 includes a first side encoder {side encoder 1} 202, a central encoder 204 and a second side encoder 206. The entire frame sequence 201 is applied in its entirety to the central encoder 204. In the present embodiment, the first subset 210 of the full frame sequence 201 constituting a subset of the even frame sequences 210 of the full frame sequence 201 is applied to the first side encoder 202. In the present embodiment, the second subset 220 of the full frame sequence 201 constituting the odd frame sequence 220 of the full frame sequence 201 is applied to the second side encoder 206.

Now we summarize the predictive encoding operation.

A. First Side Encoder 202

An odd frame sub-sequence 210 that includes a subset of the input sequence 201 is applied to the first side encoder 202. It should be noted that the first side encoder 202 may be advantageously implemented as any conventional prediction codec (eg MPEG-1 / 2/4, H.26-1 / 3). The odd frame sub-sequence 210 is encoded by the first side encoder 202 which outputs the encoded odd frame sub-sequence 211. The encoded odd frame sub-sequence 211 is included as one component to be output in the first data sub-stream 245. The encoded odd frame sub-sequence 211 may also be supplied as an input to the central encoder sub-module 230, which will be described below.

B. Second Side Encoder 206

An even frame sub-sequence 220 comprising a sub-set of the input sequence 220 is applied to the second side encoder 206. The second side encoder 206 is advantageously like any conventional prediction codec (eg, MPEG-1 / 2/4, H.26-1 / 3), similar to the first side encoder 202. It should be noted that it may be implemented. The even frame sub-sequence 220 is encoded by a second side encoder 206 that outputs the encoded even frame subsequence 212. The encoded even frame subsequence 212 is included as one component to be output in the second data sub-stream 255. The encoded even frame subsequence 212 is also supplied as an input to the central encoder sub-module 232, which will be described below.

C. Central Encoder 204

The entire frame sequence 201 is applied to the central encoder 204.

The central encoder sub-module 250 calculates the first set 214 of motion vectors and also calculates and encodes the even frame prediction sequence 215, which predicts even frames from the odd frames of the input sequence 201. Configure The central encoder sub-module 250 outputs the even frame prediction sequence 215 and the first motion vector sequence 214, both of which are supplied as input to the central encoder sub-module 230.

The central encoder sub-module 260 calculates the second set 216 of motion vectors and also calculates and encodes the odd frame prediction sequence 217, which predicts odd frames from the even frames of the input sequence 201. Configure The central encoder submodule 250 outputs the odd frame prediction sequence 217 and the second motion vector sequence 216, both of which are supplied as input to the central encoder sub-module 232.

The central encoder sub-module 230 performs two functions or processes. The first process involves encoding the first set of motion vectors 214 received from sub-module 250 to output a first set of encoded motion vectors 218. The second function or process relates to calculating the first prediction remainder 221, which may be calculated as follows.

First prediction remainder = e _c -e _s

Where e _c = even frame prediction frame sequence 215,

e _s = encoded odd frame subsequence 211.

The central encoder sub-module 230 includes a first prediction remainder 221 encoded with the first set of coded motion vectors 218. These outputs are combined with the encoded odd frame subsequence 211 (point A) and collectively output as the first data sub-stream 245.

Similarly, the second prediction remainder may be calculated as follows, for inclusion in the second data sub-stream 255.

2nd prediction remainder = e _c -e _s

Where e _c = odd frame prediction frame sequence 217,

e _s = encoded even frame subsequence 212.

The central encoder sub-module 232 includes a second prediction remainder 222 encoded with the second set of coded motion vectors 219. These outputs are combined with the encoded even frame sequence 212 (point B) and output as a second data sub-stream 255.

The foregoing description of the preferred embodiment of the present invention has been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. Such modifications and variations apparent to those skilled in the art are intended to be included within the scope of the invention as defined by the appended claims.

The present invention is generally available for multiple representation coding (MDC) of data, voice, audio, image, video and other types of signals for transmission over a network or other type of communication medium.

Claims (15)

An encoding method for encoding an input frame sequence 201, a) encoding a first sub-sequence 210 of the frame from the input frame sequence 201 to produce an encoded first sub-sequence 211 of the frame; b) encoding a second sub-sequence 220 of the frame from the input frame sequence 201 to produce an encoded second sub-sequence 212 of the frame; c) calculating a first prediction frame sequence (215) from the second sub-sequence (220) of the frame; d) calculating a second prediction frame sequence (217) from the first sub-sequence (210) of the frame; e) calculating a first set (214) of motion vectors from the first prediction frame sequence (215); f) calculating a second set (216) of motion vectors from said second prediction frame sequence (217); g) calculating a first prediction remainder as an error difference between the first prediction frame sequence (215) and the encoded first sub-sequence (211) of the frame; h) calculating a second prediction remainder as an error difference between the second prediction frame sequence (217) and the encoded second sub-sequence (212) of a frame; i) encoding said first prediction remainder, said second prediction remainder, said first set of motion vectors (214), and said second set of motion vectors (216); j) determining network conditions; k) the encoded first prediction remainder 218, the encoded first set 221 of motion vectors and the encoded first sub-sequence 211 of the frame according to the determined network conditions; Scalablely combining as a first data sub-stream 245; l) the encoded second prediction remainder 219, the encoded second set 222 of motion vectors and the encoded second sub-sequence 212 of the frame according to the determined network conditions; Scalablely combining as a second data sub-stream (255); And m) independently transmitting the first and second data sub-streams (245, 255). 2. The method of claim 1 wherein the determined network condition is channel bandwidth determination. 2. The method of claim 1, comprising a preliminary step of placing the input frame sequence (201) in a predetermined coding order prior to step (a). 2. The method of claim 1, wherein the first sub-sequence (210) of the frame comprises only odd frames from the input frame sequence (201). 2. The method of claim 1, wherein the second sub-sequence (220) of the frame comprises only even frames from the input frame sequence (201). The method of claim 1, wherein the second sub-sequence (220) of the frame comprises only frames from the input frame sequence (201) that are not included in the first sub-sequence (210) of the frame. 2. The method of claim 1, wherein the first and second sub-sequences of frames (210, 220) are selected according to user preferences. The method of claim 1, wherein the input frame sequence comprises an intra-frame (I), a prediction frame (P), and a bidirectional frame (B). An encoder 200 that encodes an input sequence 201 of a frame, a) encoding a first sub-sequence 210 of a frame from the input frame sequence 201 at a first side encoder 202; b) encoding a second sub-sequence 220 of the frame from the input frame sequence 201 at a second side encoder 206; c) calculating a first predictive frame sequence (215) from the second sequence (220) of frames at the central encoder (204); d) calculating a second predicted frame sequence (217) from the first sub-sequence (210) of the frame at the central encoder (204); e) calculating a first set of motion vectors 214 from the first prediction frame sequence 215 at the central encoder 204; f) calculating a second set of motion vectors 216 from the second prediction frame sequence 217 at the central encoder 204; g) a first prediction remainder as an error difference between the first prediction frame sequence 215 at the central encoder 204 and the encoded first sub-sequence 211 of the frame at the central encoder 204. To calculate; h) calculating a second prediction remainder as an error difference between the second prediction frame sequence (217) and the encoded second sub-sequence (212) of a frame at the central encoder (204); i) encoding, at the central encoder (204), the first prediction remainder, the second prediction remainder, the first set of motion vectors (214), and the second set of motion vectors (216); j) determining network conditions; k) the encoded first prediction remainder 218, the encoded first set 221 of motion vectors and the encoded first sub-sequence 211 of the frame according to the determined network conditions; Scalable combining as a first data sub-stream 245; l) the encoded second prediction remainder 219, the second set 222 of motion vectors, and the encoded second sub-sequence 212 of the frame, according to the determined network condition; Scalable combining as sub-stream 255; And m) independently transmitting the first and second data sub-streams (245, 255) from the encoder (200). 10. The encoder (200) of claim 9, wherein the first side encoder (202), the second side encoder (206) and the central encoder (204) are conventional predictive encoders. The encoder (200) of claim 10, wherein the first side encoder (202), the second side encoder (206) and the central encoder (204) are scalable encoders. 11. The method of claim 10, wherein the conventional predictive encoders are MPEG1, MPEG2, MPEG4, MPEG7, H.261, H.262, H.263, H.263 +, H.263 ++, H.26L, and H. Encoder 200, which is an encoder selected from the group of encoders including a 26L encoder. 10. The encoder (200) of claim 9, wherein the encoder (200) is included in a telecommunications transmitter of a wireless network. A system for encoding an input sequence 201 of a frame, Means for encoding a first sub-sequence (210) of a frame from the input frame sequence (201) to produce an encoded first sub-sequence (211) of a frame; Means for encoding a second sub-sequence (220) of the frame from the input frame sequence (201) to produce an encoded second sub-sequence (212) of the frame; Means for calculating a first predictive frame sequence (215) from the second sequence (220) of frames; Means for calculating a second prediction frame sequence (217) from the first sub-sequence (210) of the frame; Means for calculating a first set of motion vectors (214) from the first prediction frame sequence (215); Means for calculating a second set of motion vectors (216) from the second prediction frame sequence (217); Means for calculating a first prediction remainder as an error difference between the first prediction frame sequence (215) and the encoded first sub-sequence (211) of a frame; Means for calculating a second prediction remainder as an error difference between the second prediction frame sequence (217) and the encoded second sub-sequence (212) of a frame; Means for encoding the first prediction remainder, the second remainder, the first set of motion vectors (214), and the second set of motion vectors (216); Means for determining network conditions; The encoded first prediction remainder 218, the encoded first set 221 of motion vectors and the encoded first sub-sequence 211 of a frame are firstly determined according to the determined network condition. Means for scalable coupling as data sub-stream 245; The encoded second prediction remainder 219, the encoded second set 222 of motion vectors and the encoded second sub-sequence 212 of a frame, according to the determined network condition; Means for scalable coupling as data sub-stream 255; And Means for independently transmitting said first and second data sub-streams (245, 255). 15. The system of claim 14, further comprising means for placing the input frame sequence (201) in a predetermined coding order.