JP2005535219A - Method and apparatus for performing multiple description motion compensation using hybrid prediction code - Google Patents

Abstract

An improved multiple description coding (MDC) method and apparatus is provided that extends multi-description motion compensation (MDMC) by enabling multi-frame prediction and is not limited to I-frames and P-frames. In addition, the coding method of the present invention extends MDMC for use with conventional predictive codecs such as MPEG2 / 4 and H.26L. The improved MDC allows the use of conventional predictive encoders used as upper and lower predictive encoders. Further, the upper and lower prediction encoders may advantageously include B frame and multiple prediction motion compensation. Still further, any of the upper, middle and lower prediction encoders are scalable encoders (eg, data partitioning type or FGS type, temporal scalability type, etc. when motion vectors (MV) are transmitted first). be able to.

Description

  The present invention generally relates to multiple description coding (MDC) of data, voice, audio, images, video and other types of signals for transmission over a network or other type of communication medium. About.

  Most of the information that flows through today's networks is useful even in a degraded state. Examples include voice, audio, still images and video. If this information is subject to packet loss, retransmission may not be possible due to real-time constraints. Excellent performance for all transmission rates, distortions and delays can be achieved by adding redundancy to the bitstream rather than occasionally repeating lost packets.

  The surplus part can be added to the bitstream in one way by multiple description coding (MDC) so that the data is decomposed into multiple streams with some extra part between the streams To be. If all streams are received, low distortion can be ensured by bearing a slightly higher bit rate than systems designed purely for compression. On the other hand, if only some of the streams are received, the quality of the reconstruction is gracefully degraded, which is very unlikely with a system designed purely for compression. Unlike multi-resolution or hierarchical source coding, there is no hierarchy of descriptions, so multiple description coding is suitable for erasure channels or packet networks that do not give priority. ing.

  Multiple description coding can be implemented in multiple ways. One way is to separately collect the sequence of odd and even frames in the encoder and independently encode the resulting temporally subsampled sequence into an arbitrary channel subset. By dividing. Upon receipt of one of the subsampled sequences at a decoder, the video stream can be decoded at half the frame rate. Due to the correlated nature of the video stream, receiving only one of the sub-sampled sequences allows for recovery of intermediate frames using motion compensated error concealment techniques. This technique is described in detail in the document Wenger et al., “Error resilience support in H.263 +,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 867-877, November 1998.

  To achieve error resilience, the literature Wang and Lin, “Error resilient video coding using multiple description motion compensation,” IEEE Trans. Circuits and Systems for Video Technology, vol.12, no.6, pp.4348 -52, June 2002 describes one way to implement multiple description coding. According to this approach, temporal predictors allow both past even and odd frames to be used while the encoder is encoding, so only one description is the decoder. Causes a mismatch between the encoder and the decoder. Non-conformance errors are explicitly encoded to overcome this problem. The main advantage that allows the encoder to use both odd and even frame sequences for prediction relates to coding efficiency. By changing the temporal filter tap, the amount of excess can be controlled. The disclosed method provides reasonable flexibility between the amount of surplus and error recovery.

  The disadvantage of Wang and Lin's approach is that it is limited to I and P frames only (no B frames). Yet another drawback of this approach is that it does not allow multiframe prediction as employed in H.26L. These disadvantages limit the coding efficiency of MDMC and require a fully dedicated implementation instead of using available codec modules.

  The present invention provides an improved multiple description coding (MDC) method and apparatus that overcomes the aforementioned drawbacks. In particular, the coding method of the present invention extends multi-description motion compensation (MDMC) by enabling multi-frame prediction and is not limited to only I frames and P frames. In addition, the coding method of the present invention extends MDMC for use with conventional predictive codecs such as MPEG2 / 4 and H.26L.

  According to a first aspect of the present invention, an improved MDMC encoder is provided that includes three predictive encoders: an upper, middle and lower encoder. The input frame is supplied to the encoder as three different inputs. The input frame is supplied to a central encoder. In addition, the input frame is separated or divided into two substreams of the frame: a first substream having only odd frames and a second substream having only even frames. The first substream consisting of odd frames is supplied as input to be encoded by an upper encoder, resulting in an encoded odd frame sequence, and the second substream consisting of even frames is a lower code Resulting in an encoded even frame sequence supplied as input to be encoded by the encoder. Other embodiments use different criteria, such as an unbalanced division where every third frame is encoded by the upper encoder and every third frame is encoded by the lower encoder. Note that it may be used to split the frame. The original undivided input stream of frames is applied to the central encoder that computes the prediction of the odd frames from the even frames. In addition, the central encoder calculates the prediction of the even frame from the odd frame separately. A prediction residual is calculated between the central encoder and the first and second side encoders, respectively. The MDMC encoder of the present invention outputs the first calculation prediction error corresponding to the prediction of the even frame together with the output of the upper encoder, and supports the prediction of the odd frame together with the output of the lower encoder. The second calculation prediction error is output.

  According to a second aspect of the invention, there is provided a method of encoding a video signal representing a sequence of frames, the method comprising: dividing the sequence of frames into a first subsequence and a second subsequence; Adding the first sub-sequence to a one-side encoder; adding the second sub-sequence to a second side encoder; and adding the original unsegmented sequence of frames to a central encoder Calculating a first prediction error between the output of the first side encoder and the output of the central encoder; the output of the second side encoder and the central encoder; Calculating a second prediction error with the output, and combining the first prediction error and the output of the first side encoder as a first data substream. Combining the second prediction error and the output of the second encoder as a second data substream, and transmitting the first data substream and the second data substream separately. Have.

  The advantages of the present invention include:

  (1) Conventional predictive encoders can be used for the upper and lower encoders. Furthermore, the upper and lower prediction encoders may advantageously include B-frame and multiple prediction motion compensation.

  (2) The upper, middle and lower predictive encoders are scalable encoders (eg, data partitioning type, FGS type, temporal scalability type, etc. when motion vectors (MV) are transmitted first). Can do. For example, when only the intermediate encoder is a scalable encoder, the intermediate encoder transmits only the amount of information permitted by the communication path. In the extreme case where the available bandwidth is determined to be very narrow, only the information encoded by the side encoder will be transmitted. As additional bandwidth becomes available, the amount of non-conforming signal allowed by the channel will be transmitted using the scalable intermediate encoder.

  (3) In order to limit the complexity of the system, prediction of the current even / odd frame from the odd / even frame sequence to determine the non-conforming signal can be made from the B frame.

  (4) Conventionally, side prediction errors (ie, errors between the even and odd frames for the side encoder) and side prediction errors and center errors (ie, current frame and previous two) Instead of calculating and encoding the incompatibility between the prediction error from the frame), alternatively, the central error is calculated.

  Reference is now made to the drawings in which like reference numerals represent corresponding parts.

  Multiple description coding (MDC) refers to a form of compression and the goal is to encode an input signal into a number of different bitstreams, which are often referred to as multiple descriptions. These different bitstreams all have the property that they can be decoded independently of each other. In particular, if the decoder receives a single bitstream, the decoder can decode the bitstream to generate a useful signal (without requiring access to other bitstreams). MDC has the additional property that the more bitstreams are received correctly, the better the quality of the decoded signal. For example, assume that video is encoded into a total of N streams using MDC. As long as the decoder receives any one of these N streams, the decoder can decode a useful version of the video. If the decoder receives two streams, the decoder can decode an improved version of the video compared to receiving only one of the streams. This improvement in quality continues until the receiver has received all N of the streams, in which case the decoder can reconstruct the maximum quality.

  There are a number of different approaches to achieving video MDC coding. One approach is to encode different frames independently into different streams. For example, each frame of a video sequence is a video coding standard (eg MPEG-1 / 2/4, H.26-1 / 3) that uses only intra-frame coding, eg JPEG, JPEG-2000 or I-frame coding only. Can be encoded as a single frame (independent of other frames). In this case, different frames can be transmitted in different streams. For example, all the even frame sequences can be transmitted in stream 1 and all the odd frames can be transmitted in stream 2. Since each of the frames can be decoded independently of the other frames, each of the bitstreams can also be decoded independently of the other bitstreams. This simple form of MDC video coding has the properties described above, but is not very efficient with respect to compression due to the lack of interframe coding.

  Before describing FIG. 1 in detail, some definitions will be recalled regarding the prediction strategy as used in the MPEG2 standard and the hierarchical arrangement of pixels in a digitized picture. Both luminance samples and chrominance samples (pixels) are grouped into blocks each consisting of an 8 × 8 matrix (8 rows of 8 pixels each), and a specific number of luminance blocks and chrominance blocks (eg, four blocks of luminance data). And two corresponding blocks of chrominance data) form a macroblock, and the digitized picture in this case has a matrix of macroblocks, the size of the macroblock matrix being the selected profile (ie resolution) Depending on the power frequency, for example, for a 50 Hz power source, the size can range from a minimum of 18 × 32 macroblocks to a maximum of 72 × 120. A picture can have a frame structure (later rows of pixels are associated with different fields) or a field structure (all pixels are associated with the same field). As a result, the macroblock can have a frame structure or a field structure as well. The pictures are organized into groups of pictures, the first picture is always an I picture, followed by a plurality of B pictures (bi-interpolated pictures, forward or backward prediction or both Where the 'forward' means that the prediction is based on the previous reference picture, and 'backward' means that the prediction is based on the future reference picture) followed by the B picture Followed by the P picture to be encoded immediately after the I picture.

  Referring to FIG. 1, a source not shown is a sequence of frames 201 (ie a frame structure) already arranged in coding order, ie in order to make the reference pictures available before pictures used for prediction. ) To the encoder 200. A complete frame sequence 201 is a motion estimation unit (not shown) that calculates and emits one or more motion vectors and costs or errors associated with each vector for each macroblock in the picture to be encoded. ). The encoder 200 includes a first side encoder (side encoder 1) 202, a central encoder 204, and a second side encoder 206. The complete frame sequence 201 is added to the central encoder 204 as it is. The first subset 210 of the complete frame sequence 201 that constitutes the even frame sequence subset 210 of the complete frame sequence 201 in this embodiment is applied to the first side encoder 202. The second subset 220 of the complete frame sequence 201 that constitutes the odd frame sequence 220 of the complete frame sequence 201 in this embodiment is applied to the second side encoder 206.

  Predictive coding operations are summarized here.

A. First side encoder 202
An odd frame subsequence 210 having a subset of the input sequence 201 is applied to the first side encoder 202. It should be noted that the first side encoder 202 can be advantageously implemented as a conventional predictive codec (eg, MPEG-1 / 2/4, H.26-1 / 3). The odd frame subsequence 210 is encoded by the first side encoder 202 that outputs the encoded odd frame subsequence 211. The encoded odd frame subsequence 211 is included in the first data substream 245 as one element to be output. The encoded odd frame subsequence 211 is also supplied as an input to the central encoder submodule 230 described later.

B. Second side encoder 206
An even frame subsequence 220 having a subset 220 of the input sequence is applied to the second side encoder 206. Similarly to the first side encoder 202, the second side encoder 206 may be advantageously implemented as a conventional prediction codec (eg, MPEG-1 / 2/4, H.26-1 / 3). it can. The even frame subsequence 220 is encoded by the second side encoder 206 that outputs the encoded even frame subsequence 212. The encoded even frame subsequence 212 is included in the second data substream 255 as one element to be output. The encoded even frame sub-sequence 212 is also provided as input to a central encoder sub-module 232 which will be described later.

C. Central encoder 204
The complete frame sequence 201 is added to the central encoder 204.

  The central encoder sub-module 250 calculates the first set 214 of motion vectors and also calculates and encodes the even frame prediction sequence 215 that constitutes the prediction of the even frames from the odd frames of the input sequence 201. The central encoder submodule 250 outputs the even frame prediction sequence 215 and the first motion vector sequence 214, both of which are provided as inputs to the central encoder submodule 230.

  The central encoder sub-module 260 calculates the second set of motion vectors 216 and also calculates and encodes the odd frame prediction sequence 217 that constitutes the prediction of the odd frames from the even frames of the input sequence 201. The central encoder submodule 250 outputs the odd frame prediction sequence 217 and the second motion vector sequence 216, both of which are provided as inputs to the central encoder submodule 230.

The central encoder submodule 230 performs two functions or processes. The first process is intended to encode the first set 214 of motion vectors received from the submodule 250 and output a first set 218 of encoded motion vectors. The second function or process aims at calculating the first prediction error 221 and can be calculated as follows.
First prediction error = e _c −e _s (1)
Where e _c = even frame predicted frame sequence 215;
e _s = encoded odd frame subsequence 211

  The output of the central encoder sub-module 230 includes a first prediction error 221 encoded with a first set 218 of encoded motion vectors. These outputs are combined with the encoded odd frame sequence 211 (point A) and output together as a first data substream 245.

Similarly, the second prediction error is calculated for inclusion in the second data substream 255 as follows.
Second prediction error = e _c −e _s (2)
Where e _c = odd frame prediction frame sequence 217;
e _s = encoded even frame subsequence 212.

  The output of the central encoder sub-module 232 includes a second prediction error 222 that is encoded with a second set 219 of encoded motion vectors. These outputs are combined with the encoded even frame sequence 212 (point B) and output as a second data substream 255.

  The foregoing description of the preferred embodiment of the present invention has been presented for purposes of illustration and description. These examples are not intended to be exhaustive or to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. It is. Such modifications and variations that may be apparent to a person skilled in the art are intended to be included within the scope of this invention as defined by the accompanying claims.

2 illustrates an MDMC encoder according to one embodiment of the present invention.

Claims (15)

In an encoding method for encoding an input frame sequence,
a) encoding a first frame subsequence from the input frame sequence to generate an encoded first frame subsequence;
b) encoding a second frame subsequence from the input frame sequence to generate an encoded second frame subsequence;
c) calculating a first predicted frame sequence from the second frame subsequence;
d) calculating a second predicted frame sequence from the first frame subsequence;
e) calculating a first motion vector set from the first predicted frame sequence;
f) calculating a second motion vector set from the second predicted frame sequence;
g) calculating a first prediction error as an error difference between the first prediction frame sequence and the encoded first frame subsequence;
h) calculating a second prediction error as an error difference between the second prediction frame sequence and the encoded second frame subsequence;
i) encoding the first prediction error, the second prediction error, the first motion vector set, and the second motion vector set;
j) determining the network status;
k) Depending on the determined network situation, the encoded first prediction error, the encoded first motion vector set, and the encoded first frame subsequence as a first data substream. A step for scalable coupling;
l) Depending on the determined network situation, the encoded second prediction error, the encoded second motion vector set, and the encoded second frame subsequence as a second data substream. A step for scalable coupling;
m) independently transmitting the first data substream and the second data substream;
Having a method.   The method of claim 1, wherein the determined network condition is a determination of a bandwidth of a communication path.   The method of claim 1, comprising a prior step of placing the input frame sequence in a predetermined coding order prior to the step (a).   The method of claim 1, wherein the first frame subsequence has only odd frames from the input frame sequence.   The method of claim 1, wherein the second frame subsequence comprises only even frames from the input frame sequence.   The method of claim 1, wherein the second frame subsequence includes frames from the input frame sequence that are not included in the first frame subsequence.   The method of claim 1, wherein the first frame subsequence and the second frame subsequence are selected according to user preferences.   The method of claim 1, wherein the input frame sequence includes an intra frame, a prediction frame, and a bidirectional frame. In an encoder for encoding an input frame sequence,
a) means for encoding a first frame subsequence from the input frame sequence in a first side encoder;
b) means for encoding a second frame sub-sequence from the input frame sequence in a second side encoder;
c) means for calculating a first predicted frame sequence from the second frame subsequence in the central encoder;
d) means for calculating a second predicted frame sequence from the first frame subsequence in the central encoder;
e) means for calculating a first motion vector set from the first predicted frame sequence in the central encoder;
f) means for calculating a second motion vector set from the second predicted frame sequence in the central encoder;
g) means for calculating a first prediction error as an error difference between the first prediction frame sequence and the encoded first frame subsequence in the central encoder;
h) means for calculating a second prediction error as an error difference between the second prediction frame sequence and the encoded second frame subsequence in the central encoder;
i) means for encoding the first prediction error, the second prediction error, the first motion vector set, and the second motion vector set in the central encoder;
j) means for determining the network status;
k) Depending on the determined network situation, the encoded first prediction error, the encoded first motion vector set, and the encoded first frame subsequence as a first data substream. A means of scalable coupling;
l) Depending on the determined network situation, the encoded second prediction error, the encoded second motion vector set, and the encoded second frame subsequence as a second data substream. A means of scalable coupling;
m) means for independently transmitting the first data substream and the second data substream from the encoder;
An encoder.   The encoder of claim 9, wherein the first side encoder, the second side encoder, and the central encoder are conventional predictive encoders.   The encoder according to claim 10, wherein the first side encoder, the second side encoder, and the central encoder are scalable encoders.   The conventional predictive encoder includes an MPEG1 encoder, an MPEG2 encoder, an MPEG4 encoder, an MPEG7 encoder, an H.261 encoder, an H.262 encoder, and an H.262 encoder. 11. An encoder selected from the group of encoders including a .263 encoder, an H.263 + encoder, an H.263 ++ encoder, and an H.26L encoder. The encoder described in 1.   The encoder of claim 9, wherein the encoder is included in a telecommunication transmitter of a wireless network. In a system for encoding an input frame sequence,
Means for encoding a first frame subsequence from the input frame sequence and generating an encoded first frame subsequence;
Means for encoding a second frame subsequence from the input frame sequence and generating an encoded second frame subsequence;
Means for calculating a first predicted frame sequence from the second frame subsequence;
Means for calculating a second predicted frame sequence from the first frame subsequence;
Means for calculating a first motion vector set from the first predicted frame sequence;
Means for calculating a second motion vector set from the second predicted frame sequence;
Means for calculating a first prediction error as an error difference between the first prediction frame sequence and the encoded first frame subsequence;
Means for calculating a second prediction error as an error difference between the second prediction frame sequence and the encoded second frame subsequence;
Means for encoding the first prediction error, the second prediction error, the first motion vector set, and the second motion vector set;
A means of determining network status;
Depending on the determined network situation, the encoded first prediction error, the encoded first motion vector set, and the encoded first frame subsequence are scalable as a first data substream. A means of combining;
Depending on the determined network situation, the encoded second prediction error, the encoded second motion vector set, and the encoded second frame subsequence are scalable as a second data substream. A means of combining;
Means for independently transmitting the first data substream and the second data substream;
Having a system.   The system of claim 14, further comprising means for arranging the input frame sequence in a predetermined coding order.

Images