KR100626419B1 - Switching between bit-streams in video transmission - Google Patents

Abstract

The present invention relates to a method for transmitting video information, wherein at least a first bit-stream (510) and a second bit-stream are formed. The first bit-stream 510 includes at least one video frame, and the second bit-stream 520 includes at least one predictive video frame 524. At least partially different encoding parameters are used for encoding the frames of the first bit-stream 510 and the second bit-stream 520. At least one frame of the first bit-stream 510 is transmitted, and the transmission is exchanged from the first bit-stream 510 to the second bit-stream 520. When exchanging the transmission from the first bit-stream 510 to the second bit-stream 520, a second exchange frame 550 is transmitted, and the second exchange frame 550 is the first exchange. At least one reference frame from bit-stream 510 and the encoding parameters of second bit-stream 520 are encoded. The second exchange frame 550 is used as a reference frame in the reconstruction of the at least one predictive video frame 524 of a second set of video frames. The invention also relates to an encoder for encoding video information, a decoder for decoding video information, and a signal representing encoded video information.

Description

Switching between bit-streams in video transmission}

The present invention relates to a method for transmitting video information, wherein at least a first bit-stream and a second bit-stream are formed from the video information, the first bit-stream comprising at least one video frame. One set of frames, wherein the second bit-stream comprises a second set of frames comprising at least one predictive video frame, wherein at least partially different encoding parameters comprise the first bit-stream and the first Used to encode frames of two bit-streams, at least one frame of the first bit-stream is transmitted, and the transmission is exchanged from the first bit-stream to the second bit-stream. It is about.

The invention also relates to an encoder, comprising: means for forming at least a first bit-stream and a second bit-stream from video information, the first bit-stream comprising at least one video frame Means for including a second set of frames comprising at least one predictive video frame, at least partially for encoding the frames of the first bit-stream and the second bit-stream. Means for using different encoding parameters, means for transmitting at least one frame of the first bit-stream, and means for exchanging the transmission from the first bit-stream to the second bit-stream. It is about an encoder.

The invention further relates to a decoder for decoding video information from a signal, the signal comprising frames from at least a first bit-stream and a second bit-stream formed from the video information. The stream comprises a first set of frames comprising at least one video frame, and the second bit-stream comprises a second set of frames comprising at least one predictive video frame, the at least partially different encoding medium Variables relate to a decoder used for encoding the frames of the first bit-stream and the second bit-stream.

The invention further relates to a signal representing encoded video information and comprising a frame from at least a first bit-stream and a second bit-stream formed from the video information, wherein the first bit-stream is at least one. A first set of frames comprising a video frame of the second bit-stream, wherein the second bit-stream comprises a second set of frames comprising at least one predictive video frame, wherein at least partially different encoding parameters And a signal used for encoding the frames of the first bit-stream and the second bit-stream.

Recently, multimedia applications, including streaming audio and video information, are becoming more popular. Several international standardization bodies have established and proposed standards for compressing / encoding, decompressing / decoding audio and video information. The MPEG standards set by the Video Experts Group are the most widely accepted international standards for multimedia applications. VCEG is the "Video Coding Experts Group" working under the direction of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). This group works on the standard H.26L for encoding video.

A typical video stream contains a series of pictures, often referred to as frames. The frames comprise pixels arranged in a rectangle. In the current video coding standards such as H.261, H.262, H.263, H26L and MPEG-4, three major types of pictures are defined: Intra frames, prediction Predictive frames (P-frames) and Bi-directional frames (B-frames). Each picture type uses different types of redundancy in a series of images and thus different levels of compression efficiency, providing different functionality within the encoded video sequence, as described below. An intra frame is a frame of video data that is encoded using only the spatial correlation of the pixels in its frame without using any information from past or future frames. Intra frames are used as the basis for decoding / decompressing other frames and provide access points to the encoded sequence from which decoding can begin.

A predictive frame is a so-called reference frame, ie a frame that is encoded / compressed using motion compensated prediction from one or more pre / post intra frames or predictive frames available in an encoder or decoder. A bidirectional frame is a frame that is encoded / compressed by prediction from a previous intra frame or prediction frame and / or a subsequent intra frame or prediction frame.

Since adjacent frames in a typical video sequence are highly correlated, higher compression can be achieved when using bidirectional or predictive frames instead of intra frames. On the other hand, when temporal prediction encoding is used within an encoded video stream, B-frames and / or P-frames are not correctly decoded for all other previous and / or subsequent reference frames used for encoding bi-directional and predictive frames. Cannot be decrypted. The reference frame (s) used in the encoder and each reference frame (s) in the decoder are not the same because of errors during transmission or due to some deliberate operation on the sender, resulting in prediction from such a reference frame. After use, the frames cannot be reconstructed at the decoding side to produce the same decoded frame as originally encoded at the encoding side. This mismatch is not limited to a single frame but is further time propagated due to the use of motion compensated encoding.

1A-1C show types of encoded / compressed video frames used in a typical video encoding / decoding system. Preferably, prior to encoding, the pictures of the video sequence are represented by many-bit numbers of matrices, one representing the luminance (brightness) of the image pixels, and the other two each have two chrominances (colors). One of the components is shown. 1A shows how the intra frame 200 is encoded using only image information in its frame. 1B shows the configuration of the prediction frame 210. Arrow 205a illustrates the use of motion compensated prediction to generate P-frame 210. 1C shows the configuration of the bidirectional frames 220. B-frames are usually inserted between I-frames or P-frames. 2 shows a group of pictures in display order and shows how B-frames are inserted between I-frames and P-frames, as well as the direction in which motion compensation information flows. 1B, 1C, and 2, arrows 205a represent forward motion compensation prediction information needed to reconstruct P-frames 210. On the other hand, arrows 215a and 215b represent motion compensation information used to reconstruct B-frames 220 in the forward 215a and reverse 215b directions. In other words, the arrows 205a and 215a indicate the flow of information when the prediction frames are predicted from earlier frames in the display order than the frame from which they are reconstructed, and the arrows 215b indicate the display order than the frame from which the prediction frames are reconstructed. Represents the flow of information when predicted from later frames.

In motion compensated prediction, similarity between successive frames in a video sequence is used to improve coding efficiency. More specifically, so-called motion vectors are used to describe how pixels or regions of pixels move between successive frames of a sequence. Motion vectors provide error data and offset values related to past or future frames of video data having decoded pixel values that can be used with the error data to compress / encode or decompress / decode a given frame of video data. do.

The ability to decode / decompress P-frames requires the availability of a previous I- or P-reference frame, and furthermore, also requires the availability of a later I- or P-reference frame to decode B-frames. do. For example, if the encoded / compressed data stream has the following frame sequence or display order:

I ₁ B ₂ B ₃ P ₄ B ₅ P ₆ B ₇ P ₈ B ₉ B ₁₀ P ₁₁ ... P _n-3 B _n-2 P _n-1 I _n ,

The corresponding decoding order is:

I ₁ P ₄ B ₂ B ₃ P ₆ B ₅ P ₈ B ₇ P ₁₁ B ₉ B ₁₀ ... P _n-1 B _n-2 I _n .

The decoding order is different from the display order because B-frames require future I- or P-frames for decoding B-frames. 2 can be referenced to display the beginning of the frame sequence and to understand the dependencies of the frames as described above. P-frames require the previous I- or P-reference frame available. For example, P ₄ needs I ₁ to be decoded. Similarly, frame P ₆ requires that P ₄ be available to decode / decompress frame P ₆ . B-frames, such as frame B ₃ , require past and / or future I- or P- reference frames, such as P ₄ and I ₁ , to be decoded. B-frames are frames that are between I- or P-frames during encoding.

Prior art systems for encoding and decoding are shown in FIGS. 3 and 4. Referring to the encoder 300 of FIG. 3, the frame 301 I (x, y) to be encoded, referred to as the current frame, is divided into rectangular regions of KxL pixels. Coordinates (x, y) indicate the position of the pixels in the frame. Each block is encoded using intra coding (i.e., using only spatial correlation of image data in the block) or inter coding (i.e., using both spatial and temporal prediction). The following description considers the process by which inter-coded blocks are formed. Each inter-coded block is predicted 360 from one of the frames R (x, y) previously coded and transmitted in frame memory 350, referred to as a reference frame. The motion information used for prediction is obtained from the motion prediction and coding block 370 using the reference frame and the current frame 305. The motion information is represented by two-dimensional motion vectors Δx, Δy, where Δx is a horizontal displacement and Δy is a vertical displacement. In a motion compensated (MC) prediction block, motion vectors are used together with a reference frame to construct a prediction frame P (x, y).

P (x, y) = R (x + Δx, y + Δy)

Subsequently, the prediction error E (x, y), that is, the difference between the current frame and the prediction frame P (x, y) is calculated according to the following equation (307).

E (x, y) = I (x, y)-P (x, y)

In transform block 310, the prediction error for each KxL block is expressed as the weighted sum of the transform basis functions f _ij (x, y).

The weights c _err (i, j) corresponding to the basis functions are referred to as transform coefficients. These coefficients are then quantized in quantization block 320 to provide the following equation.

I _err (i, j) = Q (c _err (i, j), QP)

Where I _err (i, j) are quantized transform coefficients. Quantization operation Q () introduces a loss of information, but quantized coefficients can be represented with a smaller number of bits. The level of compression (loss of information) is controlled by adjusting the value of the quantization parameter QP.

Before the motion vectors and the quantized transform coefficients are supplied to the multiplexer 380, they are further encoded using Variable Length Codes (VLC). This reduces the number of bits needed to represent motion vectors and quantized transform coefficients. Coded motion vectors, quantized transform coefficients, as well as other side information necessary to represent each coded frame are multiplexed in multiplexer 380 and the resulting bit-stream is sent to the decoder (415). The quantized transform coefficients are also passed to inverse quantization block 330 to obtain inverse quantized transform coefficients, and further to inverse transform block 340 to obtain prediction error information E _c (x, y) for the current frame. do. The prediction error information E _c (x, y) is added to the prediction frame P (x, y) in the addition element to obtain a video frame that can be stored in the frame memory 350.

Next, decoding the video frames will be described with reference to FIG. 4. Decoder 400 receives the multiplexed video bit-stream 415 from an encoder, and demultiplexer 410 demultiplexes the bit-stream to obtain components of video frames that are decoded. These components comprise at least coded quantized prediction error transform coefficients and coded motion vectors, which are then decoded (not shown) to obtain quantized prediction error transform coefficients and motion vectors. The quantized transform coefficients are dequantized in inverse quantization block 420 and obtain dequantized transform coefficients d _err (i, j) according to the following relationship.

d _err (i, j) = Q ^-1 (I _err (i, j), QP)

In inverse transform block 430, the inverse quantized transform coefficients are inverse transformed to obtain prediction error E _c (x, y).

The pixels of the currently encoded frame find the prediction pixels in reference frame R (x, y) obtained from frame memory 440 and use the received motion vectors together with the reference frame in motion compensation prediction block 450. By reconstructing to obtain the prediction frame P (x, y). The prediction frame P (x, y) and the prediction error information E _c (x, y) are added by the addition element 435 according to the following relationship.

I _c (x, y) = R (x + Δx, y + Δy) + E _c (x, y)

These values I _c (x, y) may be filtered to obtain further decoded video frames 445. The values I _c (x, y) are also stored in the frame buffer 440. The reconstructed values I _c (x, y) may be filtered in the filtering block (not shown in FIG. 4) following the addition block 435.

Video streaming has emerged as an important application on the fixed Internet. Video streaming is also expected to become important in 3G wireless networks in the near future. In streaming applications, the sending server starts sending a pre-encoded video bit stream to the receiver via the transmission network upon request from the receiver. The receiver plays the video stream while receiving the video stream. The best effort of current networks causes variations in the effective bandwidth available to the user due to changes in network conditions. To accommodate these variations, the sending server can scale the bit rate of the compressed video. In the case of an interactive service featuring real-time encoding and point-to-point delivery, this can be achieved by adjusting the source encoding parameters in operation. Such adjustable parameters can be, for example, quantization parameters or frame rates. The adjustment is preferably based on feedback from the transport network. In typical streaming scenarios where a previously encoded video bit stream is transmitted to a receiver, the solution is not applicable.

One solution to achieving bandwidth scalability in the case of pre-coded sequences is to provide multiple independent streams with different bit rates and quality. The transport server then dynamically exchanges between the streams to accommodate variations in the available bandwidth. The following example illustrates this principle. Assume that multiple bit streams are generated independently with different coding parameters, such as quantization parameters, corresponding to the same video sequence. {P _{1, n-1} , P _{1, n} , P _{1, n + 1} } and {P _{2, n-1} , P _{2, n} , P _{2, n + 1} } are derived from bit streams 1 and 2, respectively. Assume that it represents a sequence of decoded frames. Since the encoding parameters are different for 2 bit streams, the frames reconstructed from the bit streams at the same time, for example frames P _{1, n-1} and P _{2, n-1,} are not identical. Now, assuming that the server initially transmits the encoded frames from bit stream 1 until time n and after the time n transmits the encoded frames from bit stream 2, the decoder determines that the frames {P _{1, n-2} , P _{1, n-1} , P _{2, n} , P _{2, n + 1} , P _{2, n + 2} }. In this case, P _{2, n} cannot be decoded correctly because its reference frame P _{2, n-1} has not been received. On the other hand, instead of P _{2, n-1} , the received frame P _{1, n-1} is not equal to P _{2, n-1} . The exchange between bit streams at arbitrary locations thus leads to visual artifacts due to mismatches between the reference frames used for motion compensated prediction of different sequences. These visual artifacts are not limited to frames at the point of exchange between bit streams, but propagate in time due to the motion compensated encoding that continues in the rest of the video sequence.

In current video coding standards, a complete (unmatched) exchange between bit streams is achieved at locations where current and future frames or regions of frames do not use any information before the current exchange location, ie only in I-frames. It is possible. Moreover, by placing I-frames at fixed (eg 1 second) intervals, VCR functionality such as random access or "fast forward" or "fast reverse" (which increases playback speed) is achieved for streaming video content. do. The user can jump over part of the video sequence or resume playback at any I-frame location. Similarly, increased playback speed can be achieved by sending only I-frames. The disadvantage of using I-frames in these applications is that I-frames require much more bits than P-frames at the same quality because they do not use any temporal redundancy.

It is an object of the present invention to provide a novel method and system for transmitting video images in a variable transmission environment. The present invention provides that a new type of compressed (non-mismatched) exchange between video streams forms a new type of compressed video frame and is allowed to exchange from one bit-stream to another bit-stream. It is based on the idea that it may be possible by inserting frames into video bit-streams. In this specification, the new type of compressed video frame will be generally referred to as S-frame. More specifically, S-frames can be classified into SP-frames and SI-frames. The SP-frames are formed in a decoder using motion compensated prediction from frames already decoded using motion vector information. The SI-frames are formed in a decoder using spatial (intra) prediction from already decoded neighboring pixels in the frame to be decoded. In general, an S-frame according to the present invention may be formed in a block-by-block manner and include inter-coded (SI) blocks as well as inter-coded (SP) blocks. have.

The method according to the invention mainly when exchanging transmissions from the first bit-stream to the second bit-stream, the second bit-stream comprises at least one first exchange frame, the second exchange frame being transmitted The second exchange frame is encoded using at least one reference frame from the first bit-stream and the encoding parameters of the second bit-stream, wherein the second exchange frame is a second set of video frames. Characterized in that it is used in place of the first exchange frame as a reference frame used for reconstruction of at least one prediction video frame.

The encoder according to the present invention mainly means for exchanging transmissions from the first bit-stream to the second bit-stream is adapted to enable the exchange of the transmissions from the first bit-stream to the second bit-stream. Means for encoding a second exchange frame using reference frames from one bit-stream and the encoding parameters of the second bit-stream.

The decoder according to the present invention mainly comprises a means for decoding the second exchange frame, wherein the second exchange frame comprises at least one reference frame from a first bit-stream and an encoding medium of the second bit-stream. Encoded using the variables, wherein the second exchanged frame is a reference frame used for reconstruction of at least one predictive video frame of the second set of video frames, and is added to the signal instead of the first exchanged frame. Said means for decoding comprises means for using reference frames from said first bit-stream and decoding parameters of said second bit-stream.

When a signal according to the invention mainly exchanges transmissions from a first bit-stream to a second bit-stream, the second bit-stream comprises at least one first exchange frame, the signal being the first bit. At least one reference frame from the stream and a second exchanged frame encoded using the encoding parameters of the second bit-stream, the second exchanged frame comprising at least one prediction of a second set of video frames A reference frame used for reconstruction of a video frame is used in place of the first exchange frame.

Significant advantages are achieved by the present invention over prior methods and systems. The present invention allows the exchange between bit streams to occur at the locations of SP-frames as well as at the locations of I-frames. The coding efficiency of SP-frames is much better than the coding efficiency of typical I-frames. According to the prior art, smaller bandwidth is required to transmit bit streams with SP-frames at the locations where I-frames are used, and still provide sufficient adaptability to change the transmission states. The exchange of one bit stream with another bit stream may be performed at the positions where the SP-frame according to the invention is placed in the encoded bit stream. Images reconstructed from the bit stream by the decoder do not deteriorate as a result of the change from one bit stream to another bit stream. The invention also has the advantage that random access, fast-forward and fast rewind operations can be performed on the bit stream. The system according to the present invention provides improved error recovery and resilience characteristics over the prior art solutions described above.

These and other features, aspects, and advantages of embodiments of the present invention will become apparent with reference to the following detailed description in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are not intended to define the limitations of the invention, but for the purpose of illustration only and that the limitations of the invention should be referred to in the appended claims.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1A-1C and 2 illustrate prior art video frames encoding / compression.

3 is a block diagram of a typical motion-compensated predictive video coding system (encoder).

4 is a block diagram of a typical motion-compensated predictive video coding system (decoder).

5 is a diagram illustrating the exchange between two different bit streams using S-frames in accordance with the present invention.

6 is a block diagram of a decoder according to a preferred embodiment of the present invention.

7 is a diagram of random access using S-frames.

8 is a diagram of a fast-forward process using S-frames.

9 is a block diagram of a decoder according to another preferred embodiment of the present invention.

10 is a block diagram of a decoder according to another preferred embodiment of the present invention.

11 is a block diagram of an encoder in accordance with a preferred embodiment of the present invention.

12 is a block diagram of a decoder according to another preferred embodiment of the present invention.

13 is a block diagram of a decoder according to a second embodiment of the present invention.

14 is a diagram of an error resilience / recovery process using SP-frames.

15 is a block diagram of an encoder according to a third preferred embodiment of the present invention.

16 is a block diagram of an encoder according to another preferred embodiment of the present invention.

The invention is described below in a system in which multiple bit streams are formed from video signals. The video signal can be any digital video signal comprising multiple images, i.e., an image sequence. The digital video signal is encoded in an encoder to form a plurality of bit streams. Each bit stream is encoded from the same video signal using at least partially different encoding parameters. For example, by selecting different encoding parameters, the bit rate can be changed, and in this way bit streams with different bit rates can be formed. The coding parameters may be, for example, frame rate, quantization parameters, spatial resolution or other factors affecting the size of the images, which are known per se to those skilled in the art. The encoder also inserts at least one intra frame into each bit stream. Typically, at least the first frame of each bit stream is preferably an intra frame. This may cause the decoder to begin reconstruction of the video signal. The encoder used for encoding I-frames, P-frames and B-frames can be any prior art encoder that performs the encoding of the video signal. Alternatively, there may be more than one prior art encoder, each using different encoding parameters to form multiple bit streams. However, also in order to encode a video signal comprising SP-frames and / or SI-frames according to the invention, a new function is needed in the encoder. This will be described later in more detail.

The encoder also inserts coded frames (P-frames and optionally B-frames) into the bit streams using motion compensated predictive coding. The encoder also inserts a new type of frames, referred to herein as S-frames, into each bit stream at locations where exchange between different bit streams is permitted in accordance with the present invention. The S-frames may be inserted at positions where an intra coded frame is inserted in the prior art methods. Or the S-frames may be used in addition to using intra coded frames in a video sequence. Preferably, different bit streams are stored in storage means for later use. However, it is also possible that the transmission can occur substantially immediately after encoding if it is not necessary to store the entire video sequences and it is sufficient to store the necessary reference frames. The transmission of the encoded video stream may for example be performed by a transmission server having means for retrieving said stored bit streams for transmission and / or means for receiving bit streams directly from an encoder. The transport server also has means for transmitting the bit stream to a transport network (not shown).

Hereinafter, a method according to a preferred embodiment of the present invention will be described. 5 shows a portion of the first bit stream 510 and a portion of the second bit stream 520 formed at the encoder. A few P-frames of each bit stream are shown. Specifically, the first bit stream 510 is shown to include P-frames 511, 512, 514, and 515, and the second bit stream 520 is corresponding P-frames 521, 522. , 524 and 525). Both the first bit stream 510 and the second bit stream 520 also include S-frames 513 (also denoted S ₁ ), 523 (also denoted S ₂ ) at corresponding positions. It is assumed that the two bit streams 510 and 520 correspond to the same sequence encoded at different bit rates, for example using different frame rates, different spatial resolutions or different quantization parameters. The first bit stream 510 is transmitted from the transmission server to the decoders 600, 1200 and 1300 (FIGS. 6, 12 and 13, respectively) via the transmission network, which transmits the bit rate of the transmitted video stream. It is further assumed that a request to change is received from the transport network.

As described above, S-frames are located in the bit stream during the encoding process at positions that allow for the exchange of one bit stream from another bit stream within video sequences. As can be seen in FIG. 5, in a preferred embodiment of the invention an additional S-frame 550 (also indicated as S ₁₂ ) is associated with the S-frames S ₁ and S ₂ . This S-frame is referred to as the second representation of the S-frame (or simply the second S-frame) and is transmitted only during the bit stream exchange. This second S-frame S ₁₂ uses motion compensated prediction from the reference frames of the n-th frame of the first bit stream 510 and corresponds to the corresponding S-frame 523 of the second bit stream 520. S ₂ )) using the encoding parameters to generate by spatial encoding of the n th frame of the video sequence. In the case shown in FIG. 5, S-frame S ₂ uses frames previously reconstructed from second bit stream 520 as reference frames and second S-frame S ₁₂ is used as reference frames. Note that the frames previously reconstructed from the 1 bit stream 510 are used. However, the reconstructed pixel values of both S ₂ and S ₁₂ are the same. The S-frame S ₁₂ is transmitted only if the exchange from the first bit stream 510 to the second bit stream 520 is actually performed. Therefore, it is necessary to form second S-frames only when the exchange is performed, not during the encoding step. On the other hand, it may be useful to have at least some second S-frames formed in advance at the time other bit streams are formed in order to reduce the computational burden during transmission.

When the transmission server reaches a frame of a video sequence that is encoded as an S-frame 513 (S ₁ ) in the first bit stream 510, the transmission server uses the encoded frames of the second bit stream 520. May initiate the actions necessary to continue the transmission of the video stream. At that point the transmission server has already transmitted the P-frames 511 and 512 from the first bit stream 510 and the decoders 600, 1200 and 1300 receive the respective P-frames 511 and 512. And decrypted. Thus, the frames are already stored in the frame memories 640, 1250, 1360 of the decoders 600, 1200, 1300. The frame memories 640, 1250, and 1360 include enough memory to store the necessary information for all the frames needed to reconstruct a P-frame or B-frame, i.e. all the reference frames needed for the current frame to be reconstructed.

The transmission server performs the following operations to continue the transmission of the video stream using the encoded frames of the second bit stream 520. The transmitting server finds out that, for example, by checking the type information of the frame, the current frame transmitted is an S-frame, and thus it is possible to perform exchange between bit streams. Of course, the exchange is performed only if a request is received to perform the exchange or if it is necessary to perform the exchange for some other reason. The transmitting server inputs the corresponding S-frame 523 of the second bit stream and uses it to form a second S-frame 550 (S ₁₂ ), which uses the second S-frame S ₁₂ . It transmits to the decoders 600, 1200, and 1300. The transmitting server does not transmit the S-frame S ₂ of the second bit stream but instead transmits the second S-frame S ₁₂ . By decoding the second S-frame S ₁₂ , the decoder 600 uses the same frames as the image generated when the decoder 600 uses the respective frames 521, 522 and the S-frame 523 of the second bit stream 520. The second S-frame is formed in such a way that the image can be reconstructed. After transmission of the second S-frame, the transmission server continues to transmit the encoded frames of the second bit stream 520, that is, 524, 525, and so on.

The S-frames 513, 523, 550 may include blocks encoded using only spatial correlation among pixels (intra blocks) and blocks encoded using both spatial and temporal correlation (inter blocks). have. For each inter block, the prediction P (x, y) of this block is formed at decoders 600, 1200, 1300 using the received motion vectors and the reference frame. Transform coefficients c _pred for P (x, y) corresponding to the basis functions f _ij (x, y) are calculated and quantized. Quantized values of transform coefficients c _pred are denoted as I _pred and dequantized values of quantized transform coefficients I _pred are denoted as d _pred . Quantized coefficients I _err for the prediction error are received from the encoder. _Dequantized values of these coefficients will be represented as d _err . The value of each pixel S (x, y) in the inter block will be decoded as the weighted sum of the basis functions f _ij (x, y) and the weight values d _rec will be referred to as dequantized reconstructed image coefficients. The value of d _rec must be such that the coefficient, which may be the _rec d obtained by the quantization and inverse quantization (c _rec) exists. In addition, the values d _rec must satisfy one of the following conditions.

d _rec = d _pred + d _err or

c _rec = c _pred + d _err

The values S (x, y) can be further normalized and filtered.

Next, encoding of S-frames located in a bit stream, such as, for example, S-frames 513 (S ₁ ) and 523 (S ₂ ), is described.

In general, S-frames according to the present invention, such as frames 513 and 523 of FIG. 5, are configured in a block-by-block manner. As described above, each of the blocks is such a method (intra or SI-blocks) using spatial correlations among the pixels of the image to be encoded or such using temporal correlation between blocks of pixels in successive frames of the video sequence. May be encoded in a manner (inter or SP-blocks).

Encoding of S-frames according to the present invention will be described with reference to FIG. 11, which is a block diagram of the S-frame encoder 1100 according to the first embodiment of the present invention.

A video frame encoded in S-frame format is first divided into blocks and each block is then encoded into an SP-block, an SI-block or an intra-block. The intra block is known per se from the prior art. The switch 1190 suitably operates to switch between the SI coding mode and the SP coding mode. That is, the switch 1190 is a structure used in the description of the present invention, but is not necessarily a physical device. In the SP-coding mode switch 1190 is operated to obtain a motion compensated prediction 1170 for the current block. The motion compensated prediction block 1170 forms a prediction P (x, y) for the current block of the frame that is encoded in a manner similar to that used in motion compensated prediction known from the prior art. More specifically, motion compensated prediction block 1170 is used to determine the motion vector representing the relationship between the pixels in the current block and the pixel values of the reconstructed reference frame maintained in frame memory 1146. Form a prediction P (x, y) for the current block.

In the SI-coding mode, the switch 1190 is operated to obtain a prediction for the current block of the frame to be encoded from the intra prediction block 1180. Intra prediction block 1180 forms a prediction P (x, y) for the current block of the frame that is encoded in a manner similar to that used in intra prediction known in the prior art. More specifically, intra prediction block 1180 forms a prediction P (x, y) for the current block of the frame to be encoded using spatial prediction from neighboring coded neighboring pixels in the frame to be encoded.

In SP- and SI-coding modes the prediction P (x, y) takes the form of a block of pixel values. In block 1160 a forward transform, for example a discrete cosine transform (DCT), is applied to the predicted block P (x, y) of pixel values, and the resulting transform coefficients, referred to as c _pred , are then quantized Are quantized in quantization block 1150 to form fields I _pred . Corresponding operations are also performed on the original image data. More specifically, the current block of pixel values of the original image to be encoded is applied to the transform block 1110. Here, a forward transform (eg DCT) is applied to the pixel values of the original image block to form transform coefficients c _orig . These transform coefficients are passed to quantization block 1120 and quantized to form quantized transform coefficients I _orig . The addition element 1130 receives sets of quantized transform coefficients I _preg and I _orig from the respective quantization blocks 1150 and 1120, and determines the quantized prediction error coefficients I _err according to the following relationship. Create a set.

I _err = I _orig -I _pred

The quantized prediction error coefficients I _err are passed to a multiplexer 1135. If the current block is coded in SP-form / mode, the multiplexer 1135 also receives motion vectors for the SP-coded block. When the current block is encoded in SI-form / mode, information about the intra prediction mode used to form the prediction for the SI-coded block in intra prediction block 1180 is passed to the multiplexer. Preferably, variable length coding is applied to the intra prediction mode information or the motion vector in the multiplexer 1135 and to the quantized prediction error coefficients I _err , where the bit-stream is subjected to multiplexing with various types of information. And the bit-stream thus formed are transmitted to the corresponding decoders 1200, 1300 (see Figs. 12 and 13).

S-frame encoder 1100 according to the present invention also includes a local decoding function. The quantized predictive transform coefficients I _pred formed at quantization block 1150 are supplied to addition element 1140, which also receives quantization error coefficients I _err . The addition element 1140 recombines the quantized prediction transform coefficients I _pred and the quantized prediction error coefficients I _err to form a set of reconstructed quantized transform coefficients I _rec according to the following relationship: do.

I _rec = I _pred + I _err

The reconstructed quantized transform coefficients are passed to inverse quantization block 1142 and the inverse quantization block 1142 dequantizes the reconstructed quantized transform coefficients to form dequantized reconstructed transform coefficients d _rec . . The inverse quantized reconstructed transform coefficients are further passed to inverse transform block 1144, where the inverse quantized reconstructed transform coefficients are, for example, an inverse discrete cosine transform (IDCT) or block 1160. Any other inverse transform is performed that corresponds to the transform performed in. As a result, a block of reconstructed pixel values for that image block is formed and stored in frame memory 1146. As the next blocks of the frame encoded in the S-frame format are subjected to the above-described encoding and local decoding operations, the decoded version of the current frame is gradually aggregated in the frame memory, from which the version can be accessed and the same frame It can be used in intra prediction of next blocks of or in inter (motion compensated) prediction of next frames in a video sequence.

The operation of the general S-frame decoder according to the first embodiment of the present invention will now be described with reference to FIG.

The bit-stream generated by the S-frame encoder described above in connection with FIG. 11 is received by the decoder 1200 and demultiplexed into components by the demultiplexer 1210. The decoder reconstructs the decoded version of the S-frame in a block-to-block manner. As mentioned above, the S-frame may include intra-blocks, SP-coded image blocks and SI-coded image blocks. For SP-type image blocks, the information in the received bit-stream includes VLC coded motion coefficient information and VLC coded quantized prediction error coefficients I _err . For SI-coded image blocks, the information in the received bit-stream is used to form intra prediction for the SI-coded block together with the VLC coded quantized prediction error coefficients I _err . Contains VLC coded information about the intra prediction mode.

When decoding an SP-encoded block, demultiplexer 1210 first receives the appropriate variable length decoding (VLD) bits to recover motion vector information and quantized prediction error coefficients (I _err ). -Apply to stream The demultiplexer 1210 then separates the motion vector information from the quantized prediction error coefficients I _err . The motion vector information is supplied to a motion compensation prediction block 1260, and the quantized prediction error coefficients recovered from the bit-stream are applied to one input of the addition element 1220. The motion vector information is used in conjunction with the pixel values of the previously reconstructed frame held in frame memory 1250 in motion compensation prediction block 1260 to predict P (x) in a manner similar to that employed by encoder 1100. , y).

When decoding an SI-coded block, demultiplexer 1210 applies variable length decoding suitable for received intra prediction mode information and quantized prediction error coefficients I _err . The intra prediction mode information is then separated from the quantized prediction error coefficients and fed to an intra prediction block 1270. The quantized prediction error coefficients I _err are supplied to one input of an addition element 1220. The intra prediction mode information is used together with previously decoded pixel values of the current frame maintained in the frame memory 1250 in the intra prediction block 1270 to form a prediction P (x, y) for the current block to be decoded. . Again, the intra prediction process performed by the decoder 1200 is similar to that performed by the encoder 1100 described above.

Once the prediction is made for the current block of the frame to be decoded, the switch 1280 is operated such that a prediction P (x, y) containing the predicted pixel values is supplied to the transform block 1290. Again, the switch 1280 is not necessarily a physical device but an abstract structure used in the description of the present invention. In the case of an SP-coded block, the switch 1280 is operated to connect the motion compensation prediction block 1260 to the transform block 1290, while in the case of an SI-coded block, the switch 1280 Is operated to connect the intra prediction block 1270 to the transform block 1290.                 

In block 1290, a forward transform, for example a discrete cosine transform (DCT), is applied to the predicted block P (x, y) of pixel values and the resulting transform coefficients c _pred are applied to quantization block 1295. Supplied and quantized to form quantized transform coefficients I _pred . The quantized transform coefficients I _pred are then supplied to the second input of the addition element 1220 and added to the prediction error coefficients I _err to reconstruct the quantized transform coefficients ( I _rec ).

I _rec = I _pred + I _err

The reconstructed quantized transform coefficients I _rec are further inversely quantized to be supplied to inverse quantization block 1230 to form dequantized reconstructed transform coefficients d _rec . The inverse quantized reconstructed transform coefficients d _rec are then passed to an inverse transform block 1240 to, for example, an inverse discrete cosine transform (IDCT) or any other inverse transform corresponding to the transform performed at block 1290. This is done. In this way, a block of reconstructed pixel values for the image block is formed. Reconstructed pixel values are supplied to video output and frame memory 1250. As the next blocks of the S-frame to be decoded are subjected to the above-described decoding operation, the decoded version of the current frame is gradually collected in the frame memory 1250, from which the version can be accessed and the next block of the same frame. It can be used in intra prediction of or in inter (motion compensated) prediction of the next frames in a video sequence.

Reviewing the structure and function of the S-frame encoder and the decoder according to the first embodiment of the present invention, how the S-frames according to the present invention are without mismatch errors such as mismatch errors in previous video encoding / decoding systems. It is now possible to understand whether it is possible to exchange between bit-streams. Referring again to the bit-stream exchange example shown in FIG. 5, the exchange from the first bit-stream 510 to the second bit-stream 520 results in S-frames S in each bit-stream. ₁ 513 and S ₂ 523). As described above, when the exchange is performed, the second S-frame indicated by S ₁₂ 550 is encoded and transmitted. The second S-frame S ₁₂ is encoded using the encoding parameters of the second bit-stream 520 and the reference frames of the first bit-stream 510, wherein the second frame S ₁₂ is When recoded, the reconstructed pixel values are the same as the pixel values resulting from the transmission of frame S ₂ in the second bit-stream.

I ² and I _err ² _pred have the above procedure each let represent the quantized coefficients of the prediction error and the prediction frame obtained by the encoding of the frame SP- (S _2). And, suppose that I ² _rec represents the quantized reconstructed image coefficients of the S-frame S ₂ . The encoding of the second S-frame 550 (S ₁₂ ) follows the same procedures as in the encoding of the S-frame 523 (S ₂ ) except for the following. 1) The reference frame (s) used for prediction of each block of the second S-frame S ₁₂ are reconstructed frames obtained by decoding the first bit stream 510 from the video sequence to the current nth frame. 2) The quantized prediction error coefficients are calculated as follows. I ¹² _err = I ² _rec -I ¹² _pred . Where I ¹² _pred represents the quantized prediction transform coefficients. The quantized prediction error coefficients I ¹² _err and the motion vectors are sent to the decoder 1200.

When decoding the second S-frame S ₁₂ in the decoder 1200 using frames reconstructed from the first bit stream 510 before exchange as reference frames, the coefficients I ¹² of the second S-frame. _pred ) is added and added to the received quantized prediction error coefficients I ¹² _err as described above. That is, I ¹² _rec = I ¹² _err + I ¹² _pred = I ² _rec -I ¹² _pred + I ¹² _pred = I ² _rec . This equation shows that I ¹² _rec and I ² _rec are the same. Thus, even if S ₁₂ is decoded, the S-frame S ₂ is decoded, even if the second S-frame S ₁₂ and the S-frame S ₂ of the second bit stream have different reference frames. Produces an image with reconstructed pixel values equal to the pixel values resulting from

From the above description of the encoding and decoding of S-frames according to the invention, it will be appreciated that there is a significant difference compared to the encoding and decoding of P-frames and I-frames according to the prior art. Specifically, when encoding or decoding an image block in SP or SI-form, it should be understood that the prediction P (x, y) for that block is transformed into a transform coefficient region by applying a transform such as a discrete cosine transform. . The transform coefficients thus generated are then quantized and the prediction error is determined in the quantized coefficient region. This is in contrast to the prior art predictive encoding in which the prediction error is determined in the spatial (pixel value) region.

The operation of decoder 1200 during the exchange between bit-streams 510 and 520 is described in detail below. At the location of the video sequence where the exchange from the first bit-stream to the second bit-stream 520 occurs, the decoder 1200 deletes the previous P-frames 511 and 512 of the first bit-stream 510. Already received and decrypted. Decoded frames are stored in frame memory 1250 and are therefore usable as reference frames. The second bit from the stream 510-the first bit if the exchange of a stream (520) takes place, the encoder (1100; Fig. 11) is the second S ₁₂ to configure the frame S- (S _12, 550) and encoding The encoded video information indicating s is transmitted to the decoder 1200.

As mentioned above, the encoding is performed in a block-to-block manner. Specifically, the second S-frame S ₁₂ is encoded as a combination of image blocks and generally each image block is encoded as an SP-coded block or SI-coded block or intra-block. In the SP-encoded blocks of the second S-frame S ₁₂ , the compressed video information transmitted from the encoder to the decoder takes the form of quantized prediction error transform coefficients I ¹² _err and motion vector information. do. In the SI-coded blocks of the second S-frame S ₁₂ , the compressed video information forms a prediction for the SI-coded block at the encoder and the quantized prediction error transform coefficients I ¹² _err . Contains information about the intra prediction mode used. As described above, the compressed video information is subjected to appropriate variable length coding (VLC) before being sent from the encoder to further reduce the number of bits required for display.

Compressed video information for a given image block is received by the decoder 1200 and first a suitable variable length decoding (VLD) is performed and separated by its demultiplexer 1210 into its components. The quantized prediction error coefficients I ¹² _err extracted from the received bit-stream are applied to the first input of the adder 1220 and the block of predicted pixel values P (x, y) for each image block is encoded. It is formed according to the mode SP or SI. In the case of an SP-coded block, the block of predicted pixel values P (x, y) is derived from the first bit-stream (e.g., P-frame 511 or 512) available in frame memory 1250. Is formed in the motion compensation prediction block 1260 by using the motion vector information extracted from the encoded video information of the second S-frame S ₁₂ by the reference frame and demultiplexer 1210. In the case of an SI-coded block, the block of predicted pixel values P (x, y) also uses previously decoded pixels of the second S-frame S ₁₂ stored in the frame memory 1250. Intra prediction block 1270 is formed. Intra prediction is performed by the demultiplexer 1210 according to the intra prediction mode information extracted from the received video information for the second S-frame S ₁₂ .

Once the prediction for the current block of the second S-frame is made, the predicted pixel values P (x, y) are passed to the transform block 1290. Here, a forward transform (eg, a discrete cosine transform (DCT)) is applied to the predicted pixel values P (x, y) to form a set of transform coefficients c _pred . These transform coefficients are then passed to quantization block 1295 and quantized to form quantized transform coefficients I ¹² _pred . The quantized transform coefficients I ¹² _pred are then applied to the second input of the adder 1220. The adder 1220 may _combine the quantized transform coefficients I ¹² _pred with the quantized prediction error transform coefficients I ¹² _err to form reconstructed quantized transform coefficients I ¹² _rec according to the following relationship. To combine.

I ¹² _rec = I ¹² _pred + I ¹² _err

The reconstructed quantized transform coefficients I ¹² _rec are then supplied to inverse quantization block 1230 to inverse quantize to form dequantized reconstructed transform coefficients d ¹² _rec . The inverse quantized reconstructed transform coefficients d ¹² _rec are then passed to an inverse transform block 1240 to perform an inverse transform operation (eg an inverse discrete cosine transform (IDCT)). As a result, a block of reconstructed pixel values for the current block of the second S-frame S ₁₂ is formed. Reconstructed pixel values I _c (x, y) are supplied to video output and frame memory 1250. As the next blocks of the second S-frame S ₁₂ are encoded and transmitted from the encoder 1100 to the decoder 1200 and subsequently decoded, the decoded version of the second S-frame is progressively stored in the frame memory 1250. Accumulate. From the frame memory, already decoded blocks of the second S-frame can be retrieved and predicted pixel values P (x) for the next blocks of the second S-frame S ₁₂ by the intra prediction block 1270. , y). It should be noted here that the quantized prediction error transform coefficients for each image block of the second S-frame S ₁₂ are generated at the encoder 1100 according to the following relationship.

I ¹² _err = I ² _rec -I ¹² _pred

Here, I ² _rec are quantized reconstructed transform coefficient values generated by encoding and then decoding the S-frame S ₂ in the second bit-stream. This means that the reconstructed transform coefficients I ¹² _rec generated by decoding the compressed video information for the second S-frame S ₁₂ are transmitted and decoded by the S-frame S ₂ from the second bit-stream. If it is, it means the same as the generated coefficients. As mentioned above, this is as follows.

I ¹² _rec = I ¹² _pred + I ¹² _err

= I ¹² _pred + I ² _rec -I ¹² _pred = I ² _rec

Thus, I ¹² _rec = I ² _rec .

Thus, by constructing a second S-frame S ₁₂ in accordance with the method of the present invention, transmitting it from the encoder to the decoder, and then decoding it, a mismatch-free exchange between the first and second bit-streams can be achieved. It can be seen that there is.

Consider the case where the second S-frame is an SI-frame but the S-frame in the bit stream is an SP-frame. In this case, a frame using motion-compensated prediction is represented as a frame using only spatial prediction. This special case relates to the random access and error resilience described below.

In the encoder 1100 and the decoder 1200 according to the first embodiment of the present invention described above, in the transform blocks 1160 (encoder) and 1290 (decoder) to generate quantized transform coefficients I _pred . Note that the quantization applied to the generated transform coefficients c _pred is the same as that used to generate the quantized prediction error transform coefficients I _err . More specifically, in the first embodiment of the present invention, when a block of predicted pixel values P (x, y) for an image block of an S-frame to be encoded / decoded is generated, the predicted pixel values are predicted. The quantization parameter QP used to quantize the transform coefficients c _pred corresponding to the block P (x, y) includes the quantization parameters used to generate the quantized prediction error transform coefficients I _err . Should be the same. This is desirable because the addition performed to produce the reconstructed transform coefficients I _rec is performed in the quantized transform coefficient region. In other words

I _rec = I _pred + I _err

Because of this, not using the same quantization parameters in the configuration of I _pred and I _err will be an error in the reconstructed quantized transform coefficients I _rec .

15 is a block diagram of an S-frame encoder 1500 according to a second embodiment of the present invention that provides greater flexibility in the selection of quantization parameters for generating quantized transform coefficients I _pred and I _err . Shows. As can be seen by comparing FIG. 15 with FIG. 11, the main difference between the S-frame encoder 1500 according to the second embodiment of the present invention and the S-frame encoder 1100 according to the first embodiment of the present invention. Is related to the position of the quantization blocks 1525 and 1550. The operation of the S-frame encoder 1500 according to the second embodiment of the present invention will be described in detail with reference to FIG. 15 below.

According to the second embodiment of the present invention, a video frame encoded in S-frame format is first divided into blocks and then each block is encoded into an SP-block or an SI-block. The switch 1585 is suitably operated to exchange between the SP and SI encoding modes. In the SP encoding mode, the switch 1585 is operated to obtain motion compensated prediction for the current block of the frame being encoded from the motion compensated prediction block 1575. The motion compensation prediction block 1575 is predicted for the current block of the frame to be encoded by determining a motion vector that describes the relationship between the pixels of the current block and the pixel values of the reconstructed reference frame maintained in the frame memory 1570. Form a block of pixel values P (x, y).

In the SI-coding mode, the switch 1585 is operated to obtain a prediction for the current block of the frame to be encoded from the intra prediction block 1580. Intra prediction block 1580 is used to form a block of predicted pixel values P (x, y) for the current block of a frame encoded using spatial prediction from already encoded neighboring pixels in the frame being encoded. It operates in a manner similar to that described in connection with the first embodiment.

In both SP- and SI-coding modes, a forward transform, for example a discrete cosine transform (DCT), is applied to the predicted block P (x, y) of pixel values at transform block 1590. The resulting transform coefficients c _pred are supplied to adders 1520 and 1540. The original image data containing the actual pixel values of the image block currently being encoded is passed to the transform block 1510 for further forward conversion (eg DCT). The resulting transform coefficients c _orig are then passed to an adder 1520, which adds a difference between c _orig and c _pred to produce prediction error transform coefficients c _err according to the following relationship: do.

c _err = c _orig -c _pred

Prediction error transform coefficients are supplied to quantization block 1525 and quantized using quantization parameter PQP to form quantized prediction error transform coefficients I _err , and the quantized prediction error transform coefficients I _err ) is then passed to multiplexer 1540.

When the current block is encoded in the SP-form, the multiplexer 1540 also receives information about the motion vectors used to form the motion compensated prediction P (x, y) for the SP-coded block. When the current block is encoded in SI-form, information about the intra prediction mode used to form the prediction P (x, y) for the SI-coded block is also passed to the multiplexer. Preferably, the multiplexer 1540 applies quantized prediction error transform coefficients (I _err ) and variable length coding (VLC) suitable for motion vector or intra prediction mode information, and correspondingly by multiplexing with various types of information. Form a bit-stream for transmission to the decoder.

Quantized prediction error transform coefficients I _err are passed from quantization block 1525 to inverse quantization block 1530 to form a quantization parameter PQP to form dequantized prediction error transform coefficients d _err . Inverse quantized. The dequantized prediction error transform coefficients d _err are then passed to adder 1540 and combined with transform coefficients c _pred generated from predicted pixel values P (x, y) for the current block. . More specifically, the adder 1540 adds transform coefficients c _pred and inverse quantized prediction error transform coefficients d _err to form reconstructed transform coefficients c _rec according to the following relationship.

c _rec = c _pred + d _err

The reconstructed transform coefficients c _rec are then passed to quantization block 1550 and quantized using quantization parameter SPQP to produce quantized reconstructed transform coefficients I _rec . Note that the quantization parameter SPQP used to quantize the reconstructed transform coefficients is not necessarily the same as the quantization parameter PQP used to quantize the prediction error transform coefficients c _err at quantization block 1525. do. In particular, finer quantization can be applied to the reconstructed transform coefficients c _rec and coarser quantization can be applied to the prediction error coefficients c _err . This in turn results in a smaller reconstruction error (distortion) when the decoded image is formed in the decoder.

The quantized reconstructed transform coefficients I _rec are then supplied to inverse quantization block 1560 using inverse quantization parameter SPQP to form inverse quantized reconstructed transform coefficients d _rec . Is quantized. The inverse quantized reconstructed transform coefficients d _rec are then passed to an inverse transform block 1565 to perform an inverse transform operation, for example an inverse discrete cosine transform (IDCT). As a result of this operation, a block of reconstructed pixel values I _c (x, y) for the image block is formed. The block of reconstructed pixel values I _c (x, y) is then stored in frame memory 1570. As the next blocks of the frame encoded in the S-frame format are subjected to the above-described encoding and local decoding operations, a decoded version of the current frame is gradually collected in the frame memory 1570, and the version from the frame memory can be accessed. And in intra prediction of the next blocks of the same frame or in inter (motion compensated) prediction of the next frames in the video sequence.

The operation of the S-frame decoder 1300 according to the second embodiment of the present invention will now be described with reference to FIG. The bit-stream generated by the S-frame encoder 1500 according to the second embodiment of the present invention and described above with respect to FIG. 15 is received by the decoder 1300 and demultiplexed into its components. The decoder reconstructs the decoded version of the S-frame in a block-to-block manner. As mentioned above, an S-frame generally includes SP-coded image blocks and SI-coded image blocks. For SP-encoded image blocks, the information in the received bit-stream includes VLC coded motion vector information and VLC coded quantized prediction error transform coefficients I _err . For SI-coded image blocks, the information in the received bit-stream forms intra prediction for the SI-coded block as well as the VLC coded quantized prediction error transform coefficients (I _err ). Contains VLC coded information about the intra prediction mode used.

When decoding an SP-encoded image block, the demultiplexer 1310 first applies appropriate variable length decoding (VLD) to the received bit-stream to recover motion vector information and quantized prediction error coefficients (I _err ). Apply. The demultiplexer 1310 then separates the motion vector information from the quantized prediction error coefficients I _err . The motion vector information is supplied to a motion compensation prediction block 1370, and the quantized prediction error coefficients I _err recovered from the received bit-stream are applied to an inverse quantization block 1320. The motion vector information recovered from the received bit-stream is used in conjunction with the pixel values of the previously reconstructed frame maintained in frame memory 1360 in motion compensation prediction block 1370, similar to the scheme employed in encoder 1500. Form a prediction P (x, y) for the current block to be decoded in such a manner.

When decoding the SI-coded image block, the demultiplexer 1310 applies variable length decoding suitable for the received intra prediction mode information and the quantized prediction error transform coefficients I _err . The intra prediction mode information is then separated from the quantized prediction error transform coefficients I _err and fed to an intra prediction block 1380. The quantized prediction error transform coefficients I _err are supplied to an inverse quantization block 1320. The intra prediction mode information recovered from the received bit-stream is used together with previously decoded pixel values of the current frame maintained in frame memory 1360 in intra prediction block 1380 to predict P ( x, y). Again, the intra prediction process performed at decoder 1200 is similar to that performed at corresponding encoder 1500 described above.

For SP- and SI-coded image blocks, the quantized prediction error transform coefficients I _err recovered from the received bit-stream are dequantized prediction error transform coefficients d in inverse quantization block 1320. _err ) to dequantize using a quantization parameter (PQP). The dequantized prediction error transform coefficients d _err are applied to one input of an adder 1325.

If the prediction P (x, y) is formed for the current block of the frame once decoded, either by motion compensated prediction in motion compensated prediction block 1370 or by intra prediction in intra prediction block 1380, switch 1385 ) Is suitably operated to supply the predicted pixel values P (x, y) to the transform block 1390. Here a forward transform, for example a discrete cosine transform (DCT), is applied to the predicted block P (x, y) of pixel values to form transform coefficients c _pred . The transform coefficients c _pred are then supplied to a second input of the adder 1325 to inverse quantized prediction error transform received from inverse quantization block 1320 to form reconstructed transform coefficients c _rec . Combined with coefficients. More specifically, the reconstructed transform coefficients are determined by adding transform coefficients c _pred and dequantized prediction error transform coefficients d _err according to the following relationship.

c _rec = c _pred + d _err

The reconstructed transform coefficients c _rec are then passed to quantization block 1330 and quantized using quantization parameter SPQP to generate quantized reconstructed transform coefficients I _rec . The quantized reconstructed transform coefficients I _rec are then supplied to inverse quantization block 1340 using the quantization parameter SPQP to form inverse quantized reconstructed transform coefficients d _rec . Dequantized. The inverse quantized reconstructed transform coefficients d _rec are then passed to an inverse transform block 1350 to perform an inverse transform operation, for example an inverse discrete cosine transform (IDCT). As a result of the inverse transform applied at inverse transform block 1350, a block of reconstructed image pixels I _c (x, y) for that image block is formed. The block of reconstructed pixels I _c (x, y) is supplied to a video output of a decoder and to frame memory 1360, where the pixels are stored. As the next blocks of the S-frame are decoded as described above, the decoded version of the current frame is progressively aggregated in the frame memory 1360. The version from the frame memory can be accessed and used in intra prediction of the next blocks of the same frame or in inter (motion compensated) prediction of the next frames in the video sequence.

16, an encoder according to a third embodiment of the present invention is shown. In this embodiment, transform coefficients c _pred are quantized and dequantized using the same quantization parameter (SPQP) in encoder section (blocks 1625 and 1630) and in decoder section (blocks 1692 and 1694). . Therefore, the encoder does not introduce any additional quantization error into the prediction loop, and error augmentation in the prediction loop is thus effectively prevented. Blocks 1610, 1620, 1625, 1630, 1640, 1650, 1660, 1665, 1670, 1675, 1680, 1685, 1690 are blocks 1510, 1520, 1525, 1530, 1540, 1550, respectively, shown in FIG. 15. , 1560, 1565, 1570, 1575, 1580, 1585, 1590.

6, a decoder 600 according to a preferred embodiment of the present invention is shown. Most of the components of the decoder 600 are the same as those of the decoder 1200 shown in FIG. The operational blocks of decoder 600 are arranged to decode predictive blocks of frames, where no exchange means is shown in FIG. 6. The other blocks 610, 615, 620, 630, 640, 650, 660 and 670 each have similar functionality to the blocks 1210, 1220, 1230, 1240, 1250, 1260, 1290 and 1295 shown in FIG. Have

In Fig. 9, a decoder 600 according to another preferred embodiment of the present invention is shown. The decoder 600 shown in FIG. 9 is a variation of the decoder 600 shown in FIG. The difference between the decoder shown in FIG. 9 and the decoding period shown in FIG. 12 is that a normalization block 680 is inserted between one input of the demultiplexer 610 and the addition element 615. The other blocks 610, 615, 620, 630, 640, 650, 660 and 670 each have similar functionality to the blocks 1210, 1220, 1230, 1240, 1250, 1260, 1290 and 1295 shown in FIG. Have.

In Fig. 10, a decoder 600 according to another preferred embodiment of the present invention is shown. Most of the components of the decoder 600 are the same as those of the decoder 1300 shown in FIG. The operational blocks of the decoder 600 are arranged to decode the predictive blocks of the frames, where no exchange means is shown in FIG. 10. Another difference between the decoder shown in FIG. 10 and the decoding period shown in FIG. 13 is that the normalization block 680 is used instead of the dequantization block 1230. The other blocks 610, 615, 620, 630, 640, 650, 660 and 670 each have similar functionality to the blocks 1310, 1325, 1330, 1340, 1350, 1360, 1370 and 1390 shown in FIG. 13, respectively. Have

Encoding of the video frame may be performed in a block-to-block manner such that differently encoded regions may exist in the same encoded video frame. For example, some parts of the frame may be inter coded and some other parts of the frame may be intra coded. The above procedures are suitably applied to each part of the frame according to the encoding procedure of that part.

In addition to the transmission network, a request for a change in the bit stream transmission characteristics may also be sent by other parts of the transmission system. For example, the receiver may ask the sending server to change the parameters for some reason. This request is for example delivered to the transmitting server via the transmitting network.

Although H.26L is used as an example of the standard, it is contemplated that embodiments of the present invention and certain variations and modifications are within the scope of the present invention.                 

Bit stream exchange is not the only application to which the present invention can be applied. If one of the bit streams has a low time resolution, for example 1 frame / sec, this bit stream can be used to provide fast forward functionality. Specifically, decoding from a bit stream with a low temporal resolution and then swapping into a bit stream with a normal frame rate will provide such functionality. 8 shows two bit streams. The second bit stream includes only S-frames predicted from each other at intervals greater than the frame repetition interval of the first bit-stream. Moreover, "Fast Forward" can start and stop at any point in the bit-stream. In the following, some other applications of the invention are described.

Splicing and Random Access

The bit stream-exchange example described above considered bit streams belonging to the same sequence of images. However, this is not necessarily the case in all situations where bit stream exchange is required. Examples include: exchange between bit streams arriving from different perspectives from different cameras capturing the same event or from cameras located around a building for surveillance; Embed commercials in television broadcasts, video bridging, etc. or switch to local / national programming. The general term for the process of concatenating encoded bit streams is splicing.

If an exchange occurs between bit streams belonging to a sequence of different images, this only affects the encoding of the S-frames used for the exchange between the bit streams, ie the second S-frame S ₁₂ in FIG. 5. . Specifically, the use of motion-compensated prediction of frames in one sequence of images using reference frames from a sequence of different images is not as effective as in the case where both bit streams belong to the same sequence of images. In this case, spatial prediction of the second S-frames will be more effective. This is shown in FIG. 7, where the exchange frame is an SI-frame using only spatial prediction that equally reconstructs the corresponding SP-frame S ₂ . This method can be used for the bit stream as a random access mechanism and has a closer relationship to error recovery and resilience as described below.

Error recovery

Various representations of a single frame in the form of S-frames predicted from different reference frames, eg, immediately preceding the reconstructed frame and temporally earlier reconstructed frame, increase the error resilience of the encoded video sequence and / or Or to improve recovery from errors in the bit-stream. This is shown in FIG. If packet loss occurs during the streaming of the pre-encoded bit stream, and if a frame or slice is lost, the receiver notifies the transmitter of the lost frame / slice and the transmitter indicates one of the alternative representations of the next S-frame. Respond by sending one. The alternative representation, for example frame S ₁₂ of FIG. 14, uses reference frames already correctly received by the receiver. In slice-based packetization and delivery, the transmitter can further estimate slices affected by such slice / frame loss and update only those slices in the next S-frame with their replacement representations.

Similarly, as described above in the description of splicing, a second representation of an S-frame may be generated without using any reference frames, i.e., an SI ₂ -frame as shown in FIG. In this case, the transmitter will send a _second SI-frame, ie SI ₂ , instead of S ₂ to stop the error propagation. This approach can also be extended to direct slice-based encoding / packetization. More specifically, the server sends slices in the next S-frame that are affected by packet loss from the SI-frame.

Error resilience

Encoding of the video frame may be performed in a block-to-block manner such that differently encoded regions may exist in the same encoded video frame. For example, some parts of the frame may be inter coded and some other parts of the frame may be intra coded. As mentioned above, since intra-block coding does not use any time correlation, intra-block coding stops any error propagation that may be started due to transmission corruptions.

In lossy transmission networks, intra macroblock refresh strategy can provide significant error resilience / recovery performance. In the interactive client / server scenario, the encoder at the server side is expected to be calculated based on the specific position received from the client, for example, the exact location of the lost / corrupted frame / slice / macroblock or through negotiation. Determine to encode frames / macroblocks based on network conditions or measured network conditions. This kind of intra-macroblock update strategy improves the quality of the received video by providing error resilience and error recovery. The optimal intra-macroblock update reproduction rate, ie the frequency at which macroblocks are intra-encoded, depends on the transport channel conditions, for example packet loss and / or bit error rate. However, if already encoded bit streams are transmitted, which is the case for typical streaming applications, the strategy cannot be applied directly. The sequence needs to be coded with the expected network conditions to account for the worst case or additional error resilience / recovery mechanisms are required.

From the above description regarding the use of S-frames in error recovery and slicing applications, the S-frames or slices within S-frames do not use any reference frames and still lead to the same reconstruction of the S-frame. It can be noted that it can be easily displayed as fields / slices. This feature can be used in the adaptive intra regeneration mechanism described above. First, the sequence of images is encoded with some predetermined ratio of S-macroblocks. Then, during transmission, some of the S-macroblocks are sent in a second representation, such as SI-macroblocks. The number of S-macroblocks transmitted in the SI representation can be calculated in a manner similar to the method used in the real time encoding / delivery approach described above.

Video Redundancy Coding

S-frames have other uses in applications that do not operate on behalf of I-frames. Video redundancy coding (VRC) may be given as an example. The principle of the VRC method is to divide the sequence of pictures into two or more threads in such a way that all pictures in the sequence are assigned to one of the threads in a round-robin manner. At regular intervals, all threads converge to a so-called sync frame. From this sync frame, a new thread series begins. If one of the threads is damaged due to, for example, packet loss, the remaining threads typically remain intact and can be used to predict the next sync frame. It is possible to continue decrypting the damaged thread, which degrades the image quality slightly. Alternatively, it is possible to stop decrypting the damaged thread, which reduces the frame rate. Sync frames are always predicted from one of the intact threads. This means that the number of transmitted I-frames can be kept small, because no full re-synchronization is needed. If more than one representation (P-frame) has been sent for a sync frame, each uses a reference frame from a different thread. Due to the use of P-frames, these representations are not the same. Hence mismatch is introduced if some of the representations cannot be decrypted and counterparts are used when decrypting the next threads. Using S-frames as sync frames eliminates this problem.

It is apparent that the invention is not limited to the above-described embodiments but may be modified within the scope of the appended claims.

Claims (26)

A method for transmitting video information, wherein at least a first bit-stream 510 and a second bit-stream are formed from the video information, the first bit-stream 510 comprising at least one video frame. A first set of frames, the second bit-stream 520 includes a second set of frames, including at least one predictive video frame 524, At least partially different encoding parameters are used for encoding the frames of the first bit-stream 510 and the second bit-stream 520, At least one frame of the first bit-stream 510 is transmitted, In the transmission of the video information is exchanged from the first bit-stream 510 to the second bit-stream 520, When exchanging the transmission from the first bit-stream 510 to the second bit-stream 520, the second bit-stream 520 is a first exchange frame in at least one second bit-stream. 523, wherein a second exchange frame 550 is transmitted, wherein the second exchange frame 550 includes at least one reference frame and the second bit-stream from the first bit-stream 510. Encoded using the encoding parameters of 520, wherein the second exchange frame 550 is a reference frame used for reconstruction of the at least one predictive video frame 524 of the second set of video frames. Used in place of the first exchange frame (523) in the second bit-stream. 2. The method of claim 1, wherein the first bit-stream (510) comprises a first exchange frame (513) in at least one first bit-stream. The method of claim 1, wherein the first bit-stream 510 is configured to perform one intra frame to perform a transition from one position of the video information to another position of the video information. And one second exchange frame (550). 3. The method according to claim 1 or 2, characterized in that the first bit-stream 510 comprises only intra frames and first exchange frames 513 to perform a fast forward operation on the video information. Video information transmission method. The method according to claim 1 or 2, wherein the first exchange frame 523 or the second exchange frame 550 in the second bit-stream is a prediction video frame, and the prediction information includes only intra prediction information. Video information transmission method. The method according to claim 2, wherein a first exchange frame 513 in a first bit-stream is formed such that transform coefficients c _pred are calculated and quantized to form quantized values I _pred of the transform coefficients, is the quantized prediction error coefficients (I _err) is defined, the transform coefficients in (c _rec) by a quantization from the, in the reconstructed quantized transform coefficients (I _rec) transform coefficients that are to be acquired (c _rec) is and the quantized transform coefficients to reconstruct the presence (I _rec) is defined, in the reconstruction of the quantized transform coefficients (I _rec) are the following conditions: I _rec = I _pred + I _err , or c _rec = c _pred + d _err Here, d _err is a video information transmission method, characterized in that the dequantized values of the prediction error. 7. The method of claim 6, wherein the same quantization parameters are used for the quantized prediction error coefficients (I _err ). 7. The method of claim 6, wherein different quantization parameters are used for quantization of the transform coefficients c _rec and quantization of a prediction error. A method for transmitting video information, wherein at least a bit-stream 510 is formed from the video information, the bit-stream 510 comprising a first set of frames comprising at least one video frame, and at least one A second set of frames comprising a predictive video frame 524 of At least partially different encoding parameters are used for encoding the frames of the first set of frames and the frames of the second set, At least one frame of the bit-stream 510 is transmitted, And wherein said transmission is exchanged from said first set of frames to said second set of frames. When exchanging the transmission from the first set of frames to the second set of frames, the second set of frames includes a first exchange frame 523 in at least one second bit-stream, A second exchange frame 550 is transmitted, and the second exchange frame 550 is encoded using at least one reference frame from the first set of frames and the encoding parameters of the second set of frames. And the second exchange frame 550 is a reference frame used for reconstruction of the at least one predictive video frame 524 of the second set of video frames and a first exchange frame 523 in the second bit-stream. Instead of) The second exchange frame 550 is used to recover from transmission errors, the second exchange frame 550 is a prediction video frame, and the prediction information is prediction information from video frames that are earlier than the previous frame of the prediction video frame. Video information transmission method comprising a. A method for transmitting video information, wherein at least a bit-stream 510 is formed from the video information, the bit-stream 510 comprising a first set of frames comprising at least one video frame, and at least one A second set of frames comprising a predictive video frame 524 of At least partially different encoding parameters are used for encoding the frames of the first set of frames and the frames of the second set, At least one frame of the bit-stream 510 is transmitted, And wherein said transmission is exchanged from said first set of frames to said second set of frames. When exchanging the transmission from the first set of frames to the second set of frames, the second set of frames includes a first exchange frame 523 in at least one second bit-stream, A second exchange frame 550 is transmitted, and the second exchange frame 550 is encoded using at least one reference frame from the first set of frames and the encoding parameters of the second set of frames. And the second exchange frame 550 is a reference frame used for reconstruction of the at least one predictive video frame 524 of the second set of video frames and a first exchange frame 523 in the second bit-stream. Instead of) The second exchange frame (550) is used to recover from transmission errors, the second exchange frame (550) is a predictive video frame, and the prediction information includes only intra prediction information. 11. The method of claim 10, wherein both the first exchange frame 523 and the second exchange frame 550 in the at least one second bit-stream produce the same reconstruction result of the at least one prediction video frame 524. Video information transmission method, characterized in that. 11. The method of claim 10, wherein both the first exchange frame (523) and the second exchange frame (550) in the at least one second bit-stream have the same reconstructed values. Means for forming at least a first bit-stream 510 and a second bit-stream 520 from video information, the first bit-stream comprising a first set of frames comprising at least one video frame; The second bit-stream (520) comprises means for including a second set of frames comprising at least one predictive video frame (524); Means for using at least partially different encoding parameters to encode the frames of the first bit-stream (510) and the second bit-stream (520); Means for transmitting at least one frame of the first bit-stream (510); And An encoder comprising means for exchanging said transmission from said first bit-stream 510 to said second bit-stream 520, The means for exchanging the transmission from the first bit-stream 510 to the second bit-stream 520 converts the transmission from the first bit-stream 510 to the second bit-stream 520. Means for encoding a second exchange frame 550 using the reference frames from the first bit-stream 510 and the encoding parameters of the second bit-stream 520 to be interchangeable with Encoder comprising a. 14. An encoder according to claim 13, comprising means for generating prediction information using the reference frames (1670, 1675), and means for performing quantization and inverse quantization on the prediction information (1692, 1694). . 14. An encoder according to claim 13, comprising means (1670, 1675) for generating prediction information using the reference frames, and means (1690) for converting the prediction information. A decoder for decoding video information from a signal, The signal comprises frames from at least a first bit-stream 510 and a second bit-stream 520 formed from the video information, the first bit-stream comprising at least one video frame; Includes a set of frames, the second bit-stream 520 includes a second set of frames, including at least one predictive video frame 524, In the decoder wherein at least partially different encoding parameters are used for encoding the frames of the first bit-stream 510 and the second bit-stream 520, The decoder includes means for decoding the second exchange frame 550, The second exchange frame 550 is encoded using at least one reference frame from the first bit-stream 510 and the encoding parameters of the second bit-stream 520, and the second exchange frame. Frame 550 is a reference frame used for reconstruction of the at least one predictive video frame 524 of the second set of video frames instead of the first exchange frame 523 in a second bit-stream 520. Is added to the signal, Said means for decoding a second exchange frame 550 comprises means for using reference frames from said first bit-stream 510 and decoding parameters of said second bit-stream 520. Decoder. 17. The method of claim 16, wherein the first exchanged frame 513 in the first bit-stream is regions encoded by intra prediction using only spatial correlation and inter prediction using motion compensation. Includes regions encoded by The decoder Means for using motion compensation information in the reconstruction; Means for using spatial correlation information in the reconstruction; And And switching means for performing reconstruction of each region by the means using motion compensation information or by the means using spatial correlation information, depending on the prediction method used by the encoding of each region. Decoder, characterized in that. A computer-readable recording medium representing encoded video information and storing signal data including frames from at least a first bit-stream 510 and a second bit-stream 520 formed from the video information. , The first bit-stream includes a first set of frames including at least one video frame, and the second bit-stream 520 includes a second set of at least one predictive video frame 524. Including frames, In signal data wherein at least partially different encoding parameters are used for encoding the frames of the first bit-stream 510 and the second bit-stream 520, When exchanging the transmission from the first bit-stream 510 to the second bit-stream 520, the second bit-stream 520 is defined as one of at least one second bit-stream 520. 1 exchange frame 523, The signal data comprises at least one reference frame from the first bit-stream 510 and a second interchange frame 550 encoded using the encoding parameters of the second bit-stream 520 and , The second exchange frame 550 is a reference frame used for the reconstruction of the at least one predictive video frame 524 of the second set of video frames (the first exchange frame in the second bit-stream 520). 523) A computer readable recording medium, characterized in that being used instead. A decoder for decoding video information from a signal, The signal comprises at least one video block and at least one predictive video block, the predicted video block being predicted from the at least one video block, A decoder comprising a memory for storing reference information about previously decoded blocks, and a predictor for forming a block predicted using the reference information stored in the memory. A transformer for transforming the predicted block to form a transformed predicted block, and And an adder for adding the transformed predicted block with information representing a current block to obtain added information for use in decoding the current block. 20. The decoder of claim 19, further comprising an inverse quantizer and an inverse transformer for inversely quantizing and inversely transforming the current block after the addition. 21. The decoder of claim 19 or 20, comprising a quantizer for quantizing the transformed predicted block prior to the addition. 22. The decoder of claim 21, wherein the information indicative of the current block is obtained by transforming and quantizing at least a video block, wherein the transform and quantization are the same as the transform and quantization used for the predicted block. 21. The apparatus of claim 19 or 20, wherein the information indicative of the current block is obtained by transforming and quantizing at least a video block, and the decoder includes an inverse quantizer for dequantizing the information indicative of the current block prior to the addition. Decoder, characterized in that. 21. The apparatus of claim 19 or 20, wherein the information representing the current block is obtained by transforming and quantizing at least a video block, and the decoder includes a normalization block that scales the information representing the current block prior to the addition. Decoder, characterized in that. 24. The decoder of claim 23, comprising a quantizer, an inverse quantizer, and an inverse transformer for quantizing, inverse quantizing, and inverse transforming the current block after the addition. 20. The decoder of claim 19, wherein transform basis functions are determined to be used for the transform.

Images