KR20060069527A - Grouping of image frames in video coding - Google Patents

Abstract

A method for coding video frames for forming a scalable, compressed video sequence comprising video frames coded according to at least a first and a second frame format. The video frames of the first frame format are independent video frames, and the video frames of the second frame format are predicted from at least one of the other video frames. The video sequence has a first sub-sequence determined therein, at least part of the first sub-sequence being formed by coding at least video frames of the first frame format; and with at least a second sub-sequence, at least part of which is formed by coding video frames of the second frame format, and at least one video frame of the second sub-sequence having been predicted from at least one video frame of the first sub-sequence. Frame identifier data of the second sub- sequence is determined into the video sequence.

Description

Grouping of image frames in video coding

In the following, the invention will be described in connection with the accompanying drawings and preferred embodiments.

1 illustrates a general multimedia data streaming system to which the scalable coding layer of the present invention may be applied.

2 illustrates a scalable coding layer of a preferred embodiment of the present invention.

3A and 3B illustrate embodiments of the present invention for adjusting the scalability.

4A, 4B and 4C illustrate embodiments of the present invention for adjusting image numbering.

5A, 5B and 5C illustrate embodiments of the present invention for using B-frames in one scalable coding layer.

6A, 6B and 6C illustrate preferred scalable coding layers of the present invention in connection with reference picture selection.

7 illustrates a configuration according to a preferred embodiment of the present invention for coding scene transition.

The present invention relates to the grouping of multimedia files, and more particularly to the grouping of video files in connection with streaming.

The term 'streaming' refers to the simultaneous transmission and reproduction of data, which is usually multimedia data such as audio and video files of data, and the receiver already starts playing the data before all the data to be transmitted has been received. A multimedia data streaming system includes a streaming server and terminal devices typically used by the receiver to establish a data connection to the streaming server via a communication network. Recipients from the streaming server retrieve the stored or real-time multimedia data, and then the playback of the multimedia data can be started by the streaming application included in the terminal, most preferably in real time with the data transmission.

From the point of view of the streaming server, the streaming can be done as normal streaming or progressive downloading to the terminal. The transmission of multimedia data and / or data content during normal streaming ensures that the transmission bit rate substantially matches the playback rate of the terminal device, or when the communication network used for transmission causes difficulties in data transmission. It is controlled by ensuring that it substantially matches the bandwidth available for this communication network. The transmission of multimedia and / or data content in continuous downloading does not necessarily have to be interrupted at all, but the multimedia files are usually transmitted to the receiver by transport protocol flow control. The terminals then receive, store and play the same copy of the data sent from the server, which can then be played back later on this terminal without having to start streaming again over the communication network. Since multimedia files stored on the terminal are usually very large, the transfer of the files to the terminal is time consuming and requires huge storage memory capacity. This is often why generic streams are preferred.

Video files in multimedia files have numerous still picture frames, which are displayed in quick succession (usually 15 to 30 frames per second) to create the effect of the video. Image frames usually have a number of static background objects determined by image information that remains substantially unchanged, and some dynamic objects determined by somewhat varying image information. The information organized by successively displayed image frames is generally very similar. That is, a series of image frames contain a significant amount of redundancy. The extras that appear in the video files are divided into spatial, temporal, and spectral extras. Spatial redundancy relates to the cross-correlation of adjacent image pixels, temporal redundancy relates to changes occurring in particular image objects in subsequent frames, and spectral redundancy relates to the correlation of different color components in one image frame.

In order to reduce the amount of data in the video files, the image data can be compressed into smaller forms by reducing the amount of redundant information in the image frames. In addition, during encoding, most video encoders currently used deteriorate the picture quality of image frame sections that are less important in video information. In addition, many video coding methods allow the redundancy of the coded bit stream from the image data to be reduced by efficient and lossless coding of compression parameters known as VLC (variable coding).

In addition, many video coding methods utilize the aforementioned temporal redundancy of a series of image frames. In this case, a scheme known as motion-compensated time prediction is used. That is, the contents of some (usually most) of the image frames in a video sequence are predicted from other frames in the sequence by tracking changes in specific objects or regions of subsequent image frames. The video sequence always has some compressed image frames whose image information has not been determined using motion-compensated temporal prediction. Such frames are called INTRA-frames, or I-frames. Similarly, motion-compensated video sequence image frames predicted from previous image frames are called INTER-frames, or P-frames (Predicted). Image information of P-frames is determined using one I-frame and possibly one or more previously coded P-frames. If one frame is lost, the frames that depend on that frame can no longer be decoded correctly.

An I-frame usually produces a video sequence defined as a Group of Pictures (GOP), whose P-frames can only be determined based on the I-frame and the previous P-frames of that GOP. . The next I-frame starts a new picture group (GOP), and thus the provided image information cannot be determined based on the frames of the previous GOP. That is, picture groups are not overlapped in time, and each picture group can be decoded separately. In addition, many video compression schemes use bi-predictive B-frames, which are two anchor frames (I- and P-frames, or two P-) in a picture group GOP. Frames), the image information of the B-frame is predicted from both the previous anchor frame and what follows the B-frame. Thus, B-frames provide higher quality image information than P-frames, but are not generally used as anchor frames, so removing B-frames from a video sequence does not degrade the quality of a series of images. . However, nothing prevents B-frames from being used also as anchor frames, in which case B-frames cannot be removed from the video sequence without degrading the picture quality of the frames dependent on the B-frames. .

Each video frame can be divided into what is known as macroblocks containing the color components (such as Y, U, V) of all pixels in a rectangular image area. More specifically, one macroblock consists of at least one block per color component, each of which contains color values (such as Y, U, or V) of one color level in the associated image region. The spatial resolution of the blocks may differ from the resolution of the macroblocks, for example, the U- and V components can be displayed using only half of the Y-component resolution. Macroblocks can be further grouped into slices, for example, groups of macroblocks that are generally selected in the scanning (scanning) order of an image. Temporal prediction is performed specific to a block or macroblock instead of being specific to an image frame in video coding methods.

To allow for flexible streaming of video files, many video coding systems use scalable coding, where some elements or groups of elements in a sequence affect the reconstruction of other parts of the video sequence. Can be removed without reaching. Scalability is usually implemented by grouping image frames into multiple hierarchical layers. Image frames coded with image layers of base layer scalability have only those that are forced for decoding video information at the receiving end. Thus, the base layer of each picture group GOP includes one I-frame and the required number of P-frames. One or more enhancement layers may be determined below the base layer, each of which enhances the quality of video coding compared to the upper layer. Thus, the enhancement layer includes predicted P- or B-frames based on motion-compensation from one or more higher layer images. The frames are usually numbered according to the hydraulic series.

In streaming, the transmission bit rate must be able to be controlled based on the bandwidth to be used or the maximum decoding or bit rate value of the receiver. The bit rate can be controlled by any component of the communication network, such as a streaming server, an internet router or a base station of a mobile communication network. The simplest means of a streaming server to control the bit rate is to omit B-frames with highly informational content in transmission. In addition, the streaming server may determine the number of scalable layers to be transmitted in the video stream, such that the number of scalable layers will always change when a new picture group GOP is started. It is also possible to use different video sequence coding methods. As such, B-frames may be removed from the bit stream of the communication network component like other P-frames of the enhancement layers.

The above configuration involves several defects. Many coding methods, such as the ITU-T (International Telecommunication Union, Telecommunication Standardization Sector) standard H.263, are familiar with a procedure called reference picture selection. At least a portion of the P-image in selecting the reference picture was predicted from at least one other image than the P-frame just before the time domain. The selected reference image is a coded bit stream or bit stream header field, which is specific to the image, specific to the image-segment (such as a slice or group of macroblocks), specific to a macroblock, or signal specific to the block. . The reference picture selection can be generalized so that prediction can also be made from images subsequent to the image to be coded. In addition, reference picture selection may be generalized to cover all temporally predicted frame types, including B-frames. Since it is also possible to select at least one image preceding the I-image starting the picture group GOP as the reference image, the picture group using the reference picture selection may not necessarily be decoded independently. In addition, scaling or coding schemes become difficult to adjust in streaming servers or network components, which allows video sequences to be decoded, analyzed and buffered over long periods of time to ensure that any dependencies between different image groups can be detected. Because it must be.

Another problem involves inserting one video sequence in the middle of another video sequence, which usually caused discontinuities in image numbering. Numbering of video sequence images is generally used to detect loss of image frames. However, if a separate video sequence, such as a commercial, is inserted into a video sequence, that separate video sequence is usually given as a separate image numbering, which is consistent with the ascending image numbering of the original video sequence. I never do that. Therefore, the receiving terminal interprets the mismatched image numbering as a signal of lost image frames and starts an unnecessary operation to request to reconstruct or retransmit the image frames estimated to be lost.

A similar problem can be encountered when buffering and indexing reference pictures at the encoder and decoder. As part of indexing, the encoder can intentionally remove a plurality of useless image frames as reference image frames for the temporal prediction process of any remaining image frames. There is also no process for informing the decoder of the reason for such removal: For example, if a plurality of image frames are removed, the receiving terminal may interpret this deliberate removal as unnecessary a protocol error.

It is an object of the present invention to provide an improved method and apparatus for implementing the method that enables the decoder to take into account image frames that the encoder deliberately removed.

Now, various aspects of the present invention include a method, a video system, a video decoder, an apparatus for adjusting the bit rate of a video signal, a video sequence, and preferably a streaming system component, characterized in what is stated in the independent claims. Computer programs are included.

Preferred embodiments of the invention are disclosed in the dependent claims.

The present invention is based on the concept of decoding a compressed video sequence, wherein image frames are input into a buffer memory associated with decoding. The video sequence includes an indication associated with at least one discontinuity in the numbering (numbering) of the image frames, the indication being decoded from the video sequence. Then, in response to the indication, the buffer memory is set to provide image frames corresponding to the discontinuities in the numbering of the image frames, and the image frames in the buffer memory are used in the decoding process. The indication is provided as information that at least one discontinuity is intentional in the numbering of the imposition frames in the video sequence, and it is preferred that image frames generated in the buffer memory be used in place of image frames not present in the decoder. Missing image frames cause discontinuities in image frame numbering.

According to one embodiment, after the buffer memory is set to provide image frames corresponding to discontinuities in image frame numbering, the input of the image frames included in the video sequence into the buffer memory is resumed from the correct image frame number.

According to one embodiment, filler frames corresponding to discontinuities in image frame numbering are input to the buffer memory. The supplemental frames are indicated by an identifier indicating that they do not belong to the actual video sequence.

According to one embodiment, the step of setting the buffer memory to provide image frames corresponding to discontinuities in image frame numbering is performed by a memory indication, so that no data is input into the buffer memory.

An advantage of the procedure according to the present invention is that the decoder does not start unnecessary error correction operations by providing the decoder with information that is gap and intentional in image frame numbering. Another advantage is that the buffer memories of the encoder and decoder can be kept in synch, thereby enabling a rough operation for the reference picture selection process.

In the following, a general purpose multimedia data streaming system is disclosed, the basic principles of which can be applied in connection with any telecommunication system. The present invention is described herein with particular reference to a streaming system where multimedia data is most preferably transmitted over a telecommunications network using a packet-switched data protocol, such as an IP scheme, but the present invention provides a fixed telephone network PSTN. It can be equally well implemented in circuit-switched networks such as / SDN (public switched telephone networks / integrated services digital networks) or in mobile telecommunication networks PLMNs (public land mobile networks). The invention can also be applied in the streaming of multimedia files in the form of both general streaming and continuous downloading, for example to implement a video call.

Although the invention is described herein with particular reference to streaming systems and is preferably applied within the systems, the invention is not limited to streaming systems, but the invention is not limited to how and where a video file to be decoded is downloaded. Can be applied within a video playback system. Thus, the present invention can be applied to the playback of video files to be downloaded from a DVD disc or any other computer memory carrier, for example with respect to the variable processing capacity available for video playback. In particular, the present invention is applicable to various low bit rate video codings typically used in telecommunication systems requiring bandwidth limitations. One example is the system defined in ITU-T standard H.263 and H.26L (possibly later H.264). In this regard, the present invention can be applied, for example, to mobile stations, in which case, even when the mobile station is used to execute an application other than video playback, video playback can vary both in varying transmission capacity or channel quality and in the processor power currently in use. It can be done to fit about.

For clarity, it should be further noted that the present invention will be described below by providing a description of image frame coding and temporal prediction of image frame levels. In practice, however, coding and temporal prediction typically occur at the block or macroblock level as described above.

Referring to Figure 1, a general multimedia streaming system is shown, which is a preferred system for applying the procedure of the present invention.

A multimedia data streaming system typically includes one or more multimedia sources 100, such as video images and computer graphics files stored in a video camera and microphone, or a memory carrier. Raw data from different multimedia sources 100 are combined into a multimedia file of encoder 102, which may also be called an editor. Raw data arriving from one or more multimedia sources 100 is first captured using capture means 104 contained within encoder 102, which capture means 104 typically comprise different interface cards, driver software, Or as application software that controls the function of the card. For example, video data can be obtained using a video capture card and its associated software. When a video capture card is involved, the output of the capture means 104 is usually uncompressed or slightly compressed data, such as, for example, uncompressed video frames in the YUV 4: 2: 0 format or the Motion-JPEG image format. It may be a flow.

The editor 106 links the different media flows together so that the video and audio flows can be played simultaneously as desired. The editor 106 also edits each media flow, such as a video flow, by halving the frame rate or reducing the spatial resolution. Although synchronized, the independent media flows are compressed in the compressor 108, where each media flow is compressed independently using a compressor suitable for that media flow. For example, video frames in YUV 4: 2: 0 format will be compressed using low bit rate video coding according to ITU-T Recommendation H.263 or H.26L. Individually synchronized compressed media flows are usually interleaved in the multiplexer 110, and the output obtained from the encoder 102 is a single regular bit flow that can be called a multimedia file with data consisting of a plurality of media flows. Note that creation of a multimedia file does not necessarily require multiplexing multiple media flows into one file. However, the streaming server will interleave the media flows before sending them.

The multimedia files are transmitted to the streaming server 112, and the streaming server may perform streaming in a real time streaming or continuous downloading format. In continuous downloading, the multimedia files are first stored in the memory of the server 112, and when demand arises, the multimedia files are retrieved from the memory for transmission. In real time streaming, the editor 102 sends a continuous media flow of multimedia files to the streaming server 112, which sends the flow directly to the client 114. As an additional option, real-time streaming can also be performed such that the multimedia files are stored in storage accessible from server 112, where real-time streaming can be done from that storage and a continuous media flow of multimedia files commences when demand arises. do. In this case, the editor 102 does not necessarily control streaming. The streaming server 112 performs traffic shaping of the multimedia data in terms of the available bandwidth or the maximum decoding and refresh rate of the client 114, which may omit B-frames or adjust the number of scalable layers, for example, in transmission. By doing this, the bit rate of the media flow can be adjusted. In addition, streaming server 112 transforms the header fields of the multiplexed media flow to reduce its size and wrap it with data packets suitable for transmission in a telecommunications network that is using multimedia data. The client 114 may generally coordinate the operation of the server 112 to at least some extent using an appropriate control protocol. The client 114 can control the server 112 at least in such a way that the desired multimedia file can be selected to be sent to the client, and in addition, the client can typically interrupt and interrupt the transmission of the multimedia file.

When the client 114 is receiving a multimedia file, the file is first sent to a demultiplexer 116, where the media flows that make up the multimedia file are separated. The separated, compressed media flows are then sent to a decompressor 118 where each separated media flow is decompressed by a decompressor suitable for each. The decompressed and reconstructed media flows are sent to playback unit 120 where they are rendered at the correct pace by their synchronization data and sent to display means 124. Actual display means 124 may comprise, for example, a computer or mobile station display and speaker means. The client 114 also includes a control unit for controlling both the operation of the server through the above-described control protocol and the operation of the playback unit 120 based on the command issued by the end user, which is normally controlled by the end user, usually through a user interface. 122).

It should be noted that the transmission of multimedia files from the streaming server 112 to the client 114 takes place over a telecommunications network, the transmission path of which generally includes a plurality of telecommunications network components. Thus, there may be at least some network components that can perform traffic shaping of multimedia data in terms of available bandwidth or maximum decoding and refresh rate of the client 114 in the same way as at least some of those described above in connection with streaming servers. .

Scalable coding will be described below with reference to the example shown in FIG. 2 and preferred embodiments of the present invention. FIG. 2 is a compressed video having an independently determined video frame, the first frame 200 being an intra frame, or an I-frame, whose image information is determined without using motion-compensated temporal prediction. Show part of the sequence. I-frame 200 is placed on a first scalable layer, which may be called an INTRA layer. Each scalable layer is assigned a unique identifier, such as a layer number. The intra layer may thus be given a number 0, for example another letter / number identifier, such as any one character, or an identifier in the form of a combination of letters and numbers.

Similarly, sub-sequences of groups of one or more video frames are defined for each scalable layer, and at least one of the images in the group (usually the first or last) is at least usually the other parent, Or temporally predicted from video frames of the same scalable layer, and the remaining video frames are typically temporally only from video frames of the same sub-sequence, or perhaps from one or more video frames of the second sub-sequence. Is predicted. One sub-sequence may be independently decoded independently of other sub-sequences except the second sub-sequence. Sub-sequences of each scalable layer are assigned a unique identifier using consecutive numbering, such that the first sub-sequence of that scalable layer begins with the number zero. The I-frame 200 is also formed in a separate sub-sequence manner since it can be determined independently and can be decoded independently upon receipt, independent of other image frames.

A practical aspect of the present invention is to determine each sub-sequence in terms of the sub-sequences on which the sub-sequence is dependent. That is, one sub-sequence contains information about all sub-sequences that were used immediately to predict the image frames of that sub-sequence. Since this information is signaled as a video sequence bit stream and preferably separate from the actual image information, it is easy to determine the portion of the video sequence that can be decoded and removed independently without affecting the decoding of the remaining image data. Image data of the video sequence can be preferably adjusted.

Next, within each sub-sequence, video frames of that sub-sequence are given image numbers using consecutive numbering, for example, beginning with the number 0 being given to the first video frame of the bus-sequence. Since I-frame 200 also forms an independent sub-sequence, its image number is zero. In FIG. 2, I-frame 200 is shown in type (I), which is the subsequence identifier and image number (0.0) of the frame.

2 also shows the next I-frame 202 of the intra layer, which is also a uniquely defined video frame that has been determined without using motion-compensated temporal prediction. The temporal transmission frequency of I-frames depends on many factors related to video coding, image information content and bandwidth to be used, and depending on the application or application environment, the I-frames are transmitted in video sequences at intervals of, for example, 0.5 to 10 seconds. Since I-frame 202 can be decoded independently, it also forms its own sub-sequence. Since this is the second sub-sequence of the intra layer, the consecutive number for the sub-sequence identifier of I-frame 202 is one. In addition, since the I-frame 202 also forms its own sub-sequence, that is, it is the only video frame in that sub-sequence, its image number is zero. Thus, I-frame 202 may be designated with an identifier (I.1.0). Similarly, the identifier of the next I-frame of the intra layer is (I.2.0), which continues to be identified in this way. As a result, only I-frames that are uniquely defined are not coded using motion-compensated temporal prediction, but coded within the first scalable layer, i.e., the intra layer. If the sub-sequences can be distinguished from each other, they may be determined using different kinds of numbering schemes or other identifiers.

For example, when the next scalable layer with layer number 1 is called the base layer, the layer is coded and generally only from previous image frames (ie in this case from I-frames of the upper intra layer). ) With predicted motion-compensated inter (INTER) or P-frames. Image information of the first P-frame 204 of the base layer illustrated in FIG. 2 is determined using the I-frame 200 of the intra layer. P-frame 204 starts the first sub-sequence of the base layer, such that the sub-sequence identifier of P-frame 204 becomes zero. Further, since the P-frame 204 is the first image frame of the first sub-sequence of the base layer, the image number of the P-frame 204 is zero. Therefore, the P-frame 204 may be identified as (P.0.0).

The P-frame 206 of the base layer following in time is predicted from the previous P-frame 204. P-frames 204 and 206 thus belong to the same sub-sequence, so that P-frame 206 also receives sub-sequence identifier 0. Since P-frame 206 is the second image frame of sub-sequence 0, since P-frame 206 has an image number of 1, P-frame 206 may be identified as (P.0.1).

The scalable layer with layer number 2 following the base layer is called enhancement layer 1. This layer is coded and has motion-compensated P-frames that are predicted only from previous image frames that are I-frames of the intra layer or P-frames of the base layer. 2 shows a first image frame 208 and a second image frame 210 of enhancement layer 1, both of which are predicted only from the first image frame 200 of the intra layer. P-frame 208 starts the first sub-sequence of enhancement layer 1, so that sub-sequence identifier of the P-frame becomes zero. Also, since P-frame 208 is the first image frame of the sub-sequence, P-frame 208 receives image number zero. Therefore, the P-frame 208 can be identified as (P.0.0).

Since the second image frame 210 is also predicted only from the first image frame 200 of the intra layer, the P-frame 210 starts the second sub-sequence of enhancement layer 1 and the P-frame 210. The sub-sequence identifier of is thus one. Since P-frame 210 is the first image frame of its sub-sequence, the image number of P-frame 210 is zero. The P-frame can thus be identified as (P.1.0). P-frame 212 of enhancement layer 1 that follows in time is predicted from the previous P-frame 210. Since P-frames 210 and 212 belong to the same sub-sequence, that P-frame 212 also receives sub-sequence identifier 1. Since P-frame 212 is the second image frame of sub-sequence 1, P-frame may receive image number 1 and may be identified as (P.1.1).

The temporally fourth image frame 214 of enhancement layer 1 is predicted from the first image frame 204 of the base layer. Thus, the P-frame 214 starts the third sub-sequence so that the P-frame 214 receives the sub-sequence identifier 2. Also, since P-frame 214 is the first and only image frame in its sub-sequence, the image number of P-frame 214 is zero. Therefore, the P-frame 208 can be identified as (P.2.0).

Since the temporally fifth image frame 216 of enhancement layer 1 is also predicted only from the first image frame 204 of the base layer, the P-frame 216 is adapted to the fourth sub-sequence of enhancement layer 1. Beginning, the sub-sequence identifier of P-frame 216 becomes three. Also, since the P-frame 216 is first in the sub-sequence, the image number of the P-frame 216 is zero. Therefore, the P-frame 216 may be identified as (P.3.0). The P-frame 218 of the enhancement layer 1 that follows in time is predicted from the previous P-frame 216. Since P-frames 216 and 218 belong to the same sub-sequence, the sub-sequence identifier of P-frame 218 also becomes three. Since P-frame 218 is the second image frame of sub-sequence 3, the image number of P-frame 218 is 1 and therefore the identifier of P-frame 218 is (P.3.1).

For simplicity and clarity of explanation, what is disclosed above associates only I- and P-frames. However, those skilled in the art will realize that the scalable video coding of the present invention is also implemented using the aforementioned B-frames and at least other existing image frame types such as SI-frames, SP-frames and MH-frames. It will be appreciated. SI-frames correspond to I-frames, but enable the same image reconstruction with an SP-frame. An SP-frame is a P-frame that experiences certain coding that allows the same image to be reconstructed with an SI-frame or another SP-frame. SP-frames are generally placed in a video sequence into points where one access point or scanning point is desired or where changes in coding parameters of the video stream should be possible. Frames may be used for error correction and error tolerance enhancement. SP-frames are the same as normal P-frames predicted from previous frames, except that they are specified so that they can be replaced with other video frames of the SP- or SI-type, and the decoding result for the new frame is the video This will be the same as the decoding result of the original SP-frame in the stream. That is, a new SP-frame used to replace what was in the video stream is predicted from another sequence or video stream and the reconstructed frame contains the same content. SP-frames are described, for example, in our previous application PCT / FI02 / 00004.

Similar to B-frames, macroblocks of MH (Multi Hypothesis) frames based on motion-compensated prediction are predicted from two different frames, but not necessarily located after one MH-frame. More specifically, the predicted macroblocks are calculated as the average of two macroblocks of two different frames. Instead of two frames, MH-frame macroblocks may naturally be predicted from another frame. Reference images may vary with macroblocks, ie, not all macroblocks in one and the same image are predicted using exactly the same frames.

The sub-sequence therefore covers some particular time period of the video sequence. Sub-sequences of the same layer or other layers may overlap some or all. If temporally overlapping image frames exist on the same layer, they are interpreted as alternative representations of the same image content so that any mode of image representation can be used. On the other hand, if there are temporally overlapping image frames in different layers, the image frames produce different representations for the same image content, and thus the representations differ in image quality. In other words, the image quality is better at the lower layers.

The above description with reference to FIG. 2 illustrates a scalable coding scheme, hierarchy and numbering of image frames in accordance with an embodiment of the invention. In this embodiment, only the intra layer has I-frames, and the base layer can only be decoded using information received from the intra layer. Similarly, decoding of enhancement layer 1 generally requires information from both the base layer and the intra layer.

The number of scalable layers is not limited to three as above, and any number of enhancement layers may be used if deemed necessary to provide sufficient scalability. As a result, the number of layers of enhancement layer 2 is four, and the number of layers of enhancement layer 3 is five. Some of the image frames in the above example are given the same identifier (e.g., all of the image frames 204 and 208 have their identifier (P.0.0)), so that each image frame is unique by including a layer number in the identifier. It can be identified and at the same time a dependency on other image frames of each image frame is preferably determined. Thus, each image frame is uniquely identified, and the identifier of the image frame 204 is, for example, (P.1.0.0), or simply (1.0.0), and similarly, the identifier of the image 208 is (P.2.0). .0), or (2.0.0).

According to a preferred embodiment of the present invention, the number of reference image frames is determined according to certain predetermined alphanumeric series, for example an integer between 0 and 255. When the parameter value reaches the maximum value N (eg 255) of the relevant series, the determination of the parameter value starts from the beginning, ie from the minimum value of the series (eg 0). Thus, image frames are uniquely identified within a particular sub-sequence up to the point where the same image number is used again. The sub-sequence identifier may also be determined according to a particular, certain mathematical sequence. When the value of the sub-sequence identifier reaches the maximum value N of the series, determining the identifier starts again from the beginning of the series. However, sub-sequences cannot be assigned identifiers in use (in the same layer). The series in use may also be determined in a manner other than mathematically. Another alternative is to assign random sub-sequence identifiers in consideration that the assigned identifier is not used again.

The problem of numbering for image frames occurs when the user wants to start searching for a video file in the middle of a video sequence. For example, when a user wishes to search back and forth a locally stored video file or search for a streaming file at a specific point; When the user starts playing the streaming file from one random point; Or such cases occur when it is detected that the video file to be played contains an error that prevents its playback, or when it is necessary to resume playback at a point following the error. When a video file is resumed at one random point after a previous search, discontinuities in image numbering usually occur. In general, the decoder interprets this as an unintentional loss of image frames and unnecessarily attempts to reconstruct the image frames that it assumes are lost.

According to a preferred embodiment of the invention, this can be avoided by defining, at the decoder, a starting image in the independently decodable picture group GOP, which is activated at a random point in the video file, the starting image number being zero. It is decided. Thus the independently decodable image group may be a sub-sequence of the intra layer, for example in this case an I-frame is used as its starting image, and the independently decodable image if scaling takes place in the base layer is used. The group is a sub-sequence of the base layer, in which case the first image frame of that sub-sequence, which is typically an I-frame, is usually used as the starting image. As a result, when activated at a random point, the decoder sets the first image frame, preferably the I-frame, of the independently decodable sub-sequence to zero. Since the sub-sequence to be decoded may also have other image frames with an identifier of zero (eg when the above-mentioned alpha-numeric series starts from the beginning), the beginning of the sub-sequence, ie the first image frame is part of that image frame. It may be indicated to the decoder by an individual flag added to the header field. This allows the decoder to correctly interpret the image numbers and to know the correct image frame starting the sub-sequence from the video sequence image frames.

The numbering system described above provides only one example of how to perform the unique image frame identification of the present invention and thus simultaneously indicate interdependence between image frames. However, video coding methods to which the method of the present invention can be applied, such as video coding methods according to the ITU-T standards H.263 and H.26L, use code tables that use variable length codes. When variable length codes are used for coding layer numbers, for example, a lower codeword index, i.e., a small layer number, indicates a shorter codeword. Indeed the scalable coding of the present invention will be used in most cases so that the base layer can consist of much more image frames than the intra layer. This justifies the use of a lower index, i.e., a smaller layer number, on the base layer than on the intra layer, since the amount of coded video data is preferably reduced accordingly. Eventually, the intra layer is assigned layer number 1 and the base layer is assigned layer number 0. Alternatively, the code can be made using fewer bits to code the base layer number than the intra layer number, in which case the actual layer number value is independent of the generated code length side.

Further, according to the second preferred embodiment of the present invention, when the number of scalable layers should be kept low, in particular, the first scalable layer can be coded to have both an intra layer and a base layer. From the point of view of coding hierarchical, the simplest way to plan this is to exclude the entire intra layer, and the image information is not determined using motion-compensated temporal prediction, and the image frames are identical to the independently defined I-frames. Coding frames consisting of all of the image frames expected from previous frames, which are motion-compensated P-frames predicted from the I-frames of the layer, are provided to the base layer. Thus layer number 0 can be used for the base layer, and when enhancement layers are coded in a video sequence, enhancement layer 1 is assigned layer number 1. This will be illustrated below with reference to FIGS. 3A and 3B.

3A shows a non-scalable video sequence structure in which all image frames are located on the same scalable layer, ie, the base layer. The video sequence has a first image frame 300 that starts the first sub-sequence with an I-frame (I.0.0). Image frame 300 is used to predict the second image frame 302 of that sub-sequence, ie P-frame (P.0.1), and frame 302 is the third image frame 304 of its server-sequence. In other words, it is used to predict the P-frame (P.0.2), and the frame 304 is used to predict the next image frame 306, ie the P-frame (P.0.3). The video sequence is then provided with an I-frame (I.1.0), that is, an I-frame 308, coded therein, which starts a second sub-sequence of the video sequence. This kind of non-scalable coding may be used, for example, when the application in use does not allow or need no use of scalable coding. In circuit-switched video phone applications, for example, the channel bandwidth remains constant and the video sequence is coded in real time, so there is usually no need for scalable coding.

3B shows an example of how scalability can be added to intra- and base layer combining when needed. Here too, the video sequence base layer is an I-frame (I.0.0) and comprises a first image frame 310 which starts the first sub-sequence of the base layer. The image frame 310 is used to predict the second image frame 312 of the sub-sequence, ie P-frame (P.0.1), and the second image frame 312 is the third image frame 314 of the sub-sequence. ), Ie, to predict the P-frame (P.0.2). However, enhancement layer 1 is also coded in this video sequence and includes the first sub-sequence. The only image frame 316 that is only one of the first sub-sequences is a P-frame (P.0.0), which is predicted from the first image frame 310 of the base layer. The first image frame 318 of the second sub-sequence of the enhancement layer is predicted from the second image frame 312 of the base layer, so that the P-frame is (P.1.0). The next image frame 320 of the enhancement layer is predicted again from the previous image frame 318 of the same layer and belongs to the same sub-sequence, so that the identifier is (P.1.1).

In this embodiment of the invention, even if one base layer sub-sequence is dependent on another base layer sub-sequence, the sub-sequences of the base layer can be decoded independently. Decoding of base layer sub-sequences requires information from the second sub-sequence of base layer and / or enhancement layer 1, wherein the sub-sequences of enhancement layer 2 are enhancement layer 1 and / or enhancement layer. An equation that requires information from a sub-sequence of two is given. According to an embodiment, I-frames are not limited to the base layer alone, and enhancement layers may also have I-frames.

The basic idea behind the above embodiments is that one sub-sequence for all sub-sequences that it was used to predict at least one of the image frames of the sub-sequence in question, i.e. To have information. However, according to one embodiment, all sub-sequences for all sub-sequences dependent on that sub-sequence, ie all subs with at least one image frame that were predicted using at least one image frame of that sub-sequence. It is also possible to include information about sequences. In the latter case, since the dependency is usually determined forward in time, image frame buffers may be preferably utilized in coding according to the manner described below.

In all the above embodiments the numbering of the image frames is specific to the sub-sequence. The new sub-sequence always starts numbering from the beginning. Thus identification of individual image frames requires a layer number, a sub-sequence identifier, and an image frame number to be determined. According to a preferred embodiment of the present invention, image frames may be independently numbered using a continuous numbering scheme in which consecutive reference image frames in coding order are represented as weighted numbers one by one. With regard to layer numbers and sub-sequence identifiers, the above-mentioned numbering scheme can be used. This allows each image frame to be uniquely identified whenever needed without using layer numbers and sub-sequence identifiers.

This is illustrated by the example shown in FIG. 4A where the base layer includes the first I-frame 400 (I.0.0) in time. This frame is used to predict the first image frame 402 of the enhancement layer 1, i.e. (P.0.1), the first image frame 402 belonging to the same sub-sequence (sub-sequence identifier 0). Used to predict two image frames 404, i.e. (P.0.2), second image frames 404 are used to predict third image frames 406, i.e. (P.0.3), of the same sub-sequence, The third image frame 406 is used to predict the fourth image frame 408 (P.0.4), and finally, the fourth frame is used to predict the fifth image frame 410 (P.0.5). The next video sequence image frame 412 in time is located above the base layer, where it is in the same sub-sequence as I-frame 400, but in time it is only the seventh coded image frame, so the identifier is (P. 0.6). The seventh frame is used to predict the first image frame 414 of the second sub-sequence of enhancement layer 1, i.e. (P.1.7), and the first image frame 414 is the same sub-sequence (sub-sequence). Used to predict the second image frame 416 (P.1.8) belonging to the sequence identifier 1), the second image frame 416 is used to predict the third image frame 418 (P.1.9), The third image frame is used to predict the fourth image frame 420 (P.1.10), and finally the fourth image frame is used to predict the fifth image frame 422 (P.1.11). Again, in time the next video sequence image frame 424 is located on the base layer, which is in the same sub-sequence as the I-frame 400 and the P-frame 412, but in time the thirteenth coded image. Since it is only a frame, its identifier becomes (P.0.12). For the sake of clarity of illustration, the description of the embodiment did not include layer identifiers, but in order to implement scalability, it is obvious that layer identifiers should usually be combined and signaled with the video sequence as part of the image frame identifiers. .

4B and 4C show other optional embodiments for grouping image frames of the video sequence shown in FIG. 4A. The image frames of FIG. 4B are numbered according to sub-sequences. That is, the new sub-sequence always starts numbering from the beginning (from zero). FIG. 4C otherwise uses image frame numbering corresponding to that used in FIG. 4A except that P-frames in the base layer are replaced with SP-frame pairs to enable the same reconstruction of the image information.

As mentioned above, the scheme of the present invention may be implemented using B-frames. One example of this is shown in FIGS. 5A, 5B, and 5C. 5A shows one video sequence in the time domain. The sequence includes P-frames P1, P4, and P7 with B-frames lying between them, and the dependencies of the B-frames in relation to temporal prediction are shown as arrows. FIG. 5B shows a preferred grouping of video sequence image frames, with the interdependencies shown in FIG. 5A being indicated. 5B shows image frame numbering specific to a sub-sequence, where the new sub-sequence always starts numbering image frames from zero. FIG. 5C shows image frame numbering consecutive in temporal prediction order, where subsequent reference frames always receive the next image number as the previous encoded reference frame. Image frame B1.8 (and (B2.10)) is not provided as a reference prediction frame in any other frame, and thus does not affect image frame numbering.

The above examples illustrate different optional alternatives to how the scalability of video sequence coding can be adjusted using the method of the present invention. From the perspective of the terminal device playing the video sequence, the more scalable layers are available, or the more scalable layers can be decoded, the better the picture quality. That is, an increase in the amount of image information and the bit rate used to transmit that information improves the temporal or spatial resolution or spatial quality of the image data. Likewise, a larger number of scalable layers also place even higher demands on the processing capacity of the terminal device to perform the decoding.

Again, the examples illustrate the benefits obtained by using sub-sequences. With image frame identifiers, dependencies on the other image frames of each image frame in the subsequence are indicated in a clear manner. The bus-sequence thus forms an independent entity that can be missed from the video sequence when needed without affecting the decoding of the series of image frames of the video sequence. In that case, only image frames of that sub-sequence and other sub-sequences on the lower scalable layers that are the same and / or dependent thereon are not decoded.

The image frame identifier data transmitted with the video sequence is preferably included in the video sequence header fields or in the header fields of the transport protocol to be used for video sequence transmission. That is, the identifier data of the predicted image frames are not included in the image data of the coded video sequence, and always fall into the header fields, so that the dependencies of the image frames can be detected without decoding the images of the actual video sequence. Identifier data of the image frames may be stored, for example, in a buffer memory of the streaming server when the video sequence is being coded for transmission. In addition, the sub-sequences can be decoded independently on each scalable layer, since an image frame of one sub-sequence is not dependent on other sub-sequences of the same scalable layer.

According to an embodiment of the present invention, image frames included in a sub-sequence may depend on other sub-sequences of the same scalable layer. This dependency must then be signaled to a streaming server or the like that performs traffic shaping, since independent sub-sequences located on the same layer cannot be removed separately from the video sequence to be transmitted. A preferred way of performing signaling is to list the layer-sub-sequence pairs on which the sub-sequence depends, so as to include that dependency in the image frame identifiers to be transmitted. This also provides a preferred way of indicating dependencies from other sub-sequences of the same scalable layer.

The above examples illustrate the situation in which image frames are predicted in time from previous image frames. In some coding schemes, however, reference picture selection has been further extended to include predicting image information of image frames from temporally subsequent image frames. Reference picture selection provides the most diversified means for generating temporally different scalable image frame structures and allows the error sensitivity of the video sequence to be reduced. One of the coding techniques based on reference picture selection is intra frame postponement. An intra frame does not sit in a "right" position in time within the video sequence, but that position follows in time. Video sequence image frames that are between the "right" position and the actual position of an intra frame are predicted in the direction temporally backward from the intra frame. This naturally requires that the uncoded image frames be buffered for a sufficiently long period of time so that all image frames to be displayed can be coded and placed in the order of reproduction. In accordance with the present invention, determination of intra frame transmission and its associated sub-sequences will be described below with reference to FIG. 6.

FIG. 6A shows a portion of a video sequence, where the intra frame includes a single I-frame 600, which has been moved in time to the position shown in FIG. 6, but the “right” position in the video sequence is the first image. It would have been a frame. Video sequence image frames between the "correct" position and the actual position 600 are predicted in time backwards. This is illustrated by the sub-sequence coded in enhancement layer 1 and includes a P-frame (P.0.0), which is the first predictive image frame 602 temporally backwarded. This frame is used for temporal prediction of the previous image frame 604, i.e., P-frame (P.0.1), and the image frame 604 predicts the image frame 606, i.e., P-frame (P.0.2). Finally, frame 606 is used to predict an image frame 608, i.e., a P-frame (P.0.3) at a position that would have been the "right" position of I-frame 600 in this video sequence. do. Similarly, I-frame 600 on the base layer is divided into four P-frames 610, 612, 614, 616 (P.0.0), (P.0.1), (P.0.2), and (P. 0.3) is used for prediction of the temporally advancing direction of the sub-sequence.

In this example, the fact that retrograde predictive image frames are located in a lower layer than forward predictive image layers, for purposes of illustration, retrograde predictive image frames are considered to be subjectively less important in this coding example than forward predictive image frames. It is present. Naturally, the sub-sequences can both be located in the same layer, in which case they are considered equal, and if the retrograde prediction sub-sequence is in a higher layer, the sequence can be considered subjectively more important.

6b and 6c illustrate some other options for coding the video sequence according to FIG. 6a. In FIG. 6B the forward and backward prediction sub-sequences are located above the base layer and the I-frame is located only in the intra layer. The forward prediction sub-sequence of this layer is the second sub-sequence and its sub-sequence identifier is one. In FIG. 6C, the I-frame and the forward prediction sub-sequences based thereon are located in the base layer, while the backward prediction sub-sequence is located in enhancement layer 1. FIG.

Further, according to a preferred embodiment of the present invention, the above described scalability can be utilized for coding known as scene transitions into video sequences. Video material such as news, music videos, and movie shows often include fast scene cuts between individual image composition scenes. Sometimes these scene changes are abrupt, but what's known as a transition is a common one, and the transition from one scene to another is dimming, wiping, dissolving, or image frames from previous scenes. By scrolling them and likewise providing image frames of later scenes. In terms of coding efficiency, video coding of scene transitions is often the most problematic, since the image frames that appear during the scene transition contain information about the image frames that remove and initiate the scene.

Fading, a typical scene transition, is performed by gradually decreasing the intensity or luminance of the image frames of the first scene to zero while gradually increasing the intensity of the image frames of the second scene to a maximum value. This scene transition is called a cross-faded scene transition.

Generally speaking, a computer-generated image can be thought of as layers or image objects. Each object may be defined with reference to at least three types of information. The three types of information refer to the structure of the image object, its shape and transmission, and the hierarchical order (depth) associated with the background of the image and other image objects. Shape and transmittance are often determined using what is known as the alpha plane, and opacity is measured and its value is usually set separately for each image object, usually excluding backgrounds that are considered non-transparent.

Thus, the alpha plane value of a non-transparent image object, such as a background, can be set to 1.0, while the alpha plane value of a fully transparent image object is 0.0. The values in between define the intensity of the visibility of a particular image object in the screen as compared to the background and at least partially overlapping image objects having a greater depth than the corresponding image object.

The superposition of image objects in layers according to the shape, transmittance and depth position of the image objects is called scene synthesis. In practice, the approach is based on the use of weighted averages. First, the closest image object closest to the background, i.e., the deepest image object, is placed on the background and a combined image of the two is created. The pixel values of the combined image are formed as a weighted average of the alpha plane values of the background image and the corresponding image object. The alpha plane value of the combined image is set to 1.0 and then used as the background image for the next image object. This process continues until all image objects have been added to the image.

In the following, a scheme according to a preferred embodiment of the present invention will be described, wherein the video sequence scalable layers are combined with the image objects of the above-described image frames and their information type, which also has good compression efficiency for scene transitions. Provides scalable video coding.

This embodiment of the present invention will be described below in a simplified manner, for example, using one crossover fading transition and the other abrupt transition. Image frames to be displayed during a scene change usually consist of two overlapping image frames, wherein a first image frame includes a first image scene and a second image frame includes a second image scene. One of the image frames is used as the background image and the other, called the foreground image, is located above the background image. The opacity, or non-transparency value, of the background image is constant. That is, pixel-specific alpha plane values are not adjusted.

In this embodiment of the invention, both the background and foreground images can be defined according to the scalable layer. This is illustrated in FIG. 7, which shows an example of how image frames of two different scenes can be placed on scalable layers during a scene transition of the present invention. 7 shows a first image frame 700 of a first (removal) scene located on a base layer. Image frame 700 may be an I-frame that includes image information that was not determined using motion compensated temporal prediction, or it may be a P-frame that is a motion compensated image frame predicted from previous image frames. Coding of the second (start) scene begins in time during the next image frame, and according to the invention the image frames of that scene are also located on the base layer. The image frames 702, 704 of the second (removed) scene are then located in enhancement layer 1. These image frames are usually P-frames.

In this embodiment, the image frames of the second (start) scene are located on the base layer at least during the scene change. The first image frame 706 of the scene is usually an I-frame and is used to temporally predict successive image frames of the scene. After all, the continuous image frames of the second scene are conventional P-frames, which are temporally predicted frames, such as frames 708 and 710 shown in FIG.

According to a preferred embodiment of the present invention, positioning on the scalable layers of such image frames is such that cross fading scene transitions are determined by always determining the image layer in the base layer as the maximum opacity (100%), or non-transparent background image. Can be used to implement During the transition, image frames placed on the enhancement layers are placed over the background image so that their opacity is adjusted such that the frames gradually change from non-transparent to transparent by appropriate filters.

In the video sequence of FIG. 7, no image frames exist above the lower scalable layers during the first base layer image frame 700. During this time, only the first image frame 700 is coded in the video sequence.

The next image frame 706 of the base layer initiates a new (second) scene, during which the image frame 706 is provided with a depth position that places it as a background image, with its opacity value set to maximum. In time temporally the same as the image frame 706 of the base layer, there is an image frame 702 of the removed (first) scene above enhancement layer 1. In order for cross fading cutaway to occur, the transmittance of frame 702 must be increased. The example of FIG. 7 assumes that the opacity of the image frame 702 is set to 67% and that the image frame 702 is given a depth position that determines it as a foreground image. During this moment, the image combining the image frames 706 and 702 is coded into the video sequence, where the image 706 is shown as a weaker image on the background and the image 702 has a substantially high opacity value (67). %) Is seen as a stronger image from the front.

During a subsequent image frame in time, there is a second image frame 708 of the second scene on the base layer, whereby the frame 708 is given a depth position that defines itself as the background image and its opacity value is maximized. It is decided. Enhancement layer 1 further includes the last image frame 704 of the (first) scene, which is removed simultaneously in time, with an opacity value of 33%. Image frame 704 is also given a depth position that also defines itself as the foreground image. Finally, during this moment, the image with combined image frames 708 and 704 is coded into the video sequence, where image 708 is displayed as a stronger image on the background and image 704 has a higher opacity value. Since it is not more than 33%, it is displayed as a weaker image on the foreground.

During the subsequent image frame in time, the base layer comprises a third image frame 710 of the second scene. Since the first scene has been removed, only the image frame 710 is coded in the video sequence, and the display of the second scene continues from frame 710.

The example disclosure described positioning of image frames onto scalable layers in accordance with the present invention to implement cross fading scene transitions in an advantageous manner in terms of coding efficiency. However, when a video sequence is being transmitted or decoded, a situation arises in which the bit rate of the video sequence must be adjusted according to the maximum value of the bandwidth for data transmission and / or the terminal decoding rate available. This type of bit rate control causes problems when scene transitions are implemented using conventional video coding methods.

A preferred embodiment of the present invention now allows one or more scalable layers, or independently decodable sub-sequences contained therein, to be removed from the video sequence, thereby allowing the bit rate of the video sequence to drop and at the same time video Allows the sequence to be decoded without reducing the image frequency. In positioning the image frame according to FIG. 7, this may be implemented by removing enhancement layer 1 from the video sequence. The video sequence is thus only used to display image frames 700, 706, 708, and 710 of the base layer. That is, an I-image frame 706 in which a direct transition from the first (removed) scene to the second (initiated) scene is in the form of a sharp transition, i. In the form of a direct conversion. Therefore, this transition is not a cross fading scene transition but an abrupt scene transition. Nevertheless, this transition can be performed in a desirable way without affecting the quality of the video sequence image, and the viewer usually does not experience any abrupt transitions performed in place of cross-fading transitions in any anxious or wrong way. Do not. On the contrary, since the conventional configuration does not allow the scalable layers to be removed, the viewer will find that the screen is intermittent and unstable.

Thus, the present invention relates to a multimedia data traffic shaping in a streaming server that includes information about different sub-sequences of a video sequence, that is, their average bit rate, location, duration, and interdependence information relating to the entire video sequence. It provides a preferred means to carry out. The streaming server also determines the maximum value of the bandwidth available for data transmission and / or the decoding rate of the terminal device. Based on this information, the streaming server determines how many scalable layers and which sub-sequences are transmitted as the video sequence. Then, rough adjustments to the number of scalable layers can be performed first, and then finer sub-sequence-specific adjustments can be easily made so that bit rate control can be achieved whenever needed. As the simplest one, the rate control means can make a sub-sequence-specific decision as to whether a particular sub-sequence is to be added or removed from the video sequence. When removing, it is advisable to remove the entire sub-sequences from the video sequence, since the removal of individual images can cause errors in other images of the same sub-sequence. For the same reason, all sub-sequences of the lower enhancement layer should also be removed if they were dependent on the sub-sequence of the higher layer being removed. If there are independent sub-sequences on one and the same scalable layer, sub-sequences that depend on the previous subsequence should be removed when the previous sub-sequence is removed.

When image frame identifier data is added to a video sequence to be transmitted, traffic shaping may also be used in a telecommunications network component to be used for video sequence transmission, such as, for example, an Internet router, other gateways, or a base station controller in a base station or mobile communication network. Can be performed. For network components to be able to maintain and process sub-sequence information, traffic shaping must have extra memory and processing capacity. For this reason, traffic shaping to be performed within a network is likely to be performed using simple processing methods such as DiffServ, a differentiated service that is a procedure supported by some IP-based networks. In the DiffServ scheme, each IP data packet is assigned a priority, so that higher priority data packets are delivered to the receiver faster and more securely than lower priority packets. This is preferably applied to the scalability of the present invention by setting priorities unique to the sub-sequence as well as unique to the scalable layer to enable very advanced prioritization.

There are many selectable ways of adding image frame identifier data to the video sequence to be transmitted. It is also possible not to include any identifier data in the video sequence, in which case traffic shaping is only performed on the streaming server. Identifier data may be included in header fields of a video sequence or header fields of a transport protocol to use, such as RTP (Real Time Protocol). According to a preferred embodiment, the identifier data may be transmitted using a Supplemental Enhancement Information (SEI) mechanism. SEI provides a data carrying mechanism that is transmitted in synchronization with the video data content to assist in decoding and displaying video sequences. In particular, the SEI mechanism when used to transmit layer and sub-sequence information is described in detail in ITU-T standard document ITU-T Recommendation H.264 (ISO / IEC 14496-10: 2002), Appendix D. In such cases where a separate transport protocol or mechanism is used for the identifier data transmission, traffic shaping may also be performed at one of the network components in the transmission path. In addition, the receiving terminal device can control decoding.

When an encoder or decoder supports reference picture selection, video sequence coding allows decoded image frames to be buffered prior to that coding so that the relationship between different image frames can be predicted in time from one or more other image frames. in need. Image frame buffering may be accomplished as sliding windowing or adaptive buffer memory control, at least two different ways. In sliding windowing, the last coded M image frames are used as buffers. Frames in the buffer memory are in a decoded and reconstructed format, so that they can be used as reference images in coding. As coding progresses, image frame buffering works according to the FIFO principle (Fisrt-In-Fisrt-Out, which first comes first). Images that are not used as reference images, such as conventional B-images, do not need to be stored in a buffer. Alternatively, buffering may be implemented with adaptive buffer memory control, in which case image buffering is not limited to the FIFO principle and image frames that are not needed may be removed from the buffer during the process, or similarly any image When frames are needed as reference images for later image frames, they may be stored in a buffer for a longer period of time. Known reference picture selection is implemented by indexing image frames in the buffer memory in a particular order, the image indices being used to reference an image in terms of motion compensation and the like. This indexing scheme, for example, refers to a particular image when the motion-compensated reference images are to be signaled, providing better compression efficiency compared to using image numbers in general.

The reference image indexing method is sensitive to error transmission because the buffer of the sender's encoder and the receiver's decoder contain mutually corresponding reconstructed images in the same order so that both the encoder and the decoder use the same indexing order. It should be able to guarantee the formation. If the image frames are indexed in different orders in the buffers of the encoder and decoder, the wrong reference image may be used at the decoder. To prevent this, it becomes important that the decoder be able to be controlled to account for image frames and sub-sequences that the encoder has deliberately removed from the video sequence. In this case the image frame numbering may include spaces, which the decoder typically tries to reconstruct the image frames which it interprets as missing and interprets them as lost. For this reason, it is important for the encoder to inform the decoder that the discontinuity in image numbering of transmitted image frames is intentional.

In response, and when sliding windowing is used to buffer image frames, the decoder considers the buffer memory to be multiple image frames whose contents are completely random and correspond to the number of images lost. These random image frames are designated as "invalid" to indicate that they are input buffer buffer fill frames for buffer memory management only and do not belong to the actual video sequence. The buffer fill frame can of course be implemented using only memory indicators. That is, no data is preferably entered into the buffer memory, except that memory management is used to store only references to generic "invalid" frames. The input of the image frames of the actual video sequence is resumed from the correct image frame number after the number of fill frames indicated by the missing image numbers is entered into the buffer, which is preferably the buffer memories of the encoder and decoder. To be kept in sync. When a reference to the image number is detected during decoding, and then it is found that it represents a fill frame located in the buffer, the error correction action is triggered by the decoder asking the encoder to retransmit the reference image, for example, to reconstruct the actual reference image. Begins.

In addition, the method of the present invention allows separate buffer memories to be used for different scalable layers, or likewise uniquely to sub-sequences. Each scalable layer thus includes a separate buffer memory that is conceptually separate and operates according to the sliding window principle. Similarly, each sub-sequence is also provided with a buffer memory that is conceptually separate and also operates according to the sliding window principle. This means that when one sub-sequence is interrupted, its buffer memory is always empty. Each of the buffer memories can be used in a preferred way to reduce the need for signaling in some situations where general sliding window buffering is inadequate and actually requires adaptive buffer memory management.

The H.26L standard defines the picture order count as the picture position in the output order. The decoding process specified in the H.26L standard uses picture order counts to determine the default index order for the reference pictures of B slices, vector scaling in motion vector prediction, and implicit mode weighted prediction of B slices. Picture order difference between frames and fields, and determine when successive slices belong to different pictures in decoding order. The picture order count is coded and transmitted for each picture.

In one embodiment of the invention, the decoder uses the picture order count to infer that pictures are overlapping in time, i.e., pictures with the same picture order count are overlapping in time. Preferably, the decoder outputs only the picture of the highest reception layer. In the absence of layer information, the decoder infers that the most recently duplicated picture in time in the decoding order exists on the highest receiving layer.

In the above, a method of coding video frames for the purpose of generating a scalable compressed video sequence has been described. The actual method is performed in a video encoder, which can be any existing video encoder, such as compressor 108 of FIG. For example, a video encoder according to ITU-T Recommendation H.263 or H.26L may be used, wherein the video encoder is, according to the invention, at least part of a first sub-sequence formed by coding I-frames. Generate a sub-sequence into a video sequence; Forming at least a second sub-sequence in the video sequence, wherein at least a portion of the sub-sequence is formed by coding at least P- or B-frames and predicted from at least one video frame of the first sub-sequence; And to identify identification data of at least a second sub-sequence of video frames in the video sequence.

According to the method of the invention, it is preferred that each sub-sequence of a particular scalable layer can be decoded independently, of course with dependencies from higher scalable layers and possibly other sub-sequences of the same scalable layer. This should be taken into account. A scalable video sequence, as described above, decodes a first sub-sequence of the video sequence, wherein at least a portion is formed by coding at least I-frames, and at least a portion codes at least P- or B-frames. Wherein at least one video frame formed and provided decodes at least a second sub-sequence of the video sequence that was predicted from at least one video frame of the first sub-sequence, and the video frame provided in the at least second sub-sequence of the video sequence. Is determined by determining their identifier and dependency data and reconstructing at least a portion of the video sequence based on the dependencies of the sub-sequences.

The actual decoding takes place in the video decoder, which is any known video decoder such as decompressor 118 of FIG. For example, a low bit rate video decoder in accordance with ITU-T Recommendation H.263 or H.26L is used, which in the present invention is the first sub- of the video sequence, some of which is formed by coding I-frames. Decode the sequence; Is configured to decode at least a second sub-sequence of the video sequence wherein at least a portion is formed by coding at least P- or B-frames and wherein at least one video frame including is predicted from at least one video frame of the first sub-sequence. . The video decoder is configured to determine identifier and dependency data of video frames included in at least a second sub-sequence of the video sequence and is configured to reconstruct at least a portion of the video sequence based on the dependencies of the sub-sequences.

A substantial aspect of the operation of the streaming system of the present invention is that the encoder and the decoder are positioned such that the encoder is operably connected with the streaming server and the decoder is operably connected with the receiving terminal device. However, different configurations of the streaming system, in particular terminal devices, may include functions that enable two-way transmission, ie transmission and reception of multimedia files. The encoder and decoder can then be implemented in the form of what is known as a video codec that combines both encoder and decoder capabilities.

According to the present invention, the functional components of the above-described streaming system and components such as a streaming server, a video encoder, a video decoder and a terminal are preferably implemented by software, a hardware solution, or a combination of the two. The coding and decoding method of the present invention is particularly suitable for implementation as computer software with computer readable instructions for executing the process steps of the present invention. A preferred method of implementing the encoder and decoder is to store them in storage means as program code that can be executed by a computer-like device such as a personal computer (PC) or a mobile station to provide a coding / decoding function for the device.

Another option is to implement the invention as a video signal, the video signal having a scalable video sequence, the video sequence comprising video frames coded according to at least the first and second frame formats, In this case, video frames according to the first frame format are independent of other video frames, and video frames of the second frame format are predicted from at least one of the other video frames. According to the invention, the video signal comprises at least a first sub-sequence, at least in part formed by coding video frames of at least a first frame format; A second sub-sequence, wherein at least a portion is formed by coding video frames of at least a second frame format, wherein at least one video frame comprising is predicted from at least one video frame of the first sub-sequence; And at least one data field that determines video frames belonging to the second sub-sequence.

It will be apparent to those skilled in the art that as the technology evolves, the basic idea of the present invention can be implemented in various ways. The invention and its embodiments are thus not limited to the examples described above but may be modified within the scope of the claims.

According to the present invention, by providing the decoder with information that the gap in image frame numbering is intentional, the decoder does not start unnecessary error correction operations, and synchronization of the buffer memories of the encoder and decoder is maintained. This makes it possible to perform a rough operation for the reference image selection process.

Claims (31)

A method of decoding a compressed video sequence, wherein image frames are input into a buffer memory associated with decoding. Decoding an indication relating to at least one discontinuity in the numbering of image frames from the video sequence; In response to the display, setting a buffer memory to provide image frames corresponding to discontinuities in the numbering of the image frames; And Using the image frames in the buffer memory in a decoding process. 2. The method of claim 1 wherein the indication indicates that at least one discontinuity in image frame numbering in the video sequence is intentional. The method according to claim 1 or 2, Using image frames in the buffer memory instead of image frames not present in the decoder due to a discontinuity in the numbering of the image frames. The method according to any one of claims 1 to 3, And image frames in the buffer memory are used for sliding window buffering for the image frames. The method according to any one of claims 1 to 4, Setting the buffer memory to provide image frames corresponding to discontinuities in the numbering of the image frames, and then resuming input of the image frames included in the video sequence into the buffer memory starting from the correct image frame number; Decoding method characterized in that. The method according to any one of claims 1 to 5, And inputting, into the buffer memory, filler frames corresponding to discontinuities in the numbering of the image frames. 7. The method of claim 6 wherein the supplemental frames are indicated by an identifier indicating that they do not belong to an actual video sequence. 6. The method according to any one of claims 1 to 5, wherein the setting of the buffer memory to provide image frames corresponding to discontinuities in the numbering of the image frames is performed according to the memory indication, and thus no data Decoding is not input to the buffer memory. The method according to claim 3 or 6, wherein Detecting an error in the compressed video sequence based on a reference to an image frame corresponding to a discontinuity in the numbering of image frames in the decoding process. A video decoder connected to a buffer memory to which image frames are input in connection with decoding to decode a compressed video sequence, Decode an indication associated with at least one discontinuity in the numbering of image frames from the video sequence; In response to the display, setting a buffer memory to provide image frames corresponding to discontinuities in the numbering of the image frames; And use the image frames in the buffer memory during a decoding process. 11. The video decoder of claim 10, wherein the indication indicates that at least one discontinuity in image frame numbering in the video sequence is intentional. The method according to claim 10 or 11, wherein And use the image frames in the buffer memory instead of image frames not present in the decoder due to discontinuities in the numbering of the image frames. The method according to any one of claims 10 to 12, And image frames in the buffer memory are used for sliding window buffering for the image frames. The method according to any one of claims 10 to 13, And configured to provide image frames corresponding to discontinuities in the numbering of the image frames, and then resume input of the image frames included in the video sequence into the buffer memory from the correct image frame number. Video decoder. The method according to any one of claims 10 to 14, And input, into the buffer memory, filler frames corresponding to discontinuities in the numbering of the image frames. 16. The video decoder of claim 15, wherein the supplemental frames are indicated by an identifier indicating that they do not belong to an actual video sequence. 15. The apparatus according to any one of claims 10 to 14, further configured to set the buffer memory to provide image frames corresponding to discontinuities in the numbering of the image frames in accordance with the memory indication, and thus no data And a video decoder which is not input to the buffer memory. A computer program product stored in a computer readable medium and executed through a data processing apparatus to decode a compressed video sequence when image frames of the video sequence are input to a buffer memory associated with decoding, Computer program code for decoding an indication relating to at least one discontinuity in the numbering of image frames from a video sequence; Computer program code for setting a buffer memory in response to the display to provide image frames corresponding to discontinuities in the numbering of the image frames; And Computer program code for including image frames in the buffer memory during a decoding process. 19. The computer program product of claim 18, wherein the indication indicates that at least one discontinuity in image frame numbering in the video sequence is intentional. The method of claim 18 or 19, And computer program code for including image frames in the buffer memory instead of image frames that are not present in the decoder due to discontinuities in the numbering of the image frames. The method according to any one of claims 18 to 20, And computer program code for causing the image frames in the buffer memory to be used for sliding window buffering for the image frames. The method according to any one of claims 18 to 21, The buffer memory is set to provide image frames corresponding to a discontinuity in the numbering of the image frames, and then computer program code for resuming input of the image frames included in the video sequence into the buffer memory from the correct image frame number. Further comprising a computer program product. The method according to any one of claims 18 to 22, And into the buffer memory, computer program code for inputting filler frames corresponding to discontinuities in the numbering of the image frames. The method of claim 23, wherein And computer program code for causing the supplemental frames to be represented as an identifier indicating that the supplemental frames do not belong to an actual video sequence. The method according to any one of claims 18 to 22, And further comprising computer program code for setting the buffer memory to provide image frames corresponding to discontinuities in the numbering of the image frames in accordance with the memory indication, thereby preventing any data from being input into the buffer memory. Decoding method. In a video encoder for encoding a video sequence, Encode the video sequence such that at least one image frame is excluded from the video sequence to cause discontinuity in the numbering of the image frames; And encode an indication relating to the numbering discontinuity of image frames in the video sequence. 27. The video encoder of claim 26, wherein the indication indicates that a discontinuity in image frame numbering in the video sequence is intentional. An apparatus for receiving an encoded video sequence and adjusting a bit rate of the video sequence, Exclude at least one image frame from the video sequence to cause discontinuity in the numbering of the image frames. The method of claim 28, Wherein the indication indicates that a discontinuity in image frame numbering in the video sequence is intentional. A video signal comprising a compressed video sequence with image frames, An indication relating to the numbering discontinuity of image frames in the transmitted video signal, The indication is informing the decoder to use image frames corresponding to a discontinuity in the numbering of image frames in the decoding process. 31. The video signal of claim 30, wherein the indication indicates that the discontinuity in image frame numbering in the video sequence is intentional.

Images