KR20080066784A - Efficient decoded picture buffer management for scalable video coding - Google Patents

Abstract

A system and method for enabling the removal of decoded pictures from a decoded picture buffer as soon as the decoded pictures are no longer needed for prediction reference and future output. An indication is introduced into the bifstreara as to whether a picture may be used for mter-layer prediction reference, as well as a decoded picture buffer management method which uses the indication. The present invention includes a process for marking a picture as being used for inter-layer reference or unused for inter-layer reference, a storage process of decoded pictures into the decoded picture buffer, a marking process of reference pictures, and output and removal processes of decoded pictures from the decoded picture buffer.

Description

Efficient decoded picture buffer management for scalable video coding

The present invention relates to the field of video coding. More specifically, the present invention relates to scalable video coding.

Video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) is included. In addition, efforts are currently underway to develop new video coding standards. One such specification under development is the scalable video coding (SVC) specification, which will be a scalable extension to H.264 / AVC. Another such effort involves China video coding standards.

Scalable video coding may provide scalable video bit streams. Portions of the scalable video bitstream may be extracted and decoded with degraded visual reproduction quality. In today's concept, a scalable video bitstream includes a non-scalable base layer and one or more enhancement layers. The enhancement layer may improve temporal resolution (ie, frame rate), spatial resolution, or simply the quality of video content represented by its sublayer or part thereof. In some cases, the data of the enhancement layer may be truncated after a certain position, even at arbitrary positions, and each cut position may include some additional data indicating a greatly improved visual quality. Such scalability is called fine-grained scalability (FGS). In contrast to FGS, scalability provided by a quality enhancement layer that does not provide microscale scalability is called coarse-grained scalabilty (CGS). Base layers may also be designed to have FGS scalability; No current video compression standard or draft standard implements such a concept.

The scalable hierarchy in the current SVC standard is called temporal_level, dependency_id and quality_level, which can be signaled through the bit stream or derived according to its specification. It features three variables. temporal_level is used to indicate temporal scalability or frame rate. A layer containing pictures of smaller temporal_level has a lower frame rate than a layer containing pictures of larger temporal_level. dependency_id is used to indicate the inter-layer coding dependency hierarchy. At some temporal location, a picture with a smaller dependency_id value can be used for inter-layer prediction for picture coding of a larger dependency_id value. quality_level is used to indicate the hierarchy of FGS layers. An FGS picture with the same quality_level as QL, having the same dependency_id value on any time position, is an FGS picture or base quality picture with the same quality_level value as QL-1 for inter-layer prediction (ie, QL-1 = 0 days). When using non-FGS images).

1 illustrates the temporal segment of an example scalable video stream with display values of the three variables described above. Note that the time values are relative, i.e., that time = 0 does not necessarily mean the time of the first picture in the display order on that bit stream. In this example, a typical predictive reference relationship is shown in FIG. 2, in which the solid arrows indicate the horizontal interprediction reference relationship and the dashed block arrows indicate the inter-layer predictive reference relationship. The indicated instance uses the instance in the opposite direction for predictive reference.

As discussed herein, one layer is defined as a set of pictures, each having the same temporal_level, dependency_id and quality_level values. In order to decode and play the enhancement layer, its lower layers, which typically include the base layer, should also be available, since the lower layers will be used directly or indirectly for inter-layer prediction in decoding the enhancement layer. . For example, in Figures 1 and 2, pictures with (t, T, D, Q) equal to (0, 0, 0, 0) and (8, 0, 0, 0) belong to the base layer, This can be decoded independently of any enhancement layers. Pictures with (t, T, D, Q) such as (4,1,0,0) belong to an enhancement layer that is twice the frame rate of the base layer; Decoding of this layer requires the presence of base layer pictures. Pictures with (t, T, D, Q) such as (0,0,0,1) and (8,0,0,1) are enhancement layers that improve the quality and bit rate of the base layer in the FGS scheme Belongs to; Decoding of this layer also requires the presence of base layer pictures.

In the current SVC standard draft, a coded picture in the enhancement layer of a spatial or CGS includes an inter-layer prediction reference indication (ie, base_id_plus1 syntax element in the slice header). Inter-layer prediction includes coding mode, motion information, and sample residual prediction. The use of inter-layer prediction can greatly improve the coding efficiency of enhancement layers. Interlayer prediction always uses lower layers as a reference for prediction. In other words, the upper layer is never needed for decoding the lower layer.

In a scalable video bitstream, an enhancement layer picture is free to choose which lower layer to use for inter-layer prediction. For example, when there are three layers base_layer_0, CGS_layer_1 and spatiallayer_2, and they have the same frame rate, the enhancement layer picture may select any of these layers for inter-layer prediction.

A general inter-layer prediction dependency hierarchy is shown in FIG. 3. Referring to FIG. 3, inter-layer prediction is indicated by arrows pointing in the dependency direction. Pointed-to objects need a pointed-from object for inter-layer prediction. With continued reference to FIG. 3, a pair of values to the right of each layer represent dependency_id and quality_level values as specified in the current SVC standard draft. However, one picture in spatial_layer_2 may also choose to use base_layer_0 for inter-layer prediction as shown in FIG. 4. Also, one picture in spatiallayer_2 selects base_layer_2 for inter-layer prediction, but it is possible to decide that one picture in CGS_layer_1 at the same temporal location does no inter-layer prediction at all, as shown in FIG. 5.

When FGS layers are involved, inter-layer prediction for coding mode and motion information may be obtained from the base layer rather than inter-layer prediction for sample residual information. For example, as shown in FIG. 6, in the spatial_layer_2 picture, the inter-layer prediction for coding mode and motion information takes place from the CGS_layer_1 picture, while the inter-layer prediction for sample residual information is obtained from the FGS_layer_1_1 picture. As another example, in the spatial_layer_2 picture as shown in FIG. 7, the inter-layer prediction for coding mode and motion continues to be obtained from the CGS_layer_1 picture, while the inter-layer prediction of sample residual information starts from the FGS_layer_1_0 picture. More specifically, the relationship may be expressed such that inter-layer predictions for both coding mode, motion information, and sample residual information are obtained from the same FGS layer, as shown in FIGS. 8 and 9, respectively.

In video coding standards, a bit stream is defined as being matched when it is decoded by a reference decoder under the assumption that it is conceptually connected with the output of an encoder, the reference decoder being a pre-decoder buffer, a decoder, and It includes an output / display unit. Such virtual decoders are known as hypothetical reference decoders (HRDs) of H.263 and H.264 and video buffering verifiers (VBVs) under MPEG. Appendix G of the 3GPP Packet-Switched Streaming Services Standard (3GPP TS 26.234) specifies a server buffering verifier that may be considered as an HRD, with the difference that it is conceptually connected to the output of a streaming server. Techniques such as virtual decoders and buffering verifiers will be collectively referred to as hypothetical reference decoders (HRDs) throughout this specification. The stream becomes compliant when it is decoded by HRD without buffer overflow or underflow. A buffer overflow occurs when more bits have to be placed in the buffer when the buffer is already full. Buffer underflow occurs when the buffer is empty at the point where bits must be called from the buffer for decoding / playback.

HRD parameters can be used to impose constraints on the encoded sizes of pictures and to help determine the required buffer size and startup delay.

In the preceding HRD specification prior to PSS Annex G and H.264, only the operation of the pre-decoded buffer is specified. This buffer is commonly referred to as H.264 coded picture buffer, CPB. The HRDs in PSS Appendix G and H.264 HRDs also specify the behavior of post-decoder buffers (also called decoded picture buffers (DBPs) in H.264). In addition, although earlier HRD specifications use only one HRD operating point, the HRD of PSS Appendix G and H.264 HRD enables multiple HRD operating points. Each HRD operating point corresponds to a set of HRD parameter values.

According to the SVC standard draft, decoded pictures used for prediction and subsequent output for the next coded pictures are buffered in a decoded picture buffer (DPB). In order to use the buffer memory efficiently, DPB management processes are specified that include a process of storing decoded pictures in a DPB, a marking process of reference pictures, and a process of outputting and removing decoded pictures from a DPB.

DPB management processes specified in the current draft of the SVC standard cannot efficiently handle the management of decoded pictures that need to be buffered for inter-layer prediction, especially when those pictures are non-reference pictures. This is due to the fact that DPB management processes are intended for traditional single layer coding, which at best supports temporal scalability.

In traditional single layer coding such as H.264 / AVC, decoded pictures that must be buffered for cross prediction or later output can be removed from the buffer when they are no longer needed for cross prediction and later output. The reference picture marking process so that it can be known as soon as the reference picture is no longer needed for cross prediction criteria, so that the reference picture can be removed as soon as the reference picture is no longer needed for cross prediction and future output. Is specified. However, for pictures for inter-layer prediction reference, there are currently no available mechanisms to help the decoder to get information about the picture as quickly as possible as long as it is no longer needed for inter-layer prediction reference. Do not. One such method may involve, after decoding each picture of the desired scalable layer, removing all pictures in the DPB from which the following conditions are all true: Conditions 1) the picture is a non-reference picture; 2) the picture is in the same access unit as the picture just decoded; 3) the image is in a layer below the desired scalable layer. As a result, pictures for inter-layer prediction reference will be unnecessarily buffered in the DPB, which reduces the efficiency of buffer memory utilization. For example, the required DPB will grow beyond what is technically necessary.

In addition, in scalable video coding, decoded pictures of any scalable layer below the scalable layer desired for reproduction are never output. When such pictures are needed for mutual prediction or inter-layer prediction, storing them in the DPB is simply a waste of buffer memory.

Accordingly, it will be desirable to provide a system and method for removing pictures from a DPB as soon as decoded pictures are no longer needed for predictive reference (inter prediction or inter-layer prediction) and later output.

The present invention proposes a system and method that enables the removal of decoded pictures from a DPB as soon as the decoded pictures are no longer needed for cross prediction reference, inter-layer prediction reference and future output. The system and method of the present invention include an introduction into the bitstream for an indication of whether a picture can be used for inter-layer prediction reference, and an introduction to a DPB management method using that indication. The DPB management method is a process of marking a picture, a process of storing decoded pictures into a DPB, a marking process of reference pictures, and an output of decoded pictures from a DPB unless used for inter-layer reference or for inter-layer reference. And removal processes. A new memory management control operation (MMCO) is defined to perform marking that the picture is not used for inter-layer references so that the decoder knows as soon as a picture is no longer needed for inter-layer predictive references. The corresponding signaling is specified in the bit stream.

The present invention makes it possible to provide a decoded picture buffer management process that can save the memory required for decoding scalable video bit streams. The present invention can be used in the context of scalable expansion of the H.264 / AVC video coding standard and other scalable video coding schemes.

These and other advantages and features of the present invention, as well as their operating system and manner, will be apparent from the following detailed description taken in conjunction with the accompanying drawings, and like reference throughout the several drawings to be discussed below. The symbols indicate the same component.

1 shows a temporal segment of an example scalable video stream in which values of three variables, temporal_level, dependency_id and quality_level, are indicated.

FIG. 2 is a typical predictive reference relationship in the temporal segment shown in FIG. 1.

3 is a typical inter-layer prediction dependency hierarchical representation, where an arrow indicates that the object pointed to uses the object pointed to for inter-layer prediction reference.

4 is a flow chart showing how a picture in spatial_layer_2 may choose to also use base_layer_0 for inter-layer prediction.

FIG. 5 shows an example in which a picture in spatial_layer_2 selects base_layer_0 for inter-layer prediction, while determining that a picture in CGS_layer_1 on the same temporal location does not have any inter-layer prediction.

FIG. 6 illustrates an example of how an inter-layer prediction for coding mode and motion information is output to a base layer different from the inter-layer prediction for sample residual information.

FIG. 7 shows an example of how an inter-layer prediction for sample residual information comes from an FGS_layer_1_0 picture while the spatial_layer_2 picture is derived from a CGS_layer_1 picture while the inter-layer prediction for coding mode and motion is shown.

8 illustrates an example in which inter-layer prediction for both coding mode, motion information, and sample residual information comes from an FGS_layer_1_1 picture when the coding mode and motion information are inherited from the base quality layer.

FIG. 9 shows an example where an inter-layer prediction for both coding mode, motion information, and sample residual information comes from an FGS_layer_1_0 picture when the coding mode and motion information are inherited from the base quality layer.

10 illustrates an example of a state evolution process for multiple coded pictures in an access unit, in accordance with conventionally known systems.

11 shows an example of a state evolution process of various coded pictures within an access unit, in accordance with the system and method of the present invention.

12 is a schematic diagram of a system in which the present invention may be implemented.

13 is a front view of an electronic device that may incorporate the principles of the present invention.

14 is a schematic circuit diagram representation of the electronic device of FIG. 13.

15 illustrates a general multimedia data streaming system to which the scalable coding hierarchy of the present invention can be applied.

Referring to Fig. 6, a conventional multimedia streaming system, which is one system for applying the processing procedure of the present invention, will be discussed.

A multimedia data streaming system typically includes one or more multimedia sources 100, such as video cameras and microphones, or video images or computer graphics files stored in a memory carrier. Raw data from different multimedia sources 100 will be combined into a multimedia file in encoder 102, which may also be called an editor. Raw data from one or more multimedia sources 100 is first captured using capture means 104 included in encoder 102, which typically captures different interface cards, driver software, or card functionality. It can be implemented as controlling application software. For example, video data can be captured using a video capture card and associated software. The output of the capture means 104 is usually an uncompressed or slightly compressed data flow, such as an uncompressed video frame in YUV 4: 2: 0 format or Motion JPEG image format when the video capture card is involved. It becomes.

The editor 106 links the different media flows together to synchronize the video and audio flows so that they can be played simultaneously as desired. The editor 106 may also edit each media flow, such as a video flow, by cutting the frame rate in half, reducing the spatial resolution, and the like. Although synchronized, the individual media flows are compressed in the compressor 108, where each media flow is individually compressed using a compressor suitable for that media flow. For example, video frames in YUV 4: 2: 0 format can be compressed using ITU-T Recommendation H.263 or H.264. Separately, synchronous and compressed media flows are typically interleaved within the multiplexer 110, and the output obtained from the encoder 102 may be called a multimedia file as one single bit flow containing data of a plurality of media flows. have. It will be appreciated that creating a multimedia file does not necessarily require multiplexing multiple media flows into one file, and that a streaming server can interleave them just before sending the media flows.

The multimedia files are sent to streaming server 112, which can then perform streaming in the form of real-time streaming or progressive downloading. In progressive downloading, the multimedia files are first stored in the memory of the server 112 where they can be retrieved for transmission when demand arises. In real time streaming, the editor 102 sends a continuous media flow of multimedia files to the streaming server 112, which server 112 passes the flow directly to the client 114. As an additional option, real time streaming may also be performed such that multimedia files are accessible from server 112 and from which live streaming can be derived and stored in storage where a continuous media flow of multimedia files begins when there is a demand. . In that case, the editor 102 does not necessarily control streaming by any means. The streaming server 112 performs traffic shaping of the multimedia data with respect to the available bandwidth of the client 114 or the maximum decoding and playback rate, and the streaming server removes or scales B-frames from the transmission information. By optimizing the number of layers, the bit rate of the media flow can be adjusted. Furthermore, the streaming server 112 can change the header fields of the multiplexed media flow to reduce their size and encapsulate the multimedia data in data packets suitable for transmission on the telecommunications network in use. The client 114 can typically adjust the operation of the server 112 to at least some extent using an appropriate control protocol. The client 114 can control the server 112 so that at least the desired multimedia file can be selected to be sent to the client, in addition, the client can usually stop and stop the multimedia file transfer.

The following text describes one specific embodiment of the present invention in a specific text format for the SVC standard. In this embodiment, the decoding reference picture marking syntax is as follows.

Decoding Reference Picture Marking Syntax

The slice header in the scalable extension syntax is as follows.

Slice headers in scalable extension syntax

In decoding reference picture marking semantics, "num_inter_layer_mmco" refers to the number of memory_management_control operations for marking decoded pictures in a DPB as "unused for inter-layer prediction." "dependency_id [i]" indicates dependency_id of an image to be marked "inter-layer prediction not used". dependency_id [i] is less than or equal to dependency_id of the current picture. "quality_level [i] represents the quality_level of the picture to be marked" inter-layer prediction not used. "When dependency_id [i] is equal to dependency_id, quality_level [i] is less than quality_level.It is in the same access unit as the current picture and dependency_id [ A decoded picture with dependency_id and quality_level [i], such as i], will have inter_layer_ref_flag equal to 1.

If present, the scalable extension syntax elements pic_parametr_set_id, frame_num, inter_layer_ref_flag, field_pic_flag, bottom_field_flag, idr_pic_id, pic_order_cnt_1sb, delta_pic_order_cnt_bottom, delta_pic_order_cnt [0], slice_delta_pic_slice in header, delta_pic_order_cnt [0], and delta_pic_order. Same as inside "frame_num" has the same syntax as fram_num in subclause S.7.4.3 of the current draft of the SVC standard. A value of "inter_layer_ref_flag" corresponding to 0 means that the current picture is not used for inter-layer prediction reference for decoding of any picture having a dependency_id having a value larger than the dependency_id value for the current picture. A value of "inter_layer_ref_flag" corresponding to 1 indicates that the current picture can be used as an inter-layer prediction reference for picture decoding with a dependency_id of a value larger than the current picture. "filed_pic_flag" has the same semantics as filed_pic_flag in the current subclause S.7.4.3 of the SVC standard draft.

In a series of operations of the decoding picture marking process, if the "inter_layer_ref_flag" value is equal to 1, the current picture is marked as "used for inter-layer reference".

In the process of marking a picture as "unused for inter-layer reference", this process is caused when the value of "num_inter_layer_mmco" is not zero. All pictures in the DPB where all of the following conditions are true are marked as "unused for inter-layer reference": condition (1) The picture belongs to the same access unit as the current picture; (2) the picture has a value of "inter_layer_ref_flag" corresponding to 1 and is marked as "used for inter-layer reference"; (3) the picture has a pair of dependency_id and quality_level values such as dependency_id [i] and quality_level [i] signaled via dec_ref_pic_marking () for the current picture; (4) The picture is a non-reference picture.

In operation of a decoded picture buffer, the decoded picture buffer comprises frame buffers. Each of the frame buffers is decoded as a decoded frame, marked as "used for reference" (reference pictures) or marked as "used for inter-layer reference" or held for later output (recorded or delayed pictures). complementary filed pair) or a single decoding field (not a pair). Before initialization, the DPB is empty (DPB fullness is set to zero). The steps in the following subclauses of this subclause all occur instantaneously at t _r (n), according to the listed sequence.

In the decoding of frame_num and the storage of "non-existing" frames, if applicable, gaps in frame_num are detected through the decoding process, and the generated frames are marked and inserted into the DPB as specified below. Gaps in frame_num are detected by the decoding process and generated frames are marked as specified in subclause 8.2.5.2 of the current SVC standard draft. After each frame marking created, each picture m marked "unused for reference" through the "sliding window" process is either marked as "not present" or its DPB output time is When less than or equal to the coded picture buffer (CPB) removal time of the current picture n, that is, when t _{o, dpb} (m) < = t _r (n), it is removed from the DPB. When the last field in one frame or one frame buffer is removed from the DPB, DPB fullness is reduced by one. If a "non-existing" generation frame is inserted into the DPB, the DPB fullness is increased by one.

In picture decoding and output, picture n is decoded and stored temporarily (not in the DPB). If picture n is within the desired scalable hierarchy, the text below applies. The DPB output time t _{o and dpb} (n) of the image n are _derived by t _{o, dpu} (n) = t _r (n) + t _c * dpb_output_delay (n). The current image output is specified as follows. When t _{o, dpb} (n) = t _r (n), the current image is output. Note that when the current picture is a reference picture it will be stored in the DPB. If t _{o, dpb} (n) ≠ t _r (n), then t _{o, dpb} (n)> t _r (n), and the current picture will be output later and stored in the DPB (the subclause of the current SVC standard draft). as _equal), t _o, the earlier time in _dpb (n) unless indicated as not being output by the decoding or estimating the no_output_of_prior_pics_flag corresponding to _{1, t o, dpb (n} ) output at the time specified in C.2.4 do. The output picture is cropped using a cropping rectangle specified for the sequence parameter specified for that sequence.

의 값은

으로 정의되고, 이때 n_n은 출력 순서상 화상 n 뒤에 이어지는 화상을 가리킨다.When picture n is an output picture and not the last picture of the output bit stream,

The value of

Where n _n refers to an image following image _n in the output order.

Removing pictures from the DPB before the current picture insertion proceeds according to the sequence listed as follows. If the decoded picture is an IDR picture, the following applies. All reference pictures in the DPB and with the same dependency_id and quality_level values as the current picture, respectively, as specified in subclause 8.2.5.1 of the current SVC standard draft, are marked as "unused for reference". A sequence parameter set in which the value of PicWidthInMbs, FrameHeightInMbs, or max_dec_frame_buffeing derived from the active sequence parameter set and not the first IDR picture decoded had an effect on the previous sequence with the same dependency_id and quality_level values as the current coded video sequence, respectively. When different from the values of PicWidthInMbs, FrameHeightInMbs, or max_dec_frame_buffeing derived from, no_output_of_prior_pics_flag is estimated to correspond to 1 by HRD regardless of the actual value of no_output_of_prior_pics_glag. It should be noted that the decoder configuration will attempt to handle frame or DPB resizes more gracefully than HRD in terms of PicWidthInMbs or FrameHeightInMbs changes.

When no_output_of_prior_pics_flag is 1 or estimated to be 1, all frame buffers in the DPB that contain decoded pictures with the same dependency_id and quality_level values as the current picture, respectively, are emptied without output of the pictures they contain, and the DPB fullness is empty. Is reduced by the number of buffers. Otherwise (ie, when the decoded picture is not an IDR picture), the following applies. If the slice header of the current picture includes a memory_management_control_operation value corresponding to 5, all pictures included in the DPB and each having the same dependency_id and quality_level values as the current picture are marked as "unused for reference". Otherwise (ie, the slice header of the current picture does not contain a memory_management_control_operation value corresponding to 5), the decoding reference picture marking process specified in subclause 8.2.5 of the current SVC standard draft occurs. The marking process takes place as "unused for inter-layer references" as specified in subclause 8.2.5.5 of the current draft of the SVC standard.

If the current picture is within the desired scalable layer, all decoded pictures in the DPB that meet all of the following conditions are marked as "unused for inter-layer reference". Conditions (1) the picture belongs to the same access unit as the current picture; (2) the picture is marked as "used for inter-layer reference" with the inter_layer_ref_flag value corresponding to 1; (3) The picture has a smaller dependency_id value than the current picture, or has the same dependency_id value but a smaller quality_level value than the current picture.

All pictures in the DPB for which all the following conditions are true are removed from the DPB. Conditions (1) Picture m is marked as "unused for reference" or picture m is a non-reference picture. When a picture is a reference frame, it is considered to be marked "unused for reference" only when both of its fields have been marked "unused for reference". (2) Include inter_layr_ref_flag where picture m is marked as "unused for inter-layer reference" or picture m is zero. (3) picture m is marked as "not present", not in the desired scalable layer, or its DPB output time is less than or equal to the CPB removal time of current picture n, i.e. t _{o, dpb} (m) < = t _r (n). When the last field in one frame or one frame buffer is removed from the DPB, DPB fullness is reduced by one.

The following will be a discussion of current decoding picture marking and storage. In the marking of the reference decoded picture and storage into the DPB, when the current picture is the reference picture, it is stored in the DPB as follows. When the current decoded picture is the second field (in decoding order) of the supplementary reference field pair, and the first field of the pair is still in the DPB, the current decoded picture is stored in the same frame buffer as the first field of the pair. do. Otherwise, the current decoded picture is stored in one empty frame buffer and the DPB fullness is increased by one.

In storing the non-reference picture in the DPB, the following matters apply if the current picture is a non-reference picture. If the current riverbed is not in the desired scalable layer, or if the current picture is in the desired scalable hierarchy and has t _{o, dpb} (m)> t _r (n), the picture is stored in the DPB as follows. When the current decoded picture is the second field of the supplemental dereferenced field pair (in decoding order) and the first field is still in the DPB, the current decoded picture is stored in the same frame buffer as the first field of the pair. Otherwise, the current decoded picture is stored in one empty frame buffer and the DPB fullness is increased by one.

In the above-described embodiment, an indication to discriminate whether a picture is to be used for inter-layer prediction reference is signaled through the slice header. It is signaled as the syntax element inter_layer_ref_flag. There are many other alternative ways to signal such an indication. For example, the indication can be signaled via a NAL unit header or other ways.

As long as pictures to be marked as unused as inter-layer references are identified, signaling of a memory management operation command (MMCO) may also be performed in other alternative ways. For example, the syntax element dependency_id [i] may be coded as an increment to the dependency_id value of the current picture to which the slice header belongs.

The main differences between the above-described embodiment and the original DPB management processor are as follows. (1) In the above embodiment, when inter_layer_ref_flag is 1, it is marked that decoded pictures are used for inter-layer reference. (2) The decoded picture output process of the above embodiment is specified only when the picture is within the desired scalable layer. (3) In the above embodiment, the process of marking a picture as "unused for inter-layer reference" occurs before removal of pictures from the DPB before possible insertion of the current picture. (4) In the above embodiment, the condition of the pictures to be removed from the DPB before the possible insertion of the current picture is whether the picture is marked as "unused for inter-layer reference" or has an inter_layer_ref_flag corresponding to 0, and the picture has a desired scale change. It is changed to take into account whether you are in the hierarchy. (5) The condition of pictures to be stored in the DPB is changed in the above embodiment in consideration of whether the picture is within a desired scalable hierarchy.

FIG. 10 shows an example of a state advancement process for various coded pictures in an access unit according to conventionally known systems, and FIG. 11 shows such an example according to the present invention. In the conventional system shown in FIG. 10, the DPB state advancement process is as follows (assuming Layer 4 is the desired scalable layer for decoding and playback). Pictures from previous decoding access units will also be stored in the DPB, but for simplicity these pictures will not be considered below. After decoding the layer 0 picture and the corresponding DPB management process, the DPB includes only the picture from layer 0. After decoding the layer 1 picture and the corresponding DPB management process, the DPB includes 2 pictures from layer 0 and 1 respectively. After decoding the layer 2 picture and the corresponding DPB management process, the DPB includes three pictures from layer 0-2, respectively. After decoding the layer 3 picture and the corresponding DPB management process, the DPB includes 4 pictures from layer 0-3, respectively. After decoding the layer 4 picture and the corresponding DPB management process, the DPB includes two pictures from layers 0 and 4, respectively.

The DPB state advancement process as shown in FIG. 11 is as follows (assuming Layer 4 is the desired scalable layer for decoding and playback). Pictures from previous decoding access units may also be stored in the DPB, but these pictures will not be considered below for simplicity purposes. After decoding the layer 0 picture and the corresponding DPB management process, the DPB contains only the picture from layer 0. After decoding the layer 1 picture and the corresponding DPB management process, the DPB includes two pictures from layer 0 and 2, respectively. After decoding the layer 2 picture and the corresponding DPB management process, the DPB includes two pictures from layer 0 and 2, respectively. After decoding the layer 3 picture and the corresponding DPB management process, the DPB includes two pictures from layers 0 and 3, respectively. After decoding the layer 4 picture and the corresponding DPB management process, the DPB includes two pictures from layers 0 and 4, respectively.

As can be seen in FIG. 11, the present invention can reduce the requirement for a buffer memory. In the example shown in FIG. 11, the buffer memory for two decoded pictures can be saved.

Figure 12 shows a system 10 in which the present invention may be utilized, which includes several communication devices capable of communicating over a network. The system 10 may include any combination of wired and wireless networks, including but not limited to mobile telephone networks, wireless local area networks, Bluetooth personal area networks, Ethernet LANs, token ring LANs, wide area networks, the Internet, and the like. It may include. System 10 may include both wired and wireless communication devices.

By way of example, the system 10 shown in FIG. 12 includes a mobile telephone network 11 and the Internet 28. Connections to the Internet 28 include, but are not limited to, long distance wireless connections, short range wireless connections and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and the like.

The communication devices of the example system 10 include a mobile telephone 12, a composite PDA and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20. , And notebook computer 22, but are not limited to those listed above. The communication devices may be stationary or mobile such as carried by a moving individual. Communication devices may also be placed in a traffic mode, including but not limited to cars, trucks, taxis, buses, boats, airplanes, bicycles, motorcycles, and the like. Some or all of the communication devices may originate and receive calls and messages and communicate with the service providers to the base station 24 via a wireless connection 25. The base station 24 may be connected to a network server 26 that enables communication between the mobile telephone network 11 and the Internet 28. System 10 may include additional communication devices and other types of communication devices.

Communication devices include Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), and Transmission Control (TCP / IP). Communicate using a variety of transport technologies including, but not limited to, Protocol / Internet Protocol (SMS), Short Messaging Service (SMS), Multimedia Messaging Service (MMS), Email, Instant Messaging Service (IMS), Bluetooth, IEEE 802.11, and so forth. Can be. Communication devices can communicate using a variety of media, including but not limited to radio, infrared, laser, cable connections, and the like.

13 and 14 illustrate one representative mobile phone 12 in which the present invention may be implemented. It should be understood, however, that the present invention is not intended to be limited to one particular type of mobile phone 12 or other electronic device. The mobile phone 12 of FIGS. 13 and 14 has a housing 30, a display 32 in the form of a liquid crystal display, a keypad 34, a microphone 36, earphones 38, a battery 40, an infrared port 42. , An antenna 44, a smart card 46 in the form of a UICC according to an embodiment of the present invention, a card reader 48, a radio interface circuit 52, a codec circuit 54, a controller 56 and a memory ( 58). The individual circuits and components are all of a type well known in the art, for example those belonging to Nokia's mobile telephones.

The invention has been described as steps of a general method that can be implemented as an embodiment via a program product comprising computer executable instructions such as program code executed by computers under a network environment.

Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing the steps of the methods disclosed herein. The particular sequence of such executable instructions or related data structures represents examples of corresponding acts for implementing the functions described in such steps.

The software and web implementations of the present invention can be implemented through standard programming techniques with rule-based logic and other logic to perform various database search steps, correlation steps, comparison steps and decision steps. It is noted that the terms "component" and "module" as used in the specification and claims are intended to cover implementations that use one or more lines of software code, and / or hardware configuration, and / or a manual input receiving device. Should be.

The foregoing description of the embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to be limited to the particular forms disclosed, and various modifications and substitutions are possible in light of the above concepts or may be acquired from practice of the invention. The embodiments have been selected and described in order to explain the principles of the invention and its practical application which enable those skilled in the art to utilize the invention through various modifications tailored to the specific embodiments contemplated by the various embodiments.

Claims (52)

A method of managing a decoded picture buffer for scalable video coding, Receiving a first decoded picture belonging to a first layer on a bit stream into the decoded picture buffer; Receiving a second decoded image belonging to a second layer; Determining whether the first decoded picture is needed for inter-layer prediction reference in view of the reception of the second decoded picture; And Removing the first decoded picture from the decoded picture buffer when the first decoded picture is no longer needed for inter-layer prediction reference and subsequent output. The method of claim 1, Conveying information about possible inter-layer predictive reference indications of subsequent pictures in the decoding order signaled over the bit stream. 3. The method of claim 2, wherein the possible inter-layer prediction reference indication is signaled in a slice header. 3. The method of claim 2, wherein the possible inter-layer prediction reference indication is signaled by being enclosed in a Network Abstraction Layer (NAL) unit header. 3. The method of claim 2, wherein determining whether the first decoded picture is required for an inter-layer prediction reference comprises selectively marking the first decoded picture as "unused for inter-layer reference." How to feature. 6. The method of claim 5, wherein the first decoded picture is marked as "unused for inter-layer reference" when the first picture belongs to the same access unit as the second picture. 7. The method of claim 6, wherein the determination as to whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling through the bit stream. 6. The method of claim 5, wherein the first decoded picture is marked as "unused for inter-layer reference" when the first picture is marked as "used for inter-layer reference" with a possible possible inter-layer prediction reference indication. How to feature. 9. The method of claim 8, wherein the determination of whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling through the bit stream. 6. The method of claim 5, wherein the first decoded image has a dependency_id of which the first image has a smaller value than the second image, or has a dependency_id of the same value as the second image, but has a quality_level of a value smaller than the second image. Case, marked as "unused for inter-layer reference". 12. The method of claim 10, wherein determining whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling over the bit stream. 3. The method of claim 2, wherein the first picture is marked as "unused for reference" or is an unreferenced picture; The first picture is marked as "unused for inter-layer reference" or the possible inter-layer prediction reference indication is negative; The first decoding when the first picture is marked as “not present” or not in the desired scalable layer, or when the decoded picture buffer output time is less than or equal to the coded picture buffer removal time of the second picture. And it is determined that the picture is no longer needed for inter-layer prediction reference. 13. The method of claim 12, wherein if the first decoded picture is a reference frame, the first decoded picture is considered to be marked as "unused in reference" only when all the fields of the first decoded picture are marked as "unused in reference." How to feature. 2. The method of claim 1, wherein the first decoded picture is not needed for future output unless it is within the desired scalable hierarchy for playback. The bit stream of claim 1, wherein the bit stream includes a first sub bit stream and a second sub bit stream, the first sub bit stream includes coded pictures belonging to a first layer, and And comprising two layers of images. A decoder for decoding an encoding stream of a plurality of pictures, The plurality of pictures are defined as reference pictures or non-reference pictures, and when information related to the decoding order and the output order of one picture is determined for the pictures of the picture stream, the decoder is adapted to perform the method of claim 1. Decoder configured. A computer program product for managing a decoded picture buffer for scalable video coding, Computer code for receiving a first decoded picture belonging to a first layer on a bit stream into the decoded picture buffer; Computer code for receiving a second decoded picture belonging to a second layer; Computer code for determining whether the first decoded picture is needed for inter-layer prediction reference in light of the reception of the second decoded picture; And And computer code for removing the first decoded picture from the decoded picture buffer when the first decoded picture is no longer needed for inter-layer prediction reference and subsequent output. The method of claim 17, And computer code for conveying information about possible inter-layer predictive reference indications of subsequent pictures in the decoding order signaled through the bit stream. 19. The computer program product of claim 18, wherein the possible inter-layer prediction reference indication is enclosed in a slice header and signaled. 19. The computer program product of claim 18, wherein the possible inter-layer prediction reference indication is signaled by being enclosed in a Network Abstraction Layer (NAL) unit header. 19. The computer-readable medium of claim 18, wherein determining whether the first decoded picture is required for inter-layer prediction reference comprises selectively marking the first decoded picture as "unused for inter-layer reference." Program product. 22. The computer program product according to claim 21, wherein the first decoded picture is marked as "unused for inter-layer reference" when the first picture belongs to the same access unit as the second picture. 23. The computer program product of claim 22, wherein the determination of whether the first decoded picture is marked "unused for inter-layer reference" is based on signaling over the bit stream. 22. The method according to claim 21, wherein the first decoded picture is marked as "unused for inter-layer reference" when the inter-layer predictive reference indication capable of the first picture is marked as "used for inter-layer reference" with a positive. Computer program product characterized in that. 25. The computer program product of claim 24, wherein the determination of whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling via the bit stream. 22. The method of claim 21, wherein the first decoded image has a dependency_id of which the first image has a smaller value than the second image, or has a dependency_id of the same value as the second image, but has a quality_level of a value smaller than the second image. If marked as "unused for inter-layer reference." 27. The computer program product of claim 26, wherein the determination of whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling via the bit stream. 18. The method of claim 17, wherein the first picture is marked as “unused for reference” or is an unreferenced picture; The first picture is marked as "unused for inter-layer reference" or the possible inter-layer prediction reference indication is negative; The first decoding when the first picture is marked as “not present” or not in the desired scalable layer, or when the decoded picture buffer output time is less than or equal to the coded picture buffer removal time of the second picture. A computer program product, characterized in that it is determined that a picture is no longer needed for inter-layer prediction reference. 29. The method of claim 28, wherein if the first decoded picture is a reference frame, the first decoded picture is considered to be marked as "unused in reference" only when all the fields of the first decoded picture are marked as "unused in reference." Computer program product characterized. 17. The computer program product of claim 16, wherein the first decoded picture is not needed for future output unless it is within the desired scalable layer for playback. 17. The method of claim 16, wherein the bit stream includes a first sub bit stream and a second sub bit stream, the first sub bit stream includes coded pictures belonging to a first layer, and the second sub bit stream includes a first sub bit stream. A computer program product comprising two layers of images. In an electronic device, A processor; And A memory coupled to interoperate with the processor, the memory including a computer program product for managing a decoded picture buffer for scalable video coding, The computer program product, Computer code for receiving a first decoded picture belonging to a first layer on a bit stream into the decoded picture buffer; Computer code for receiving a second decoded picture belonging to a second layer; Computer code for determining whether the first decoded picture is needed for inter-layer prediction reference in view of the reception of the second decoded picture; And And computer code for removing the first decoded picture from the decoded picture buffer when the first decoded picture is no longer needed for inter-layer prediction reference and subsequent output. 33. The electronic device of claim 32, wherein the memory unit further comprises computer code for conveying information about a possible inter-layer prediction reference indication of a next picture in decoding order signaled via the bit stream. . 34. The electronic device of claim 33, wherein the possible inter-layer prediction reference indication is signaled by being put in a slice header. 34. The electronic device of claim 33, wherein the possible inter-layer prediction reference indication is signaled by being put in a network abstraction layer (NAL) unit header. 34. The electronic device of claim 33, wherein determining whether the first decoded picture is required for an inter-layer prediction reference comprises selectively marking the first decoded picture as "unused for inter-layer reference." device. 37. The electronic device according to claim 36, wherein the first decoded picture is marked as "unused for inter-layer reference" when the first picture belongs to the same access unit as the second picture. 38. The electronic device of claim 37, wherein determining whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling through the bit stream. 37. The method of claim 36, wherein the first decoded picture is marked as "unused for inter-layer reference" when the first picture-enabled inter-layer prediction reference mark is marked as "used for inter-layer reference" with a positive. Electronic device characterized in that the. 40. The electronic device of claim 39, wherein determining whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling through the bit stream. 37. The method of claim 36, wherein the first decoded image has a dependency_id of which the first image has a smaller value than the second image, or has a dependency_id of the same value as the second image but has a quality_level of a value smaller than the second image The electronic device, if marked as "unused for inter-layer reference." 42. The electronic device of claim 41, wherein the determination of whether the first decoded picture is marked as "unused for inter-layer reference" is based on signaling through the bit stream. 37. The method of claim 36, wherein the first picture is marked “unused for reference” or is an unreferenced picture; The first picture is marked as "unused for inter-layer reference" or the possible inter-layer prediction reference indication is negative; The first decoding when the first picture is marked as “not present” or not in the desired scalable layer, or when the decoded picture buffer output time is less than or equal to the coded picture buffer removal time of the second picture. An electronic device, characterized in that it is determined that a picture is no longer needed for inter-layer prediction reference. 44. The method of claim 43, wherein if the first decoded picture is a reference frame, then the first decoded picture is considered to be marked as "unused in reference" only when all the fields of the first decoded picture are marked as "unused in reference." An electronic device characterized by the above-mentioned. 33. The electronic device of claim 32, wherein the first decoded picture is not needed for future output if it is not within the desired scalable hierarchy for playback. 33. The method of claim 32, wherein the bit stream comprises a first sub bit stream and a second sub bit stream, the first sub bit stream including coded pictures belonging to a first layer, and the second sub bit stream being a second sub bit stream. An electronic device comprising two layers of images. 33. The electronic device of claim 32, wherein the electronic device comprises a decoder configured to read syntax elements of possible reference indication and memory management control operations from the bit stream. An encoder for forming an encoding stream of pictures, the encoder comprising: The pictures are defined as reference pictures or non-reference pictures, and when the decoding order and output order related information of a picture is defined in the pictures of the stream, Wherein the encoder places in the stream syntax elements for possible reference indication and memory management control operations, the syntax elements being generated by an electronic device according to claim 32. In the bit stream, And a syntax element that provides an indication to selectively remove the first decoded picture of the first layer from the decoded picture buffer in the light of the second decoded picture of the second layer. 49. A computer device comprising an encoder for generating a bit stream according to claim 48. In the bit stream, A syntax element for providing an indication to selectively remove the first decoded picture of the first layer from the decoded picture buffer in the light of the second decoded picture of the second layer; And wherein said syntax element is set according to the method of claim 1. A method of managing a decoded picture buffer for scalable video coding, Receiving a first decoded picture belonging to a first layer on a bit stream into the decoded picture buffer; Receiving a second decoded image belonging to a second layer; Determining whether the first decoded picture is required for inter-layer prediction reference, cross-prediction reference, and future output in view of receiving the second decoded picture; And Removing the first decoded picture from the decoded picture buffer if the first decoded picture is no longer needed for inter-layer prediction reference, cross prediction reference, and future output.

Images