Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
  • H04N19/89—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder
  • H04N19/895—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder in combination with error concealment
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/46—Embedding additional information in the video signal during the compression process
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
  • H04N19/89—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder

Definitions

• the invention relates to scalable video coding (SVC) .
• SVC scalable video coding
• the invention relates to a method and an apparatus for packet loss detection and virtual packet generation at decoders for SVC bit-streams.
• Error concealment methods have been introduced for JSVM4 1 . Exemplarily, four methods are described in JVT-Q046 2 .
• This known solution for error concealment is targeting on fixed GOP (group-of-pictures) size, where the base-layer is H.264 compliant with spatial and FGS (Fine Grain Scalability) scalable layers.
• Temporal scalability in SVC is supported by a hierarchical B frame GOP structure. In this structure, bi-directionally predicted frames (B frames) within a GOP have different temporal levels according to a defined scheme.
• B frames bi-directionally predicted frames within a GOP have different temporal levels according to a defined scheme.
• there is a need to further improve the robustness and correctness of error concealment.
• supporting error concealment for any kind of scalability and combined scalability is desirable.
• the invention provides a method for detecting errors in received packetized video data, wherein the video data
• MPEG & ITU-T VCEG, Nice, October 2005 contain a spatial or spatio-temporal base-layer (BL) and at least one enhancement-layer (EL) , and each layer is organized in frames or slices with a respective header and a GOP structure, and different temporal levels exist within layers.
• a packet is assumed to contain a frame or a slice.
• the method according to the invention comprises the steps of reading/extracting header information relating to a layer, generating from the header information a scheme, the scheme comprising at least display order number (that is picture order count, POC) , number of reference frames (that is frame_num) and temporal level for the frames within a complete GOP of the respective layer, extracting from received video data frames parameters including POC number, frame num and temporal level, and detecting that one or more frames or packets are missing, wherein the received parameters are compared with the parameters of expected frames according to said scheme and a missing packet or frame is determined by a mismatch between received and expected parameters.
• a scheme comprising at least display order number (that is picture order count, POC) , number of reference frames (that is frame_num) and temporal level for the frames within a complete GOP of the respective layer
• POC picture order count
• frame_num number of reference frames
• temporal level for the frames within a complete GOP of the respective layer
• predefined schemes may be stored, or generated on-the-fly using an appropriate algorithm.
• the method further comprises the steps of generating for each missing packet or frame a default packet, wherein each default packet gets the display order number (POC), number of reference frames (frame num) and temporal level of the respective expected packet, and inserting for each missing packet the generated default packet into the data stream before the data stream is parsed.
• POC display order number
• frame num number of reference frames
• the GOP structure contains a plurality of complete GOPs and not more than one incomplete GOP (that is a GOP with less pictures/frames than the others)
• the method according to the invention further comprises the step of determining for a missing frame if it belongs to a complete GOP or to the incomplete GOP.
• said inserting of default packets is only performed for missing frames that belong to complete GOPs.
• the step of determining for a missing frame if it belongs to a complete GOP or to the incomplete GOP in the previous embodiment can be further specified in comprising the steps of estimating the maximum GOP size for the incomplete GOP, and not generating or suppressing default packets for the incomplete GOP for display order numbers (POC) that are above the estimated GOP size.
• POC display order numbers
• the method according to the invention can be further specified in that said header information contains an indication of the difference between the POC numbers of successive frames of a defined spatio-temporal or spatial layer.
• the previous embodiment can be further specified in that said defined spatio-temporal or spatial layer is the BL.
• said indication is the logarithm to the basis two (Iog2) of said difference.
• the invention provides a signal containing packetized video data, wherein the video data are organized in frames or slices and have a respective header and a group-of-picture (GOP) structure containing a plurality of GOPs, and wherein each video frame or slice comprises at least an indication of its POC number, and wherein said header contains an indication (Iog2 min poc increase) of the difference between the display order numbers (POC) of successive frames.
• GOP group-of-picture
• the video signal can be further specified in that it has a spatial or spatio-temporal base-layer and at least one enhancement-layer, and the indication of the difference between the display order numbers (POC) of successive frames refers to the frames of one defined layer of these layers .
• POC display order numbers
• the invention provides an apparatus for generating such signal.
• An apparatus for detecting errors in received packetized video data wherein the video data contain a base-layer and at least one enhancement-layer and each layer is organized in frames or slices with a respective header and a group- of-picture (GOP) structure, comprises means for reading and extracting header information relating to a layer, means for generating from the extracted header information a scheme, or means for selecting a predefined scheme according to the extracted header information (SEI), the scheme comprising at least display order number (POC) , number of reference frames (frame num) and temporal level for the frames within a complete GOP of the respective layer, means for extracting from received video data frames parameters including POC number, frame_num and temporal level, and means for detecting that one or more frames or packets are missing, wherein the received parameters are compared with the parameters of expected frames according to said scheme and a missing packet or frame is determined by a mismatch between received and expected parameters.
• POC display order number
• frame num number of reference frames
• temporal level for the frames within a complete GOP
• the apparatus further comprises means for generating for each missing packet or frame a default packet, wherein each default packet gets the display order number (POC) , number of reference frames (frame_num) and temporal level of the respective expected packet, and means for inserting for each missing packet the generated default packet into the data stream before the data stream is parsed.
• POC display order number
• frame_num number of reference frames
• the apparatus further comprises means for determining for a missing frame if it belongs to a complete GOP or to the incomplete GOP, and means for performing said inserting of default packets only for missing frames that belong to complete GOPs.
• Fig.3 a GOP with inserted virtual packets
• Fig.4 an abnormal GOP.
• Fig.l shows the NAL (network abstraction layer) packet structure of an SVC bit-stream.
• the sequence of packets contains Supplementary Enhancement Information (SEI), sequence parameter set (SPS) , sequence parameter sets in scalable extension (SPS_S) , picture parameter sets (PPS) , picture parameter sets in scalable extension (PPS_S) , coded slice of an IDR picture (SLICE_IDR) and coded slice of an IDR picture in scalable extension (SLICE_IDR_S) .
• SEI Supplementary Enhancement Information
• SPS sequence parameter set
• SPS_S sequence parameter sets in scalable extension
• PPS picture parameter sets
• PPS_S picture parameter sets in scalable extension
• coded slice of an IDR picture SLICE_IDR
• SLICE_IDR_S coded slice of an IDR picture
• SLICE_IDR_S coded slice of an IDR picture in scalable extension
• GOPs groups of pictures
• Pictures are either normal non-IDR pictures (SLICE_NOIDR) or non-IDR pictures in scalable extension (SLICE_NOIDR_S) .
• the bit stream has two spatial layers and the GOP size is 16, so that one line (16 BL packets + 16 EL packets) makes a complete GOP.
• the number of complete GOPs in a stream may be very large, e.g. several thousands.
• the last picture in each GOP is an I- or P-frame (intra-coded or predicted) serving as key picture, while the others are B-frames (bi- directionally predicted) .
• Fig.2 shows temporal levels within hierarchical B frame GOPs of different size.
• the frames on the lowest temporal level TLO are called key pictures and are either I or P frames, and one of them (denoted as 0) belongs actually to the previous GOP.
• frames on higher temporal levels than TLO are bi-directionally predicted from two or more reference frames.
• Reference frames can be frames on lower temporal levels or (for frames that are not on the highest temporal level) previously decoded frames of the same temporal level.
• Temporal levels are decoded in increasing order, i.e. first TLO, then TLl, then TL2 etc.
• the invention provides packet loss detection and virtual frame generation, in particular for packets with non-IDR pictures (denoted Slice_NOIDR/Slice_NOIDR_S in Fig.l).
• the transmission of packets belonging to SEI Message, SPS, PPS and the first IDR picture is assumed to be guaranteed.
• a first phase of packet loss detection is followed by lost packet analysis, virtual packet generation and insertion, parsing and decoding.
• the packet loss detection uses the picture_order_count (POC) values, frame_num values and temporal level values.
• POC is an enumeration of the frames in display order, as used in Fig.2 a) -d) .
• frame_num is a parameter of a frame that indicates the number of (potential) reference frames that it has.
• POC_Increase we usually assume for the highest spatial layer a POC_Increase of 1, i.e. all frames are displayed. However, sometimes this is not true. E.g.
• one aspect of the present invention is to add an indication Iog2_min_poc_increase of the minimum POC Increase to the SEI message, as exemplarily shown in Tab.l and described in the following.
• the POC_Increase is 2 log2 - min - poc - increase for the spatial layer with the maximum MTL (MTL max ) .
• MTL max the maximum temporal level of MTL Layer
• a virtual packet can be generated for the parser and the decoder. This is done in the decoder and before the real parsing. Always when a packet is sent to the parser, no matter if it is a correctly received packet or a virtual packet, the number of decoded pictures (decoded_index) will be incremented. Therefore it is advantageous to insert the virtual packets before the parser, so that the numbering of subsequent packets is correct.
• the temporal level of each layer is defined in the SVC Scalability Information SEI message syntax (see JVT_Q201).
• MTL the maximum temporal level of all the layers belonging to the same dependency level (spatial resolution layer) as MTL, so the size for the normal GOP is
• GopSize 2 MTL .
• the relative POCs in a GOP are constructed using the hierarchical B frame structure.
• the relative POC, relative frame num and temporal level values are listed by decoding order in the following tables, cf . Fig.2 a) -d) . They can be easily generated for a specific GOP size. Alternatively, the following schemes can be predefined, and a scheme corresponding to the received GOPSize is selected.
• the real frame_num actually might be modified by modulus operation with the maximum frame number in real applications .
• the first frame in the decoding order actually is not exactly with an order of 0. Instead, it is usually more accurate for this frame to have a decoding order value of - (GopSize-1) , because it has been coded at the beginning of the previous GOP.
• Frames or frame information can be ordered by various criteria, e.g. by POC, by decoding order, by frame_num etc,
• This feature is helpful for packet loss detection and virtual packet generation.
• the POC increase of two successive displayed frames might be 1 or 2, or even 2 n when the number of spatial scalable layers is more than two and each layer has a different frame rate.
• the POC increase needs to be considered.
• the frame num should be increased by 1 (except for the frames at the highest temporal level) . If the value
• the frame wi l l not cause frame_num to increase; we call it odd frame here.
• these odd frames will have the highest temporal level.
• the POC_Increase may be larger; generally it is 2 n , where n is the difference between MTL of the current layer and the MTL of the highest temporal layer.
• the highest (spatial/ spatio-temporal) layer will have the maximum number of temporal levels.
• the BL can have half the frame rate of the EL.
• the MTL of the BL is 5.
• the GOP sizes for BL and EL are 2 5 and 2 6 respectively.
• each spatial or FGS layer will be composed by GopSize-n +d packets.
• n complete GOPs There are n complete GOPs and then maybe an incomplete GOP with a size d, where d is less than GopSize .
• "Incomplete” however refers only to the fact that it contains fewer frames than "complete" GOPs, while an "incomplete" GOP is syntactically correct and the properties (see below) for each frame are adapted to the actual GOP size.
• A can be ordered by decoding order, by POC, by frame_num, or even first ordered by temporal_level and then within the same temporal level by POC.
• the elements in A are updated, usually with a POC increase of GopSize and a frame num increase of GopSize/2.
• (some) POC numbers of subsequent packets may be regarded.
• these are ⁇ 1,3,5,7,9,11,13, 15 ⁇ , which matches the subsequently expected packets of the GOP. Therefore it can be concluded that we are in a normal GOP, and only two packets are missing, namely 6 and 10. As shown in Fig.3b, these packets are then replaced by virtual packets il, i2.
• a GOP being an abnormal GOP contains only 9 frames.
• abnormal GOP process Once a packet loss in an abnormal GOP is detected, the following "abnormal GOP process" may be entered.
• the GopSize is changed for the abnormal GOP, because it is different. Since the whole decoding process is near the end of the bit-stream, the GopSize value will be changed soon anyway.
• an upper bound for GopSize is determined. If current frame f c is an odd frame, we can know the upper bound of the GopSize with a possible error of 1 ( ⁇ and >> are shift operators) :
• GopSize abnrmax ( (frame_num-l) % (GopSize»l) «1) +1 (eq.2)
• the iterative algorithm to follow is to start from the difference in frame_num between f" and f c with both the same POC number, and decrease the value for each frame in A that has higher POC number but lower temporal_level than the current frame sfC until the value is zero.
• the POC number of the last frame gives an upper bound for the GOP size :
• GopSize abn ,ma ⁇ f t -ipoc (i- e - the POC value of the frame i-1)
• Gopsize abn ,ma ⁇ is the upper bound of GOP size of the abnormal GOP of the current spatial layer.
• each spatial layer may have its own list of elements A.
• the frame_num values in A are updated.
• the updated A we give a new order for A: it is first ordered by temporal level, and within the same temporal level it is ordered by POC.
• f c f r ame num means: frame_num of frame f c Actually this order is like the decoding order of a normal GOP. For simplicity, we will still call it decoding order. However, we use an ordering index t. So we have a newly ordered A
• frame_num frame _ num%Max _ Frame _ Num
• the GopSize of the truncated A may be stored for the next process of packet loss detection and packet virtualization .
• decoded_index we can always find an element in A if it is ordered by decoding order: fdecoded_inde ⁇ +i • I f the f i s equal to -fdecoded_ ⁇ nde ⁇ +i r no packet is lost and the current received packet is sent to the parser .
• Else a virtual packet is constructed using ⁇ decoded xndex+i • In both cases decoded_index is increased by 1
• a new reference lists construction method is helpful for BLSkip mode, as described in the following.
• virtual syntax creation is described.
• the correct values for the virtual packets can be generated since the pic_order_cnt_lsb and frame_num is known from the slice header. If the temporal level is not zero, the virtual packet is specified as B frame with nal_ref_idc of 1 or 0 (if it is the highest temporal level, it will be 0), else it is specified as P frame with a nal ref idc of 3.
• MMCO memory management control operation
• RPLR and reference index numbers need to be handled. Conventionally only the key picture needs to add RPLR commands, forcing the key picture to refer to the previous key picture.
• the BL packet values of num_ref_idx_10_active_minusl and num ref idx 11 active minusl are set for the spatial EL packet .
• match means that if we have the same ref_idx_lx values of list_X in the BL and EL, the corresponding reference pictures of the BL and EL will have the same display time (or POC) .
• the conventional JSVM encoder guarantees this by using RPLR for BL, and the initialisation of reference lists construction to match the BL at the EL. So it is not necessary to generate RPLR commands for lost non-key pictures, but we still need to generate RPLR for key pictures.
• the invention provides the following advantages.
• Any packet loss can be handled, except for the packets belonging to SEI Message, SPS, PPS and IDR.
• Each resolution can have one or more FGS layers.
• An FGS layer will be intentionally dropped if its corresponding lower quality level packet (with the same dependence level and temporal level) is lost or has been intentionally dropped.
• Insertion of default packets before the parser has the advantage that the parser needs not handle packet loss situations, which e.g. may require packet renumbering, so that conventional simple parsers can be used.
• the payload content of such packets is not relevant.
• the disclosed concept of the invention can also be used in cases where packets contain more than one frame or slice.
• the invention can be used for video decoding products or for video encoding products, particularly if a base-layer and at least one enhancement-layer are included and have a plurality of temporal levels defined.
• the spatial layers may have different frame rates.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=37735133

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (23)

Citations (2)

Family Cites Families (25)

• 2006
  • 2006-02-27 EP EP06300164A patent/EP1827023A1/en not_active Withdrawn
• 2007
  • 2007-02-14 WO PCT/EP2007/051451 patent/WO2007096288A1/en not_active Ceased
  • 2007-02-14 CN CN2007800067870A patent/CN101390400B/zh not_active Expired - Fee Related
  • 2007-02-14 JP JP2008555755A patent/JP5227193B2/ja not_active Expired - Fee Related
  • 2007-02-14 US US12/224,239 patent/US8249170B2/en not_active Expired - Fee Related
  • 2007-02-14 EP EP07704585A patent/EP1989884A1/en not_active Ceased

Patent Citations (2)

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events