RU2776357C1 - Video stream concept - Google Patents

Abstract

FIELD: video data processing technologies.

SUBSTANCE: inventions group relates to video data processing technologies. A device for decoding a data stream for recovering video content is proposed. The device contains a decoder configured to extract from the data stream a sequence of packets representing sub-parts of video content and organized into a plurality of access blocks, each of the plurality of access blocks refers to an image of the video content and includes one or more decoding blocks, and each block decoding includes at least one payload packet, and each of the sub-parts corresponds to a slice. Extraction involves entropy decoding of slices across slice boundaries according to a parallel wavefront processing technique. The decoder is also configured to identify, in each of the plurality of access blocks, one or more timing control packets corresponding to one or more decoding blocks, respectively, and placed in the access block, each of the timing control packets indicating the extraction time from the decoder buffer at which the contents of the corresponding decoding block must be retrieved from the buffer.

EFFECT: ensuring low end-to-end delay when encoding/decoding video content images.

18 cl, 39 dwg

Description

The present application relates to video stream concepts which are particularly advantageous in connection with low latency applications.

HEVC [2] allows various means of signaling the High Level Syntax to the application layer. Such means are the NAL block header, Parameter Sets, and Supplementary Enhancement Information (SEI) messages. The latter are not used in the decoding process. Other means of High Level Syntax signaling derive from relevant transport protocol specifications, such as MPEG2 Transport Protocol [3] or Real Time Transport Protocol [4], and their specific payload specifications, such as recommendations for H.264/AVC [5], scalable video coding (SVC) [6] or HEVC [7]. Such transport protocols may introduce High Level signaling that uses similar structures and mechanisms as High Level signaling of a conforming application layer codec specification, such as HEVC [2]. One example of such signaling is the Payload Content Scalability Information (PACSI) NAL block as described in [6], which provides additional information to the transport layer.

For parameter sets, HEVC includes a Video Parameter Set (VPS), which collects the most important stream information to be used by the application layer, in a single and central location. In earlier approaches, it was necessary that this information be collected from several Parameter Sets and NAL block headers.

Prior to this application, the status of the standard with respect to the operations of the Coded Picture Buffer (CPB) of the Hypothetical Reference Decoder (HRD) and all related syntax provided in the Sequence Parameter Set (SPS)/Video Applicability Information (VUI), SEI Image Timing, SEI Buffer Period, as well as the definitions of the decoding block describing the sub-image and the syntax of Dependent Slices as being present in the slice header, as well as the Picture Parameter Sets (PPS), were as follows.

In order to enable low latency CPB operations at the sub-picture level, CPB sub-pictures operations have been proposed and integrated into the draft HEVC 7 standard JCTVC-11003 [2]. Here especially, the decoding block was defined in section 3 in [2] as:

decoding block: An access block or a subset of an access block. If SubPicCpbFlag is 0, the decoding block is an access block. Otherwise, the decode unit consists of one or more VCL NAL units in the access unit and associated non-VCL NAL units. For the first VCL NAL unit in the access unit, the associated non-VCL NAL units are the padding data NAL units, if any, immediately following the first VCL NAL unit and the non-VCL NAL units in the access unit that precede the first VCL NAL unit. For a VCL NAL that is not the first VCL NAL in an access unit, the associated non-VCL NALs are the padding data NAL, if any, immediately following the VCL NAL.

In the standard defined up to that time, "Decoding Block Removal Timing and Decoding Block Decoding" was described and added to Appendix C "Hypothetical Reference Decoder". In order to signal sub-picture timing, the buffer period SEI message and the picture timing SEI message, as well as the HRD parameters in the VUI, have been extended to support decoding blocks as sub-picture blocks.

The syntax [2] of the buffer period SEI message is shown in FIG.

When NalHrdBpPresentFlag or VclHrdBpPresentFlag is equal to 1, a buffer period SEI message may be associated with any access block in the bitstream, and a buffer period SEI message will be associated with each RAP access block and each access block associated with the recovery point SEI message.

For some applications, the frequent presence of a buffer period SEI message may be desirable.

The buffer period was defined as the set of access blocks between two buffer period SEI message objects in decoding order.

The semantics were as follows:

seq_parameter_set_id defines a sequence parameter set that contains the HRD attributes of the sequence. The value of seq_parameter_set_id will be equal to the value of seq_parameter_set_id in the picture parameter set referenced by the primary encoded picture associated with the buffer period SEI message. The value of seq_parameter_set_id must be between 0 and 31, inclusive.

rap_cpb_params_present_flag equal to 1 specifies the presence of the initial_alt_cpb_removal_delay[ SchedSelIdx ] and initial_alt_cpb_removal_delay_offset[ SchedSelIdx ] syntax elements. When not present, the value of rap_cpb_params_present_flag is assumed to be 0. When the associated picture is neither a CRA picture nor a BLA picture, the value of rap_cpb_params_present_flag will be 0.

initial_cpb_removal_delay[ SchedSelIdx ] and initial_alt_cpb_removal_delay[ SchedSelIdx ] define the initial CPB removal delays for the SchedSelIdxth CPB. The syntax elements have a length in bits given by initial_cpb_removal_delay_length_minus1 + 1 and are expressed in units of a 90 kHz clock signal. Syntax element values will not be 0 and will not exceed 90000 * ( CpbSize[ SchedSelIdx ] ÷ BitRate[ SchedSelIdx ]), the time equivalent of the CPB size in units of 90 kHz clock.

initial_cpb_removal_delay_offset[ SchedSelIdx ] and initial_alt_cpb_removal_delay_offset[ SchedSelIdx ] are used for the SchedSelIdx-th CBP to set the initial delivery time of encoded data blocks to the CPB. The syntax elements have a length in bits given by initial_cpb_removal_delay_length_minus1 + 1 and are expressed in units of a 90 kHz clock signal. These syntax elements are not used by decoders and may only be needed by the delivery scheduler (HSS).

Across the encoded video sequence, the sum of initial_cpb_removal_delay[ SchedSelIdx ] and initial_cpb_removal_delay_offset[ SchedSelIdx ] must be constant for each SchedSelIdx value, and the sum of initial_alt_cpb_removal_delay[ SchedSelIdx ] and initial_alt_cpb_removal_delay_offset[ SchedSelIdx ] must be constant for each SchedIdxSelIdx value.

The syntax of the picture timing SEI message from [2] is shown in FIG.

The syntax of the picture timing SEI message was dependent on the content of the sequence parameter set that is active for the encoded picture associated with the picture timing SEI message. However, if the IDR or BLA picture timing SEI message is not preceded by a buffer period SEI message in the same access unit, activating the associated sequence parameter set (and, for IDR or BLA pictures that are not the first picture in the bitstream, determining that the encoded picture is an IDR picture or a BLA picture) does not occur until the first encoded NAL unit of the encoded picture slice is decoded. Since the encoded NAL unit of the encoded picture slice follows the picture timing SEI message in NAL unit order, there may be cases in which it is necessary for the decoder to store the RBSP containing the picture timing SEI message before determining the sequence parameter parameters that will be active for the encoded picture, and then parse the image timing SEI message.

The presence of the picture timing SEI message in the bitstream was specified as follows:

- If CpbDpbDelaysPresentFlag is equal to 1, one picture timing SEI message will be present in each access block of the encoded video sequence.

- Otherwise (CpbDpbDelaysPresentFlag is 0), the picture timing SEI message will not be present in every access block of the encoded video sequence.

The semantics has been defined as follows:

cpb_removal_delay specifies how many clock cycles to wait after the access block associated with the most recent buffer period SEI message in the previous access block is removed from the CPB before the access block data associated with the picture timing SEI message is removed from the buffer. This value is also used to calculate the earliest possible time of arrival of access unit data in the CPB for the HSS. The syntax element is a fixed length code whose length in bits is given by cpb_removal_delay_length_minus1 + 1. cpb_removal_delay is the remainder of the counter modulo 2^(cpb_removal_delay_length_minus1 + 1).

The cpb_removal_delay_length_minus1 value, which specifies the length (in bits) of the cpb_removal_delay syntax element, is the cpb_removal_delay_length_minus1 value encoded in the sequence parameter set that is active for the primary encoded picture associated with the picture timing SEI message, although cpb_removal_delay specifies the number of clock cycles with respect to time deleting the previous access block containing the buffering period SEI message, which may be an access block of another encoded video sequence.

dpb_output_delay is used to calculate the output time of a DBP image. It determines how many clock cycles to wait after the last decoding block in the access block is removed from the CPB before the decoded picture is output from the DPB.

The image is not removed from the DBP at its render time while it is still marked as "in use for short-term link" or "in use for long-term link".

Only one dpb_putput_delay is defined per decoded picture.

The length of the dpb_output_delay syntax element is given in bits by dpb_output_delay_length_minus1 + 1. When sps_max_dec_pic_buffering[ max_temporal_layers_minus1 ] is 0, dpb_output_delay shall be 0.

The output time obtained from the dpb_output_delay of any picture that is output from the satisfying decoder output time must precede the output time obtained from the dpb_output_delay of all pictures in any subsequent encoded video sequence in decoding order.

The order in which images are displayed, set by the values of this syntax element, will be the same order as set by the values of PicOrderCntVal.

For pictures that are not output by the "peep" process because they precede in decoding order an IDR or BLA picture with no_output_of_prior_pics_flag equal to 1 or implied equal to 1, the output times obtained from dpb_output_delay will be incremented with increasing value PicOrderCntVal with respect to all images in the same encoded video sequence.

num_decoding_units_minus1 plus 1 specifies the number of decoding units in the access unit with which the picture timing SEI message is associated. The value of num_decoding_units_minus1 must be between 0 and PicWidthInCtbs * PicHeightInCtbs - 1, inclusive.

num_nalus_in_du_minus1[ i ] plus 1 specifies the number of NAL units in the i-th decode unit of the access unit with which the picture timing SEI message is associated. The value of num_nalus_in_du_minus1[ i ] plus 1 must be between 0 and PicWidthInCtbs * PicHeightInCtbs - 1, inclusive.

The first decoding block of the access block consists of the first num_nalus_in_du_minus1[ 0 ] + 1 consecutive NAL blocks in decoding order in the access block. The i-th (where i is greater than 0) access block decode block consists of num_nalus_in_du_minus1[ i ] + 1 consecutive NAL blocks immediately following the last NAL block in the preceding access block decode block, in decoding order. Each decoding unit will have at least one VCL NAL unit. All non-VCL NAL units associated with a VCL NAL unit will be included in the same decoding unit.

du_cpb_removal_delay[ i ] specifies how many sub-picture clock cycles to wait after removing from the CPB the first decoding block in the access block associated with the most recent buffering period SEI message in the previous access block, before removing the i-th decoding block in the access block from the CPB, associated with the picture timing SEI message. This value is also used to calculate the earliest possible time of arrival of the decoding block data in the CPB for the HSS. The syntax element is a fixed length code whose length in bits is given by cpb_removal_delay_length_minus1 + 1. du_cpb_removal_delay[ i ] is the remainder of the counter modulo 2^(cpb_removal_delay_length_minus1 + 1).

The cpb_removal_delay_length_minus1 value that specifies the length (in bits) of the du_cpb_removal_delay[ i ] syntax element is the cpb_removal_delay_length_minus1 value encoded in the sequence parameter set that is active for the encoded picture associated with the picture timing SEI message, although du_cpb_removal_delay[ i ] specifies the number of clocks synchronization of the sub-picture with respect to the removal time of the first decoding block in the previous access block containing the buffering period SEI message, which may be an access block of another encoded video sequence.

Some information was contained in the VUI syntax [2]. The syntax of the VUI parameters from [2] is shown in Fig.3. The syntax of HDR parameters from [2] is shown in Fig.4. The semantics has been defined as follows:

sub_pic_cpb_params_present_flag equal to 1 specifies that the sub-picture layer CPB deletion delay parameters are present and the CPB may operate at the access unit layer or the sub-picture layer. sub_pic_cpb_params_present_flag equal to 0 specifies that sub-picture layer CPB deletion delay parameters are not present and that the CPB operates at the access unit layer. When sub_pic_cpb_params_present_flag is not present, its value is assumed to be 0.

num_units_in_sub_tick represents the number of time units of a clock operating on a Hz frequency time scale that corresponds to one increment (called the sub-picture clock tick) of the sub-picture clock tick counter. num_units_in_sub_tick will be greater than 0. The sub-picture clock tick is the minimum time interval that can be represented in encoded data when sub_pic_cpb_params_present_flag is equal to 1.

sub_pic_cpb_params_present_flag equal to 1 indicates that each picture parameter set that is active in the encoded video sequence has the same value of the num_tile_columns_minus1, num_tile_rows_minus1, uniform_spacing_flag, column_width[ i ], row_height[ i ] and loop_filter_across_tiles_enabled_flag syntax elements when they are present. tiles_fixed_structure_flag equal to 0 indicates that tile syntax elements in different image parameter sets may or may not have the same value. When the tiles_fixed_structure_flag syntax element is not present, its value is assumed to be 0.

Signaling tiles_fixed_structure_flag equal to 1 is a guarantee to the decoder that each picture in the encoded video sequence has the same number of tiles distributed in the same way, which can be useful for workload allocation in case of multi-threaded decoding.

The padding data in [2] was signaled using the filter data RBSP syntax shown in FIG.

The hypothetical Reference Decoder from [2] used to check the bitstream and decoder match was defined as follows:

The two types of bitstreams are subject to HRD compliance testing for this Recommendation | International Standard. The first such type of bitstream, referred to as a Type I bitstream, is a NAL unit stream containing only VCL NAL units and stuffing data NAL units for all access units in the bitstream. The second type of bitstream, referred to as a Type II bitstream, contains, in addition to VCL NAL units and stuffing data NAL units for all access units in the bitstream, at least one of the following:

- additional non-VCL NAL units other than padding data NAL units,

- all leading_zero_8bits, zero_byte, start_code_prefix_one_3bytes, and trailing_zero_8bits syntax elements that form a byte stream from the NAL unit stream.

6 shows the bitstream type matching points checked by HRD in [2].

Two types of HRD parameter sets are used (NAL HRD parameters and VCL HRD parameters). The HRD parameter sets are signaled via video applicability information, which is part of the syntax structure of the sequence parameter set.

All sequence parameter sets and picture parameter sets referred to in the VCL NAL units and the corresponding picture buffering period and picture timing SEI messages shall be transmitted to the HRD in a timely manner, either in the bitstream or by other means.

The specification for the "presence" of non-VCL NAL units is also satisfied when those NAL units (or only some of them) are conveyed to decoders (or HRDs) by other means not defined by this Recommendation | International Standard. For the purposes of bit counting, only the corresponding bits that are actually present in the bitstream are counted.

As an example, synchronization of a non-VCL NAL unit transmitted by means other than presence in the bitstream, along with NAL units that are present in the bitstream, can be achieved by specifying two points in the bitstream between which the non-VCL NAL blocks would be present in the bitstream if the encoder chose to transmit them in the bitstream.

When the content of the non-VCL NAL of the unit is submitted for use by some means other than being present in the bitstream, the presentation of the content of the non-VCL NAL of the unit is not required to use the same syntax defined in this annex.

It should be noted that when information is contained within a bitstream, it is possible to verify that the bitstream complies with the requirements of this subclause solely on the basis of the information contained in the bitstream. When HRD information is not present in the bitstream, as is the case for all "standalone" Type I bitstreams, conformance can only be checked when the HRD data is supplied by some other means not specified in this Recommendation | International Standard.

The HRD contains a coded picture buffer (CPB), an instant decoding process, a decoded picture buffer (DPB), and an output framing as shown in FIG.

The CPB size (number of bits) is CpbSize[ SchedSelIdx ]. The DBP size (number of image storage buffers) for a temporal layer X is sps_max_dec_pic_buffering[ X ] for each X in the range 0 to sps_max_temporal_layers_minus1, inclusive.

The variable SubPicCpbPreferredFlag is either externally set or, when not externally set, set to 0.

The SubPicCpbFlag variable is obtained as follows:

SubPicCpbFlag = SubPicCpbPreferredFlag && sub_pic_cpb_params_present_flag

If SubPicCpbFlag is equal to 0, the CPB operates at the access block level, and each decoding block is an access block. Otherwise, the CPB operates at the sub-picture level, and each decoding block is a subset of the access block.

HRD works as follows. Data associated with decoding units that arrive at the CPB according to a set arrival schedule are delivered by the HSS. The data associated with each decoding block is removed and instantaneously decoded by the instantaneous decoding process at the CPB removal times. Each decoded picture is placed in the DPB. The decoded picture is removed from the DBP at a later time from the DPB output time or the time when one becomes no longer needed for the inter prediction reference.

The HRD is initialized as indicated by the SEI buffer period. The timing of removing decoding blocks from the CPB and the timing of outputting decoded pictures from the DPB are indicated in the Picture Timing SEI message. All timing information related to a particular decoding block must arrive before the decoding block CPB removal time.

HRD is used to check if bitstreams and decoders match.

While conformance is guaranteed provided that all frame rates and clocks used to generate the bitstream exactly match the values signaled in the bitstream, in a real system, each of these can vary from the transmitted or given value.

All arithmetic operations are performed on real values, so no rounding errors can propagate. For example, the number of bits in the CPB immediately before or after decoding block deletion is not necessarily an integer.

The variable tc is obtained as follows and is called the clock cycle:

tc = num_units_in_tick ÷ time_scale

The variable tc_sub is obtained as follows and is called the sub-picture clock cycle:

tc_sub = num_units_in_sub_tick ÷ time_scale

The following is defined to express constraints:

- Let access block n be the nth access block in decoding order, with the first access block being access block 0.

- Let picture n be the encoded picture or the decoded picture of the access unit n.

- Let decoding block m be the m-th decoding block in decoding order, with the first decoding block being decoding block 0.

In [2], the slice header syntax allows for so-called dependent slices.

8 shows the slice header syntax in [2].

The slice header semantics has been defined as follows:

dependent_slice_flag equal to 1 indicates that the value of each slice header syntax element that is not present is assumed to be equal to the value of the corresponding slice header syntax element in the preceding slice containing the coding tree block for which the coding tree block address is SliceCtbAddrRS - 1. In the case when not present, the dependent_slice_flag value is assumed to be 0. The dependent_slice_flag value shall be 0 when SliceCtbAddrRS is 0.

slice_address specifies the address in the slice granularity resolution at which the slice begins. The length of the slice_address syntax element is ( Ceil( Log2( PicWidthInCtbs * PicHeightInCtbs ) ) + SliceGranularity ) bits.

The SliceCtbAddrRS variable specifying the coding tree block in which the slice starts in the bitmap scan order of the coding tree block is obtained as follows.

SliceCtbAddrRS = ( slice_address >> SliceGranularity )

The variable SliceCtbAddrZS, which defines the address of the first coding block in the slice in the minimum granularity of the coding block in the z-order of the scan, is obtained as follows.

SliceCbAddrZS = slice_address

<< ( ( log2_diff_max_min_coding_block_size - SliceGranularity ) << 1 )

Slice decoding starts with the largest possible coding block at the slice start coordinate.

first_slice_in_pic_flag indicates whether the slice is the first slice of an image. If first_slice_in_pic_flag is equal to 1, the variables SliceCbAddrZS and SliceCtbAddrRS are both set to 0, and decoding starts from the first coding tree block in the picture.

pic_parameter_set_id specifies the set of image parameters to use. The value of pic_parameter_set_id must be between 0 and 255, inclusive.

num_entry_point_offsets specifies the number of the entry_point_offset[ i ] syntax element in the slice header. When tiles_or_entropy_coding_sync_idc is 1, num_entry_point_offsets must be in the range 0 to ( num_tile_columns_minus1 + 1 ) * ( num_tile_rows_minus1 + 1) - 1, inclusive. When tiles_or_entropy_coding_sync_idc is 2, num_entry_point_offsets must be in the range 0 to PicHeightInCtbs - 1, inclusive. When not present, the value of num_entry_point_offsets is assumed to be 0.

Offset_len_minus1 plus 1 specifies the length, in bits, of entry_point_offset[ i ] syntax elements.

entry_point_offset[ i ] specifies the i-th entry point offset, in bytes, and must be represented by offset_len_mmusl plus 1 bits. The encoded slice data after the slice header consists of num_entry_point_offsets + 1 subsets, with subset index values ranging from 0 to num_entry_point_offsets, inclusive. Subset 0 consists of bytes 0 to entry_point_offset[ 0 ] - 1, inclusive, of encoded slice data, subset k, where k is in the range 1 to num_entry_point_offsets - 1, inclusive, consists of bytes entry_point_offset[ k-1 ] to entry _point_offset[ k ] + entry_point_offset[ k-1 ] - 1, inclusive, of encoded slice data, and the last subset (with subset index equal to num_entry_point_offsets) consists of the remaining bytes of encoded slice data.

When tiles_or_entropy_coding_sync_idc is 1 and num_entry_point_offsets is greater than 0, each subset must contain all encoded bits of exactly one tile, and the number of subsets (i.e., the value of num_entry_point_offsets + 1) must be equal to or less than the number of tile in the slice.

When tiles_or_entropy_coding_sync_idc is equal to 1, each slice must include either a subset of a single tile (in which case entry point signaling is optional) or an integer number of complete tile.

When tiles_or_entropy_coding_sync_idc is 2 and num_entry_pomt_offsets is greater than 0, each subset of k, where k is in the range 0 to num_entry_point_offsets - 1, inclusive, shall contain all encoded bits of exactly one line of coding tree blocks, the last subset (with subset index equal to num_entry_point_offsets) must contain all encoded bits of the remaining coding blocks included in the slice, where the remaining coding blocks consist of either exactly one line of coding tree blocks or a subset of one line of coding tree blocks, and the number of subsets (i.e. the value of num_entry_point_offsets + 1) must be equal to the number of coding tree block lines in the slice, where a subset of one coding tree block line in the slice is also counted.

When tiles_or_entropy_coding_sync_idc is equal to 2, the slice may include a number of coding tree blocks and a subset of a row of coding tree blocks. For example, if a slice includes two and a half lines of coding tree blocks, the number of subsets (i.e. num_entry_point_offsets + 1) should be 3.

9 shows the RBSP syntax of a picture parameter set in [2], wherein the RBSP semantics of a picture parameter set in [2] is defined as:

dependent_slice_enabled_flag equal to 1 specifies the presence of the dependent_slice_flag syntax element in the slice header for encoded pictures belonging to the picture parameter set. dependent_slice_enabled_flag equal to 0 indicates the absence of the dependent_slice_flag syntax element in the picture header for encoded pictures belonging to the picture parameter set. When tiles_or_entropy_coding_sync_idc is 3, dependent_slice_enabled_flag must be 1.

tiles_or_entropy_coding_sync_idc equal to 0 indicates that there should be only one tile in each image related to the image parameter set, there will be no specific synchronization process for context variables called prior to decoding the first coding tree block of the string of coding tree blocks in each image related to to a picture parameter set, and the cabac_independent_flag and dependent_slice_flag values for encoded pictures related to the picture parameter set shall not both be equal to 1.

When cabac_independent_flag and dependent_slice_flag are both equal to 1 for a slice, the slice is an entropy slice.

tiles_or_entropy_coding_sync_idc equal to 1 indicates that there can be more than one tile in each image related to the image parameter set, there will be no specific synchronization process for context variables called prior to decoding the first coding tree block of the string of coding tree blocks in each image related to to a picture parameter set, and the cabac_independent_flag and dependent_slice_flag values for encoded pictures related to the picture parameter set shall not both be equal to 1.

tiles_or_entropy_coding_sync_idc equal to 2 indicates that there should be only one tile in each picture belonging to the set of image parameters, a certain synchronization process for context variables should be called before decoding the first block of the coding tree of the string of blocks of the coding tree in each image belonging to the set image parameters, and a defined memory process for context variables shall be invoked after decoding two coding tree blocks of a line of coding tree blocks in each image belonging to the image parameter set, and cabac_independent_flag and dependent_slice_flag values for encoded images belonging to the image parameter set shall not be both equal to 1.

tiles_or_entropy_coding_sync_idc equal to 3 indicates that there should be only one tile in each image belonging to the image parameter set, there will be no specific synchronization process for context variables called prior to decoding the first coding tree block of the string of coding tree blocks in each image belonging to to the picture parameter set, and the cabac_independent_flag and dependent_slice_flag values for encoded pictures related to the picture parameter set may both be 1.

When dependent_slice_enabled_flag shall be 0, tiles_or_entropy_coding_sync_idc must not be 3.

This is a bitstream conformance requirement that the value of tiles_or_entropy_coding_sync_idc must be the same for all picture parameter sets that are enabled in the encoded video sequence.

For each slice belonging to a picture parameter set, when tiles_of_entropy_coding_sync_idc is equal to 2 and the first coding block in the slice is not the first coding block in the first coding tree block of a coding tree block line, the last coding block in the slice must belong to the same coding tree block line, as the first coding block in the slice.

num_tile_columns_minus1 plus 1 specifies the number of tile columns that divide the image.

num_tile_rows_minus1 plus 1 specifies the number of tile rows that separate the image. When num_tile_columns_minus1 is 0, num_tile_rows_minus1 must not be 0.

uniform_spacing_flag equal to 1 indicates that the column borders and similarly the row borders are evenly spaced across the image. uniform_spacing_flag equal to 0 indicates that column boundaries and similarly row boundaries are not evenly spaced across the image, but are signaled explicitly using the column_width[ i ] and row_height[ i ] syntax elements.

column_width[ i ] specifies the width of the i-th tile column in units of coding tree blocks.

row_height[ i ] defines the height of the i-th row of the tile in units of coding tree blocks.

The colWidth[ i ] vector defines the width of the i-th tile column in CTB units, with the i-column ranging from 0 to num_tile_columns_minus1, inclusive.

The vector CtbAddrRStoTS[ ctbAddrRS ] defines the conversion from a CTB address in raster scan order to a CTB address in scan order with an index ctbAddrRS ranging from 0 to (picHeightInCtbs * picWidthInCtbs) - 1, inclusive.

The CtbAddrTStoRS[ ctbAddrTS ] vector defines the conversion from a CTB address in tile scan order to a CTB address in raster scan order with a ctbAddrTS index ranging from 0 to (picHeightInCtbs * picWidthInCtbs) - 1, inclusive.

The TileId[ ctbAddrTS ] vector defines the conversion from the CTB address in tile scan order to a tile id with ctbAddrTS ranging from 0 to (picHeightInCtbs * picWidthInCtbs) - 1, inclusive.

The values colWidth, CtbAddrRStoTS, CibAudrTSloRS and TileId are obtained by invoking the CTB raster transformation and tile scanning process as defined in subclause 6.5.1 with PicHeightInCtbs and PicWidthInCtbs as input, and the output is assigned to colWidth, CtbAddrRStoTS and TileId.

The ColumnWidthInLumaSamples[ i ] values that define the width of the i-th image fragment column in luma sampling units are set to colWidth[ i ] << Log2CtbSize.

An array of MinCbAddrZS[ x ][ y ] specifying the conversion from location ( x, y ) in units of minimum CBs to the minimum address of CBs in z-order scan, with x being in the range 0 to picWidthInMinCbs - 1, inclusive, and y being in the range 0 to picHeightInMinCbs - 1, inclusive, is obtained by calling the scan order Z array initialization process as defined in subclause 6.5.2 with Log2MinCbSize, Log2CtbSize, PicHeightInCtbs, PicWidthInCtbs, and a CtbAddrRStoTS vector as input, and the output is assigned MinCbAddrZS.

loop_filter_across_tiles_enabled_flag equal to 1 indicates that in-loop filtering operations are performed across tile boundaries. loop_filter_across_tiles_enabled_flag equal to 0 indicates that in-loop filtering operations are not performed across tile boundaries. The in-loop filtering operations include the deblocking filter, adaptive sample bias, and adaptive loop filter operations. When not present, the value of loop_filter_across_tiles_enabled_flag is assumed to be 1.

cabac_independent_flag equal to 1 specifies that the CABAC decoding of the coding blocks in the slice is independent of any state of the previously decoded slice. cabac_independent_flag equal to 0 specifies that the CABAC decoding of coding blocks in a slice depends on the states of the previously decoded slice. When not present, the value of cabac_independent_flag is assumed to be 0.

The derivation process for the availability of a coding block with a minimum coding block address has been described as follows.

The input for this process is as follows

- minimum address minCbAddrZS of the encoding block in the z-order of scanning

- current minimum address currMinCBAddrZS of coding block in scan z-order

The output of this process is the availability of the coding block with the minimum address cbAddrZS of the coding block in the z-order of the cbAvailable scan.

Note that the availability value is determined when the process is invoked.

It should be noted that any coding block, regardless of its size, is associated with a minimum coding block address, which is the address of the coding block with the minimum coding block size in the z-order scan.

- If one or more of the following conditions are true, cbAvailable is set to FALSE.

- minCbAddrZS less than 0

- minCbAddrZS is greater than currMinCBAddrZS

- the coding block with the minimum address minCBAddrZS of the coding block belongs to a different slice than the coding block with the current minimum address currMinCBAddrZS of the coding block, and the dependent_slice_flag of the slice containing the coding block with the current minimum address currMinCBAddrZS of the coding block is equal to 0.

- the coding block with the minimum address minCBAddrZS of the coding block is contained in a different image fragment than the coding block with the current minimum address currMinCBAddrZS of the coding block.

- Otherwise, cbAvailable is set to TRUE.

The CABAC parsing process for slice data in [2] was as follows.

The process is invoked when parsing syntax elements with descriptor ae(v).

The input to this process is a request for the value of a syntax element and the value of previously parsed syntax elements.

The output of this process is the value of the syntax element.

When you start parsing slice data, the CABAC parsing process initialization process is called.

The minimum coding block address of the coding tree block containing the spatial neighboring block T (FIG. 10a), ctbMinCbAddrT, is obtained using the location ( x0, y0 ) of the top left luma sample of the current coding tree block as follows.

x = x0 + 2 << Log2CtbSize - 1

y = y0 - 1

ctbMinCbAddrT = MinCbAddrZS[ x >> Log2MinCbSize ][ y >> Log2MinCbSize ]

The availableFlagT variable is obtained by calling the coding block availability get process with CtbMinCbAddrT as input.

When starting encoding tree parsing, the following ordered steps apply.

1. The arithmetic decoding engine is initialized as follows.

- If CtbAddrRS is equal to slice_address, dependent_slice_flag is equal to 1, and entropy_coding_reset_flag is equal to 0, the following applies.

- The CABAC parser synchronization process is called with TableStateIdxDS and TableMPSValDS as input.

- The decoding process for binary solutions is called before completion is called, followed by the initialization process for arithmetic decoding.

- Otherwise, if tiles_or_entropy_coding_sync_idc is 2 and CtbAddrRS % PicWidthInCtbs is 0, the following applies.

- When availableFlagT is equal to 1, the CABAC parsing process synchronization process is called with TableStateIdxWPP and TableMPSValWPP as input.

- The decoding process for binary solutions is called before terminating, followed by the initialization process for the arithmetic decoding engine.

2. When cabac_independent_flag is 0 and dependent_slice_flag is 1, or when tiles_or_entropy_coding_sync_idc is 2, the learning process is applied as follows.

- When tiles_or_entropy_coding_sync_idc is equal to 2 and CtbAddrRS % PicWidthInCtbs is equal to 2, the CABAC parsing process store process is called with TableStateIdxWPP and TableMPSValWPP as output.

- When cabac_independent_flag is 0, dependent_slice_flag is 1, and end_of_slice_flag is 1, the CABAC parsing process store process is called with TableStateIdxDS and TableMPSValDS as output.

The parsing of syntax elements continues as follows:

For each requested value of the syntax element, a binarization (conversion to a binary representation) is obtained.

The binarization of the syntax element and the sequence of parsed containers determines the course of the decoding process.

For each syntax element binarization container that is indexed by the binIdx variable, the ctxIdx index is obtained.

For each ctxIdx, the arithmetic decoding process is called.

The resulting sequence (b ₀ ..b _binIdx ) of the parsed containers is compared to the set of container strings given by the binarization process after decoding each container. When a sequence matches a container string in a given set, the corresponding value is assigned to the syntax element.

In case a syntax element value request is processed for a pcm-flag syntax element and the decoded pcm-flag value is 1, the decoding engine is initialized after decoding any pcm_alignment_zero_bit, num_subsequent_pcm, and all pcm_sample_luma and pcm_sample_chroma data.

In the design of the infrastructure described so far, the following problem arose.

It is necessary to know the timing of decoding blocks before encoding and sending data in low latency scenarios where NAL blocks will already be sent by the decoder while the decoder is still encoding parts of the picture, i.e. other sub-picture decoding blocks. This is because the order of the NAL units in the access unit only allows SEI messages to precede the VCL (Video Encoding NAL units) in the access unit, but in such a low latency scenario, the non-VCL NAL units must be already on the wire, i.e. . sent if the encoder starts encoding decoding blocks. Fig.10b illustrates the structure of the block access, as defined in [2]. [2] had not yet determined the end of the flow sequence, so their presence in the access block was tentative.

In addition, the number of NAL units associated with a sub-picture must also be known in advance in a low latency scenario, since the picture timing SEI message contains this information and must be compiled and sent before the encoder starts encoding the actual picture. An application designer reluctant to insert padding data NAL units, potentially without padding data corresponding to the NAL unit number, as signaled to a decode unit in the picture SEI timing needs a means to signal this information at the sub-picture level. The same holds true for the sub-picture timing, which is currently fixed in the existence of the access block by the parameters specified in the timing SEI message.

Additional disadvantages of the Draft Specification [2] include numerous sub-image level signaling that is required for certain applications, such as ROI signaling or sub-image measurement signaling.

The problems outlined above are not specific to the HEVC standard. Rather, this problem occurs also in connection with other video codecs. 11 shows, more broadly, a video transmission landscape where a pair of encoder 10 and decoder 12 are connected via network 14 to transmit video 16 from encoder 10 to decoder 12 with short end-to-end delay. The problem already outlined above is the following. Encoder 10 encodes a sequence of frames 18 of video 16 in accordance with a specific decoding order, which essentially, but not necessarily, follows the playback order 20 of frames 18, and each frame 18 passes through the frame area of frames 18 in some specific manner, such as, for example, raster scanning method with or without partitioning of image fragments of frames 18. The decoding order controls the availability of information for the encoding methods used by the encoder 19, such as, for example, predictive and/or entropy coding, i.e. the availability of information relating to spatially and/or temporally adjacent portions of video 16 available to serve as a basis for prediction or context selection. Even though encoder 10 may be able to use parallel processing to encode frames 18 of video 16, encoder 10 necessarily needs some time to encode a particular frame 18, such as the current frame. 11, for example, illustrates the point in time when the encoder 10 has already finished encoding part 18a of the current frame 18, while the other part 18b of the current frame 18 has not yet been encoded. Because encoder 10 has not yet encoded part 18b, encoder 10 may not anticipate how the available data rate for encoding the current frame 18 must be distributed spatially over the current frame 18 to achieve an optimal value in terms of, for example, rate/distortion. Accordingly, encoder 10 has only two options: either encoder 10 estimates a near-optimal distribution of the available data rate for the current frame 18 into slices into which the current frame 18 is spatially subdivided in advance, respectively, assuming that the estimate may be incorrect, or encoder 10 completes the encoding of the current frame 18 before the packets containing the slices are transmitted from the encoder 10 to the decoder 12. In any case, in order to be able to take advantage of any transmission of slice packets of the current encoded frame 18 before its encoding is completed, the network 14 must be informed about the data rates associated with each slice packet, in the form of encoded picture buffer draw times. However, as mentioned above, although the decoder 10, in accordance with the current version of HEVC, has the ability to change the data rate distributed across frames 18 by using the determination of the encoded picture buffer extraction times for the sub-picture areas separately, the encoder 10 needs to transmit or send such information via the network 14 to the decoder 12 at the beginning of each access block, collecting all the data related to the current frame 18, thereby causing the encoder 10 to choose from the two alternatives just described, one of which results in a lower delay, but a poorer speed/distortion ratio, the other results in an optimal speed/distortion ratio but with increased end-to-end delay.

Thus, until now, there is no video codec that makes it possible to achieve such a low delay that the encoder would be able to start transmitting packets related to parts 18a of the current frame before encoding the remaining part 18b of the current frame, while the decoder is able to use this intermediate transmission of packets related to the preliminary parts 18a by the network 16, which is subject to the decoding buffer extraction timing transmitted in the video data stream sent from the encoder 12 to the decoder 14. Applications that would, for example, take advantage of such low delay roughly cover industrial applications such such as a workpiece or supervision of production for the purpose of automation or verification or the like. So far, there is also no satisfactory solution for informing the decoding side about the association of packets with the image fragments on which the current frame is structured and the regions of interest (regions of interest) of the current frame, so that intermediate network entities in the network 16 are able to collect such information from the data stream. without the need to deeply study the internals of the packages, i.e. frame syntax.

Accordingly, it is an object of the present invention to provide a video stream coding concept that is more efficient in providing low end-to-end delay and/or brings the identification of parts of the data stream to a region of interest or certain sub-pictures more easily.

This goal is achieved by the subject matter of the independent claims.

One of the ideas on which the present application is based is that decoder extraction timing information, ROI information, and fragment identification information should be transmitted in the video data stream at a level that allows easy access by network entities such as MANEs or decoders, and that in order to achieve such a level, information of these types must be transmitted in the video data stream by means of packets placed in packets of access units of the video data stream. According to an embodiment, the placed packages are of the type of the package being removed, i. e. the removal of these placed packets maintains the decoder's ability to fully recover the video content transmitted over the video stream.

In accordance with an aspect of the present application, achieving low end-to-end delay is more efficient by using nested packets to convey information about decoder buffer extraction times for decoding blocks generated by payload packets that follow the corresponding timing control packet in the video stream in the current access block. . With this measure, the decoder is able to determine the decoder buffer extraction times on the fly during the encoding of the current frame, thereby being able, during the encoding of the current frame, to continuously determine the data rate actually spent on the portion of the current frame already encoded into payload packets. and transmitted or sent prefixed with timing control packets on the one hand, and accordingly adjust the distribution of the remaining data rate for the current frame over the remainder of the current frame not yet encoded. By this measure, the available data rate is efficiently used and the delay is nevertheless kept shorter because the decoder does not have to wait to complete encoding of the current frame completely.

In accordance with a further aspect of the present application, packets placed in access unit payload packets are used to convey region of interest information, thereby allowing, as outlined above, easy access to this information by network entities, since they do not should check intermediate payload packets. In addition, the encoder is still free to determine the packets belonging to the ROI during encoding of the current frame on the fly, without having to determine the division of the current frame into sub-parts and the corresponding payload packets in advance. In addition, according to an embodiment where the placed packets are of the type of dropped packets, the ROI information may be ignored by recipients of the video stream not interested in the ROI information or unable to process it.

Similar thoughts are used in the present application in accordance with another aspect, in accordance with which the placed packets convey information that belongs to certain packets of sub-images in the access unit.

Useful implementations of the present invention are the subject of the dependent claims. Preferred embodiments of the present invention are the subject of the dependent claims.

FIGS. 1-10b show the current status of HEVC, wherein FIG. 1 shows the buffering period SEI message syntax, FIG. 2 shows the picture timing SEI message syntax, FIG. 3 shows the VUI parameter syntax, FIG. 4 shows the HRD parameter syntax, FIG. 5 shows the syntax of RBSP padding data, FIG. 6 shows the structure of byte streams and NAL unit streams for HRD conformance checks. Fig. 7 shows the HRD buffer model, Fig. 8 shows the syntax of a slice header, Fig. 9 shows the syntax of a picture parameter set RBSP, Fig. 10a shows a schematic illustration of a spatially adjacent coding tree block T, possibly used to invoke a process of obtaining the availability of a coding tree block relative to the current block of the coding tree, and Fig. 10b shows the definition of the access block structure;

Fig. 11 schematically shows a pair of encoder and decoder connected via a network to illustrate the problems encountered in the transmission of a video data stream;

12 shows a schematic block diagram of an encoder according to an embodiment using timing control packets;

Fig. 13 is a block diagram illustrating an operation mode of the encoder in Fig. 12 according to the embodiment;

Fig. 14 shows a block diagram of an embodiment of a decoder in such a way as to explain its functionality in connection with the video data stream generated by the encoder of Fig. 12;

15 shows a schematic block diagram illustrating an encoder, a network entity, and a video data stream in accordance with a further embodiment using ROI packets;

16 shows a schematic block diagram illustrating an encoder, a network entity, and a video data stream in accordance with a further embodiment using identification packets;

17 shows the structure of an access block according to an embodiment. The dotted line reflects the case of the optional slice prefix NAL unit;

18 shows the use of sub-pictures in ROI signaling;

19 shows the first simple syntax/version 1;

20 shows extended syntax/version 2 including tile_id signaling, decoding block start identifier, slice prefix ID, and slice header data, excluding the SEI message concept;

21 shows the NAL unit type code and NAL unit type classes;

22 shows a possible syntax for a slice header where certain syntax elements present in the slice header according to the current version are shifted to a lower hierarchy syntax element called slice_header_data();

23 shows a table where all syntax elements removed from a slice header are signaled via the syntax element slice header data;

24 shows the syntax for reporting additional extended information;

25 shows an adapted SEI payload syntax to represent new slice or sub-picture SEI message types;

26 shows an example for a sub-picture buffering SEI message;

Fig. 27 shows an example for a sub-picture timing SEI message;

FIG. 28 shows what a sub-picture slice information SEI message may look like.

Fig. 29 shows an example for reporting sub-picture fragment information SEI;

30 shows an example of a syntax for reporting sub-picture fragment size information to the SEI;

31 shows a first variant of an example syntax for an SEI message region of interest, where each ROI is signaled in a separate SEI message;

32 shows a second variant of an example syntax for an SEI message region of interest where all ROIs are signaled in one SEI message;

33 shows a possible syntax for a timing control packet according to a further embodiment;

34 shows a possible syntax for a sub-picture identification packet according to an embodiment;

35-38 show possible subdivisions of an image according to different subdivision settings according to an embodiment; and

39 shows an example of a portion of a video stream according to an embodiment using timing control packets placed between access block payload packets.

FIG. 12 describes an encoder 10 according to an embodiment of the present invention and its mode of operation. Encoder 10 is configured to encode video content 16 into stream 22 of video data. The encoder is configured to do this in blocks of sub-frames/images 18 of video content 16, where the sub-parts may, for example, be slices 24 into which images 18 are divided, or some other spatial segments, such as, for example, sub-pictures 26 or WPP substreams 28, all of which are illustrated in FIG. 12 for illustrative purposes only, and not to suggest that encoder 10 should be able to support WPP parallel processing, for example, or that sub-parts should be slices.

In encoding video content 16 in blocks of sub-parts 24, encoder 10 may obey a decoding order - or encoding order - determined among sub-parts 24, which, for example, bypasses images 18 of video 16 according to an image decoding order, which, for example, is not necessarily matches the playback order 20 defined among the pictures 18 and traverses in each picture 18 the blocks into which the pictures 18 are divided according to the raster scan order, the sub-parts 24 representing continuous passes of such blocks according to the decoding order. In particular, the decoder 10 may be configured to obey order in determining the availability of spatially and/or temporally neighboring parts to be currently encoded in order to use attributes describing the parts in predictive coding and/or entropy coding, such as, for example to determine the predictive and/or entropy context: Only previously visited (encoded/decoded) parts of the video are available. Otherwise, the attributes just mentioned are set to their default values, or some other replacement action is taken.

On the other hand, the encoder 10 does not have to sequentially encode the sub-parts 24 according to the decoding order. Conversely, encoder 10 may use parallel processing to speed up the encoding process, or to be able to perform more complex encoding in real time. Similarly, encoder 10 may or may not be configured to transmit or send data encoding sub-parts according to decoding order. For example, the encoder 10 may output/transmit the encoded data in some other order, such as, for example, according to the order in which the encoding of the sub-parts is completed by the encoder 10, which may, due to parallel processing, for example, deviate from just mentioned decoding order.

To generate encoded versions of sub-parts 24 suitable for transmission over the network, encoder 10 encodes each sub-part 24 into one or more payload packets of video stream 22 packet sequences. In case the sub-parts 24 are slices, the decoder 10 may, for example, be configured to put the data of each slice, i. e. each encoded slice, into one payload packet, such as a NAL unit. This packetization may serve to form a video data stream 22 suitable for transmission over the network. Accordingly, the packets may represent the smallest blocks in which the video stream 22 can take place, ie. the smallest blocks that can be individually sent by the encoder 10 for transmission over the network to the recipient.

In addition to the payload packets and timing control packets placed between them and discussed below, other packets, i.e. Other types of packets may also exist, such as padding data packets, picture or sequence parameter set packets for conveying infrequently changing syntax elements, or EOF (End of File) or AUE (End of Access Unit) packets or the like.

The encoder performs encoding into payload packets in such a way that the sequence of packets is divided into a sequence of access units 30, and each access unit collects payload packets 32 related to one image 18 of video content 16. That is, the sequence 34 of packets forming a stream 22 of video data, divided into non-overlapping portions called access blocks 30, each associated with a respective one of the pictures 18. The sequence of access blocks 30 may follow the decoding order of the pictures 18 to which the access blocks 30 belong. 12 illustrates, for example, that an access unit 30 located in the middle of a portion of the illustrated data stream 22 contains one payload packet 32 per sub-part 24 into which the image 18 is subdivided. That is, each payload packet 32 carries a corresponding sub-part 24. Encoder 10 is configured to place timing control packets 36 in packet sequence 34 such that timing control packets subdivide access blocks 30 into decode blocks 38 such that at least some access units 30, such as the middle one shown in FIG. 12, are subdivided into two or more decode units 38, with each timing control packet signaling a decoder buffer extraction time to decode unit 38 whose payload packets 32 follow the corresponding timing control packets in sequence 34 packets. In other words, the encoder 10 prefixes subsequences of the sequence of payload packets 32 in one access block 30 with the corresponding timing control packet 36 signaling for the corresponding subsequence of payload packets prefixed with the corresponding timing control packet 36 and generating the decoding block 38, time decoder buffer extraction. Fig.12, for example, illustrates the case where every second the packet 32 represents the first packet of the payload block 38 of the decoding block 30 of the access. As illustrated in FIG. 12, the amount of data or data rate spent for each decoding block 38 varies, and decoder buffer extraction times can be correlated with this change in data rate among decoding blocks 38 in that the decoder buffer extraction time of block 38 decoding may follow the decoder buffer extraction time signaled by the timing control packet 36 of the immediately preceding decoding block 38, added to the time interval corresponding to the data rate spent for that immediately preceding block 38.

That is, encoder 10 may operate as shown in FIG. In particular, as mentioned above, the encoder 10 may, in step 40, subject the current sub-part 24 of the current picture 18 to encoding. As already noted, encoder 10 may sequentially pass through sub-parts 24 in the aforementioned decoding order, as illustrated by arrow 42, or encoder 10 may use some parallel processing, such as WPP and/or sub-picture processing, to simultaneously encode multiple "current sub-parts" 24. Regardless of whether parallel processing is used or not, the encoder 10 generates a decoding block from one or more sub-parts just encoded in step 40, and proceeds to step 44, where the encoder 10 sets the buffer extraction time decoder for that decoder and transmits that decoder prefixed with a timing packet signaling the decoder buffer extraction time just set for that decoder. For example, encoder 10 may determine the decoder buffer fetch time at step 44 based on the data rate spent to encode sub-parts encoded into payload packets forming the current decoding block, including, for example, all additional intermediate packets in that block. decoding, if any, i.e. "prefixed packets".

Then, at step 46, the encoder 10 may adapt the available bit rate based on the bit rate spent for the decoding block just transmitted in step 44. If, for example, the image content in the decoding block just transmitted in step 44 was sufficiently complex in terms of compression rate, encoder 10 may reduce the available data rate for the next decoding block in order to comply with some externally set data rate determined, for example, based on the current bandwidth situation encountered in connection with the network. , transmitting the stream 22 of video data. Then steps 40-46 are repeated. With this measure, the images 18 are encoded and transmitted, i. e. are sent, in units of decoding blocks, each prefixed with a corresponding time control packet.

In other words, the encoder 10, while encoding the current image 18 of the video content 16, encodes 40 the current sub-part 24 of the current image 18 into the current payload packet 32 of the current decoding unit 38, transmits 44, in the data stream, the current decoding unit 38 provided with a prefix as the current timing control packet 36, along with setting the decoder buffer extraction time signaled by the current timing control packet (36) at the first time, and encodes 44, by returning to the beginning of the loop from step 46 to step 40, additional sub-part 24 of the current image 18 at the second time - the second time visiting step 40 - later than the first time - the first time visiting step 44.

Since the encoder is able to send a decoding block before encoding the remainder of the current picture to which this decoding block belongs, the encoder 10 is able to reduce the end-to-end delay. On the other hand, the encoder 10 does not need to waste the available data rate, since the encoder 10 is able to respond to the specific nature of the content of the current picture and its spatial distribution of complexity.

On the other hand, intermediate network entities responsible for transmitting the video stream 22 further from the encoder to the decoder are able to use timing control packets 36 to ensure that any decoder receiving the video stream 22 receives decoding blocks in time to have the ability to take advantage of the per-packet coding and transmission decoding by the encoder 10. See, for example, FIG. 14 showing an example for a decoder to decode a video stream 22. The decoder 12 receives the video stream 22 in the CPB encoded picture buffer 48 via the network through which the encoder 10 transmitted the video stream 22 to the decoder 12. In particular, since the network 14 is expected to be capable of supporting low latency applications, the network 10 checks the buffer extraction times decoder to forward the sequence 34 of video stream 22 packets to the encoded picture buffer 48 of the decoder 12 so that each decoding block is present in the encoded picture buffer 48 until the decoder buffer retrieval time, signaled by a timing control packet that is the prefix of the corresponding decoding block. With this measure, the decoder is able, without stopping, i.e. without exhausting the available payload packets in the encoded picture buffer 48, use the decoder buffer extraction times in the timing control packets to empty the decoder encoded picture buffer 48 in units of decoding blocks instead of filling access blocks. 14, for example, shows for illustrative purposes a processing unit 50 coupled to the output of an encoded picture buffer 48 whose input receives a video stream 22 . Like encoder 10, decoder 12 may be capable of performing parallel processing such as, for example, using sub-picture parallel processing/decoding and/or WPP parallel processing/decoding.

As will be described in more detail below, the decoder buffer extraction times mentioned so far do not necessarily refer to the extraction times relating to the encoded picture buffer 48 of the decoder 12. Rather, the timing control packets may additionally, or alternatively, control the extraction of the already decoded picture data of the corresponding decoded image buffer of the decoder 12. FIG. stored and output, in units of decoded versions of decoding blocks. The decoded image buffer 22 of the decoder can thus be connected between the output of the decoder 12 and the output of the processing unit 50 . By being able to set the extraction times for output decoded versions of the decoding blocks from the decoded picture buffer 52, the encoder 10 is given the ability, on the fly, i.e. during the encoding of the current image, to control the playback or end-to-end playback delay of the video content on the decoding side even at a detail lower than the image rate or frame rate. Obviously, excessive segmentation of each image 18 into a huge number of sub-parts 24 on the encoding side would have a negative effect on the data rate for transmitting the video stream 22, although, on the other hand, end-to-end delay can be minimized since the time required for encoding and transmission and decoding and output of such a decoding unit would be minimized. On the other hand, increasing the sub-parts 24 increases the end-to-end delay. Accordingly, a compromise must be found. Using the decoder buffer extraction times just mentioned to control the output timing of the decoded versions of the sub-parts 24 in units of decoding units allows the encoder 10 or some other unit on the encoding side to adapt this trade-off spatially across the content of the current picture. With this measure, it would be possible to control the end-to-end delay in such a way that it varies spatially depending on the content of the current images.

In the implementation of the embodiments described above, it is possible to use packets of the packet type to be removed as timing control packets. Packets of the packet type to be removed are not necessary in order to restore the video content on the decoding side. In the following, such packages are referred to as SEI packages. Additional drop type packets may also exist, ie, drop packets of a different type, such as, when streamed, redundancy packs. As another alternative, the timing control packets may be packets of a particular type of packets to be removed, additionally carrying, however, a particular SEI Packet Type field. For example, timing control packets may be SEI packets, where each SEI packet carries one or more SEI messages, and only those SEI packets that contain a certain type of SEI message form the aforementioned timing control packets.

Thus, the embodiment described so far with reference to FIGS. 12-14 according to a further embodiment is applied to the HEVC standard, thereby forming a possible concept for generating HEVC more efficiently in achieving lower end-to-end delay. Meanwhile, the aforementioned packets are formed by NAL units, and the aforementioned payload packets are VCL NAL units of a NAL unit flow with slices forming the aforementioned sub-parts.

Before such a description of a more detailed embodiment, however, additional embodiments are described that are the same as the above embodiments in that the placed packets are used to convey in an efficient manner information describing the video data stream, but the kind of information differs from the above embodiments. , where the timing control packets conveyed decoder buffer extraction timing information. In the embodiments described further below, the kind of information conveyed through placed packets placed in payload packets belonging to an access unit refers to region of interest (ROI) information and/or sub-picture identification information. The embodiments described further below may or may not be combined with the embodiments described with respect to FIGS. 12-14.

FIG. 15 shows an encoder 10 that operates similarly to the encoder explained above with respect to FIG. 12, except for the placement of timing control packets and the functionality described above with respect to FIG. 13, which is optional for encoder 10 in FIG. 15. However, encoder 10 in FIG. 15 is configured to encode video content 16 into video stream 22 in units of sub-parts 24 of images 18 of video content 16 in the same manner as explained above with respect to FIG. In encoding the video content 16, the encoder 10 is interested in transmitting, together with the video data stream 22, information about the region of interest ROI 60 to the decoding side. ROI 60 is a spatial sub-region of the current picture 18 to which the decoder must, for example, pay special attention. The spatial position of the ROI 60 may be an input to the encoder 10 from the outside, as illustrated by dashed line 62, for example, via user input, or may be determined automatically by the encoder 10 or some other entity, on the fly during encoding of the current picture 18. At any In this case, the encoder 10 faces the following problem: indicating the location of the ROI 60 is not in principle a problem for the encoder 10. To do this, the encoder 10 can easily indicate the location of the ROI 60 in the video stream 22. However, to make this information readily available, the encoder 10 of FIG. 15 uses the placement of ROI packets between access unit payload packets so that the encoder 10 is free, on-line, to select the segmentation of the current picture 18 into sub-parts 24 and/or the number payload packets into which sub-parts 24 are packetized, spatially outside and spatially within the ROI 60. Using the placed ROI packets, any network entity can easily identify the payload packets that belong to the ROI. On the other hand, in case of using drop type packets for these ROI packets, the same can be easily ignored by any network entity.

15 shows an example for placing ROI packets 64 between payload packets 32 of access unit 30 . The ROI packet 64 indicates where in the packet sequence 34 of the video data stream 22 the encoded data is contained that relates, i.e. ROI 60 is encoded. How ROI packet 64 indicates the location of ROI 60 can be implemented in various ways. For example, the net existence/occurrence of ROI packet 64 may indicate the inclusion of encoded data related to ROI 60 in one or more of the following payload packets 32 in sequential order of sequence 34, i. belonging to prefixed payload packets. Alternatively, a syntax element within the ROI packet 64 may indicate whether one or more of the following payload packets 32, ie, at least partially encodes, belongs to the ROI 60 or not. A large amount of hesitation also comes from possible changes regarding the "scope" of the respective ROI package 64, ie. the number of prefixed payload packets prefixed as a single packet 64 ROI. For example, an indication of the inclusion or non-inclusion of any encoded data related to ROI 60 in one ROI packet may refer to all payload packets 32 following in sequential order of sequence 34 until the next ROI packet 64 occurs, or may only refer to directly the next payload packet 32, i. e. payload packet 32 immediately following the corresponding ROI packet 64 in sequential order of sequence 34. In FIG. inclusion of any encoded data related to ROI 60, or non-relevance-ROI, ie. the absence of any encoded data related to the ROI 60 for all payload packets 32 occurring downstream of the corresponding ROI packet 64 until the next ROI packet 64 occurs or the current access block 30 ends, whichever occurs earlier in the packet sequence 34. In particular, FIG. 15 illustrates the case in which the ROI packet 64 has a syntax element inside that indicates whether or not the payload packets 32 following in sequential order of the packet sequence 34 have any encoded data related to the ROI 60 inside. . Such an embodiment is also described below. However, another possibility, as just mentioned, is that each ROI packet 64 indicates merely by its presence in the packet sequence 34 that the payload packet(s) 34 belonging to the "scope" of the corresponding ROI packet 64 have related to ROI 60 data inside, i.e. data related to the ROI 60. According to an embodiment described in more detail below, the ROI packet 64 even indicates the location of the portion of the ROI 60 encoded in the payload packet(s) 32 belonging to its "scope".

Any network entity 68 receiving video stream 22 may use an ROI relevance indication, as implemented by using ROI packets 64, in order to process, for example, ROI-relevant portions of packet sequence 34 with higher priority than other portions of packet sequence 34, such as . Alternatively, the network entity 68 could use the ROI relevance information to perform other tasks, such as transmitting the video stream 22, for example. The network entity 68 may be, for example, a MANE or a decoder for decoding and playing video content 60 as it is transmitted over the video stream 22. 28. In other words, the network entity 68 may use the result of the ROI packet identification to select transmission tasks related to the video data stream. Transfer tasks may contain retransmission requests regarding defective packets. The network entity 68 may be configured to process the region of interest 70 with increased priority and assign a higher priority to the ROI packets 72 and their associated payload packets, i. e. packets for which it is a prefix, which are signaled as overlapping the region of interest, as compared to ROI packets and their associated payload packets, which are signaled as non-overlapping ROIs. Network entity 68 may first request retransmission of payload packets having a higher priority assigned to them before requesting any retransmission of payload packets having a lower priority assigned to them.

The embodiment in FIG. 15 can easily be combined with the embodiment described earlier with respect to FIGS. 12-14. For example, the ROI packets 64 mentioned above may also be SEI packets having a particular type of SEI message contained therein, namely a SEI ROI message. That is, the SEI packet may, for example, be a timing control packet and an ROI packet at the same time, that is, if the corresponding SEI packet contains both timing control information and ROI indication information. Alternatively, the SEI packet may be one of a timing control packet and an ROI packet and not the other, or may be neither an ROI packet nor a timing control packet.

In accordance with the embodiment shown in FIG. 16, placing packets between access block payload packets is used to indicate, in a manner readily available to network entities 68 processing video stream 22, which picture or pictures of the current picture 18 to which the current access block 30 is overridden by any sub-part encoded in any of the payload packets 32 for which the respective packets serve as a prefix. In FIG. 16, for example, the current image 18 is, as shown, subdivided into four sub-images 70, here roughly formed by four quadrants of the current image 18. The subdivision of the current image 18 into sub-images 70 may, for example, be signaled in the video data stream in blocks, containing sequences of images, such as in VPS or SPS packages, also placed in a sequence of 34 packages. As will be described in more detail below, the subdivision into sub-images of the current image 18 may be a standard subdivision of the image 18 into columns and rows of sub-images. The number of columns and the number of rows, as well as the column width and row height of sub-images can vary. In particular, the width and height of the columns/rows of sub-images may be different for different rows and different columns, respectively. 16 further shows an example where sub-parts 24 are slices of image 18. Slices 24 subdivide image 18. As will be described in more detail below, subdivision of image 18 into slices 24 may be subject to restrictions whereby each slice 24 can either completely contained to one single fragment 70 of the image, or completely cover two or more fragments 70 of the image. 16 illustrates a case in which the image 18 is subdivided into five slices 24. The first four of these slices 24 in the above decoding order cover the first two sub-pictures 70, while the fifth slice completely covers the third and fourth sub-pictures 70. In addition, FIG. 16 illustrates the case where each slice 24 is separately encoded into a corresponding payload packet 32. In addition, FIG. 16 illustrates the case where each payload packet 32 is prefixed with the previous sub-picture identification packet 72 as an example. Each tile identification packet 72 in turn indicates the immediately following payload packet 32 as to which of the tile 70 the sub-part 24 encoded in that payload packet 32 overlaps. Accordingly, while the first two sub-picture identification packets 72 in the access block 30 related to the current image 18 indicate the first sub-picture, the third and fourth sub-picture identification packets 72 indicate the second sub-picture 70 of the image 18, and the fifth sub-picture identification packet 72 indicate the third and fourth fragments 70 of the image. With regard to the embodiment in FIG. 16, the same variations are possible as described above with respect to FIG. 15, for example. That is, the "scope" of the sub-picture identification packets 72 may, for example, only include the first immediately following payload packet 32 or the immediately following payload packets 32 until the next sub-picture identification packet occurs.

With respect to sub-images, encoder 10 may be configured to encode each sub-image 70 such that no spatial prediction or no context selection occurs across boundaries. Encoder 10 may, for example, encode sub-picture 70 in parallel. Similarly, any decoder, such as network entity 68, may decode sub-pictures 70 in parallel.

The network entity 68 may be a MANE or a decoder, or some other device between the encoder 10 and the decoder, and may be configured to use the information transmitted by subpicture identification packets 72 to select specific transmission tasks. For example, the network entity 68 may process a particular sub-image of the current image 18 of the video 16 with a higher priority, i. e. may forward corresponding payload packets indicated as related to such sub-picture previously or using more secure FEC protection or the like. In other words, the network entity 68 may use the identification result to select transmission tasks related to the video stream. Transmission tasks may contain retransmission requests regarding packets received in a defective state - i.e. exceeding any FEC protection of the video stream, if any. The network entity may process, for example, different sub-images 70 with different priorities. To this end, the network entity may assign a higher priority to the sub-picture identification packets 72 and their payload packets, ie. packets prefixed with them relating to higher priority tile than the tile identification packets 72 and their lower priority tile payload packets. The network entity 68 may, for example, first request retransmission of payload packets having a higher priority assigned to them before requesting any retransmission of payload packets having a lower priority assigned to them.

The embodiments described so far may be embedded in the HEVC infrastructure as described in the introductory part of the description of the present application, as described in the following.

In particular, SEI messages may be assigned to decoding block slices in the case of a CPB/HRD sub-picture. That is, a buffering period and timing SEI messages may be assigned to NAL units containing decoding unit slices. This can be achieved by a new type of NAL unit, which is a non-VCL NAL unit that is allowed to immediately precede one or more slice/VCL NAL units of a decoding unit. This new NAL unit may be called a slice prefix NAL unit. 17 illustrates the structure of an access unit, omitting any pre-NAL units for terminating a sequence and a stream.

17, the access block 30 is explained as follows: in the sequential order of the packets of the packet sequence 34, the access block 30 may start with the occurrence of a special type of packet, namely the access block delimiter 80. Then, one or more SEI packets 82 of the SEI packet type related to the entire access block may follow in the access block 30 . Both types 80 and 82 packages are optional. That is, packets of this type may not occur at access block 30 . This is followed by a sequence of decoding blocks 38 . Each decoding block 38 optionally begins with a slice prefix NAL block 84 including, for example, timing control information or, according to the embodiment of FIG. 15 or 16, ROI information or sub-picture information, or even more broadly corresponding sub-image SEI message 86. The actual slice data 88 then follows in the respective payload packets or VCL NAL units, as shown at 88. Thus, each decode unit 38 contains a sequence of slice prefix NAL unit 84 followed by the corresponding slice data NAL unit(s) 88 . The bypass arrow 90 in FIG. 17 bypassing the slice prefix NAL unit indicates that if there is no division into decoding units of the current access unit 30, there may not be a slice prefix NAL unit 84 .

As noted above, all information signaled in the slice prefix and the associated sub-picture SEI messages may either be valid for all VCL NAL units in the access unit, or until the second NAL of the prefix occurs, or for the next VCL-NAL unit in order. decoding, depending on the flag specified in the slice prefix NAL unit.

The slice VCL NAL unit for which the information signaled in the slice prefix is valid are hereinafter referred to as prefixed slices. Prefixed slices associated with a single prefixed slice do not necessarily form a complete decoding block, but may be part of it. However, a single slice prefix cannot be valid for multiple decoding blocks (sub-pictures), and the start of a decoding block is signaled in the slice prefix. If the means for signaling is not specified via the slice prefix syntax (as in the "simple syntax"/version 1 below), the occurrence of the slice prefix NAL unit signals the start of a decoding unit. Only certain SEI messages (identified by payloadType in the syntax description below) can be sent exclusively at the sub-picture level in the slice prefix NAL unit, while some SEI messages can be sent either in the slice prefix NAL unit at the sub-picture level, or as standard SEI message at the access block level.

As discussed above with respect to FIG. 16, additionally or alternatively, the sub-picture ID SEI message/sub-picture ID signaling may be implemented in high-level syntax. In earlier HEVC designs, the slice header/slice data contained an identifier for the sub-image contained in the corresponding slice. For example, the slice data semantics read:

tile_idx_minus_l defines the TileID in raster scan order. The first tile in the image must have a TileID of 0. The value of tile_idx_minus_1 must be between 0 and ( num_tile_columns_minus1 + 1 ) * ( num_tile_rows_minus1 + 1) - 1.

This parameter, however, is not considered useful because this ID can be easily derived from the slice address and slice dimensions, as signaled in the picture parameter set, if tiles_or_entropy_coding_sync_idc is equal to 1.

Although the sub-picture ID can be obtained implicitly during the decoding process, knowing this parameter at the application level is also important for various use cases, such as, for example, in a video conference scenario, where different sub-pictures have a different priority for playback (these sub-pictures are usually , form a region of interest that contains the speaker in a conversational use case) may take precedence over other sub-pictures. In case of loss of network packets when transmitting multiple sub-pictures, these network packets containing the sub-pictures representing the area of interest can be retransmitted with a higher priority in order to keep the quality of the experience at the receiver terminal higher than in the case of retransmission of sub-pictures without any or order of precedence. Another use case could be to assign image tiles, if the sizes and their positions are known, to different screens, for example in a videoconference scenario.

To allow such an application layer to process tiles with a certain priority in transfer scenarios, the tile_id may be provided as a sub-picture or slice-specific SEI message either in a special NAL unit before one or more tile NAL units or in a special NAL unit header section, belonging to the image fragment.

As described above with respect to FIG. 15, region-of-interest SEI messages may also additionally or alternatively be provided. Such an SEI message could enable region of interest (ROI) signaling, in particular signaling the ROI to which a particular tile_id/tile belongs. The message might allow the ROI ID plus ROI priority to be specified.

18 illustrates the use of sub-pictures in region-of-interest signaling;

In addition to what has been described above, slice header signaling can be implemented. The slice prefix NAL unit may also contain a slice header for subsequent dependent slices, i. e. slices prefixed with the corresponding slice prefixes. If a slice header is only provided in the slice prefix NAL unit, the actual slice type must be obtained by the type of the NAL unit containing the corresponding dependent slice, or by a flag in the slice prefix signaling whether subsequent slice data belongs to the slice type that serves as a random access point. .

In addition, the slice prefix NAL unit may carry slice or sub-picture specific SEI messages to convey optional information such as sub-picture timing or sub-picture identifier. The transmission of optional sub-image messages is not supported in the HEVC specification described in the introductory part of the description of this application, but is essential for certain applications.

The following describes a possible syntax for implementing the concept of slice prefixes described above. In particular, it is described what changes may be sufficient at the slice level when using the HEVC status, as described in the introductory part of the description of the present application as a basis.

In particular, two versions of a possible slice prefix syntax are presented below, one with SEI messaging functionality only, and one with extended slice header part signaling functionality for subsequent slices. The first simple syntax/version 1 is shown in FIG.

As a preliminary note, FIG. 19 shows a possible implementation for implementing any of the embodiments described above with respect to FIGS. 11-16. the placed packets shown there can be interpreted as shown in FIG. 19, and this is described in more detail with specific implementation examples below.

The extended syntax/version 2 including tile_id signaling, decoding block start identifier, slice prefix ID, and slice header data, excluding the SEI messaging concept, is shown in the table in FIG.

The semantics could be defined as follows:

rap_flag with a value of 1 indicates that the access unit containing the slice prefix is a RAP image. rap_flag with a value of 0 indicates that the access unit containing the slice prefix is not a RAP picture.

decoding_unit_start_flag points to the start of a decoding unit in an access unit, such that subsequent slices up to the end of an access unit or the start of another decoding unit belong to the same decoding unit.

single_slice_flag with a value of 0 indicates that the information provided in the slice header NAL unit and the associated sub-picture SEI messages are valid for all subsequent VCL-NAL units until the start of the next access unit, another slice prefix, or another full slice header. single_slice_flag with a value of 1 indicates that all information provided in the slice prefix NAL unit and associated sub-picture SEI messages is only valid for the next VCL-NAL unit in decoding order.

tile_idc specifies the number of tiles to be present in the next slice. tile_idc equal to 0 indicates that no tiles are used in the subsequent slice. tile_idc equal to 1 indicates that a single tile is used in the subsequent slice, and its tile ID is signaled accordingly. tile_idc with a value of 2 indicates that multiple tiles are used in the next slice, and the number of tiles and the ID of the first tiles are signaled accordingly.

prefix_slice_header_data_present_flag indicates that slice header data corresponding to slices next in decoding order is signaled at the given slice prefix.

slice_header_data() is defined later in the text. It contains relevant slice header information that is not covered by the slice header if dependent_slice_flag is set to 1.

It should be noted that the decoupling of the slice header and the actual slice data allows for more flexible schemes for transmitting the slice header and data.

num_tiles_in_prefixed_slices_minus1 indicates the number of tiles used in the next decoding block minus 1.

first_tile_id_in_prefixed_slices indicates the tile ID of the first tile in the next decoding block.

For the simple syntax/version 1 slice prefix, the following syntax elements can be set to their default values as follows if they are not present:

- decoding_unit_start is equal to 1, i.e. the slice prefix always indicates the start of a decoding block.

- single_slice_flag is 0, i.e. the slice prefix is valid for all slices in the decode block.

It is proposed that the slice prefix NAL unit has NAL unit type 24, and the NAL unit type lookup table has been extended according to FIG.

That is, briefly summing up Figs. 19-21, the syntax details shown therein show that a particular packet type can be assigned to the above-identified placed packets, here NAL unit type 24 as an example. In addition, especially the syntax example in FIG. 20 makes it clear that the alternatives described above regarding the "scope" of placed packets, the switching mechanism controlled by the corresponding syntax element in these placed packets themselves, here by way of example single_slice_flag, can be used to manage this scope, i.e. switch between different alternatives to define this scope, respectively. Furthermore, it has been made clear that the embodiments of FIGS. 1-16 described above can be extended in that the placed packets also contain common slice header data for the slices 24 contained in the packets belonging to the "scope" of the respective placed packets. That is, there may be a mechanism controlled by a corresponding flag within these placed packets that indicates whether common slice header data is contained in the corresponding placed packet or not.

Of course, the concept that just introduced which portion of the slice header data shifts into the slice header prefix requires changes to the slice headers as defined in the current version of HEVC. The table in FIG. 22 shows a possible syntax for such a slice header where certain syntax elements present in the slice header according to the current version are shifted to a lower hierarchy syntax element called slice_header_data(). The slice header syntax and slice header data only apply to the option that uses the slice header prefix extended NAL unit.

In FIG. 22, slice_header_data_present_flag indicates that the slice header data for the present slice is to be predicted from the values signaled in the last slice prefix NAL unit in the access unit, i. e. the most recent slice prefix NAL block.

All syntax elements removed from the slice header are signaled via the syntax element slice header data as given in the table in FIG.

That is, by extending the concept of FIGS. 22 and 23 to the embodiments of FIGS. 12-16, the nested packets described herein can be extended by the concept of including in these nested packets part of the slice header syntax of slices (sub- parts) 24 encoded into payload packets, i.e. VCL NAL blocks. The inclusion may be optional. That is, the corresponding syntax element in the placed package may indicate whether such slice header syntax is contained in the corresponding placed package or not. If present, the corresponding slice header data included in the corresponding placed package may apply to all slices contained in the package belonging to the "scope" of the corresponding placed package. Whether slice header data contained in a placed packet is adapted by a slice encoded in any of the payload packets belonging to the scope of that placed packet may be signaled by an appropriate flag, such as slice_header_data_present_flag in FIG. With this measure, the headers of slices encoded into packets belonging to the "scope" of the corresponding placed packet can be reduced in size accordingly using the flag just mentioned in the slice header, and any decoder receiving the video stream, such as the network objects shown 12-16 above would respond to the just-mentioned flag in the slice header so as to copy the slice header data included in the placed packet into the slice header encoded in the payload packet belonging to the scope of that placed packet, in the case of a corresponding flag in that slice, signaling an offset of the slice header data to the slice prefix, i.e. corresponding to the placed package.

Continuing with the example syntax for implementing the embodiments in FIGS. 12-16, the SEI message syntax may be as shown in FIG. To represent slice or sub-picture SEI message types, the SEI payload syntax can be adapted as shown in the table in FIG. Only a SEI message with payloadType in the range 180 to 184 can be sent exclusively at the sub-picture level in the slice prefix NAL unit. In addition, region-of-interest SEI messages with payloadType equal to 140 may be sent either in the slice prefix NAL unit at the sub-picture level or in a regular SEI message at the access unit level.

That is, in transferring the details shown in FIGS. 25 and 24 to the embodiments described above with respect to FIGS. 12-16, the placed packets shown in these embodiments in FIGS. 12-16 can be implemented by using slice prefix NAL units with a specific NAL unit type, eg 24, containing a specific SEI message type signaled by payloadType at the beginning of each SEI message in the slice prefix NAL unit, eg. In a specific embodiment of the syntax now described, payloadType = 180 and payloadType = 181 results in a timing control packet according to the embodiment in FIGS. 11-14, while payloadType = 140 results in an ROI packet according to the embodiment in FIG. 15 and payloadType = 182 results in a sub-picture identification packet according to the embodiment in FIG. The specific syntax example described herein below may contain only one or a subset of the payloadType options just mentioned. In addition, Fig.25 shows that any of the above-described embodiments in Fig.11-16 can be combined with each other. Still further, Fig. 25 shows that any of the above embodiments in Figs. 12-16, or any combination of them, can be extended by an additional placed package, subsequently explained with payloadType = 184. As already described above, the extension described below with respect to payloadType = 183, ends up with the possibility that any placed packets could include common slice header data for slice headers encoded in any payload packet belonging to its scope.

The tables in the following figures define the SEI messages that can be used at the slice or sub-picture level. The SEI message of interest is also present, which can be used at the sub-picture and access block level.

26, for example, shows an example for a sub-picture buffering SEI message occurring whenever the NAL unit slice prefix type 24 NAL unit has an SEI message type 180 included therein, thus forming a timing control packet.

The semantics could be defined as follows:

seq_parameter_set_id defines a sequence parameter set that contains the HRD attributes of the sequence. The value of seq_parameter_set_id will be equal to the value of seq_parameter_set_id in the picture parameter set referenced by the primary encoded picture associated with the buffer period SEI message. The value of seq_parameter_set_id must be between 0 and 31, inclusive.

initial_cpb_removal_delay[ SchedSelIdx ] and initial_alt_cpb_removal_delay[ SchedSelIdx ] define the initial CPB removal delays for the SchedSelIdx th CPB of the decoding block (sub-picture). The syntax elements have a length in bits given by initial_cpb_removal_delay_length_minus1 + 1 and are expressed in units of a 90 kHz clock signal. Syntax element values will not be 0 and will not exceed 90000 * ( CpbSize[ SchedSelIdx ] ÷ BitRate[ SchedSelIdx ]), the time equivalent of the CPB size in units of 90 kHz clock.

По всей закодированной видеопоследовательности сумма initial_cpb_removal_delay[ SchedSelIdx ] и initial_cpb_removal_delay_offset[ SchedSelIdx ] на блок декодирования (под-изображения) должна быть постоянной для каждого значения SchedSelIdx, и сумма initial_alt_cpb_removal_delay[ SchedSelIdx ] и initial_alt_cpb_removal_delay_offset[ SchedSelIdx ] должна быть постоянной для каждого значения SchedSelIdx.

27 also shows an example for a sub-picture timing SEI message, where the semantics could be described as follows:

du_cpb_removal_delay specifies how many clock cycles to wait after removing from the CPB the decoding block (sub-picture) associated with the most recent SEI message of the sub-picture buffering period in the previous access block in the same decoding block (sub-picture), if present, in otherwise, the buffering period associated with the most recent SEI message in the previous access block, before decoding block (sub-picture) data associated with the sub-picture timing SEI message is removed from the buffer. This value is also used to calculate the earliest possible time of arrival of the decoding block (sub-picture) data in the CPB for the HSS (Hypothetical Flow Scheduler [2]0). The syntax element is a fixed length code whose length in bits is given by cpb_removal_delay_length_minus1 + 1. cpb_removal_delay is the remainder of the counter modulo 2^(cpb_removal_delay_length_minus1 + 1).

du_dpb_output_delay is used to calculate the DBP output time of the decoding block (sub-picture). It determines how many clock cycles to wait after a decoded decoding block (sub-picture) is removed from the CPB before the decoding block (sub-picture) is output from the DPB.

Note that this allows sub-image updates. In such a scenario, the non-updated decoding blocks may remain unchanged from the last decoded picture, ie. they remain visible.

Summarizing Figs. 26 and 27 and transferring certain details contained therein to the embodiment of Figs. 12-14, it can be said that the decoder buffer extraction time for the decoding block can be signaled in the associated timing control packet differently encoded way, namely, incremented with respect to another decoding buffer extraction time. That is, in order to obtain the decoder buffer extraction time for a specific decoding block, the decoder receiving the video stream adds the decoder extraction time obtained from the timing control packet that is a prefix for the specific decoding block to the decoder extraction time of the immediately preceding decoding block, i.e. . block preceding a particular decoding block, and continuing in this manner with subsequent decoding blocks. At the beginning of the encoded video sequences of multiple pictures of each or parts thereof, the timing control packet may additionally or alternatively comprise a decoder buffer extraction time value encoded absolutely rather than differentially with respect to any decoder buffer extraction time of the preceding decoding unit.

FIG. 28 shows what a sub-picture slice information SEI message may look like. The semantics could be defined as follows:

slice_header_data_flag with a value of 1 indicates that slice header data is present in the SEI message. The slice header data provided in the SEI is valid for all slices in decoding order until the end of the access block, the appearance of the slice data in another SEI message, the slice NAL unit, or the slice prefix NAL unit.

29 shows an example for reporting sub-picture fragment information SEI, where the semantics could be defined as follows:

tile_priority indicates the priority of all tiles in the prefixed slices in decoding order. The value of tile_priority must be between 0 and 7 inclusive, where 7 indicates the highest priority.

multiple_tiles_in_prefixed_slices_flag with a value of 1 indicates that there are more than 1 tiles in prefixed slices to follow in decoding order. multiple_tiles_in_prefixed_slices_flag with a value of 0 indicates that the following prefixed slices contain only one tile.

num_tiles_in_prefixed_slices_minus1 indicates the number of tiles in prefixed slices in decoding order.

first_tile_id_in_prefixed_slices indicates the tile_id of the first tile in the prefixed slices following in decoding order.

That is, the embodiment in FIG. 16 could be implemented using the syntax in FIG. 29 to implement the sub-picture identification packets referred to in FIG. 16. As shown there, a specific flag, here multiple_tiles_in_prefixed_slices_flag, can be used to signal in a placed tile identification packet whether only one tile or more than one tile is covered by any sub-part of the current picture 18 encoded in any of the payload packets. load belonging to the scope of the corresponding placed image fragment identification package. If the flag signals the overlap of more than one tile, an additional syntax element is contained in the corresponding placed package, here as an example num_tiles_in_prefixed_slices_minus1 indicating the number of tiles overlapped by any sub-part of any payload package belonging to the scope of the corresponding placed fragment identification package Images. Finally, an additional syntax element, here as an example first_tile_id_in_prefixed_slices, specifies the tile ID among the set of tile specified by the currently placed tile identification packet which is first according to the decoding order. Transferring the syntax of Fig. 29 to the embodiment of Fig. 16, the slice identification packet 72 being the prefix for the fifth payload packet 32 could, for example, have all three syntax elements just discussed with multiple_tiles_in_prefixed_slices_flag set to 1, num_tiles_in_prefixed_slices_minus1, set to 1, thereby indicating that two tiles belong to the current scope, and the tile identification packet 72 starts on the third tile (having tile_id = 2).

29 also shows that the tile identification packet 72 may optionally indicate tile_priority, ie. priority of image fragments belonging to its scope. Similar to the ROI aspect, network entity 68 may use such priority information to manage transmission tasks such as requesting retransmission of certain payload packets.

30 shows an example syntax for reporting sub-picture fragment size information to the SEI, where the semantics could be defined as:

multiple_tiles_in_prefixed_slices_flag with a value of 1 indicates that there are more than 1 tiles in prefixed slices to follow in decoding order. multiple_tiles_in_prefixed_slices_flag with a value of 0 indicates that the following prefixed slices contain only one tile.

num_tiles_in_prefixed_slices_minus1 indicates the number of tiles in prefixed slices in decoding order.

tile_horz_start[ i ] specifies the start in the horizontal direction of the i-th tile within the image.

tile_width[ i ] specifies the width of the i-th tile in pixels within the image.

tile_vert_start[ i ] specifies the start in the horizontal direction of the i-th tile within the image.

tile_height[ i ] specifies the height of the i-th tile in pixels within the image.

It should be noted that the size SEI message can be used in display operations, for example, to assign a sub-image to a screen in a multi-screen display scenario.

30 thus shows that the example implementation of the syntax of FIG. 29 with respect to the sub-picture identification packets of FIG. of the current image 18, not their sub-picture ID. That is, instead of signaling the sub-picture ID of the first sub-picture in decoding order covered by the corresponding sub-parts encoded in any of the payload packets belonging to the scope of the corresponding placed sub-picture identification packet, for each sub-picture belonging to the current sub-picture identification packet , its position could be signaled by signaling, for example, the position of the upper left corner of each tile i, here by way of example tile_horz_start and tile_vert_start, and the width and height of the tile i, here by way of example by tile_width and tile_height.

An example syntax for reporting the SEI of a region of interest is shown in FIG. To be more precise, Fig.32 shows the first option. In particular, the SEI message of the region of interest may, for example, be used at the access unit level or the sub-picture level to signal one or more regions of interest. 32, an individual ROI is signaled once per ROI SEI message, instead of signaling all the scope ROIs of the corresponding ROI packet within a single ROI SEI message if multiple ROIs are in the current scope.

31, the ROI SEI message signals each ROI separately. The semantics could be defined as follows:

roi_id specifies the ID of the region of interest.

roi_priority indicates the priority of all tiles that belong to the region of interest in prefixed slices or all slices following in decoding order depending on whether the SEI message is sent at the sub-picture level or at the access unit level. The value of roi_priority must be between 0 and 7 inclusive, where 7 indicates the highest priority. If both roi_priority in the roi_info SEI message and tile_priority in the sub-tile tile information SEI messages are specified, the highest value of both is valid for the priority of the individual tiles.

num_tiles_in_roi_minus1 indicates the number of tiles in prefixed slices in decoding order that belong to the region of interest.

roi_tile_id[ i ] indicates the tile_id of the i-th tile that belongs to the region of interest in the prefixed slices following in the decoding order.

That is, Fig. 31 shows that the ROI packet, as shown in Fig. 15, could signal in it the ID of the region of interest to which the corresponding ROI packet and the payload packet belonging to its scope belong. Optionally, the ROI priority index could be signaled along with the ROI ID. However, both syntax elements are optional. Further, the num_tiles_in_roi_minus1 syntax element may indicate the number of tiles in the scope of the corresponding ROI package belonging to the corresponding ROI 60. Further, roi_tile_id indicates the tile ID of the i-th tiles belonging to the ROI 60. Imagine, for example, that the image 18 would be subdivided into sub-pictures 70 in the manner shown in FIG. 16 and that ROI 60 in FIG. 15 would correspond to the left half of image 18 would be generated by the first and third sub-pictures in decoding order. Further, the ROI packet may be placed before the first payload packet 32 of the access unit 30 of FIG. 16, followed by an additional ROI packet between the fourth and fifth payload packets 32 of that access unit 30. The first ROI packet would then have num_tile_in_roi_minus1 set to 0 with roi_tile_id[0] set to 0 (thereby referring to the first tile in decoding order), where the second ROI packet before the fifth payload packet 32 would have num_tiles_in_roi_minus1 set to 0 , with roi_tile_id[0] set to 2 (thereby designating the third tile in the decoding order in the bottom left quarter of image 18).

According to the second option, the syntax of the SEI message of the region of interest could be as shown in Fig.32. Here, all ROIs are signaled in one SEI message. In particular, the same syntax as discussed above with respect to FIG. 31 would be used, but by multiplying the syntax elements for each of the ROIs among the ROIs to which the corresponding ROI SEI message or ROI packet belongs, by the number signaled by the syntax element, here as an example num_rois_minus1. Optionally, an additional syntax element, here as an example roi_presentation_on_separate_screen, could signal for each ROI whether the corresponding ROI is suitable for being presented on a separate screen.

The semantics could be as follows:

num_rois_minus1 indicates the number of ROIs in prefixed slices or regular slices in decoding order.

roi_id[ i ] indicates the ID of the i-th region of interest.

roi_priority[ i ] indicates the priority of all tiles that belong to the region of interest in prefixed slices or all slices following in decoding order depending on whether the SEI message is sent at the sub-picture level or at the access block level. The value of roi_priority must be between 0 and 7 inclusive, where 7 indicates the highest priority. If both roi_priority in the roi_info SEI message and tile_priority in the sub-tile tile information SEI messages are specified, the highest value of both is valid for the priority of the individual tiles.

num_tiles_in_roi_minus1[ i ] indicates the number of tiles in prefixed slices in decoding order that belong to the i-th region of interest.

roi_tile_id[ i ][ n ] indicates the tile_id of the n-th tile that belongs to the i-th region of interest in the prefixed slices following in the decoding order.

roi_presentation_on_seperate_screen [ i ] indicates that the region of interest associated with the i-th roi_id is suitable for presentation on a separate screen.

Thus, briefly summarizing the various embodiments described so far, an additional high-level syntax signaling strategy has been introduced that allows the use of SEI messages, as well as additional high-level syntax elements beyond those included in the slice-level NAL unit header. . Therefore, the inventors have described the slice prefix NAL unit. The syntax and semantics of the slice prefix and slice_level/sub-picture level SEI messages have been described along with use cases for low latency/sub-picture CPB operations, sub-picture signaling, and ROI signaling. An extended syntax has been introduced to signal the slice header portion of the following slices in the slice prefix optionally.

For the sake of completeness, FIG. 33 shows an additional example for a syntax that could be used for a timing control packet according to the embodiment of FIGS. 12-14. The semantics could be:

du_spt_cpb_removal_delay_increment specifies the duration, in units of sub-clocks, between the nominal CPB time of the last decoding block in decoding order in the current access block and the decoding block associated with the decoding block information SEI message. This value is also used to calculate the earliest possible time of arrival of the decoding block data in the CPB for the HSS as defined in Annex C. The syntax element is represented by a fixed length code whose length in bits is given by du_cpb_removal_delay_increment_length_minus1 + 1. When the decoding block associated with the SEI message decoding block information is the last decoding block in the current access block, du_spt_cpb_removal_delay_increment shall be 0.

dpb_output_du_delay_present_flag equal to 1 specifies the presence of the pic_spt_dpb_output_du_delay syntax element in the decoding block information SEI message. dpb_output_du_delay_present_flag equal to 0 specifies the absence of the pic_spt_dpb_output_du_delay syntax element in the decoding block information SEI message.

pic_spt_dpb_output_du_delay is used to calculate the output time of the DPB picture when SubPicHrdFlag is equal to 1. It determines how many clock sub-clocks to wait after removing the last decode block in the access block from the CPB before the decoded picture is output from the DPB. When not present, the value of pic_spt_dpb_output_du_delay is assumed to be pic_dpb_output_du_delay. The length of the pic_spt_dpb_output_du_delay syntax element is given in bits by dpb_output_delay_du_length_minus1 + 1.

The bitstream conformance requirement is that all decoding block information SEI messages that are associated with the same access block apply to the same operation point, and have dpb_output_du_delay_present_flag equal to 1, must have the same pic_spt_dpb_output_du_delay value. The output time obtained from pic_spt_dpb_output_du_delay of any image that is output from a satisfying decoder output time must precede the output time obtained from pic_spt_dpb_output_du_delay of all images in any subsequent CVS in decoding order.

The order in which images are displayed, set by the values of this syntax element, will be the same order as set by the values of PicOrderCntVal.

For pictures that are not output by the "peep" process because they precede in decoding order an IRAP picture with NoRaslOutputFlag equal to 1, which has no_output_of_prior_pics_flag equal to 1, or implied to be 1, the output times obtained from pic_spt_dpb_output_du_delay, should be incrementing with increasing PicOrderCntVal value relative to all images in the same CVS. For any two images in CVS, the difference between the output times of the two images when SubPicHrdFlag is 1 must be identical to the same difference when SubPicHrdFlag is 0.

In addition, FIG. 34 shows an additional example for ROI area signaling using ROI packets. 34, the ROI packet syntax contains only one flag indicating whether all sub-parts of the image 18 encoded in any payload packet 32 belonging to its scope belong to the ROI or not. The "scope" is extended to the occurrence of an ROI packet or an SEI region_refresh_info message. If the flag is 1, the area is indicated as being encoded into the corresponding subsequent payload packet(s), and if 0, the reverse applies, i.e. the corresponding sub-parts of image 18 do not belong to ROI 60.

Before discussing some of the above embodiments again in other words with additional explanation of some of the terms used above such as sub-picture, slice and WPP sub-stream sub-stream, it should be noted that the above embodiment High Level signaling may alternatively be defined in transport specifications such as [3-7]. In other words, the packets mentioned above and forming the sequence 34 may be transport packets, some of which have application layer sub-parts such as slices included, for example, packaged in full or fragmented there, some being placed between the latter in a manner and with the intent, discussed below. In other words, the above placed packets are not limited to be defined as SEI messages of other NAL unit types defined in the application layer video codec, but could alternatively be an additional transport packet defined in transport protocols.

In other words, in accordance with one aspect of the present description, the above embodiments have shown a video data stream having video content encoded therein in blocks of sub-parts (see coding tree blocks or slices) of video content images, with each sub-part correspondingly encoded into one or more payload packets (see VCL NAL units) of a sequence of packets (NAL units) of a video stream, the sequence of packets is divided into a sequence of access units such that each access unit collects payload packets related to a corresponding image of the video content, where the sequence of packets placed timing control packets (slice prefix) there, so that timing control packets subdivide access blocks into decode blocks, so that at least some access blocks are subdivided into two or more decode blocks, with each timing control packet signaling a buffer extraction time a decoder-ra for a decoder whose payload packets follow the corresponding timing control packet in the packet sequence.

As described above, the area in relation to which the video content is encoded into the video data stream in units of sub-pictures may cover syntax elements related to predictive coding, such as coding modes (such as intra mode, inter mode, division information, and the like) , prediction parameters (such as motion vectors, extrapolation directions, or the like), and/or residual data (such as transform coefficient levels), wherein these syntax elements are associated with local parts of the picture, such as coding tree blocks, prediction blocks, and residual blocks. (such as transformations), respectively.

As described above, payload packets may each include one or more slices (in whole, respectively). The slices may be independently decodable or may show relationships that prevent them from being independently decodable. For example, entropy slices may be independently entropy decodable, but prediction beyond slice boundaries may be prohibited. Dependent slices may allow WPP processing, ie. encoding/decoding using entropy and predictive coding beyond slice boundaries with the possibility of parallel encoding/decoding of dependent slices in a time overlapping manner, however, with a stepped start of the encoding/decoding procedure of individual dependent slices and the slice/slices referenced by dependent slices.

The sequential order in which the access block payload packets are ordered in the corresponding access block may be known to the decoder in advance. For example, the encoding/decoding order may be determined among the sub-parts of the images, such as the scanning order among the encoding tree blocks in the above examples.

See, for example, the figure below. The currently encoded/decoded image 100 may be divided into sub-images, which in FIGS. 35 and 36, for example, correspond to four quadrants of the image 110 by way of example and are indicated by numerals 112a-112d. That is, the whole image 110 may form one sub-image, as in the case of FIG. 37, or may be segmented into more than one sub-image. The sub-image segmentation can be limited to conventional segmentation in which sub-images are arranged in columns and rows only. Various examples are presented below.

As can be seen, the image 110 is further subdivided into encoding (tree) blocks 114 (small rectangles in the figure and called CTBs above), among which an encoding order 116 is defined (here, raster scan order, but can also be different). The subdivision of the image into sub-images 112a-d may be constrained such that the sub-images are non-overlapping sets of blocks 114. In addition, both blocks 114 and sub-images 112a-d may be constrained to a standard arrangement in columns and rows.

If there are tiles (i.e., more than one) of the image, then the de(encoding) order 116 bitmap scans the first complete image fragment, then proceeds - also in the bitmap scanning order of the image fragment - to the next image fragment in the order of the image fragment.

Because the sub-pictures are independent in terms of encoding/decoding from each other due to the lack of crossing of sub-picture boundaries by spatial predictions and context choices derived from the spatial neighborhood, encoder 10 and encoder 12 can encode/decode an image subdivided into sub-pictures 112 (previously listed as 70) in parallel, independently of each other - except, for example, in-loop or post-filtering, which may be allowed to cross sub-image boundaries.

Image 110 may be further subdivided into slices 118a-d, 180, previously referred to using reference numeral 24. A slice may contain only part of a sub-image, one complete sub-image, or more than one entire sub-image. Thus, slicing can also subdivide the sub-images, as in the case of FIG. Each slice contains at least one coding block 114 in its entirety and consists of consecutive coding blocks 114 in coding order 116 such that the order is determined among the slices 118a-d, after which figure indices have been assigned. The slicing in Figs. 35-37 has been chosen for purposes of illustration only. The tile boundaries may be signaled in the data stream. Image 110 may form a single sub-image as shown in FIG.

Encoder 10 and decoder 12 may be configured to satisfy sub-picture boundaries conditions in that spatial prediction is not applied across sub-picture boundaries. Context adaptation, i.e. probabilistic adaptations of various entropy (arithmetic) contexts, can continue in whole parts. However, whenever a slice crosses - in accordance with encoding order 116 - a sub-picture boundary (if present within the slice), as for example in FIG. 36 with respect to slices 118a,b, then the slice is in turn subsections (substreams or image fragments) with a slice containing pointers (c.p. entry_point_pffset) pointing to the start of each subsection. In the decoder loop, filters may be allowed to cross sub-picture boundaries. Such filters may include one or more of a deblocking filter, an Adaptive Sample Shift (SAO) filter, and an Adaptive Loop Filter (ALF). The latter can be applied to the borders of a sub-image/slice if enabled.

Each optional second and subsequent subsections may have their beginnings byte-aligned in a slice with a pointer indicating the offset from the beginning of one subsection to the beginning of the next subsection. The subsections are arranged within the slices in scan order 116 . Fig.38 shows an example with slices 180c in Fig.37, which is subdivided into subsections 119i as an example.

With regard to the figures, it should be noted that the slices forming the sub-pieces of the sub-images do not need to end with a line in the sub-image 112a. See, for example, slice 118a in Figures 37 and 38.

The figure below shows an exemplary portion of the data stream related to the block of data associated with image 110 of FIG. 38 above. Here, each payload packet 122a-d - previously indicated by numeral 32 - includes only one slice 118a by way of example. Two packets 124a,b - previously indicated by reference numeral 36 - are shown as placed in access block 120 for illustrative purposes: 124a precedes packet 122a in packet order 126 (corresponding to the de/coding time axis), and 124b precedes packet 122c. Accordingly, the access block 120 is divided into two decoding blocks 128a,b - previously indicated by reference numeral 38 - the first of which contains packets 122a,b (along with optional padding data packets (following the first and second packets 122a,b, respectively) and optional leading access block SEI packets (preceding the first packet 122a)), and the second of which contains packets 118c,d (along with optional padding data packets (following packets 122c,d, respectively)).

As described above, each package in a sequence of packages can be assigned to exactly one package type from a plurality of package types (nal_unit_type). Payload packets and timing control packets (and optional padding and SEI data packets) have, for example, a different packet type. Instances of packages of a particular package type in a package sequence may be subject to certain restrictions. These constraints may define the order among packet types (see FIG. 17) that packets in each access block must obey, such that access block boundaries 130a,b are discoverable, and remain in the same position in the packet sequence even if packets of any packet type to be removed are removed from the video stream. For example, payload packets are packets of a non-removable packet type. However, timing control packets, padding data packets, and SEI packets may, as discussed above, be of a removable packet type, i. e. they may be non-VCL NAL units.

In the example above, timing control packets have been explicitly exemplified above via the slice_prefix_rbsp() syntax.

By using such placement of timing control packets, the encoder is able to control buffering scheduling on the decoder side during the encoding of individual pictures of the video content. For example, the encoder has the ability to optimize buffer scheduling to minimize end-to-end delay. In this regard, the encoder has the ability to take into account a separate distribution of encoding complexity over the video content image area for individual video content images. In particular, the encoder may continuously output the sequence of packets 22, 122a-d, 122a-d _1-3 on a per-burst basis (ie, once the current packet has been completed, it is output). By using timing control packets, the encoder is able to adjust the buffer scheduling on the decoding side at times when some of the sub-parts of the current picture have already been encoded into the corresponding payload packets with the remaining sub-parts, however, have not yet been encoded.

Accordingly, an encoder for encoding into video content a video data stream in sub-parts (see coding tree blocks, picture tiles, or slices) of video content images, with corresponding encoding of each sub-part into one or more payload packets (see VCL NAL blocks) sequence of packets (NAL units) of the video data stream, so that the sequence of packets is divided into a sequence of access units, and each access unit collects payload packets related to the corresponding image of the video content, can be configured to put timing control packets (slice prefix) in the packet sequence , so that timing control packets subdivide access units into decode units, such that at least some access units are subdivided into two or more decode units, with each timing control packet signaling a decoder buffer extraction time for the decode unit, payload packets to followed by the corresponding timing control packet in the packet sequence.

Any decoder receiving the video data stream just described is free to use the scheduling information contained in the timing control packet or not. However, while the decoder is free to use the information, the decoder corresponding to the codec level must be able to decode the data following the specified timing. If there is usage, the decoder feeds its decoder buffer and empties the decoder buffer in units of decode blocks. The "decoder buffer" may, as described above, include a decoded picture buffer and/or an encoded picture buffer.

Accordingly, a decoder for decoding a video data stream having video content encoded therein in units of sub-parts (see coding tree blocks, sub-pictures or slices) of images, with each sub-part respectively encoded into one or more payload packets (see .VCL NAL units) sequence of packets (NAL units) of the video data stream, the sequence of packets is divided into a sequence of access units, so that each access unit collects payload packets related to the corresponding video content image, can be configured to search for timing control packets placed in a sequence of packets, subdividing access blocks into decode blocks in timing control packets such that at least some access blocks are subdivided into two or more decode blocks, deriving from each timing control packet the decoder buffer extraction time for the decode block, payload packets whose loads follow the corresponding timing control packet in the sequence of packets, and decode block extractions from the decoder buffer scheduled at times determined by the decoder buffer extraction times for the decode blocks.

The search for timing control packets may include a decoder examining the NAL unit header and the syntax element contained there, namely nal_unit_type. If the value of the last flag is equal to some value, i.e. according to the examples above, 124, then the packet currently being checked is the timing control packet. That is, the timing control packet would contain or convey the information explained above with respect to pseudo-code sub-picture buffering as well as subpic_timing. That is, the timing control packets may convey or determine the initial CPB removal delays for the decoder, or determine how many clock cycles to wait after the corresponding decoding block is removed from the CPB.

To allow repeated transmission of timing control packets without inadvertently further dividing the access unit into additional decoding units, a flag in the timing control packets may explicitly signal whether the current timing control packet participates in the division of the access unit into coding units or not (compare decoding_unit_start_flag = 1 indicating decoding unit start, and decoding_unit_start_flag = 0, signaling the opposite).

The use aspect of the sub-picture identification information related to the placed decoding unit differs from the use aspect of the timing control packets related to the placed decoding unit in that the sub-picture identification packets are placed in the data stream. The above timing control packets may additionally be placed in the data stream, or the decoder buffer extraction times may be transmitted along with the sub-picture identification information explained below in the same packet normally. Accordingly, the details provided in the above section can be used to clarify the issues in the description below.

A further aspect of the present description, derived from the above embodiments, shows a video data stream having video content encoded therein, using predictive and entropy coding, in units of slices into which images of the video content are spatially subdivided, using an encoding order among slices with predictive encoding limiting predictions, and /or entropy encoding to the interior of the image fragments into which the video content images are spatially subdivided, where the sequence of slices in the order of encoding is packaged into payload packets of the sequence of packets (NAL units) of the video data stream in the order of encoding, while the sequence of packets is divided into a sequence of access blocks, so that each access unit collects the payload packets packaged in it into slices related to the corresponding image of the video content, where the sequence packets has the sub-picture identification packets placed there, identifying the sub-pictures (potentially only one) that are overlapped by slices (potentially only one), packaged in one or more payload packets immediately following the sub-picture identification packet in the sequence of packets.

See, for example, the immediately preceding figure showing the flow of data. Packets 124a and 124b will now present sub-picture identification packets. Either by explicit signaling (cf. single_slice_flag = 1) or by convention, a slice identification packet can only identify slices that are overlapped by slices packaged in the immediately following payload packet 122a. Alternatively, by explicit signaling or by convention, the sub-picture identification packet 124a may identify the sub-pictures that are overlapped by slices packaged in one or more payload packets immediately following the corresponding sub-picture identification packet 124a in the packet sequence, up to the earlier of the end. 130b of the current access block 120 and the beginning of the next decoding block 128b, respectively. See, for example, Fig. 35: if each slice 118a-d _1-3 was separately packetized into a corresponding packet 122a-d _1-3 , subdivided into decoding blocks such that the packets are grouped into three decoding blocks according to {122a _1-3 }, {122b _1-3 } and {122c _1-3 , 122d _1-3 }, then slices {118c _1-3 , 118d _1-3 } are bundled into packets {122c _1-3 , 122d _1-3 } of the third decoding block, for example, the overlay image tiles 112c and 112d and the corresponding slice prefix would indicate, for example, when referring to the complete decoding block, "c" and "d", i.e. these fragments 112c and 112d of the image.

Thus, the network entity, mentioned further below, can use this explicit signaling or convention to correctly associate each sub-picture identification packet with one or more payload packets immediately following the identification packet in the packet sequence. The way in which an identification can be signaled has been roughly described above with the subpic_tile_info pseudo-code. Associated payload packets have been referred to above as "prefixed slices". Naturally, the example can be changed. For example, the "tile_priority" syntax elements can be left out. In addition, the order among the syntax elements can be switched, and the descriptor regarding the possible bit lengths and encoding principles of the syntax elements are only illustrative.

A network entity that receives a video stream (i.e., a video stream having video content encoded therein using predictive and entropy coding, in units of slices into which video content images are spatially subdivided, using coding order among slices, with predictive coding limiting predictions and/or entropy encoding to the interior of the sub-images into which video content images are spatially subdivided, where a sequence of slices in coding order is packetized into payload packets of a sequence of video data stream packets (NAL units) in coding order, wherein the sequence of packets is divided into a sequence of access blocks , so that each access unit collects payload packets packaged in it into slices related to the corresponding image of the video content, where the sequence of packets has fragment identification packets from fragments placed there) can be configured to identify, based on sub-picture identification packets, sub-pictures that are overlapped by slices packaged in one or more payload packets immediately following the corresponding sub-picture identification packet in the burst sequence. The network entity may use the identification result to select transmission tasks. For example, a network object may process different sub-images with different playback priorities. For example, in the event of a packet loss, those payload packets related to higher priority tile may be preferred for retransmission over payload packets related to lower priority tile. That is, the network entity may first request retransmission of lost payload packets related to higher priority image tiles. Only if there is sufficient time remaining (depending on the transmission rate) does the network entity proceed to request retransmission of lost payload packets related to sub-images of lower priority. The network entity may, however, also be a renderer that is capable of assigning sub-pictures or payload packets related to certain sub-pictures to different screens.

With regard to the aspect of using the placed region of interest information, it should be noted that the ROI packets mentioned below could coexist with the above mentioned timing control and/or sub-picture identification packets, either by combining their information content with common packets, as described above with respect to to slice prefixes, or in the form of separate packets.

The aspect of using placed region of interest information as described above shows, in other words, a video data stream having video content encoded in it using predictive and entropy coding, in units of slices into which video content images are spatially subdivided using coding order among slices, with limiting predictions of predictive coding and/or entropy coding to the interior of the sub-images into which video content images are divided, where a sequence of slices in coding order is packetized into payload packets of a sequence of packets (NAL units) of a video stream in coding order, wherein the sequence of packets is divided into a sequence of access units, such that each access unit collects payload packets packaged therein into slices related to the corresponding video content image, where the sequence of packets has pack The ROI sets placed there identifying the image fragments that belong to the image ROI, respectively.

With respect to ROI packets, similar comments are valid as those provided earlier with respect to image fragment identification packets: ROI packets can only identify image fragments belonging to an image ROI among those image fragments that are overlapped by slices contained in one or more packets. payloads to which the corresponding ROI packet refers by immediately preceding one or more payload packets, as described above with respect to "prefixed slices".

ROI packets may allow the identification of more than one ROI per prefixed slice, with the identification of associated sub-pictures for each of these ROIs (c.p. num_rois_minus1)). A priority can then be passed for each ROI, allowing the placement of the ROI in terms of priority (c.p. roi_priority[ i ]). To allow "tracking" of an ROI over time over the course of a sequence of video images, each ROI can be indexed with an ROI index so that the ROIs specified in the ROI packets can be linked to each other beyond/across image boundaries, i.e. in time (c.p. roi_id[ i ]).

A network entity that receives a video stream (i.e., a video stream having video content encoded therein using predictive and entropy coding, in units of slices into which video content images are spatially subdivided, using coding order among slices, with predictive coding limiting predictions to the inside of the image fragments into which the video content images are divided, while continuing the probabilistic adaptation of entropy coding across all slices, where the sequence of slices in coding order is packaged into payload packets of the sequence of packets (NAL units) of the video data stream in coding order, while the sequence of packets divided into a sequence of access units such that each access unit collects payload packets packaged therein into slices related to a corresponding video content image) may be configured to identification, based on image fragment identification packets, packets that package slices that overlap image fragments that belong to the image ROI.

The network entity may use the information conveyed by the ROI packet in a manner similar to that explained above in this previous section regarding sub-picture identification packets.

Regarding the current section, as well as the previous section, it should be noted that any network entity, such as a MANE or a decoder, is able to determine which image fragment or fragments are covered by the slice or slices of the payload packet currently being examined, simply by polling the slice order. images and interrogating the progress of the portion of the current image these slices cover with respect to the position of the image tiles in the image, which may be explicitly signaled in the data stream as explained above, or may be known to the encoder and decoder by convention. Alternatively, each slice (except the first one in the picture in the scan order) can be provided with an indication/index (slice_address, measured in coding tree block units) of the first coding block (e.g. CTB) the same refers to (the same codes), so that the decoder can place each slice (its recovery) in the picture from this first coding block in the slice order direction. Accordingly, it may be sufficient if the aforementioned tile information packets only contain the index of the first tile (first_tile_id_in_prefixed_slices) overlapped by any slice of the associated one or more payload packets immediately following the corresponding tile identification packet, since it is clear to the network entity upon encountering the next sequencing sub-picture identification packet such that if the index transmitted by the last sub-picture identification packet differs from the previous one by more than one, then the payload packets between the two sub-picture identification packets cover the sub-pictures having a sub-picture index between them. This is true if, as mentioned above, both the sub-image subdivision and the coding unit subdivision are based, for example, on a per-line/per-column subdivision having a raster scan order defined among them, which is, as for sub-images , and for coding blocks, line by line, for example, i.e. the sub-picture index increases in this bitmap scan order as well as the slices follow each other according to the slice order according to this bitmap scan order among the coding blocks.

The packetized and placed slice header signaling aspect described as derived from the above embodiments is also combinable with any of the above aspects or any combination thereof. Previously explicitly described slice prefixes, such as those according to version 2, combined all of these aspects. An advantage of the present aspect is that slice header data can be made more easily accessible to network entities as it is transmitted in stand-alone packets external to the prefixed slices/payload packets, and repeated transmission of slice header data is possible.

Accordingly, a further aspect of the present description is an aspect of packetized and placed slice header signaling and can be, in other words, considered as showing a video data stream having video content encoded therein, in units of sub-parts (see coding tree blocks or slices) of pictures. video content, with each sub-part respectively encoded into one or more payload packets (see VCL NAL blocks) of a sequence of packets (NAL units) of the video data stream, with the packet sequence divided into a sequence of access units, such that each access unit collects packets payloads related to the corresponding video content image, where a sequence of packets is placed in slice header packets (slice prefix) carrying slice header data for, and missing from, one or more payload packets that follow the corresponding slice header packet in the sequence packages.

A network entity that receives a video stream (i.e., a video stream having video content encoded therein, in units of sub-parts (see coding tree blocks or slices) of video content images, with each sub-part respectively encoded into one or more than payload packets (see VCL NAL units) of a sequence of packets (NAL units) of a video data stream, wherein the sequence of packets is divided into a sequence of access units such that each access unit collects payload packets related to the corresponding image of the video content where the sequence of packets is placed into slice header packets) may be configured to read slice header along with payload data for slices from packets, however, obtaining slice header data from slice header packets and skipping slice header reading for one or more payload packets that follow corresponding slice header packet in n sequence of packets, but instead adapting a slice header derived from a slice header packet followed by one or more payload packets.

As was true with the above aspects, it is possible that packets, here slice header packets, could also have the functionality of indicating any network entity, such as a MANE or a decoder, the start of a decoding block, or the start of executions of one or more payload packets prefixed with the appropriate package. Accordingly, the network entity according to the present aspect can identify payload packets for which the packet header reading should be skipped based on the aforementioned syntax elements in that packet, namely single_slice_flag, in combination with, for example, decoding_unit_start_flag, of which the last flag makes possible, as discussed above, retransmission of copies of certain slice header packets in the decoding block. This is useful, for example, since the slice header in one decoding unit may change along the sequence of slices, and accordingly, while slice header packets at the beginning of decoding blocks may have decoding_unit_start_flag set (equal to one), slice header packets located between them may have this flag is not set in order to prevent any network entity from misinterpreting the appearance of this slice header packet as a new decoding unit.

Although some aspects have been described in the context of a device, it is clear that these aspects also represent a description of the corresponding method, where the block or device corresponds to a method step or method step feature. Likewise, the aspects described in the context of a method step also provide a description of the corresponding block or element or feature of the corresponding device. Some or all of the steps of the method may be performed by (or using) a hardware device such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some one or more of the most important steps of the method may be performed by such a device.

The inventive video data stream may be stored in a digital storage medium or may be transmitted over a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

Depending on certain implementation requirements, embodiments of the invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium such as floppy disk, DVD, Blu-Ray, PROM (PROM, Programmable Read Only Memory), EPROM (EPROM, Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory). device) or FLASH memory having electronically readable control signals stored therein, which interacts (or is capable of interacting) with a programmable computer system such that the corresponding method is performed. Therefore, the digital storage medium can be machine readable.

Some embodiments in accordance with the invention comprise a storage medium having electronically readable control signals that are capable of interfacing with a programmable computer system such that one of the methods described herein is performed.

Typically, embodiments of the present invention may be implemented as a computer program product with program code, wherein the program code is capable of performing one of the methods when the computer program product is executed on the computer. The program code may, for example, be stored on a computer-readable medium.

Other embodiments include a computer program for performing one of the methods described herein, stored on a computer-readable medium.

In other words, an embodiment of the inventive method is therefore a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

An additional embodiment of the inventive methods, therefore, is a storage medium (or digital storage medium, or computer-readable medium) containing a computer program recorded thereon for performing one of the methods described herein. The storage medium, digital storage medium, or recorded medium is typically tangible and/or non-temporal.

An additional embodiment of the inventive method is therefore a stream of data or a sequence of signals representing a computer program for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted over a data connection, such as the Internet.

An additional embodiment comprises computing means, such as a computer, or a programmable logic device capable of or adapted to perform one of the methods described herein.

An additional embodiment includes a computer with a computer program installed on it for performing one of the methods described in the materials of this application.

An additional embodiment, in accordance with the invention, comprises a device or system configured to transmit (eg, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, or the like. The device or system may, for example, include a file server for transferring a computer program to the receiver.

In some embodiments, a programmable logic device (eg, field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, the field-programmable gate array may communicate with the microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

The claimed invention, in particular, covers the following objects:

1. A video data stream having video content (16) encoded in it, in units of sub-parts (24) of images (18) of video content (16), with each sub-part (24) respectively encoded in one or more packets (32 ) payload of a sequence (34) of packets of a stream (22) of video data, wherein the sequence (34) of packets is divided into a sequence of blocks (30) of access, so that each block (30) of access collects packets (32) of the payload related to the corresponding image (18) of the video content (16), wherein the sequence (34) of packets contains the timing control packets (36) placed therein, so that the timing control packets (36) subdivide access blocks (30) into decoding blocks (38), so that at least At least some access blocks (30) are subdivided into two or more decoding blocks (38), with each timing control packet (38) signaling the decoder buffer extraction time for the decoding block (38), payload packets (32) followed by the corresponding timing control packet (38) in the sequence (34) of packets.

2. The video data stream of claim 1, wherein the sub-parts (24) are slices and each payload packet (32) includes one or more slices.

3. The video data stream of claim 2, wherein the slices comprise independently decodable slices and dependent slices allowing, for WPP processing, decoding using entropy and predictive decoding beyond slice boundaries.

4. The stream of video data according to any one of paragraphs. 1-3, in which each packet of the sequence (34) of packets can be assigned to exactly one type of packet from a plurality of packet types, while the payload packets (32) and timing control packets (36) have different packet types, where the occurrences of packets from the set packet types in a sequence (34) of packets are subject to certain constraints that define the order among the packet types that the packets in each access block (30) must obey, such that access block boundaries are discoverable using constraints by detecting instances where there are constraint objections. , and remain in the same position in the packet sequence even if packets of any packet type to be removed are removed from the video stream, payload packets (32) being the non-removable packet type and timing control packets (36) being the packet type to be removed.

5. The stream of video data according to any one of paragraphs. 1-4, in which each package contains a part of a syntax element indicating the package type.

6. The video data stream according to claim 5, wherein the packet type part of the syntax element contains a packet type field in the packet header of the corresponding packet, the content of which differs between payload packets and timing control packets, and for timing control packets, the SEI packet type field, different between timing control packets on the one hand, and other type SEI packets on the other hand.

7. An encoder for encoding video content (16) video data into a stream (22) in units of sub-parts (24) of images (18) of video content (16), with the corresponding encoding of each sub-part (24) into one or more packets (32) payload of a sequence (34) of packets of a stream (22) of video data, so that the sequence (34) of packets is divided into a sequence of blocks (30) of access, and each block (30) of access collects packets (32) of the payload related to the corresponding image (18 ) video content (16), while the encoder is configured to place timing control packets (36) in a sequence (34) of packets, so that timing control packets (36) subdivide access blocks (30) into decoding blocks (38), so that according to at least some access blocks (30) are subdivided into two or more decoding blocks (38), with each timing control packet (36) signaling the decoder buffer extraction time for decoding block (38), payload packets (32) and which is followed by the corresponding timing control packet (36) in the sequence (34) of packets

8. The encoder according to claim 7, in which, in the materials of the present application, the encoder is configured to, during encoding of the current image of the video content,

encode the current sub-part (24) of the current image (18) into the current packet (32) of the payload of the current decoding block (38),

transmit in the data stream the current decoding block (38) prefixed with the current timing control packet (36), setting the decoder buffer extraction time signaled by the current timing control packet (36) to the first time point; and

encode an additional sub-part of the current picture at a second time later than the first time.

9. Method for encoding video content (16) video data stream (22) in units of sub-parts (24) of images (18) of video content (16), with corresponding encoding of each sub-part (24) into one or more payload packets (32) loads a sequence (34) of packets of a stream (22) of video data, so that a sequence (34) of packets is divided into a sequence of access blocks (30), and each access block (30) collects payload packets (32) related to the corresponding image (18) video content (16), wherein the method comprises the step of placing timing control packets (36) in a sequence (34) of packets, so that timing control packets (36) subdivide access blocks (30) into decoding blocks (38), so that at least some access blocks (30) are subdivided into two or more decoding blocks (38), with each timing control packet (36) signaling the decoder buffer extraction time for decoding block (38), payload packets (32) and which is followed by the corresponding timing control packet (36) in the sequence (34) of packets.

10. A decoder for decoding a stream (22) of video data having video content (16) encoded in units of sub-parts (24) of images (18) of video content (16), with each sub-part respectively encoded in one or more packets (32) payload of a sequence (34) of packets of a stream (22) of video data, a sequence (34) of packets is divided into a sequence of access blocks (30), so that each access block (30) collects payload packets (32) related to the corresponding image (18) video content (16), while the decoder contains a buffer for buffering the video data stream or recovering the video content obtained from there by decoding the video data stream, and configured to search for timing control packets (36) placed in a packet sequence, subdivisions of blocks (30) access to decoding blocks (38) in timing control packets (36), so that at least some access blocks are subdivided into two or and more decoding blocks, and buffer underruns in units of decoding blocks.

11. The decoder according to claim 10, wherein the decoder is configured, when searching for timing control packets (36), to check in each packet the packet type-indicating part of the syntax element and, if the value of the packet-type-indicating part of the syntax element is equal to a predefined value, to recognize the corresponding packet as timing control package (36).

12. A method for decoding a stream (22) of video data having video content (16) encoded in units of sub-parts (24) of images (18) of video content (16), with each sub-part respectively encoded in one or more packets ( 32) the payload of a sequence (34) of packets of a stream (22) of video data, the sequence (34) of packets is divided into a sequence of blocks (30) of access, so that each block (30) of access collects packets (32) of the payload related to the corresponding image ( 18) video content (16), wherein the method uses a buffer for buffering the video data stream or restoring the video content obtained from it by decoding the video data stream, and comprises the steps of searching for the timing control packets (36) placed in the packet sequence, subdivide the blocks (30) accesses to decoding blocks (38) in timing control packets (36), so that at least some access blocks are subdivided into two or more decoding blocks, and empty the buffer in units of decoding blocks.

13. A network object for transmitting a stream (22) of video data having video content (16) encoded in it in units of sub-parts (24) of images (18) of video content (16), with each sub-part respectively encoded into one or more packets (32) of the payload of a sequence (34) of packets in the stream (22) of video data, while the sequence (34) of packets is divided into a sequence of blocks (30) of access, so that each block (30) of access collects packets (32) of the payload, related to the corresponding image (18) of the video content (16), wherein the decoder is configured to search for timing control packets (36) placed in a sequence of packets, dividing access blocks into decoding blocks in timing control packets (36), so that at least some blocks (30) of access are subdivided into two or more blocks (38) of decoding, receiving from each packet (36) timing control of decoder buffer extraction time for block (38) decoder , the payload packets (32) of which follow the corresponding timing control packet (36) in the packet sequence (34), and performing the transmission of a video data stream dependent on the decoder buffer extraction times for the decoding units (38).

14. A method for transmitting a stream (22) of video data having video content (16) encoded in it in units of sub-parts (24) of images (18) of video content (16), with each sub-part respectively encoded in one or more packets ( 32) payload of a sequence (34) of packets in a stream (22) of video data, wherein the sequence (34) of packets is divided into a sequence of access blocks (30), so that each access block (30) collects payload packets (32) related to corresponding image (18) of the video content (16), wherein the method comprises the steps of searching for timing control packets (36) placed in a sequence of packets, subdividing access blocks into decoding blocks in timing control packets (36), so that at least At least some access blocks (30) are subdivided into two or more decoding blocks (38), receive from each timing control packet (36) the decoder buffer extraction time for the decoder block (38). the payload packets (32) of which follow the corresponding timing control packet (36) in the packet sequence (34) and perform the transmission of a video data stream dependent on decoder buffer extraction times for decoding units (38).

15. A video data stream having video content (16) encoded therein, using predictive and entropy coding in units of slices (24) into which video content images (18) are spatially subdivided, using coding order among slices (24) with restrictive predictions of predictive coding and/or entropy encoding to the interior of the fragments (70) of the image, into which the video content images are spatially subdivided, where the sequence of slices (24) is packaged into payload packets (32) of the sequence of video data stream packets in coding order, while the sequence (34) of packets subdivided into a sequence of access blocks (30), so that each access unit collects payload packets (32) packaged therein into slices (24) related to the corresponding image (18) of video content (16), where the sequence (34) of packets has packets (72) identifying a fragment of the image, according to placed in it between payload packets of one access block, identifying one or more fragments (70) of the image overlapped by any slice (24) packaged in one or more payload packets (32) immediately following the corresponding packet (72) identification of an image fragment in a sequence (34) of packets.

16. The video data stream of claim 15, wherein the subpicture identification packets (72) identify one or more slices (70) overlaid by any slice (24) packaged in exactly the next payload packet.

17. The video data stream according to claim 15, in which the fragment identification packets (72) identify one or more fragments (70) of the image overlapped by any slice (24) packaged in one or more payload packets (32) directly following the respective fragment identification packet (72) in the packet sequence (34), up to the earlier end of the current access block (30) and the next fragment identification packet (72), respectively, in the packet sequence (34).

18. A network object configured to receive a video stream according to any one of paragraphs. 15-16 and identifying, based on sub-image identification packets (72), the sub-images (70) that are overlapped by slices (24) packaged in one or more payload packets (72) immediately following the corresponding packet (72) identification of an image fragment in a sequence of packets.

19. The network entity of claim 18, wherein the network entity is configured to use the identification result to select transmission tasks related to the video data stream.

20. The network entity of claim 18 or 19, wherein the transmission tasks comprise retransmission requests regarding defective packets.

21. The network object according to claim 18 or 19, where the network object is configured to process different fragments (70) of the image with different priority by assigning a higher priority to the packets (72) identification of the fragment of the image and payload packets immediately following the corresponding packet ( 72) identifying a sub-picture having slices stacked therein that overlap the sub-pictures (70) identified by corresponding sub-picture identification packets for which priority is higher than the sub-picture identification packets (72) and payload packets immediately following the respective a sub-picture identification packet (72) having slices stacked therein that overlap the sub-pictures (70) identified by corresponding sub-picture identification packets for which the priority is lower.

22. The network entity of claim 21, wherein the network entity is configured to first request retransmission of payload packets having a higher priority assigned to them prior to requesting any retransmission of payload packets having a lower priority assigned to them. .

23. A method comprising the steps of receiving a stream of video data according to any one of paragraphs. 15-16 and identify, based on the fragment identification packets (72), the fragments (70) of the image that are overlapped by slices (24) packaged in one or more payload packets (72) immediately following the corresponding packet (72) identification of an image fragment in a sequence of packets.

24. A video data stream having video content (16) encoded in it, in units of sub-parts (24) of images (18) of video content (16), with each sub-part (24) respectively encoded in one or more packets (32 ) payload of a sequence (34) of packets of a stream (22) of video data, wherein the sequence (34) of packets is divided into a sequence of blocks (30) of access, so that each block (30) of access collects packets (32) of the payload related to the corresponding image (18) video content (16), where at least some access blocks have a sequence (34) placed in ROI packets (64), so that timing control packets (64) subdivide access blocks (64) into decoding blocks (38), so that at least some access units (30) have ROI packets (64) placed between payload packets (32) related to the image of the corresponding access unit, with each ROI packet related to one or more following packets payload in the sequence (34) of packets following the corresponding ROI packet and identifying whether the sub-parts (24) encoded in one or more payload packets to which the corresponding ROI packet belongs overlap the video content region of interest.

25. The video data stream of claim 24, wherein the sub-parts are slices and the video content is encoded into the video data stream using predictive and entropy constrained predictive and/or entropy coding to the inside of the sub-images into which the video content images are divided, wherein each ROI packet further identifies sub-images in which sub-parts (24) encoded in any of the one or more payload packets to which the corresponding ROI packet belongs overlap the region of interest.

26. The video data stream of claim 24 or 25, wherein each ROI packet relates exclusively to the immediately following payload packet.

27. The video data stream of claim 24 or 25, wherein each ROI packet refers to all payload packets immediately following the corresponding ROI packet in the sequence of packets up to the earlier end of the access block in which the corresponding ROI packet is located and the next ROI packet , respectively.

28. A network object configured to receive a video stream according to any one of paragraphs. 24-27, and identifying, based on the ROI packets, the ROI of the video content.

29. The network entity of claim 27, wherein the network entity is configured to use the identification result to select transmission tasks related to the video data stream.

30. The network entity of claim 28 or 29, wherein the transmission tasks comprise retransmission requests regarding defective packets.

31. The network entity according to claim 28 or 29, wherein the network entity is configured to process the area of interest (70) with increased priority by assigning a higher priority to the ROI packets (72) and one or more payload packets following the corresponding packet ( 72) The ROI to which the corresponding ROI packet belongs, whose ROI packet signals the overlap of the region of interest with sub-parts (24) encoded in any one or more payload packets to which the corresponding ROI packet belongs, than to the ROI packets and one or more payload packets following the corresponding ROI packet (72) to which the corresponding ROI packet belongs, whose ROI packets signal a lack of overlap.

32. The network entity of claim 31, wherein the network entity is configured to first request retransmission of payload packets having a higher priority assigned to them prior to requesting any retransmission of payload packets having a lower priority assigned to them. .

33. A method comprising the steps of receiving a stream of video data according to any one of paragraphs. 23-26 and identify, based on the ROI packets, the ROI of the video content.

34. A computer program having a program code for implementing, when executed on a computer, the method according to PP. 9, 12, 14, 23 or 33.

The embodiments described above are merely illustrative of the principles of the present invention. It is clear that modifications and changes in the schemes and details described in the materials of this application will be obvious to experts in this field of technology. Therefore, it is intended to be limited only by the scope of the forthcoming claims, and not by the specific details provided through the description and explanation of the embodiments throughout the present application.

Bibliographic list

[1] Thomas Wiegand, Gary J. Sullivan. Gisle Bjontegaard, Ajay Luthra, "Overview of the H.264/AVC Video Coding Standard", IEEE Trans. Circuits Syst. Video Technol., vol. 13, N7, July 2003.

[2] JCT-VC, "High Efficiency Video Coding (HEVC) Textual Specification Working Draft 7", JCTVC-I1003, May 2012.

[3] ISO/IEC 13818-1: Specification for MPEG-2 systems.

[4] IETF RFC 3550 - Real Time Transport Protocol.

[5] Y.-K. Wang et al., "RTP Payload Format for H.264 Video", IETF RFC 6184, http://tools.ietf.org/html/.

[6] S. Wenger et al., "RTP Payload Format for Scalable Video Coding", IETF RFC 6190, http://tools.ietf.org/html/rfc6190.

[7] T. Schierl et al., "RTP Payload Format for High Efficiency Video Coding", IETF Internet Draft, http://datatracker.ietf.org/doc/draft-schierl-payload-rtp-h265/.

Claims (28)

1. A device for decoding a data stream for restoring video content, comprising: a decoder configured to extract from the data stream a sequence of packets representing sub-parts of the video content and organized into a plurality of access blocks, wherein each of the plurality of access blocks relates to an image of the video content and includes one or more decoding blocks, and each decoding block includes including at least one payload packet, and each of the sub-parts corresponds to a slice, wherein the extraction includes entropy decoding slices across slice boundaries in accordance with a parallel wavefront processing technique; identification in each of the plurality of access blocks of one or more timing control packets corresponding to one or more decoding blocks, respectively, and placed in the access block, each of the timing control packets indicating the extraction time from the decoder buffer at which the contents of the corresponding decoding block should be retrieved from the buffer; retrieving the content buffer of each decoding block according to the decoder buffering time associated with the decoding block, wherein each packet from the sequence of packets includes a packet type field in the packet header of the corresponding packet, and a packet type field for at least one packet payload differs from the packet type field for one or more timing control packets; and decoding one or more decoding blocks using predictive decoding to reconstruct the video content. 2. The apparatus of claim 1, wherein at least one payload packet is a non-removable type packet and each of the one or more timing control packets is a removable type packet. 3. The apparatus of claim 1, wherein the sequence of packets contains Supplemental Enhancement Information (SEI) packets of various types, each of the SEI packets includes an SEI packet type field, wherein the content of the SEI packet type field of the timing control packet is different from the packet type field. SEI of another type of SEI package. 4. The apparatus of claim. 1, wherein at least one payload packet contains encoded sub-image information of the video content. 5. The apparatus of claim 1, wherein the various types of packages in the package sequence are arranged according to a predetermined organization, and the organization of the packages in the package sequence can be determined according to the predetermined organization when one or more removable packages are removed from the package sequence. 6. The apparatus of claim 5, wherein the different types of SEI packets include Region of Interest (ROI) packets. 7. A device for encoding video content into a data stream, comprising: an encoder configured to encode sub-parts of the video content based on the video content into a sequence of packets organized into a plurality of access units, wherein each of the plurality of access units relates to an image of the video content and includes one or more decoding units, and each of the decoding units includes including at least one payload packet, and each of the sub-parts corresponds to a slice, the coding including entropy coding of slices across slice boundaries in accordance with a parallel wavefront processing technique; and placing in each of the plurality of access blocks one or more timing control packets corresponding to one or more decoding blocks, respectively, with each timing control packet indicating a decoder buffer extraction time at which the contents of the corresponding decoding block should be retrieved from the buffer, each a packet from a sequence of packets includes a packet type field in the packet header of the corresponding packet, wherein the packet type field for at least one payload packet is different from the packet type field for one or more timing control packets. 8. The apparatus of claim 7, wherein at least one payload packet is a non-removable type packet and each of the one or more timing control packets is a removable type packet. 9. The device according to claim 7, wherein the sequence of packets contains Supplemental Enhancement Information (SEI) packets of various types, each of the SEI packets includes an SEI packet type field, and the content of the SEI packet type field of the timing control packet is different from the content of the packet type field SEI of another type of SEI package. 10. The apparatus of claim 7, wherein at least one payload packet contains encoded sub-image information of the video content. 11. The apparatus of claim 7, wherein the different types of packages in the package sequence are arranged according to a predetermined organization, and the organization of the packages in the package sequence can be determined according to the predetermined organization when one or more removable packages are removed from the package sequence. 12. The apparatus of claim 11, wherein the various types of SEI packets include Region of Interest (ROI) packets. 13. A method for decoding a data stream for restoring video content, comprising: extracting from the data stream a sequence of packets representing sub-parts of the video content and organized into a plurality of access blocks, wherein each of the plurality of access blocks relates to an image of the video content and includes one or more decoding blocks, and each decoding block includes at least one payload packet, and each of the sub-parts corresponds to a slice, wherein the extraction includes entropy decoding slices across slice boundaries according to a wavefront parallel processing technique; identifying in each of the plurality of access blocks one or more timing control packets corresponding to one or more decoding blocks, respectively, and placed in the access block, each of the timing control packets indicating the decoder buffer extraction time at which the contents of the corresponding decoding block should be retrieved from the buffer; extraction from the buffer of the contents of each decoding block in accordance with the decoder buffer extraction time associated with the decoding block, each packet from the sequence of packets includes a packet type field in the packet header of the corresponding packet, and a packet type field for at least one packet payload differs from the packet type field for one or more timing control packets; and decoding one or more decoding blocks using predictive decoding to reconstruct the video content. 14. The method of claim 13, wherein at least one payload packet is a non-removable type packet and each of the one or more timing control packets is a removable type packet. 15. The method of claim 13, wherein the sequence of packets comprises Supplemental Enhancement Information (SEI) packets of different types, each of the SEI packets includes an SEI packet type field, wherein the content of the SEI packet type field of the timing control packet is different from the content of the packet type field SEI of another type of SEI package. 16. The method of claim 13, wherein at least one payload packet contains encoded sub-image information of the video content. 17. The method of claim 13, wherein the different types of packages in the package sequence are arranged according to a predetermined organization, and the organization of the packages in the package sequence can be determined according to the predetermined organization when one or more removable packages are removed from the package sequence. 18. The method of claim 17, wherein the various types of SEI packets include Region of Interest (ROI) packets.

Images