KR20220045231A - Signaling of sub picture structure - Google Patents

Abstract

A method is provided for signaling a subpicture structure for coded video. A video decoder receives from a bitstream data to be decoded as a sequence of video pictures. A video decoder receives, from a bitstream, subpicture specifications for one or more subpictures in a sequence of video pictures. The subpicture specification identifies the location and size for each subpicture by providing an index that identifies the coding tree unit (CTU) for the subpicture. The video decoder reconstructs each sub-picture for a sequence of video pictures according to the sub-picture specification.

Description

Signaling of sub picture structure

CROSS-REFERENCE TO RELATED PATENT APPLICATION(S)

This disclosure is part of a non-provisional application claiming priority to U.S. Provisional Patent Applications Nos. 62/898,127 and 62/898,620, filed on September 10, 2019 and September 11, 2019, respectively. The contents of the applications listed above are incorporated herein by reference.

This disclosure relates generally to video coding. In particular, the present disclosure relates to a method of signaling subpicture structures.

Unless otherwise indicated herein, the approach described in this section is not prior art to the claims listed below and is not admitted as prior art by inclusion in this section.

High-efficiency video coding (HEVC) is the latest international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). An input video signal is predicted from a reconstructed signal derived from a coded picture region. The prediction residual signal is processed by a linear transformation. The transform coefficients are quantized and entropy coded along with other side information of the bitstream. A reconstructed signal is generated from the prediction signal and the reconstructed residual signal after inverse transform for the inverse quantized transform coefficients. The reconstructed signal is further processed by in-loop filtering to remove coding artifacts. The decoded picture is stored in a frame buffer to predict future pictures of the input video signal.

A coded picture in HEVC is divided into non-overlapping square block regions represented by associated coding tree units (CTUs). A coded picture may be represented as a collection of slices each including an integer number of CTUs. Individual CTUs in a slice are processed in raster scan order. A bi-predictive (B) slice can be decoded using either intra prediction or inter prediction, which predicts the sample value of each block using up to two motion vectors and a reference index. A predictive (P) slice is decoded using either intra prediction or inter prediction using at most one motion vector and a reference index to predict the sample values of each block. An intra (I) slice is decoded using only intra prediction.

A CTU can be divided into multiple non-overlapping coding units (CUs) using a recursive quadtree (QT) structure to adapt to various local motion and texture characteristics. A CTU may also be partitioned into one or multiple smaller size CUs by a quadtree with nested multi-type trees using binary and ternary partitioning. The resulting CU segmentation may be square or rectangular in shape.

One or more prediction units (PUs) are specified for each CU. A prediction unit operates as a basic unit for signaling predictor information together with an associated CU syntax. The specified prediction process is used to predict the value of the associated pixel sample inside the PU. The CU may be further partitioned using a residual quadtree (RQT) structure to represent the associated prediction residual signal. Leaf nodes of the RQT correspond to a transform unit (TU). A transform unit is a transform block (TB) of luma samples of size 8x8, 16x16, or 32x32 or four transform blocks of luma samples of size 4x4 and two of chroma samples of a 4:2:0 color format image. Includes a corresponding transform block. An integer transform is applied to the transform block, and the level values of the quantized coefficients are entropy-coded in the bitstream together with other side information.

The terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) refer to CTU, CU, PU, and TU, respectively. It is defined to specify a 2D sample array of one color component involved. Therefore, the CTU consists of 1 luma CTB, 2 chroma CTBs, and related syntax elements. A similar relationship holds for CUs, PUs and TUs. Tree splitting is usually applied to both luma and chroma simultaneously, with an exception when a certain minimum size for chroma is reached.

The following summary is illustrative only and is not intended to be limiting in any way . That is, the following summary is provided to introduce concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected implementations, but not all implementations, are further described in the Detailed Description below. The following summary is not intended to identify essential features of the claimed subject matter, but to be used in determining the scope of the claimed subject matter. not for

Some embodiments provide a method for signaling a subpicture structure for coded video. A video decoder receives data from a bitstream to be decoded as a sequence of video pictures. A video decoder receives, from a bitstream, subpicture specifications for one or more subpictures in a sequence of video pictures. The subpicture specification identifies the location and size for each subpicture by providing an index that identifies the coding tree units (CTUs) within the picture in raster scan order for that subpicture. The video decoder reconstructs each sub-picture for a sequence of video pictures according to the sub-picture specification.

In some embodiments, a syntax element of a sequence parameter set (SPS) of a sequence of video pictures indicates that one or more subpictures are present in the sequence of video pictures. The SPS may also include a syntax element that specifies the number of subpictures and an identifier for each subpicture in the sequence of video pictures. The identifier of a sub-picture may be signaled in a sequence of video pictures in a slice header of the video picture and/or in a picture parameter set (PPS). In some embodiments, a syntax element in a PPS of a video picture in a sequence of video pictures indicates that all slices of the video picture are rectangular.

In some embodiments, the CTU identified in the raster scan within the picture is in a corner (eg, top left or bottom right) of the subpicture. In some embodiments, an index is assigned to a subpicture grid in a raster scan within a picture, and different subpicture grids are assigned different indices. In some embodiments, the index identifies a subpicture grid defined as or corresponding to one CTU such that the boundary of the subpicture grid is defined along the boundary of the CTU. In some embodiments, the index of sub picture position and size is signaled in the SPS of the video picture sequence.

The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this disclosure. The drawings illustrate implementations of the disclosure, and together with the description serve to explain the principles of the disclosure. It will be understood that the drawings are not necessarily to scale, as some components may be drawn to scale in an actual implementation to clearly illustrate the concepts of the present disclosure.
1A-1E conceptually illustrate CTB or CTU-based grid units used to specify subpictures of a video sequence.
2 shows a CTU or CTB based subpicture grid that is indexed in raster scan order within a picture to specify a subpicture.
3 shows an example video encoder supporting subpictures.
4 conceptually shows a part of a video encoder that implements signaling for a subpicture.
5 conceptually illustrates a process for providing a sub-picture specification in a video encoder.
6 shows an example video decoder supporting subpictures.
7 conceptually shows a part of a video decoder that implements signaling for a subpicture.
Fig. 8 conceptually shows a process for processing a sub-picture specification in a video decoder.
9 conceptually illustrates an electronic system in which some embodiments of the present disclosure are implemented.

In the following detailed description, numerous specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. Any modifications, derivatives and/or extensions based on the teachings described herein fall within the protection scope of the present disclosure. In some instances, well-known methods, procedures, components, and/or circuits relating to one or more example implementations disclosed herein have been presented at a relatively high level without detail in order to avoid unnecessarily obscuring aspects of the teachings of this disclosure. can be explained.

I. Signaling of sub picture structure

A sub-picture is a rectangular region of one or more slices within a picture and a slice is made up of one or more tiles/bricks. Each tile/brick is ordered by CTU. When there are sub pictures in the picture, the number of sub pictures may be two or more. A slice forming a sub-picture may be rectangular. In some embodiments, a grid in CTB units is used to specify the subpicture structure within a picture by indicating the bottom right subpicture grid index in raster scan order within the picture for each subpicture.

In some embodiments, the video encoder may signal the specification of the subpicture (and the video decoder may receive the specification of the subpicture). Table 1a below is Here is an exemplary syntax table for a sequence parameter set (SPS) raw byte sequence payload (RBSP) signaling subpicture information:

[Table 1a]

SPS signaling sub picture information

Table 1b below is another example syntax table for a Sequence Parameter Set (SPS) Raw Byte Sequence Payload (RBSP) that provides a specification of a subpicture.

[Table 1b]

sub picture to specify SPS

Table 1c below is Another example syntax table for the Sequence Parameter Set (SPS) Raw Byte Sequence Payload (RBSP) that provides the specification of the subpicture.

[Table 1c]

sub picture to specify SPS

Table 1d below is another example syntax table for a Sequence Parameter Set (SPS) Raw Byte Sequence Payload (RBSP) that provides a specification of a subpicture.

[Table 1d]

sub picture to specify SPS

The syntax element subpics_present_flag equal to 1 indicates that the subpicture parameter currently exists in the SPS RBSP syntax. In some embodiments, when subpics_present_flag is equal to 1, the value of rect_slice_flag is set to 1. The syntax element subpics_present_flag equal to 0 indicates that the subpicture parameter does not currently exist in the SPS RBSP syntax. In some embodiments, the value of subpics_present_flag is set to 1 in the RBSP of the SPS when the bitstream is the result of the sub-bitstream extraction process and contains only a subset of the sub-pictures of the input bitstream for the sub-bitstream extraction process. can be

The syntax element max_subpics_minus2 plus 2 specifies the maximum number of subpictures that can exist in a coded video sequence (CVS). In some embodiments, max_subpics_minus2 is constrained to be in the range of 0 to 254. The value 255 is reserved for future use.

The syntax element num_subpics_minus1 plus 1 specifies the number of subpictures that can exist in CVS. In some embodiments, the value of num_subpics_minus1 is constrained to be in the range of 0 to 254. The value 255 is reserved for future use. As illustrated in Table 1d, the number of subpictures present in the CVS is signaled directly in the SPS via the syntax element num_subpics_minus1 .

The syntax element num_subpics_minus2 plus 2 specifies the number of subpictures that can exist in CVS. In some embodiments, the value of num_subpics_minus2 is constrained to be in the range of 0 to 254. The value 255 is reserved for future use.

The syntax element subpic_grid_col_width_minus1 plus 1 specifies the width of each element of the subpicture identifier grid except for the rightmost grid column of the picture in units of CtbSizeY. The length of the syntax element is Ceil( Log2( pic_width_max_in_luma_samples /CtbSizeY) ) bits. If not present, subpic_grid_row_width_minus1 is inferred to be 0.

The syntax element subpic_grid_row_height_minus1 plus 1 specifies the height of each element of the sub-picture identifier grid except for the lower grid row of the picture in CtbSizeY units. The length of the syntax element is Ceil(Log2(pic_height_max_in_luma_samples /CtbSizeY)) bits. If not present, subpic_grid_row_height_minus1 is inferred to be 0.

The variable NumSubPicGridRows is derived as follows:

NumSubPicGridRows =

(pic_height_max_in_luma_samples　+

subpic_grid_row_height_minus1　*　CtbSizeY　+　CtbSizeY - 1 ) /

(subpic_grid_row_height_minus1　*　CtbSizeY　+　CtbSizeY)

The syntax element bottom_right_subpic_grid_idx_length_minus1 plus 1 specifies the number of bits used to represent the syntax element bottom_right_subpic_grid_idx_delta[ i ]. The value of bottom_right_subpic_grid_idx_length_minus1 must be in the range of 0 to Ceil( Log2( NumSubPicGridRows * NumSubPicGridCols ) ) - 1 (inclusive of these two numbers).

The variable bottom_right_subpic_grid_idx_delta [i] specifies the difference between the sub-picture grid index of the lower right corner of the i-th sub-picture and the sub-picture grid index of the lower right corner of the (i - 1)-th sub picture when i is greater than 0. The variable bottom_right_subpic_grid_idx_delta[ 0 ] specifies the sub picture grid index of the lower right corner of the 0th sub picture.

The syntax element subpic_grid_idx_delta_sign_flag [i] equal to 1 indicates a positive sign for bottom_right_subpic_grid_idx_delta[i]. sign_bottom_right_subpic_grid_idx_delta[ i ] equal to 0 indicates a negative sign for bottom_right_subpic_grid_idx_delta[ i ].

1A-1E conceptually illustrate that a CTB or CTU-based grid unit is used to specify a subpicture of a video sequence. 1A shows a video sequence (CVS) 100 comprising several video pictures. For example, a video picture of a sequence 100 such as picture 110 is divided into CTUs. Fig. 1B shows that the pictures of the sequence are divided into subpicture grids to specify subpictures. Each sub picture grid 120 corresponds to an integer number of CTUs such that each sub picture grid is defined based on a boundary of a CTU or CTB. 1C shows an example in which each sub-picture grid 120 corresponds to exactly one CTU or CTB. 1D shows an example in which each sub-picture grid 120 corresponds to a 2×1 CTU or CTB. FIG. 1E shows an example in which each sub-picture grid 120 corresponds to a 2×3 CTU or CTB. In some embodiments, for example, SPS syntax elements such as subpic_grid_col_width_minus1 and subpic_grid_row_height_minus1 define a subpicture grid in terms of a CTU or CTB. In addition, the size of each CTU or CTB is signaled to the SPS in the syntax element log2_ctu_size_minus5.

2 shows a CTU or CTB based subpicture grid that is indexed to specify subpictures. Each subpicture grid of picture 110 corresponds to an index that can be used when the video codec signals the size (eg, width, height) and/or position of the subpicture. The specified elements of the subpicture identifier grid are indexed in the order of the raster scan. In some embodiments, the subpictures are also indexed in the order of the raster scan.

As shown, a picture of sequence 100, eg, picture 110, has four subpictures 210, 220, 230, and 240 defined using a CTU or CTB based subpicture grid. is defined The index associated with the subpicture grid is used to specify the size and position of the subpictures 210-240.

In some embodiments, the location of each subpicture is specified based on an index associated with the subpicture grid of a corner (eg, bottom right corner or top left corner) of the subpicture. In this figure, the position of the sub picture 210 is index 27, the position of the sub picture 220 is index 36, the position of the sub picture 230 is index 79, and the position of the sub picture 240 is index 84 am. In some embodiments, for example, SPS syntax elements such as bottom_right_subpic_grid_idx_delta[i] and subpic_grid_idx_delta_sign_flag[i] are used to specify the bottom right corner position of one subpicture by referencing the bottom right corner position of another subpicture. do. Alternatively, in some embodiments, each CTU/CTB based subpicture grid may be associated with an X-index and a Y-index, the position of each subpicture being that of the subpicture grid in the upper left corner of the subpicture. It can be specified by an X index and a Y index.

In some embodiments, the maximum number of subpictures (in CVS) may be specified in a Video Parameter Set (VPS). Table 3 below is an exemplary syntax table of VPS that specifies the maximum number of subpictures in CVS.

[Table 3]

in CVS sub picture VPS specifying the maximum number

The syntax element vps_max_subpics_minus2 plus 2 (or vps_max_subpics_minus1 plus 1) specifies the maximum number of subpictures allowed in each CVS referring to the VPS. In some embodiments, the syntax element vps_max_subpics_minus2 is constrained to be in the range of 0 to 254. The value 255 is reserved for future use.

In some embodiments, sub-picture related parameters are signaled in a Picture Parameter Set (PPS). Table 4 below illustrates an exemplary syntax table of a PPS including sub-picture information.

[Table 4]

sub picture PPS specifying ID

In some embodiments, the syntax element rect_slice_flag equal to 0 in the PPS specifies that the bricks in each slice are in raster scan order and no slice information is signaled in the PPS. The syntax element rect_slice_flag equal to 1 specifies that the brick in each slice covers a rectangular area of the picture and slice information is signaled in the PPS. When subpics_present_flag is equal to 1, the value of rect_slice_flag must be equal to 1. In some embodiments, when brick_splitting_present_flag is equal to 1, the value of rect_slice_flag is set to 1. If not present, the syntax element rect_slice_flag is inferred to be equal to 1.

The syntax element signaled_subpic_id_flag equal to 1 specifies that the subpicture ID for each subpicture is signaled. The syntax element signaled_subpic_id_flag equal to 0 specifies that the subpicture ID is not signaled. If the value of signaled_subpic_id_flag is not present, it is inferred to be equal to 0.

The syntax element signaled_subpic_id_length_minus1 plus 1, if present, specifies the number of bits used to indicate the syntax element subpic_id[i] and the syntax element subpicture_id of the slice header. In some embodiments, the value of signaled_subpic_id_length_minus1 is constrained to be in the range of 0 to 7 (inclusive of these two numbers). The value of signaled_subpic_id_length_minus1 is inferred to be equal to Ceil(Log2( Max( 2, num_subpics_minus1 + 1 ) ) - 1 if not present.

The syntax element subpic_id [i] specifies the sub picture ID of the i-th sub picture. The length of the subpuic_id[i] syntax element is signaled_subpic_id_length_minus1 + 1 bits. The value of subpic_id[ i ], if not present, is inferred to be equal to i for each i in the range 0 to num_subpics_minus1 (inclusive of these two numbers). Slices with the same sub picture ID collectively form a sub picture area.

The syntax element subpicture_id specifies the subpicture ID to which the current slice belongs. The length of the subpic_id syntax element is Ceil(Log2( num_subpics_minus1 + 1 )) bits. The value of subpicture_id is a mapping to subpic_id[ i ] specified in the PPS. The signaled subpicture ID subpic_id[i] for the i-th subpicture may be added to the PPS, and the signaled subpicture ID subpicure_id may be mapped to the subpic_id[i] of the PPS and added to the slice header. Table 5 below illustrates an exemplary syntax table of a slice header including sub-picture information.

[Table 5]

sub picture Slice header specifying ID

In some embodiments, assuming that the subpicture ID is not changed for the CVS, the subpicture ID may alternatively be signaled in the slice header, PPS, and/or SPS of the CVS. For example, subpic_id[ i ], which is the signaled subpicture ID for the i-th subpicture, may be signaled in SPS (not PPS) according to Table 6 below:

[Table 6]

sub picture specifying the ID SPS

II. Exemplary video encoder

3 shows an example video encoder 300 supporting subpictures. As illustrated, a video encoder 300 receives an input video signal from a video source 305 and encodes the signal into a bit stream 395 . The video encoder 300 includes a transform module 310 , a quantization module 311 , an inverse quantization module 314 , an inverse transform module 315 , an intra picture estimation module 320 , an intra prediction module 325 , and a motion compensation module. 330 , motion estimation module 335 , in-loop filter 345 , reconstructed picture buffer 350 , MV buffer 365 , MV prediction module 375 , entropy encoder 390 . It has several components or modules for encoding the signal from the video source 305 , including at least some components selected from The motion compensation module 330 and the motion estimation module 335 are part of the inter-prediction module 340 .

In some embodiments, modules 310 - 390 are modules of software instructions that are executed by one or more processing units (eg, processors) of a computing device or electronic device. In some embodiments, modules 310 - 390 are modules of hardware circuitry implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 310-390 are described as separate modules, some modules may be combined into a single module.

A video source 305 provides a raw video signal representing the pixel data of each video frame without compression. The subtractor 308 calculates the difference between the raw video pixel data of the video source 305 and the predicted pixel data 313 from the motion compensation module 330 or intra-prediction module 325 . The transform module 310 transforms the difference (or residual pixel data or residual signal 309 ) into transform coefficients (eg, by performing a discrete cosine transform or DCT). The quantization module 311 quantizes the transform coefficients into quantized data (or quantized coefficients) 312 , which is encoded into a bit stream 395 by the entropy encoder 390 .

The inverse quantization module 314 inversely quantizes the quantized data (or quantized coefficient) 312 to obtain a transform coefficient, and the inverse transform module 315 performs an inverse transform on the transform coefficient to generate a reconstructed residual 319 . . The reconstructed residual 319 is added along with the predicted pixel data 313 to generate reconstructed pixel data 317 . In some embodiments, reconstructed pixel data 317 is temporarily stored in a line buffer (not shown) for intra picture prediction and spatial MV prediction. The reconstructed pixels are filtered by the in-loop filter 345 and stored in the reconstructed image buffer 350 . In some embodiments, the reconstructed picture buffer 350 is storage external to the video encoder 300 . In some embodiments, the reconstructed picture buffer 350 is storage internal to the video encoder 300 .

The intra picture estimation module 320 generates intra prediction data by performing intra prediction based on the reconstructed pixel data 317 . The intra prediction data is provided to an entropy encoder 390 to be encoded into a bit stream 395 . The intra prediction data is also used by the intra prediction module 325 to generate predicted pixel data 313 .

The motion estimation module 335 performs inter prediction by generating an MV for referring to pixel data of a previously decoded frame stored in the reconstructed picture buffer 350 . These MVs are provided to the motion compensation module 330 to generate predicted pixel data.

The video encoder 300 generates a predicted MV using MV prediction instead of encoding the full real MV in the bitstream, and the difference between the MV used for motion compensation and the predicted MV is encoded into the residual motion data to generate the bitstream ( 395) is stored.

The MV prediction module 375 generates a predicted MV based on a reference MV previously generated to encode the video frame, ie, a motion compensation MV used to perform motion compensation. The MV prediction module 375 retrieves a reference MV from the previous video frame from the MV buffer 365 . The video encoder 300 stores the MV generated for the current video frame in the MV buffer 365 as a reference MV for generating the predicted MV.

The MV prediction module 375 generates a predicted MV using the reference MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The difference between the predicted MV and the motion compensated MV (MC MV) of the current frame (residual motion data) is encoded by the entropy encoder 390 into a bit stream 395 .

Entropy encoder 390 encodes various parameters and data into bit stream 395 using entropy coding techniques such as, for example, context-adaptive binary arithmetic coding (CABAC) or Huffman encoding. Entropy encoder 390 encodes the residual motion data as various header elements, flags, and syntax elements along with quantized transform coefficients 312 into bitstream 395 . The bit stream 395 is eventually stored in a storage device or transmitted to a decoder, for example, via a communication medium such as a network.

The in-loop filter 345 performs a filtering or smoothing operation on the reconstructed pixel data 317 to reduce coding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the performed filtering operation includes a sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

4 is Conceptually depicts a portion of a video encoder 300 that implements signaling for subpictures. As illustrated, entropy encoder 390 receives signaling from video source 305 specifying parameters 410 for subpictures to be present in current coded sequence 420 of video pictures. These parameters 410 may indicate the number of sub-pictures present in the current coded sequence. The parameters 410 may also indicate the location and geometry (height, width, and size) of each sub-picture. The quantized coefficients 312 of the different subpictures are provided in the data path of the encoder 300 .

Based on the parameters for the subpicture, the entropy encoder 390 generates a syntax element in the bitstream 395 that serves as a specification for the subpicture. These syntax elements may include identifiers of subpictures, number of subpictures, as well as subpicture positions and geometry specified in terms of a CTB/CTU based subpicture grid. These syntax elements may be stored in the SPS of the currently coded video sequence 420 , the PPS of individual pictures of the video sequence, the slice headers of individual slices within pictures of the sequence, and/or the VPS of the entire video. Examples of such syntax elements are described with reference to Tables 1a to 1D and 2 to 6 above.

5 conceptually illustrates a process 500 for providing a subpicture specification in a video encoder. In some embodiments, one or more processing units (eg, processors) of a computing device implementing encoder 300 perform process 500 by executing instructions stored on a computer-readable medium. In some embodiments, an electronic device implementing decoder 300 performs process 500 .

The encoder receives (at block 510 ) data to be encoded as a sequence of video pictures in a bitstream. The encoder signals (at block 520 ) subpicture specifications for one or more subpictures of the sequence of video pictures in the bitstream. In some embodiments, a syntax element within an SPS of a sequence of video pictures indicates that one or more subpictures are present in the sequence of video pictures. The SPS may also include a syntax element that specifies the number of subpictures and an identifier for each subpicture in the sequence of video pictures. The identifier of a subpicture may be signaled in the sequence of video pictures in a slice header and/or PPS of a video picture. In some embodiments, a syntax element in a PPS of a video picture in a sequence of video pictures indicates that all slices of the video picture are rectangular.

The encoder identifies (at block 530 ) the location and size for each subpicture by providing an index identifying the CTU for the subpicture 830 .

In some embodiments, the identified CTU is in a corner (eg, top left or bottom right) of the subpicture. In some embodiments, an index is assigned to a subpicture grid, and different subpicture grids are assigned different indices. In some embodiments, the index identifies a subpicture grid defined as or corresponding to one CTU such that the boundary of the subpicture grid is defined along the boundary of the CTU. In some embodiments, the index of the sub-picture position is signaled in the SPS of the video picture sequence.

The encoder encodes (at block 540 ) each subpicture for the sequence of video pictures according to the subpicture specification.

III. Exemplary video decoder

6 shows an example video decoder 600 that supports subpictures. As illustrated, video decoder 600 is an image decoding or video decoding circuit that receives bit stream 695 and decodes the contents of the bit stream into pixel data of a video frame for display. The video decoder 600 includes an inverse quantization module 611 , an inverse transform module 610 , an intra prediction module 625 , a motion compensation module 630 , an in-loop filter 645 , a decoded picture buffer 650 , an MV buffer It has several components or modules for decoding the bitstream 695 , including some components selected from 665 , an MV prediction module 675 , and a parser 690 . The motion compensation module 630 is part of the inter prediction module 640 .

In some embodiments, modules 610 - 690 are modules of software instructions being executed by one or more processing units (eg, processors) of a computing device. In some embodiments, modules 610 - 690 are modules of hardware circuitry implemented by one or more ICs of an electronic device. Although modules 610-690 are described as separate modules, some modules may be combined into a single module.

Parser 690 (or entropy decoder) receives bit stream 695 and performs initial parsing according to syntax defined in a video coding or image coding standard. The parsed syntax elements include various header elements, flags, as well as quantized data (or quantized coefficients) 612 . Parser 690 parses the various syntax elements using entropy coding techniques such as, for example, context adaptive binary arithmetic coding (CABAC) or Huffman encoding.

The inverse quantization module 611 inversely quantizes the quantized data (or quantized coefficient) 612 to obtain a transform coefficient, and the inverse transform module 610 performs an inverse transform on the transform coefficient 616 to perform an inverse transformation on the reconstructed residual signal 619 . ) is created. Reconstructed residual signal 619 is added along with predicted pixel data 613 from intra-prediction module 625 or motion compensation module 630 to generate decoded pixel data 617 . The decoded pixel data is filtered by the in-loop filter 645 and stored in the decoded picture buffer 650 . In some embodiments, decoded picture buffer 650 is storage external to video decoder 600 . In some embodiments, decoded picture buffer 650 is storage internal to video decoder 600 .

The intra prediction module 625 receives the intra prediction data from the bit stream 695 , and accordingly generates predicted pixel data 613 from the decoded pixel data 617 stored in the decoded picture buffer 650 . In some embodiments, decoded pixel data 617 is also stored in a line buffer (not shown) for intra picture prediction and spatial MV prediction.

In some embodiments, the contents of decoded picture buffer 650 are used for display. The display device 655 directly retrieves the contents of the decoded picture buffer 650 for display, or retrieves the contents of the decoded picture buffer into the display buffer. In some embodiments, the display device receives pixel values from decoded picture buffer 650 via pixel transport.

The motion compensation module 630 generates predicted pixel data 613 from the decoded pixel data 617 stored in the decoded picture buffer 650 according to the motion compensation MV (MC MV). These motion compensated MVs are decoded by adding the residual motion data received from the bit stream 695 with the predicted MVs received from the MV prediction module 675 .

The MV prediction module 675 generates a predicted MV based on a reference MV generated to decode a previous video frame, eg, a motion compensation MV used to perform motion compensation. The MV prediction module 675 retrieves the reference MV of the previous video frame from the MV buffer 665 . The video decoder 600 stores the motion compensation MV generated for decoding the current video frame in the MV buffer 665 as a reference MV for generating the predicted MV.

The in-loop filter 645 performs a filtering or smoothing operation on the decoded pixel data 617 to reduce coding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the performed filtering operation includes a sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

7 conceptually illustrates a portion of a video decoder 600 that implements signaling for subpictures. As shown, entropy decoder 690 provides quantized coefficients 612 to the data path of video decoder 600 , which in turn provides pixel data to be displayed on display device 655 for different subpictures. to create The display device may display the received pixel data according to the sub-picture parameter 710 for a sub-picture to be present in the sequence of the currently coded picture 720 . The parameter 710 may also indicate the number of subpictures that will be present in the current coded sequence. These parameters may also indicate the position and geometry (size, height, width) of each sub-picture. Entropy decoder 690 provides sub-picture parameters based on syntax elements decoded from bitstream 695 .

As illustrated, the entropy decoder (parser) 690 receives from the bitstream 695 a syntax element that serves as a specification for the subpicture. These syntax elements may include identifiers of subpictures, number of subpictures, as well as subpicture positions and geometry specified in terms of a CTB/CTU based subpicture grid. These syntax elements may be stored in the SPS of the currently coded video sequence 720 , the PPS of individual pictures of the video sequence, the slice header of individual slices within pictures of the sequence, and/or the VPS of the entire video. Examples of such syntax elements are described with reference to Tables 1a to 1D and 2 to 6 above.

8 conceptually illustrates a process 800 for processing a subpicture specification in a video decoder. In some embodiments, one or more processing units (eg, processors) of a computing device implementing decoder 600 perform process 800 by executing instructions stored on a computer-readable medium. In some embodiments, an electronic device implementing decoder 600 performs process 800 .

The decoder receives (at block 810 ) data from the bitstream to be decoded as a sequence of video pictures. The decoder (at block 820 ) receives, from the bitstream, subpicture specifications for one or more subpictures in the sequence of video pictures. In some embodiments, a syntax element within an SPS of a sequence of video pictures indicates that one or more subpictures are present in the sequence of video pictures. The SPS may also include a syntax element that specifies the number of subpictures and an identifier for each subpicture in the sequence of video pictures. The identifier of a subpicture may be signaled in the sequence of video pictures in a slice header and/or PPS of a video picture. In some embodiments, a syntax element in a PPS of a video picture in a sequence of video pictures indicates that all slices of the video picture are rectangular.

The decoder identifies (at block 830 ) the location and size for each subpicture by providing an index that identifies the CTU for the subpicture. In some embodiments, the identified CTU is in a corner (eg, top left or bottom right) of the subpicture. In some embodiments, an index is assigned to a subpicture grid, and different subpicture grids are assigned different indices. In some embodiments, the index identifies a subpicture grid defined as or corresponding to one CTU such that the boundary of the subpicture grid is defined along the boundary of the CTU. In some embodiments, the index of the sub-picture position is signaled in the SPS of the video picture sequence.

The decoder reconstructs (at block 840 ) each subpicture for the sequence of video pictures according to the subpicture specification.

IV. Exemplary Electronic System

Many of the features and applications described above are implemented as software processes specified in sets of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computational or processing unit(s) (eg, one or more processors, processor cores, or other processing units), these instructions cause the processing unit(s) to perform the operations indicated in the instructions. Examples of computer-readable media include CD-ROMs, flash drives, random-access memory (RAM) chips, hard drives, erasable programmable read only memories (EPROM), electrically erasable including, but not limited to, electrically erasable programmable read-only memory (EEPROM) and the like. Computer readable media does not include carrier waves and electronic signals passing through wireless or wired connections.

As used herein, the term “software” is meant to include firmware resident in read-only memory or applications stored in magnetic storage that can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions may be implemented as sub-parts of a larger program while maintaining separate software inventions. In some embodiments, multiple software inventions may also be implemented as separate programs. Finally, any combination of separate programs that together implement the software invention described herein is within the scope of the present disclosure. In some embodiments, the software program defines one or more specific machine implementations that, when installed to operate on one or more electronic systems, execute and perform the operations of the software program.

9 conceptually illustrates an electronic system 900 in which some embodiments of the present disclosure are implemented. Electronic system 900 may be a computer (eg, desktop computer, personal computer, tablet computer, etc.), telephone, PDA, or any other type of electronic device. Such electronic systems include various types of computer readable media and interfaces for various other types of computer readable media. Electronic system 900 includes bus 905 , processing unit(s) 910 , graphics-processing unit (GPU) 915 , system memory 920 , network 925 , read-only memory 930 . , a persistent storage device 935 , an input device 940 , and an output device 945 .

Bus 905 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of electronic system 900 . For example, bus 905 communicatively connects processing unit(s) 910 with GPU 915 , read-only memory 930 , system memory 920 , and persistent storage device 935 .

From these various memory units, the processing unit(s) 910 retrieves instructions to execute and data to process in order to execute the process of the present disclosure. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. Some instructions are passed to GPU 915 and executed by GPU 915 . GPU 915 may offload various computations or supplement image processing provided by processing unit(s) 910 .

Read-only memory (ROM) 930 stores static data and instructions used by processing unit(s) 910 and other modules of the electronic system. On the other hand, persistent storage device 935 is a read and write memory device. The device is a non-volatile memory unit that stores instructions and data even when the electronic system 900 is off. Some embodiments of the present disclosure use a mass storage device (eg, a magnetic or optical disk and a corresponding disk drive) as the persistent storage device 935 .

Another embodiment uses a removable storage device (eg, a floppy disk, a flash memory device, etc. and a corresponding disk drive) as the persistent storage device. Like persistent storage device 935 , system memory 920 is a read and write memory device. However, unlike storage device 935 , system memory 920 is a volatile read and write memory, such as, for example, random access memory. The system memory 920 stores some of the instructions and data used by the processor at runtime. In some embodiments, processes according to the present disclosure are stored in system memory 920 , persistent storage device 935 , and/or read-only memory 930 . For example, various memory units contain instructions for processing multimedia clips in accordance with some embodiments. From these various memory units, processing unit(s) 910 retrieves instructions to execute and data to process in order to execute the processes of some embodiments.

Bus 905 is also connected to input and output devices 940 and 945 . Input device 940 allows a user to communicate information and select commands to the electronic system. Input device 940 includes an alphanumeric keyboard and pointing device (also referred to as a “cursor control device”), a camera (eg, a webcam), a microphone, or similar device for receiving voice commands, and the like. The output device 945 displays or otherwise outputs data generated by the electronic system. Output devices 945 include printers and display devices such as, for example, cathode ray tubes (CRTs) or liquid crystal displays (LCDs), as well as speakers or similar audio output devices. Some embodiments include, for example, a device such as a touch screen that functions as both an input device and an output device.

Finally, as shown in FIG. 9 , bus 905 also couples electronic system 900 to network 925 via a network adapter (not shown). In this way, a computer is a network of computers (eg, a local area network (“LAN”), a wide area network (“WAN”), or an intranet, or a network of networks, such as, for example, the Internet. ) can be part of Any or all components of electronic system 900 may be used in conjunction with the present disclosure.

Some embodiments provide, for example, a microprocessor storing computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). , electronic components such as storage and memory. Some examples of such computer readable media are RAM, ROM, read-only compact disc (CD-ROM), recordable compact disc (CD-R), rewritable compact disc (CD-RW), Read-only digital versatile discs (eg DVD-ROM, dual-layer DVD-ROM), various writable/rewritable DVDs (eg DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (eg SD card) , mini SD card, micro SD card, etc.), magnetic and/or solid state hard drives, read-only and writable Blu-ray® disks, ultra-dense optical disks, any other optical or magnetic media, and floppy disks. The computer readable medium may store a computer program executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include, for example, files containing machine language code, such as generated by a compiler, and high-level code executed by a computer, electronic component, or microprocessor using an interpreter. .

While the above discussion primarily refers to microprocessors or multi-core processors executing software, many of the features and applications described above may include one or more performed by integrated circuits. In some embodiments, such integrated circuits execute instructions stored in the circuit itself. Additionally, some embodiments execute software stored in a programmable logic device (PLD), ROM, or RAM device.

As used herein and in any of the claims herein, the terms “computer,” “server,” “processor,” and “memory,” all refer to an electronic or other technological device. These terms exclude people or groups of people. For the purpose of the specification, the terms displaying or displaying mean displaying on an electronic device. The terms "computer readable medium," "computer readable media," and "machine readable medium," as used herein and in any claims of this application, refer exclusively to tangible storage of information in a form readable by a computer. limited to physical objects. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.

Although the present disclosure has been described with reference to numerous specific details, those skilled in the art will recognize that the present disclosure may be embodied in other specific forms without departing from the spirit of the present disclosure. Also, a number of drawings (including FIGS. 5 and 8 ) conceptually illustrate the process. Certain acts in these processes may not be performed in the exact order shown and described. A specific operation may not be performed as one continuous series of operations, and different specific operations may be performed in different embodiments. Also, a process may be implemented using multiple sub-processes or as part of a larger macro process. Accordingly, those skilled in the art will understand that the present disclosure should not be limited by the foregoing exemplary details, but should be defined by the appended claims.

Additional notes

The subject matter described herein sometimes exemplifies different components included in or connected to different other components. It should be understood that such depicted architectures are merely examples and in practice many other architectures may be implemented that achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Accordingly, any two components herein combined to achieve a particular functionality may be viewed as “associated with” each other such that the desired functionality is achieved, regardless of architecture or intermediate component. Likewise, any two components so associated can also be viewed as being “operably connected” or “operably coupled” to each other to achieve a desired function, and any two components that may be so associated with the desired function may be viewed as “operably combinable” with each other to achieve Specific examples of operatively combinable are physically combinable and/or physically interacting components and/or wirelessly interactable and/or wirelessly interacting components and/or logically interacting and/or or components that can logically interact with each other.

Also, for use of substantially any plural and/or singular term described herein, one may translate from plural to singular and/or from singular to plural, as appropriate to the context and/or application. Various singular/plural permutations may be expressly set forth herein for clarity.

Also, that terminology used in this specification generally and in particular in the appended claims (eg, body parts of the appended claims) is generally intended to be "open" terms, eg, the term "comprising" should be construed as "including, but not limited to," the term "having" should be construed as "having at least It will be understood by those skilled in the art that it should be construed as "not". It will also be understood by those skilled in the art that if a certain number of introductory claim recitations are intended, such intent will be expressly recited in the claims and no such intent would exist in the absence of such recitation. For example, as an aid to understanding, the appended claims below may contain use of the introductory phrases “at least one” and “one or more” to introduce claim recitation. However, even when the same claim contains the introductory phrases "one or more" or "at least one" and "a singular indefinite article, use of these phrases indicates that the introduction of claim recitation by the indefinite article It should not be construed to mean limiting any particular claim, including an introduced claim recitation, to implementations comprising only one such recitation, which holds even for the use of the definite article used to introduce the claim recitation. Furthermore, even if a particular number of introduced claim recitations are explicitly recited, those skilled in the art will recognize that such recitations should be construed to mean at least the recited number, e.g., "two recitations without other modifiers." It will be appreciated that a mere description of "means at least two descriptions, or two or more descriptions. Further, in instances in which a clause similar to "at least one of A, B, and C, etc." is used, it will generally be Configuration is intended for those skilled in the art to understand this convention, for example, "a system having at least one of A, B, and C" means A only, B only, C only, A and B together, A and C including, but not limited to, systems having together, B and C together, and/or A, B and C together, etc. In this example, "at least one of A, B, or C, etc." In instances where conventions similar to " are used, generally such configurations are intended for those skilled in the art to understand these conventions; for example, "a system having at least one of A, B, or C" means that only A, only B , C only, A and B together, A and C together, B and C together, and/or A, B and C together, etc. Wherever in the detailed description, claims, or drawings, any alternative word and/or phrase presenting two or more alternative terms refers to one of the terms, either one of the terms, or both terms. containing It will also be understood by those skilled in the art that it should be understood in fact to be considered possible. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B”.

From the foregoing, it will be appreciated that various implementations of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Accordingly, the various implementations disclosed herein are not intended to be limiting, the true scope and spirit being indicated by the following claims.

Claims (12)

A video decoding method comprising:
receiving from a bitstream data to be decoded as a sequence of video pictures;
receiving from the bitstream a subpicture specification for one or more subpictures in the sequence of video pictures, wherein the subpicture specification is an index identifying a coding tree unit (CTU) for the respective subpicture. identifying a location and size for each subpicture of the one or more subpictures by providing ; and
reconstructing each subpicture of the one or more subpictures for the sequence of video pictures according to the subpicture specification;
A video decoding method comprising: According to claim 1,
wherein different CTUs correspond to different subpicture grids assigned different indices. 3. The method of claim 2,
and a boundary of the subpicture grids is defined using a boundary of a CTU. According to claim 1,
and the CTU identified by the provided index is at a corner of the subpicture. According to claim 1,
and the index is provided by a sequence parameter set (SPS) of the sequence of video pictures. 6. The method of claim 5,
and the SPS further comprises a syntax element specifying a number of subpictures of the sequence of video pictures. 6. The method of claim 5,
and the identifier for the sub-picture is signaled in the SPS. According to claim 1,
and the identifier for the subpicture is signaled in a slice header of a slice. According to claim 1,
and the identifier for the sub-picture is signaled in a picture parameter set (PPS) of a video picture of the sequence of video pictures. According to claim 1,
A syntax element of a sequence parameter set (SPS) of the sequence of video pictures indicates that one or more subpictures are present in the sequence of video pictures, and a picture parameter of a video picture in the sequence of video pictures. wherein a syntax element of a picture parameter set (PPS) indicates that all slices of the video picture are rectangular. A video encoding method comprising:
receiving data to be encoded as a sequence of video pictures in a bitstream;
signaling in the bitstream a subpicture specification for one or more subpictures in the sequence of video pictures, wherein the subpicture specification is an index identifying a coding tree unit (CTU) for the respective subpicture. identifying a location and size for each subpicture of the one or more subpictures by providing ; and
encoding each subpicture of the one or more subpictures for the sequence of video pictures according to the subpicture specification;
A video encoding method comprising: In an electronic device,
A video decoder circuitry configured to perform operations comprising:
receiving from a bitstream data to be decoded as a sequence of video pictures;
receiving, from the bitstream, a subpicture specification for one or more subpictures in the sequence of video pictures, wherein the subpicture specification is an index identifying a coding tree unit (CTU) for each of the subpictures. identifying a location and size for each subpicture of the one or more subpictures by providing ; and
reconstructing each subpicture of the one or more subpictures for the sequence of video pictures according to the subpicture specification.
The electronic device comprising a.

Images